Job Lifecycle

The states a job moves through, the retry loop, cooperative cancellation, and the append-only Transition Log that records every transition.

Every job lives inside a small state machine. It is enqueued into one of three entry states, may be claimed and run, may retry, and eventually settles in a terminal state it never leaves on its own. There are exactly seven states and a fixed set of legal transitions between them. Knowing them, and the force behind each edge, is enough to reason about what any job in your system is doing and why.

The seven states#

A job is always in one of seven states. Three are non-terminal, meaning the job still has work ahead of it. Four are terminal, meaning it has settled and will not move again on its own.

Leased is the only in-flight state. A job is only ever running while it is Leased, and a terminal job never transitions again on its own. The one thing that moves a terminal job is an explicit Operator Action, covered at the end of this page.

Two terminal states are easy to confuse. A job is DeadLettered when it ran and kept failing until it exhausted its attempt ceiling. A job is Quarantined when it could not be routed at all, because its Wire Name has no registered handler or its payload no longer deserializes. Dead-lettering is a failure of the work; quarantine is a failure to even start it.

The enum member is spelled DeadLettered, one word. In prose the domain term is written Dead-Lettered.

Entering the lifecycle#

A job enters at enqueue in one of three states, decided entirely by its dependencies at that moment:

• Scheduled, when it has no parents, or its parents are already terminal and satisfied. It waits for its due time, then becomes claimable.
• AwaitingParent, when one or more parents are not yet terminal. ParentsRemaining is set to the pending count.
• Cancelled, directly, when it is an on-success child whose gating parent had already reached a non-Succeeded terminal state by the time it was enqueued. There is no point running it, so it goes straight to terminal.

AwaitingParent resolves as parents terminate. The transition to Scheduled is not triggered by any one parent finishing. It requires the last gating parent to resolve, dropping ParentsRemaining to zero, and for on-success dependencies every parent to have actually Succeeded. If an on-success parent reaches a non-Succeeded terminal state, the waiting child is Cancelled instead, and that cancellation cascades to its own children. The full dependency model is on the Dependencies page.

Claiming a job#

A worker picks up a due Scheduled job and leases it. A single claim atomically leases a batch of due jobs to the calling worker, ordered by due time. For each one it flips the state to Leased, records which worker holds the lease and when the lease expires, and raises the Attempt count. A paused queue or a saturated Concurrency Limit yields nothing.

The claim is what raises the Attempt, because claiming is the start of an attempt. A fresh job sits at Attempt 0, and the first claim makes it 1. This is the number the handler sees as JobContext.Attempt, which starts at 1 for the first try. The Attempt does not rise when a job fails; it already rose when the job was claimed.

The Lease is a time-bounded, heartbeat-renewed claim, not a lock. A lock implies indefinite ownership, while a Lease expires on its own if the holder stops renewing it. That expiry is the mechanism behind crash recovery and at-least-once delivery, both below. The Lease mechanics have their own page, Execution Guarantee.

Running and reporting an outcome#

Once a job is Leased, BackWave runs its handler for exactly one Attempt, inside a fresh DI scope opened for that Attempt so injected dependencies resolve anew each try. Returning normally is success; throwing is failure.

The handler's result becomes a JobOutcome, which the worker reports back to the store. That write is fenced by the worker id and the attempt number, so the outcome applies only when the caller still holds the live Lease for exactly that Attempt. If the state is not Leased, the lease owner does not match, the attempt does not match, or the lease has already expired, the write is rejected as stale and nothing changes.

Each outcome maps to a target state, and a Failure turns on whether a retry remains:

• Success sends the job to Succeeded. Any emitted Job Output is persisted atomically with this transition.
• Failure with a retry still available sends it back to Scheduled, due at its backoff time. This is a retry.
• Failure with no retries left sends it to DeadLettered. The ceiling is exhausted.
• Cancelled sends it to terminal Cancelled.
• Unroutable sends it to Quarantined. Routing failure surfaces here, when BackWave tries to run a claimed job, not at enqueue.

This fence is the chokepoint that makes the recorded outcome Effect-Once even though the handler body runs at-least-once. Two nodes executing the same job concurrently is legal at-least-once behavior, not a bug; only a stale store write is fenced out. The full treatment of that boundary is on the Execution guarantee page.

Retries and the attempt ceiling#

Whether a failure retries or dead-letters is decided above the store by the RetryPolicy, then handed to the store as plain data, a next due time that is either set or null. The store never runs policy code.

MaxAttempts defaults to 10. The default backoff grows exponentially, two to the power of the attempt number in seconds, capped at five minutes. Once a job has failed its last allowed attempt, the policy hands back a null next due time, and that null is exactly what dead-letters the job instead of rescheduling it. Configuring this is covered on the Retries and Error Handling page.

Lease expiry and crash recovery#

The retry path above assumes the worker lived long enough to report an outcome. When a node crashes or is cut off from the network, it simply stops heartbeating and its Lease lapses. A maintenance sweep finds every Leased job whose lease has expired and either reschedules it or dead-letters it, using the same logic that turns a used-up retry budget into a dead-letter.

A lease expiry counts as an Attempt exactly like a thrown exception. The claim that started the attempt already counted it, so expiry just disposes the lease without counting again. This means Leased has two distinct paths back to Scheduled and two to DeadLettered, one pair driven by a reported outcome and one pair driven by lease expiry. The expiry pair is the crash-recovery story, and it is invisible if you only model reported outcomes.

A live worker keeps its claim by heartbeating, which extends the lease on the jobs it still holds. A worker that no longer holds a job learns so on its next heartbeat, and that is its signal to abandon execution rather than keep applying effects it no longer owns.

Cooperative cancellation#

Cancelling a running job never kills a thread. CancelJobAsync on a Leased job only sets a cancel-requested flag and reports that cancellation was requested. A job that has not started yet, Scheduled or AwaitingParent, has nothing to interrupt, so it transitions straight to terminal Cancelled and cascades cancellation to its on-success children.

For a running job, that flag travels back to the worker on its next heartbeat. A still-held lease carrying the flag fires the handler's CancellationToken; a lease the worker has already lost tells it to abandon the attempt instead. The handler observes the token like any other cooperative cancellation and stops promptly. The job reaches Cancelled only once the handler returns and the worker reports a cancelled outcome.

No operator action moves a Leased job straight to Cancelled. Cancellation of a running job is always cooperative. The full path is on the Cancel a Job page.

The Transition Log#

Every state-changing store operation appends exactly one JobTransition row, atomically with the change. Each row carries an Ordinal, a Timestamp, the new State, the Attempt, and an optional FailureDetail. FailureDetail is populated only on a failing transition, and only when capture is enabled by the Job History Policy; every other transition records null. The log is append-only, bounded per job, and deleted with the job under retention.

This is the timeline the Monitor surfaces for a job. It records that a job moved between states, never the steps inside a handler. BackWave does not record and replay handler internals.

The Transition Observer#

The Transition Log is also how host-supplied code reacts to lifecycle changes. A Transition Observer is egress-only code BackWave invokes through its one OnTransitionAsync method when a job reaches a declared state.

An observer must be idempotent, because delivery is at-least-once and the same transition may reach it more than once. Delivery walks the append-only Transition Log under a lease rather than emitting in-process on the acting node. That matters because the transitions operators most want alerts for, like a job dead-lettered through a lease expiry, happen on a node that has crashed and cannot emit anything. Walking the log sees every transition, including those crash-path ones.

Observer delivery is at-least-once and is not Effect-Once. The fire is a new side effect outside the (workerId, attempt) fence, so the same transition may be delivered more than once and your reaction must be idempotent, the same contract a handler carries. An observer can only observe. It never vetoes or rewrites a transition, and a slow or throwing observer is bounded by a timeout and retried without ever stopping the worker. Registration and delivery details are on the Transition Observer Internals page.

Operator Actions on terminal jobs#

A terminal job never moves on its own. The only thing that changes that is an explicit Operator Action. RequeueAsync accepts a DeadLettered or Quarantined job and returns it to Scheduled, due now, with Attempt reset to 0. The reset gives the requeued job a fresh full budget; its next claim makes the Attempt 1 again. CancelJobAsync, covered above, cancels a non-terminal job.

This is the deliberate seam in the lifecycle. Automatic transitions carry a job toward a terminal state, and a human decides whether a job that landed in DeadLettered or Quarantined gets another run, typically after fixing the bug or deploying the missing handler.

Where to go next#

• Execution Guarantee: the Leased state, heartbeat renewal, lease expiry, and the (workerId, attempt) fence in depth.
• Retries and Error Handling: configuring RetryPolicy, backoff, and what sends a job to Dead-Lettered.
• Dependencies: AwaitingParent, ParentsRemaining, on-success versus any-terminal, and the cancellation cascade.
• Cancel a Job: cooperative cancellation through the CancellationToken and the heartbeat-delivered flag.
• Transition Observer Internals: the Transition Log and at-least-once observer delivery end to end.