Job States & Transitions

The canonical enumeration of every job state, the full transition table with triggers, and the recorded terminal causes.

A job is always in exactly one of seven states. Three are non-terminal, meaning the job still has work ahead of it, and four are terminal, meaning it has settled and will not move again on its own. This page is the canonical reference for that state machine: the enum itself, where the current state lives on the job row, every legal transition and the event that drives it, and the exact strings recorded as the cause when a job goes terminal. The Job Lifecycle page tells the same story as narrative; this page is the enumeration.

The seven states#

The state is the JobState enum. Its members, in declaration order, are the persisted ordinals, so the order is load-bearing for stored data and never changes.

public enum JobState
{
    Scheduled,       // 0
    AwaitingParent,  // 1
    Leased,          // 2
    Succeeded,       // 3
    Cancelled,       // 4
    DeadLettered,    // 5
    Quarantined,     // 6
}

The enum member is spelled DeadLettered, one word. In prose the domain term is written Dead-Lettered.

Leased is the only in-flight state; a job is only ever running while it is Leased. Two terminal states are easy to confuse. A job is DeadLettered when it ran and kept failing until it used up its attempt ceiling. A job is Quarantined when it could not be routed at all, because its Wire Name has no registered handler or its payload no longer deserializes. Dead-lettering is a failure of the work; quarantine is a failure to even start it.

Terminal versus non-terminal#

Terminality is available as an extension on the enum:

public static bool IsTerminal(this JobState state);

It returns true for Succeeded, Cancelled, DeadLettered, and Quarantined, and false otherwise. The guarantee behind it is absolute: a terminal job never transitions again on its own. Only an explicit operator action, such as a requeue, can move it, and only from two of the four terminal states. Everything else in the table below is an automatic transition BackWave applies for you.

Where the state lives on the job row#

The current state and the facts that go with it live on the JobRecord. These fields are how you read a job's position in the state machine.

TerminalAt and TerminalCause are null while the job is live and set on the terminal transition, with one exception: a Succeeded job stamps TerminalAt but leaves TerminalCause null. LeaseOwner and LeaseExpiry are non-null only while Leased.

The transition table#

Every legal transition, the event that triggers it, and its effect on Attempt. A job enters the machine from (none) at enqueue or when a schedule mints an instance, moves through the non-terminal states, and settles in a terminal state. Each state-changing operation also appends one entry to the job's Transition Log.

A few points the table compresses:

• The claim raises the Attempt. A fresh job sits at Attempt 0, and the first claim makes it 1, which is the number the first real attempt runs under. A failure does not raise the Attempt again; the claim already did. A lease expiry disposes an attempt that was already counted at claim, so it does not re-increment either.
• Leased has two paths to Scheduled and two to DeadLettered. One pair is driven by a reported outcome, the other by lease expiry when a worker crashes or is cut off. The expiry pair is the crash-recovery path, using the same retry-or-dead-letter decision as a reported failure.
• Reported outcomes are fenced. A reported outcome applies only when the caller still holds the live lease for exactly that attempt: the state must be Leased, the lease owner and attempt number must match, and the lease must not have expired. A stale write changes nothing. This is what makes the recorded outcome exactly-once even though a handler body can run more than once.
• Terminal transitions cascade. Any transition that sends a job to a terminal state resolves the latches of its waiting children in the same atomic step, which may release or cancel them, recursively.
• Succeeded is where output is stored. Job Output is persisted atomically with the transition to Succeeded and only there. Output over the size limit is rejected rather than truncated.
• Requeue is the only terminal-to-active edge. It accepts only DeadLettered or Quarantined. A requeue returns the job to Scheduled, due now, with Attempt reset to 0, clearing the lease, the cancel flag, TerminalAt, and TerminalCause. Succeeded and Cancelled are not requeueable.

Whether the parent gate releases a child to Scheduled or cancels it is set by the job's DependencyMode: OnSuccess (the default) releases only if every parent Succeeded and cancels the child if any parent reaches another terminal state, while OnAnyTerminal releases once every parent is terminal whatever the outcome. The Dependencies page owns the full model.

Retry and the attempt ceiling#

Whether a failed Leased job goes back to Scheduled or on to DeadLettered is decided above the store by the RetryPolicy, then handed to the store as a plain next-due-time that is either set (retry) or null (dead-letter). The store never runs policy code.

public sealed record RetryPolicy
{
    public int MaxAttempts { get; init; } = 10;
    public Func<int, TimeSpan> Backoff { get; init; } = DefaultBackoff;
 
    public static RetryPolicy Default { get; }
 
    public static TimeSpan DefaultBackoff(int attempt)
        => TimeSpan.FromSeconds(Math.Min(Math.Pow(2, attempt), 300));
}

MaxAttempts defaults to 10, so a job dead-letters when its 10th attempt fails. The default backoff is two to the power of the attempt number in seconds, capped at five minutes. Configuring retries is covered on the Retries and Error Handling guide.

Recorded terminal causes#

When a job goes terminal, TerminalCause records one short reason on the job row. It is not the same thing as Failure Detail: TerminalCause is why the terminal state was reached, a single string on the job; Failure Detail is the full per-attempt exception text captured separately on a Transition Log entry. The recorded value depends on the terminal state and the path taken.

For a parent-failure cancel, the interpolated {parentState} is the failing parent's JobState name, so the concrete recorded values are parent-failure:DeadLettered, parent-failure:Cancelled, and parent-failure:Quarantined. A succeeded parent satisfies the gate and never produces a cancel. This cause is recorded both when the failure is already visible at enqueue and when a parent fails later while the child waits.

Only an operator-requested cancel of a running job produces a Cancelled terminal. A handler's own OperationCanceledException, such as an HttpClient timeout, is reported as an ordinary failure and retries like any other.

Terminal retention classes#

Terminal jobs are retained and purged by class, not one state at a time. TerminalStateClass groups the four terminal states into two, each purged on its own keep window.

public enum TerminalStateClass
{
    SucceededOrCancelled,
    DeadLetteredOrQuarantined,
}

Each class has its own keep window and is purged independently. A job's Transition Log is deleted with the job.

Operator-action results#

The operator actions that touch the state machine return a result telling you what the machine did, so a caller never has to re-read the job to find out whether anything changed.

Cancelling a running job is always cooperative: CancelResult.CancellationRequested sets the flag, and the job reaches Cancelled only once the handler observes its CancellationToken and returns. The full cancellation path is on the Cancel a Running Job guide.

Where to go next#

• Job Lifecycle: the same state machine as narrative, with the retry loop, crash recovery, and the Transition Log.
• Dependencies: AwaitingParent, ParentsRemaining, on-success versus any-terminal, and the cancellation cascade.
• Execution Guarantee: the Leased state, heartbeat renewal, lease expiry, and the lease fence in depth.
• Retries and Error Handling: configuring RetryPolicy and what sends a job to Dead-Lettered.
• Cancel a Running Job: cooperative cancellation through the CancellationToken and the heartbeat-delivered flag.
• Limits & Defaults: the default MaxAttempts, backoff cap, and other defaults in one place.

Found a problem on this page? Report an issue