Retries & Error Handling

Set a Worker Group’s retry policy, understand dead-lettering versus quarantine, and write handlers that are safe under at-least-once execution.

Set a Worker Group’s retry policy, understand dead-lettering versus quarantine, and write handlers that are safe under at-least-once execution.

BackWave runs your Handler under At-Least-Once Execution, which means two things you have to plan for. A Handler that throws gets tried again, on a backoff, until it runs out of budget. And a Handler body can run more than once for the same Job, so it has to be safe to repeat. This guide covers both: how to shape the Retry Policy on a Worker Group, what happens when a Job finally gives up, and how to write Handlers that survive a second run.

There are two distinct ways a Job stops trying, and they land in two different terminal states. A Job that ran and kept failing becomes Dead-Lettered. A Job that could not be routed to a Handler at all becomes Quarantined. They look similar in the dashboard, but they have opposite causes and opposite fixes, so keep them separate in your head from the start.

Where retries are configured#

The Retry Policy lives on the Worker Group, not on the Job type and not on the enqueue call. The group that claims a Job decides how many times it runs and how long it waits between tries. Two groups serving different queues can carry different policies, and the same Job type inherits whichever group picks it up.

Program.cs
backwave.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "strict-priority",
    Policy = new DispatchPolicy.Strict(["critical", "bulk", "limited"]),
    PollInterval = TimeSpan.FromMilliseconds(250),
    RetryPolicy = new RetryPolicy { MaxAttempts = 5, Backoff = _ => TimeSpan.FromMilliseconds(500) },
});
 
backwave.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "weighted-fair",
    Policy = new DispatchPolicy.Weighted([("high", 6), ("low", 1)]),
    PollInterval = TimeSpan.FromMilliseconds(250),
    RetryPolicy = new RetryPolicy { MaxAttempts = 3, Backoff = _ => TimeSpan.FromMilliseconds(500) },
});

RetryPolicy has exactly two knobs. MaxAttempts is an int, and Backoff is a Func<int, TimeSpan> that takes the number of the attempt that just failed and returns the delay before the next one. Both examples above use a fixed delay by ignoring the attempt argument (_ => TimeSpan.FromMilliseconds(500)), which is the idiomatic shape for a constant backoff. There is no separate MaxRetries, no RetryDelay, and no enum to pick "linear versus exponential". You supply the curve as a delegate.

If you leave RetryPolicy off the group, it defaults to RetryPolicy.Default.

MaxAttempts is a ceiling on attempts, not retries#

This is the one number people get wrong, so it is worth nailing down. MaxAttempts counts total executions, not extra tries after the first. With MaxAttempts = 3, the Handler runs at most three times (Attempts 1, 2, 3), and the third failure dead-letters the Job. With MaxAttempts = 1, there are no retries at all: the first failure dead-letters immediately. Do not read it as "number of retries", or every count will be off by one.

The default backoff curve#

RetryPolicy.Default allows 10 attempts and uses an exponential backoff: 2^attempt seconds, capped at 300 seconds. The cap matters once you are several failures deep, because without it the delays would run away.

failed attempt   delay before next
1                2 s
2                4 s
3                8 s
4                16 s
5                32 s
6                64 s
7                128 s
8                256 s
9                300 s (5-minute cap)

With the default 10 attempts, attempts 1 through 9 retry on the schedule above, and the 10th failure dead-letters. The backoff is pure powers of two with a ceiling. There is no randomization, so do not describe it as backoff with jitter.

One constraint on any Backoff you write: keep it a pure function of its argument. When a Worker's Lease expires, the store, not your process, has to schedule the next attempt, and the store cannot run a C# delegate. So BackWave evaluates your Backoff once per attempt up front and hands the store a precomputed table of delays. A Backoff that reads the clock or returns different values for the same input would make lease-expiry retries diverge from in-process-failure retries. Same input, same delay, every time.

How attempts are counted#

The Attempt number is 1-based and increments when a Worker claims the Job, not when the Handler fails. The first run executes as Attempt 1, and the Handler reads that number from its Job Context.

SampleJobs.cs
[Job("flaky", Queue = "low")]
public Task FlakyAsync(string label, JobContext context, CancellationToken cancellationToken)
{
    logger.LogWarning("flaky: {Label} failing on attempt {Attempt}", label, context.Attempt);
    throw new InvalidOperationException($"flaky '{label}' always fails (attempt {context.Attempt})");
}

When a Handler throws, BackWave catches the exception at the execution boundary and turns it into a failed Attempt. The Retry Policy then decides: schedule the next Attempt at the backoff instant, or, if the attempt ceiling is reached, mark the Job Dead-Lettered. The terminal cause records the last exception's message, and the full exception text (type, message, and stack) is captured as Failure Detail on the failing entry of the Transition Log. The Core never reads that text; it is diagnostic only, viewing it in the dashboard is gated behind a permission, and it can be turned off entirely on PII-sensitive hosts.

A Handler that recovers on a later try succeeds normally. Because the Attempt number is visible, a Handler can branch on it.

ChargeCardHandler.cs
public Task HandleAsync(ChargeCard job, JobContext context, CancellationToken cancellationToken)
{
    if (context.Attempt >= recorder.SucceedOnAttempt)
        return Task.CompletedTask;
    throw new InvalidOperationException($"gateway timeout (attempt {context.Attempt})");
}

A lapsed lease is a failed attempt#

A Worker claims a Job under a Lease, a time-bounded hold (60 seconds by default) that it renews with a heartbeat while it works. If the Worker dies, stalls, or runs past the Lease without heartbeating, the Lease expires and the Job becomes claimable again. That expiry counts as a failed Attempt, exactly like a thrown exception. It consumes one unit of attempt budget, and the same Retry Policy applies: retry at the backoff instant, or dead-letter if the ceiling is reached.

This is the practical face of At-Least-Once Execution. A Handler that hangs for longer than the Lease duration burns an attempt even though it never threw, and the Job may be picked up and run again by another Worker. Size your Lease duration above your real worst-case Handler runtime, and make sure long Handlers heartbeat.

One related case to know: if your Handler's own code raises an OperationCanceledException that nobody requested, an HttpClient timeout, say, or an internal deadline, BackWave treats it as a plain failure that retries. It does not become the terminal Cancelled state. Only an operator cancel produces Cancelled. A host-shutdown cancellation reports nothing at all: the Lease lapses and another node inherits the Job.

Dead-lettered versus quarantined#

Both are terminal, both set a Job aside for inspection, and both are recoverable only by an operator. The difference is whether the Handler ever ran.

A Dead-Lettered Job ran and kept failing. It exhausted its attempt budget through thrown exceptions or lapsed Leases, and its terminal cause is the last failure message. This is the end of the retry path described above.

A Quarantined Job never ran. On first claim, BackWave could not route it to a Handler, so it went straight to terminal without ever invoking your code. There are two triggers, both deployment problems rather than runtime failures:

  • The Job's Wire Name has no registered Handler. You enqueued a type the worker process does not know about, or removed a Handler that still has Jobs in flight.
  • The stored payload no longer decodes. The payload shape drifted and the bytes on disk no longer deserialize into the registered type.

Quarantine is deliberate and loud. Neither case retries, because retrying a missing handler or an undecodable payload would spin a tight failure loop that never resolves on its own. So deploy drift surfaces immediately as a single Quarantined Job you can see, rather than as a retry storm. The Attempt shows 1 because the claim counted it, but the Handler body never executed. If you are evolving payload shapes, Evolve job payloads covers how to avoid the decode failure in the first place.

Writing handlers that survive re-execution#

Retries, lease expiry, and redelivery all mean the same thing for your code: the Handler body may run more than once for a single Job. BackWave guarantees Effect-Once for the recorded outcome and the state transitions that follow from it, enforced by a fence on (workerId, attempt), so a Job cannot be marked succeeded twice. It does not guarantee your side effects run once. That part is on you.

The Attempt number is the natural key to make a side effect idempotent. Combine it with the Job id and you have a stable dedup key you can write before doing irreversible work.

ChargeCardHandler.cs
public async Task HandleAsync(ChargeCard job, JobContext context, CancellationToken cancellationToken)
{
    var idempotencyKey = $"charge:{context.JobId}";
    if (await ledger.AlreadyCharged(idempotencyKey, cancellationToken))
        return; // a prior attempt already committed this side effect
 
    await gateway.ChargeAsync(job.CardToken, job.Amount, idempotencyKey, cancellationToken);
    await ledger.RecordCharge(idempotencyKey, cancellationToken);
}

BackWave gives every Attempt a fresh DI scope, so a scoped dependency like a DbContext backing a dedup table resolves cleanly per run. Lean on that. The general rule: write Handlers so that running them twice produces the same end state as running them once. If a step is genuinely not repeatable, guard it behind a durable check keyed on the Job id.

Recovering a terminal job#

A Dead-Lettered or Quarantined Job stays queryable so an operator can inspect it. Requeue is the recovery path. It resets the state to Scheduled, sets the due time to now, and resets the attempt budget to zero, so a requeued Job gets a fresh full set of attempts and re-runs from scratch.

Program.cs
ops.MapPost("/jobs/{id:guid}/requeue", async (BackWaveOperator op, Guid id) =>
    Results.Ok(new { result = (await op.RequeueAsync(id, actor)).ToString() }))
    .WithSummary("Requeue a Dead-Lettered or Quarantined job.");

Requeue only works on Dead-Lettered or Quarantined Jobs. Call it on anything else and it returns NotRequeueable and changes nothing. It also does not resume from a partial point: it re-runs the Handler from the top under At-Least-Once, which is another reason your Handler needs to be safe to repeat. And for a Quarantined Job, requeue before you fix the root cause re-quarantines it. Deploy the missing Handler or correct the payload first, then requeue.

By default a terminal Job stays queryable for 14 days before retention sweeps it away. Inspect and recover within that window.

What the retry policy does not do#

A few things are deliberately absent, and knowing the boundary saves you from looking for an API that is not there.

  • There is no per-Job-type or per-enqueue retry override. The [Job] attribute carries only the Wire Name, Queue, and labels, and the enqueue call takes no retry argument. Retries are a Worker Group setting, full stop.
  • There is no allow-list or deny-list of exception types, and no "fail fast on this exception" switch. Every thrown exception is a failed Attempt and follows the same backoff. The one structural exception is the unrequested OperationCanceledException, which still retries like any other failure.
  • There is no retry callback, OnRetry, or OnFailure hook, and no way for a Handler to tell BackWave "do not retry this one" from inside its body. The Handler signals success by completing its Task and failure by throwing. There is no return value or result type. To react to a terminal outcome, observe the transition externally, as in React to job outcomes; an observer can notify but cannot veto or alter the transition.
  • Dead-Lettered is a terminal state on the same Job row, not a separate queue you dequeue from. You inspect and requeue it through the Monitor and operator actions, not by reading a dead-letter topic.

Where to go next#

Previous← Guard Wire CompatibilityNextSchedule Recurring Jobs →