Health, Fail-Stop & Draining

Running in production: fail-stop Worker Group semantics, the health check that goes red without crashing the host, lease inheritance, and graceful draining.

In production a Worker Group has to make a choice every time a store operation goes wrong: keep going or stop. BackWave draws a hard line between a transient fault it can ride out and an invariant it cannot. A transient fault leaves the group running but degraded; an unclassifiable fault fail-stops the group, halting its pumps for good while leaving the host process and every sibling group untouched. Health surfaces through a public state object and a ready-made health check you wire into your endpoint, so an orchestrator can see red without the process ever crashing. When a node dies, its in-flight Leases lapse and other nodes inherit the work. When a node stops cleanly, it hands its Leases back so other nodes pick the work up at once.

Fail-stop Worker Group semantics#

Each Pump runs a claim → dispatch → report cycle. When a store operation throws, BackWave classifies the fault into one of two outcomes.

This is fail-closed on the unknown. An exception BackWave cannot positively identify as transient halts the pump rather than being retried silently against a store that may be in an undefined state.

What a fail-stop does#

When a Pump fail-stops it logs a single Critical entry (event 2001) naming the group and the invariant trigger, then records the halt in health state. Every always-on check has a name from the public InvariantTrigger enum, and the exception it throws, InvariantViolationException, carries that name in its Trigger property. A halt that no named check raised reads unclassified. The same counter, backwave.invariant.violations, counts every named violation with its trigger and whether it halted or degraded. It stops claiming, stops reporting outcomes, and stops heartbeating. Every in-flight job's CancellationToken is signalled so cooperative handlers can unwind, but no outcomes are reported; the in-flight Leases simply lapse, and healthy nodes inherit that work (see Lease inheritance below). The poll/heartbeat ticker throwing is treated the same way: the group fail-stops loudly rather than idling forever with no polls.

A fail-stop is permanent for that group. Nothing in the runtime un-halts it. Recovery means fixing the underlying invariant and restarting the process.

The host stays up#

A .NET background service that throws normally takes the whole host down. BackWave's pump catches every fault and converts it into logged output plus recorded health state, so that default never fires. One group fail-stopping never takes down the host, the HTTP server, or any sibling Worker Group. The process keeps serving; only the affected group's pumps stop.

Transient faults and recovery#

A transient fault logs a Warning and marks that pump degraded. The poll interval is the backoff cadence; a skipped cycle costs latency only, never correctness, because polling is the sole correctness mechanism. The next clean cycle clears that pump's degraded mark. Degraded is a running state: the group keeps claiming and executing throughout.

Three invariant triggers degrade instead of halting: OutcomeFenceRejected, ObserverReportFenceRejected, and ObserverCursorRegressed. The Pump raises OutcomeFenceRejected and logs Warning 2003. The storage adapter raises the other two and logs Warning 1601. Each names the trigger and leaves the group running. The health check does not change. See Always-On Assertions & Fail-Stop for the full trigger list.

Whole-group versus partial halts#

A Worker Group runs Pumps independent pumps (default 1). Health is tracked per pump but reported at group altitude:

• A group counts as wholly halted only once every one of its pumps has halted.
• Until then it is partially halted: surviving pumps keep claiming and executing, and their Leases inherit the halted pump's work.

With the default Pumps = 1, any halt is a whole-group halt, which is the common case. The partial-halt distinction only matters once you raise Pumps; treat it as an advanced concern and defer pump tuning to Scaling & Throughput Tuning.

A halt supersedes that pump's degraded mark. Degraded and recovery marks only ever touch the calling pump's own state, never a sibling's.

Inspecting health programmatically#

AddBackWave automatically registers a public singleton, BackWaveHealth, in the BackWave.Hosting namespace. Resolve it from DI to read live health state.

HaltState is a public record carrying ExceptionType (the exception's full type name), Message, and Trigger (an InvariantTrigger?, null when no named check raised the halt). Its ToString() renders "ExceptionType: Message", the full type name rather than a friendly label, which is what an operator sees.

var health = app.Services.GetRequiredService<BackWaveHealth>();
 
if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}

To watch the same state live in the dashboard, including the Executing screen with Lease-expiry countdowns and the Overview health rollup, see The Dashboard.

The health check that goes red without crashing#

AddBackWave also registers BackWaveHealthCheck, a public IHealthCheck in the BackWave.Hosting namespace, as a singleton. You do not register the check type yourself; you only wire it into the health-check pipeline.

builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

The check reads in-memory state and never blocks. It reports:

• Healthy ("All Worker Groups running.") while no group is halted or degraded.
• Degraded when some pumps of a group have halted, or a group is on a transient store fault. The message begins "Worker Group degraded:".
• Unhealthy once a group has wholly halted. The message begins "Worker Group fail-stop:", for example "Worker Group fail-stop: reports (BackWave.Diagnostics.InvariantViolationException)".

Each description names the affected groups and the exception type only. The endpoint goes red only on a wholly-halted group, because ASP.NET maps Degraded to 200 OK by default. If your probe treats anything other than Healthy as a failure, a transient fault or a partial halt will trip it. DegradedGroups and PartiallyHaltedGroups carry the detail for diagnostics.

Because the host process stays up and keeps serving HTTP, a fail-stop shows as a red /health endpoint that an orchestrator, load balancer, or pager can act on, while the process itself remains alive to report it.

Lease inheritance when a node dies#

There is no takeover API. Inheritance is an emergent property of the Lease and its expiry. When a node dies, the sequence is:

1. A claim takes a Lease and counts the Attempt. Claiming leases a due job to exactly one Worker and increments its Attempt. The job stays invisible to other claimers until its Lease lapses or it reaches a terminal state.
2. Heartbeats renew the Lease. While a job runs, the owning Pump heartbeats. A node that no longer holds a job — because its Lease lapsed — is told to stop applying that job's effects.
3. A dead node stops heartbeating. After LeaseDuration elapses with no renewal, the Lease lapses and the job becomes claimable again.
4. The maintenance sweep disposes the lapsed Lease. Because the original claim already counted the Attempt, the expired job is either rescheduled at its retry backoff instant or, once it reaches its attempt ceiling, Dead-Lettered. Each expired Lease is disposed exactly once even when several nodes sweep at the same time, and the sweep is scoped to the caller's served Queues so the job's own retry policy governs regardless of which node sweeps it.
5. Another node re-claims it on its next poll once it is due again, and the handler runs again.

This is At-Least-Once Execution. A Lease that lapsed because a node died is a failed Attempt, so the recovered run is retried like any other failed Attempt: the same backoff schedule and the same attempt ceiling. If the job has already exhausted its ceiling, the sweep Dead-Letters it instead of rescheduling.

Why running twice stays safe#

A job may briefly execute on two nodes after inheritance, the slow original and the new claimant. Only one of them can record a terminal outcome. Every outcome write is fenced to the exact Worker and Attempt that holds the live Lease, so a late report from a node that was isolated past its Lease expiry changes nothing: it is rejected as stale. The terminal state, the dependency latch decrement, and the Concurrency-Limit slot release all flow from that single fenced write. This is the Effect-Once property: At-Least-Once execution with exactly-once effect. Idempotency of the handler body itself remains the handler author's responsibility.

Cadence you control#

All of these live on WorkerGroupOptions and set the timing of inheritance:

Tuning these for throughput and recovery latency belongs to Scaling & Throughput Tuning. Here they matter only as the timing of inheritance.

Graceful draining on shutdown#

When the host shuts down it cancels the Pump's stopping token, and the Pump:

1. Stops claiming. No new claims, no new polls.
2. Waits for running handlers. Each Pump gets ShutdownBudget (default 5 seconds) for this step and the two that follow. One fifth of the budget is held back for step 4, on its own clock, so a handler that ignores cancellation cannot starve the hand-back.
3. Reports its buffered outcomes. Work that finished inside the budget is recorded, not re-run.
4. Hands its remaining Leases back. Every Lease the Pump still holds is relinquished, and the count is logged at Information (event 1205) when it is above zero. The job returns to Scheduled at "now" with its Attempt unchanged, so another node picks it up on its next poll instead of after the Lease lapses. A job already at its Attempt ceiling is Dead-Lettered instead.

When the budget runs out#

If handlers are still running when ShutdownBudget ends, the Pump hands their Leases back anyway and logs Warning 1207. Those jobs re-run elsewhere under At-Least-Once Execution. A handler that ignores cancellation keeps running until it finishes on its own or the host's shutdown timeout tears the process down. If the store cannot complete the hand-back, the Pump logs Warning 1206 and the Leases lapse as they do after a crash.

The effective budget is clamped to four fifths of the standard .NET HostOptions.ShutdownTimeout (default 30 seconds), so raise the host timeout before you raise a budget past 24 seconds. Set ShutdownBudget to TimeSpan.Zero to skip the hand-back and let Leases lapse.

builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(30));

Shutdown cancellation is not an operator cancel#

Three sources of cancellation produce deliberately different outcomes:

Wiring it together#

A minimal end-to-end registration that touches every option this page discusses:

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
        PoolSize = 16,
        LeaseDuration = TimeSpan.FromSeconds(60),
        // HeartbeatInterval defaults to LeaseDuration / 3
        // MaintenanceInterval defaults to 5s
    }));
 
builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

The WorkerGroupOptions fields that bear on health, inheritance, and draining:

Where to go next#

• The Dashboard: watch Lease-expiry countdowns and the health rollup live, and requeue Dead-Lettered work or cancel jobs — including the difference between an operator cancel and a shutdown cancellation.
• Scaling & Throughput Tuning: tune Pumps, PoolSize, and the Lease/heartbeat/maintenance intervals for throughput and recovery latency.

Found a problem on this page? Report an issue