Health, Fail-Stop & Draining

Running in production: fail-stop Worker Group semantics, the health check that goes red without crashing the host, lease inheritance, and graceful draining.

In production a Worker Group has to make a choice every time a store operation goes wrong: keep going or stop. BackWave draws a hard line between a transient fault it can ride out and an invariant it cannot. A transient fault leaves the group running but degraded; an unclassifiable fault fail-stops the group, halting its pumps for good while leaving the host process and every sibling group untouched. Health surfaces through a public state object and a ready-made health check you wire into your endpoint, so an orchestrator can see red without the process ever crashing. When a node dies or shuts down, its in-flight Leases simply lapse and other nodes inherit the work.

Fail-stop Worker Group semantics#

Each Pump runs a claim → dispatch → report cycle. When a store operation throws, BackWave classifies the fault into one of two outcomes.

This is fail-closed on the unknown. An exception BackWave cannot positively identify as transient halts the pump rather than being retried silently against a store that may be in an undefined state.

What a fail-stop does#

When a Pump fail-stops it logs a single Critical entry naming the group and the cause, then records the halt in health state. It stops claiming, stops reporting outcomes, and stops heartbeating. Every in-flight job's CancellationToken is signalled so cooperative handlers can unwind, but no outcomes are reported; the in-flight Leases simply lapse, and healthy nodes inherit that work (see Lease inheritance below). The poll/heartbeat ticker throwing is treated the same way: the group fail-stops loudly rather than idling forever with no polls.

A fail-stop is permanent for that group. Nothing in the runtime un-halts it. Recovery means fixing the underlying invariant and restarting the process.

The host stays up#

A .NET background service that throws normally takes the whole host down. BackWave's pump catches every fault and converts it into logged output plus recorded health state, so that default never fires. One group fail-stopping never takes down the host, the HTTP server, or any sibling Worker Group. The process keeps serving; only the affected group's pumps stop.

Transient faults and recovery#

A transient fault logs a Warning and marks that pump degraded. The poll interval is the backoff cadence; a skipped cycle costs latency only, never correctness, because polling is the sole correctness mechanism. The next clean cycle clears that pump's degraded mark. Degraded is a running state: the group keeps claiming and executing throughout.

Whole-group versus partial halts#

A Worker Group runs Pumps independent pumps (default 1). Health is tracked per pump but reported at group altitude:

• A group counts as wholly halted only once every one of its pumps has halted.
• Until then it is partially halted: surviving pumps keep claiming and executing, and their Leases inherit the halted pump's work.

With the default Pumps = 1, any halt is a whole-group halt, which is the common case. The partial-halt distinction only matters once you raise Pumps; treat it as an advanced concern and defer pump tuning to Scaling & Throughput Tuning.

A halt supersedes that pump's degraded mark. Degraded and recovery marks only ever touch the calling pump's own state, never a sibling's.

Inspecting health programmatically#

AddBackWave automatically registers a public singleton, BackWaveHealth, in the BackWave.Hosting namespace. Resolve it from DI to read live health state.

HaltState is a public record carrying ExceptionType (the exception's full type name) and Message. Its ToString() renders "ExceptionType: Message", the full type name rather than a friendly label, which is what an operator sees.

var health = app.Services.GetRequiredService<BackWaveHealth>();
 
if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}

To watch the same state live in the dashboard, including the Executing screen with Lease-expiry countdowns and the Overview health rollup, see The Dashboard.

The health check that goes red without crashing#

AddBackWave also registers BackWaveHealthCheck, a public IHealthCheck in the BackWave.Hosting namespace, as a singleton. You do not register the check type yourself; you only wire it into the health-check pipeline.

builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

The check reads in-memory state and never blocks. It reports:

• Healthy ("All Worker Groups running.") while IsHealthy is true.
• Unhealthy otherwise, with a message listing each wholly-halted group and its cause, for example "Worker Group fail-stop: reports (InvalidOperationException: ...)".

The key threshold: the endpoint goes red only on a wholly-halted group. A transient fault (degraded) and a partial halt (some pumps still serving) do not turn the endpoint red — they are visible through DegradedGroups and PartiallyHaltedGroups for diagnostics. A connection blip never pages anyone.

Because the host process stays up and keeps serving HTTP, a fail-stop shows as a red /health endpoint that an orchestrator, load balancer, or pager can act on, while the process itself remains alive to report it.

Lease inheritance when a node dies#

There is no takeover API. Inheritance is an emergent property of the Lease and its expiry. When a node dies, the sequence is:

1. A claim takes a Lease and counts the Attempt. Claiming leases a due job to exactly one Worker and increments its Attempt. The job stays invisible to other claimers until its Lease lapses or it reaches a terminal state.
2. Heartbeats renew the Lease. While a job runs, the owning Pump heartbeats. A node that no longer holds a job — because its Lease lapsed — is told to stop applying that job's effects.
3. A dead node stops heartbeating. After LeaseDuration elapses with no renewal, the Lease lapses and the job becomes claimable again.
4. The maintenance sweep disposes the lapsed Lease. Because the original claim already counted the Attempt, the expired job is either rescheduled at its retry backoff instant or, once it reaches its attempt ceiling, Dead-Lettered. Each expired Lease is disposed exactly once even when several nodes sweep at the same time, and the sweep is scoped to the caller's served Queues so the job's own retry policy governs regardless of which node sweeps it.
5. Another node re-claims it on its next poll once it is due again, and the handler runs again.

This is At-Least-Once Execution. A Lease that lapsed because a node died is a failed Attempt, so the recovered run is retried like any other failed Attempt: the same backoff schedule and the same attempt ceiling. If the job has already exhausted its ceiling, the sweep Dead-Letters it instead of rescheduling.

Why running twice stays safe#

A job may briefly execute on two nodes after inheritance, the slow original and the new claimant. Only one of them can record a terminal outcome. Every outcome write is fenced to the exact Worker and Attempt that holds the live Lease, so a late report from a node that was isolated past its Lease expiry changes nothing: it is rejected as stale. The terminal state, the dependency latch decrement, and the Concurrency-Limit slot release all flow from that single fenced write. This is the Effect-Once property: At-Least-Once execution with exactly-once effect. Idempotency of the handler body itself remains the handler author's responsibility.

Cadence you control#

All of these live on WorkerGroupOptions and set the timing of inheritance:

Tuning these for throughput and recovery latency belongs to Scaling & Throughput Tuning. Here they matter only as the timing of inheritance.

Graceful draining on shutdown#

There is no BackWave-specific drain timeout. Draining rides entirely on the standard .NET Generic Host shutdown of each Pump. When the host shuts down it cancels the Pump's stopping token, and the Pump:

1. Stops claiming. The event loop ends and the Pump returns. No new claims, no new polls.
2. Signals in-flight handlers to cancel. Every in-flight job's CancellationToken is cancelled. Each handler runs with a token linked to the shutdown token, so a cooperative handler that observes its token stops promptly.
3. Reports nothing for cancelled in-flight work. A cancellation caused by shutdown yields no outcome. The Lease simply lapses and another node inherits the job per the section above.

Draining releases work, it does not finish it#

BackWave's drain releases in-flight work rather than blocking shutdown to finish it. The Pump does not wait for in-flight handlers to complete and commit outcomes. The consequences:

• A cooperative handler that honors its CancellationToken unwinds quickly; its job's Lease lapses and another node re-runs it under At-Least-Once Execution.
• A handler that ignores cancellation keeps running until it finishes on its own or the host's shutdown timeout forcibly tears the process down.

The lever for giving handlers more time to wind down is the standard .NET HostOptions.ShutdownTimeout (default 30 seconds), a host-level setting rather than a BackWave option.

builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(30));

Shutdown cancellation is not an operator cancel#

Three sources of cancellation produce deliberately different outcomes:

Wiring it together#

A minimal end-to-end registration that touches every option this page discusses:

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
        PoolSize = 16,
        LeaseDuration = TimeSpan.FromSeconds(60),
        // HeartbeatInterval defaults to LeaseDuration / 3
        // MaintenanceInterval defaults to 5s
    }));
 
builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

The WorkerGroupOptions fields that bear on health, inheritance, and draining:

Where to go next#

• The Dashboard: watch Lease-expiry countdowns and the health rollup live, and requeue Dead-Lettered work or cancel jobs — including the difference between an operator cancel and a shutdown cancellation.
• Scaling & Throughput Tuning: tune Pumps, PoolSize, and the Lease/heartbeat/maintenance intervals for throughput and recovery latency.