# Health, Fail-Stop & Draining

In production a Worker Group has to make a choice every time a store operation goes wrong: keep going or stop. BackWave draws a hard line between a transient fault it can ride out and an invariant it cannot. A transient fault leaves the group running but degraded; an unclassifiable fault fail-stops the group, halting its pumps for good while leaving the host process and every sibling group untouched. Health surfaces through a public state object and a ready-made health check you wire into your endpoint, so an orchestrator can see red without the process ever crashing. When a node dies or shuts down, its in-flight Leases simply lapse and other nodes inherit the work.

## Fail-stop Worker Group semantics

Each Pump runs a claim → dispatch → report cycle. When a store operation throws, BackWave classifies the fault into one of two outcomes.

| Fault class | What it covers | Pump behavior |
|---|---|---|
| Transient | A `DbException` the adapter flags as transient (connection reset, failover, deadlock victim, command timeout), a `TimeoutException`, or any provider-specific transient fault the storage adapter recognizes. | Stays running, marks the pump **degraded**, retries on the next tick. |
| Anything else | An invariant violation, or any fault not classifiable as transient. | **Fail-stops**: the pump halts permanently. |

This is fail-closed on the unknown. An exception BackWave cannot positively identify as transient halts the pump rather than being retried silently against a store that may be in an undefined state.

### What a fail-stop does

When a Pump fail-stops it logs a single `Critical` entry naming the group and the cause, then records the halt in health state. It stops claiming, stops reporting outcomes, and stops heartbeating. Every in-flight job's `CancellationToken` is signalled so cooperative handlers can unwind, but **no outcomes are reported**; the in-flight Leases simply lapse, and healthy nodes inherit that work (see [Lease inheritance](#lease-inheritance-when-a-node-dies) below). The poll/heartbeat ticker throwing is treated the same way: the group fail-stops loudly rather than idling forever with no polls.

A fail-stop is permanent for that group. Nothing in the runtime un-halts it. Recovery means fixing the underlying invariant and restarting the process.

### The host stays up

A .NET background service that throws normally takes the whole host down. BackWave's pump catches every fault and converts it into logged output plus recorded health state, so that default never fires. One group fail-stopping never takes down the host, the HTTP server, or any sibling Worker Group. The process keeps serving; only the affected group's pumps stop.

### Transient faults and recovery

A transient fault logs a `Warning` and marks that pump degraded. The poll interval is the backoff cadence; a skipped cycle costs latency only, never correctness, because polling is the sole correctness mechanism. The next clean cycle clears that pump's degraded mark. Degraded is a running state: the group keeps claiming and executing throughout.

### Whole-group versus partial halts

A Worker Group runs `Pumps` independent pumps (default `1`). Health is tracked per pump but reported at group altitude:

- A group counts as **wholly halted** only once **every** one of its pumps has halted.
- Until then it is **partially halted**: surviving pumps keep claiming and executing, and their Leases inherit the halted pump's work.

With the default `Pumps = 1`, any halt is a whole-group halt, which is the common case. The partial-halt distinction only matters once you raise `Pumps`; treat it as an advanced concern and defer pump tuning to [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning).

A halt supersedes that pump's degraded mark. Degraded and recovery marks only ever touch the calling pump's own state, never a sibling's.

## Inspecting health programmatically

`AddBackWave` automatically registers a public singleton, `BackWaveHealth`, in the `BackWave.Hosting` namespace. Resolve it from DI to read live health state.

| Member | Type | Meaning |
|---|---|---|
| `IsHealthy` | `bool` | `true` while no group has **wholly** halted. Degraded and partially-halted do **not** flip it. |
| `HaltedGroups` | `IReadOnlyDictionary<string, HaltState>` | Wholly-halted groups (every pump down), keyed by group name, with one halting cause each. |
| `PartiallyHaltedGroups` | `IReadOnlyDictionary<string, HaltState>` | Groups with at least one halted pump but others still serving. Diagnostic only. |
| `DegradedGroups` | `IReadOnlyDictionary<string, string>` | Groups with at least one pump on a transient fault; still running and claiming. The value is a short description. |

`HaltState` is a public record carrying `ExceptionType` (the exception's full type name) and `Message`. Its `ToString()` renders `"ExceptionType: Message"`, the full type name rather than a friendly label, which is what an operator sees.

```csharp title="HealthInspection.cs"
var health = app.Services.GetRequiredService<BackWaveHealth>();

if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}
```

To watch the same state live in the dashboard, including the Executing screen with Lease-expiry countdowns and the Overview health rollup, see [The Dashboard](/docs/dashboard-operations/overview).

## The health check that goes red without crashing

`AddBackWave` also registers `BackWaveHealthCheck`, a public `IHealthCheck` in the `BackWave.Hosting` namespace, as a singleton. You do not register the check type yourself; you only wire it into the health-check pipeline.

```csharp title="Program.cs"
builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");

var app = builder.Build();
app.MapHealthChecks("/health");
```

The check reads in-memory state and never blocks. It reports:

- **Healthy** (`"All Worker Groups running."`) while `IsHealthy` is `true`.
- **Unhealthy** otherwise, with a message listing each wholly-halted group and its cause, for example `"Worker Group fail-stop: reports (InvalidOperationException: ...)"`.

The key threshold: the endpoint goes red **only on a wholly-halted group**. A transient fault (degraded) and a partial halt (some pumps still serving) do not turn the endpoint red — they are visible through `DegradedGroups` and `PartiallyHaltedGroups` for diagnostics. A connection blip never pages anyone.

Because the host process stays up and keeps serving HTTP, a fail-stop shows as a red `/health` endpoint that an orchestrator, load balancer, or pager can act on, while the process itself remains alive to report it.

## Lease inheritance when a node dies

There is no takeover API. Inheritance is an emergent property of the Lease and its expiry. When a node dies, the sequence is:

1. **A claim takes a Lease and counts the Attempt.** Claiming leases a due job to exactly one Worker and increments its Attempt. The job stays invisible to other claimers until its Lease lapses or it reaches a terminal state.
2. **Heartbeats renew the Lease.** While a job runs, the owning Pump heartbeats. A node that no longer holds a job — because its Lease lapsed — is told to stop applying that job's effects.
3. **A dead node stops heartbeating.** After `LeaseDuration` elapses with no renewal, the Lease lapses and the job becomes claimable again.
4. **The maintenance sweep disposes the lapsed Lease.** Because the original claim already counted the Attempt, the expired job is either rescheduled at its retry backoff instant or, once it reaches its attempt ceiling, Dead-Lettered. Each expired Lease is disposed exactly once even when several nodes sweep at the same time, and the sweep is scoped to the caller's served Queues so the job's own retry policy governs regardless of which node sweeps it.
5. **Another node re-claims it** on its next poll once it is due again, and the handler runs again.

This is At-Least-Once Execution. A Lease that lapsed because a node died *is* a failed Attempt, so the recovered run is retried like any other failed Attempt: the same backoff schedule and the same attempt ceiling. If the job has already exhausted its ceiling, the sweep Dead-Letters it instead of rescheduling.

### Why running twice stays safe

A job may briefly execute on two nodes after inheritance, the slow original and the new claimant. Only one of them can record a terminal outcome. Every outcome write is fenced to the exact Worker and Attempt that holds the live Lease, so a late report from a node that was isolated past its Lease expiry changes nothing: it is rejected as stale. The terminal state, the dependency latch decrement, and the Concurrency-Limit slot release all flow from that single fenced write. This is the Effect-Once property: At-Least-Once *execution* with exactly-once *effect*. Idempotency of the handler body itself remains the handler author's responsibility.

### Cadence you control

All of these live on `WorkerGroupOptions` and set the timing of inheritance:

| Option | Default | Role |
|---|---|---|
| `LeaseDuration` | `60s` | The detection window — how long an unrenewed Lease survives before lapsing, i.e. how long after a node dies before its work can be inherited. |
| `HeartbeatInterval` | `null` → `LeaseDuration / 3` | Renewal cadence; a Lease survives a couple of missed heartbeats before lapsing. |
| `MaintenanceInterval` | `5s` | How often the expiry sweep runs, separately from claim polling. A missed sweep only delays maintenance by this much and never affects correctness. |
| `RetryPolicy` | `RetryPolicy.Default` | The backoff schedule and the attempt ceiling that decide reschedule-versus-Dead-Letter for a lapsed Lease. |

> Tuning these for throughput and recovery latency belongs to [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning). Here they matter only as the timing of inheritance.

## Graceful draining on shutdown

There is no BackWave-specific drain timeout. Draining rides entirely on the standard .NET Generic Host shutdown of each Pump. When the host shuts down it cancels the Pump's stopping token, and the Pump:

1. **Stops claiming.** The event loop ends and the Pump returns. No new claims, no new polls.
2. **Signals in-flight handlers to cancel.** Every in-flight job's `CancellationToken` is cancelled. Each handler runs with a token linked to the shutdown token, so a cooperative handler that observes its token stops promptly.
3. **Reports nothing for cancelled in-flight work.** A cancellation caused by shutdown yields no outcome. The Lease simply lapses and another node inherits the job per the section above.

### Draining releases work, it does not finish it

BackWave's drain **releases** in-flight work rather than blocking shutdown to finish it. The Pump does not wait for in-flight handlers to complete and commit outcomes. The consequences:

- A **cooperative** handler that honors its `CancellationToken` unwinds quickly; its job's Lease lapses and another node re-runs it under At-Least-Once Execution.
- A handler that **ignores cancellation** keeps running until it finishes on its own or the host's shutdown timeout forcibly tears the process down.

The lever for giving handlers more time to wind down is the standard .NET `HostOptions.ShutdownTimeout` (default 30 seconds), a host-level setting rather than a BackWave option.

```csharp title="ShutdownTimeout.cs"
builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(30));
```

### Shutdown cancellation is not an operator cancel

Three sources of cancellation produce deliberately different outcomes:

| Cancellation source | Outcome |
|---|---|
| Host shutdown | No outcome reported; the Lease lapses and the job is re-run on another node. Not terminal. |
| Operator cancel | The job is driven to the terminal `Cancelled` state. |
| The handler's own cancellation (e.g. an `HttpClient` timeout) | A plain Failure; retries like any other failed Attempt. |

## Wiring it together

A minimal end-to-end registration that touches every option this page discusses:

```csharp title="Program.cs"
builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
        PoolSize = 16,
        LeaseDuration = TimeSpan.FromSeconds(60),
        // HeartbeatInterval defaults to LeaseDuration / 3
        // MaintenanceInterval defaults to 5s
    }));

builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");

var app = builder.Build();
app.MapHealthChecks("/health");
```

The `WorkerGroupOptions` fields that bear on health, inheritance, and draining:

| Option | Default | Relevance here |
|---|---|---|
| `Name` (required) | — | Appears in logs and health; the key in `HaltedGroups` and `DegradedGroups`. |
| `Policy` (required) | — | Which Queues the group serves and sweeps. `DispatchPolicy.Strict(...)` or `DispatchPolicy.Weighted(...)`. |
| `PoolSize` | `20` | The number of in-flight slots cancelled on shutdown. |
| `Pumps` | `1` | The whole-versus-partial halt threshold; must be at least `1`. |
| `LeaseDuration` | `60s` | Inheritance detection window. |
| `HeartbeatInterval` | `null` → `LeaseDuration / 3` | Renewal cadence. |
| `MaintenanceInterval` | `5s` | Expiry-sweep cadence. |
| `RetryPolicy` | `RetryPolicy.Default` | Backoff and ceiling for an inherited (lapsed) Lease. |

## Where to go next

- [The Dashboard](/docs/dashboard-operations/overview): watch Lease-expiry countdowns and the health rollup live, and requeue Dead-Lettered work or cancel jobs — including the difference between an operator cancel and a shutdown cancellation.
- [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning): tune `Pumps`, `PoolSize`, and the Lease/heartbeat/maintenance intervals for throughput and recovery latency.