# Always-On Assertions & Fail-Stop

BackWave borrows TigerStyle's discipline of asserting aggressively and crashing on a violated invariant, but it translates the blast radius for a library that runs as a guest inside someone else's host. Assertions are real runtime checks that survive into production release builds, never compiled out, and a violated invariant **fail-stops** the offending **Worker Group** instead of taking down the host. This page explains that translation: what an always-on assertion actually is in the source, how the **Pump** enacts a fail-stop, why the classifier fails closed on anything it cannot prove transient, and how the result surfaces through health and draining. It is the internal counterpart to the operational treatment in [Health, Fail-Stop & Draining](/docs/dashboard-operations/health-and-fail-stop).

## Assert in production, halt on violation

TigerStyle's rule, assert aggressively and crash on violation, is written for a process that owns itself. BackWave is a guest library, so the rule survives but the blast radius is redrawn along two concrete lines.

First, an **Always-On Assertion** is an ordinary runtime check that runs in release builds, not a `Debug.Assert` that the compiler strips out of production. In practice these appear inline as guards that throw when a state the code believes it already validated turns out to be violated. For example, the in-memory store throws an `InvalidOperationException` at a commit site it annotates as an "always-on assertion: validated above". The property that would only catch a corrupted state in a debug test build instead catches it live and converts it into a controlled halt.

Second, an **Invariant Violation** fail-stops the Worker Group, not the host process. A fail-stop has four observable parts:

- claiming halts, permanently, for that group;
- in-flight **Leases** lapse, so healthy nodes inherit the work;
- the health check goes red;
- one high-severity log names the dead invariant.

The host app keeps serving traffic and the enqueue client keeps writing new work throughout, because workers are a separate failure domain from the enqueue path. A halted group never blocks jobs from being enqueued, and every sibling Worker Group keeps running.

Two alternatives were rejected on purpose. Crashing the host process is unforgivable for a library; the host owns that decision, not its dependency. Swallowing or retrying past an invariant violation is worse, because continuing to schedule jobs on corrupted assumptions is how job systems double-charge customers. The load-bearing rule for anyone extending the runtime is that wrapping an invariant violation in a retry is not resilience; it is the exact bug the discipline exists to prevent.

There is no central `Assert` or `Guard` helper in the source. Always-on assertions are simply throws that the Pump's fail-stop catch treats as invariant violations; the mechanism lives in the catch, not in a dedicated assertion API.

## Fail-closed classification: transient versus everything else

An early over-broad reading of the rule would halt a group for a dropped TCP connection. A later amendment narrows fail-stop to invariant violations, because a **Transient Store Fault** (a connection reset, a failover blip, a deadlock victim, a command timeout) is not a corrupted assumption; it is the network. The classifier runs in the **Shell** at the command-execution boundary and sorts every fault into one of two buckets.

The transient set is deliberately small. For the built-in adapters it is a `DbException` whose `IsTransient` flag is `true`, plus a `TimeoutException`. That one flag and that one exception type are the whole set. Everything else is the default bucket: an invariant violation, or any exception that cannot be positively identified as transient, halts the Pump.

| Fault | How it is recognized | Response | Log level | Health effect |
|---|---|---|---|---|
| Transient store fault | `DbException` with `IsTransient = true`, a `TimeoutException`, or an adapter's `IStoreFaultClassifier` returning `true` | Group stays running; retries on the next pump tick (the poll interval is the backoff) | `Warning` | Marked degraded; does not flip `IsHealthy` |
| Invariant violation / anything unclassifiable | The default bucket: any exception not positively classified as transient | Fail-stop; the Pump halts permanently, reports no outcomes, sends no heartbeats; Leases lapse | `Critical` (full type, message, stack) | Halted; flips `IsHealthy` once the whole group is down |
| Dead poll/heartbeat ticker | A non-cancellation throw out of the timer loop completes the event channel with the exception | Fail-stop, rather than idling forever with no polls | `Critical` | Halted |
| Normal host shutdown | `OperationCanceledException` while the stopping token is set | Graceful drain; stop claiming, cancel in-flight, report nothing; Leases lapse | (none) | No effect |

This is fail-closed on the unknown. An exception BackWave cannot prove is transient is treated as a possible invariant violation and halted, rather than retried silently against a store that may be in an undefined state. The fail-stop catch is a catch-all: it does not inspect the exception to confirm it is "really" an invariant; it simply halts on anything non-transient and non-shutdown that escapes the loop.

### Extending the transient set

Providers surface faults that the generic `IsTransient` flag never sets, like SQLite's `SQLITE_BUSY` and `SQLITE_LOCKED`. An adapter can teach the classifier about those through the optional `IStoreFaultClassifier` capability. The Shell detects it at runtime (the store either implements the interface or it does not) and consults it without taking a dependency on any provider package. The capability is one method, `IsTransientFault(Exception)`, which a storage adapter (never consumer app code) returns true from only for a fault it recognizes as transient; false defers to the host default.

An implementation must be pure inspection and must never mistake a permanent fault for a transient one. A false positive is worse than a missed one: it turns a fatal error into an endless retry, which is exactly the retry-past-an-invariant trap the amendment exists to avoid. This is the only public extension point in the classification path; the Shell's own transient probe is internal and not callable API. For where this seam sits among the others, see [Extension Seams](/docs/advanced/extension-seams).

## How the Pump enacts a fail-stop

The per-node Pump is an internal hosted service running a claim → dispatch → report loop. It feeds events to the sans-I/O [Node Driver](/docs/advanced/node-driver) and executes the driver's commands through the **Storage Contract**; all I/O, clocks, and threads live in the Shell, while the driver only decides. The fail-stop behavior lives in how that loop handles a throw.

An inner catch handles the transient case: a fault the classifier recognizes as a transient store fault logs a `Warning`, records the Pump as degraded, and lets the loop continue to the next tick. A skipped cycle costs only latency, never correctness, because polling is the sole correctness mechanism, and the next clean cycle clears this Pump's degraded mark.

An outer catch is the fail-stop path: any exception that escapes the loop and is not a transient store fault logs a single `Critical` entry naming the group, then records the halt with the exception. Normal shutdown, an `OperationCanceledException` raised because the stopping token fired, is caught separately and is not a fail-stop.

A dead ticker is surfaced as a fail-stop on purpose. If the poll/heartbeat timer loop throws for any reason other than cancellation, it completes the Pump's event channel *with* that exception; the exception then escapes the pump loop and halts the group loudly, rather than leaving it alive but silent with no polls ever arriving.

On a fail-stop, no outcomes are reported and no more heartbeats are sent. A `finally` block cancels every in-flight job's `CancellationToken` so cooperative handlers can unwind, but reports nothing; the Leases lapse and healthy nodes inherit the work. The loop catches every fault and converts it into logged output plus recorded health state for a defensive reason: a .NET `BackgroundService` that throws normally takes the host down, and BackWave's pump exists to guarantee that default never fires.

### Contrast: the Observer pump is fail-soft

The transition-observer dispatch pump is deliberately quieter. A non-cancellation fault stops that pump but logs at `Error` rather than `Critical`, turns nothing in health red, and self-heals when its cursor Lease lapses and another node re-claims it, which is explicitly lower severity than a Worker Group invariant halt. The two are not the same mechanism. A broken invariant in the scheduling path is meant to be louder than a hiccup in the observation path. The observer side is covered in [Transition Observer Internals](/docs/advanced/transition-observer-internals).

## Surfacing through health

`AddBackWave` auto-registers two public singletons in the `BackWave.Hosting` namespace: `BackWaveHealth`, the live state object you resolve from DI, and `BackWaveHealthCheck`, an `IHealthCheck` you wire into the standard pipeline yourself.

```csharp title="Program.cs" {1-3}
builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");

var app = builder.Build();
app.MapHealthChecks("/health");
```

Health is bookkept per Pump but surfaced at group altitude. Keying by `(group, pump)` stops one Pump's clean cycle from clearing a sibling's degraded mark, and stops one Pump's halt from reading as the whole group being down.

| Member | Type | Meaning |
|---|---|---|
| `IsHealthy` | `bool` | `true` while no group has *wholly* halted; degraded and partially-halted do not flip it. |
| `HaltedGroups` | `IReadOnlyDictionary<string, HaltState>` | Wholly-halted groups (every pump down), keyed by group name, one halting cause each. |
| `PartiallyHaltedGroups` | `IReadOnlyDictionary<string, HaltState>` | Groups with at least one halted pump but others still serving; diagnostic only. |
| `DegradedGroups` | `IReadOnlyDictionary<string, string>` | Groups with at least one pump on a transient fault; still running. The value is a short description. |

A group counts as wholly halted only once every one of its Pumps has stopped on an invariant violation; until then it is partially halted and still serving through its surviving pumps. `HaltState` is a public record carrying `ExceptionType` (the exception's full type name) and `Message`; its `ToString()` renders `"ExceptionType: Message"`. Operators see the full type name rather than a friendly label, deliberately, so the dead invariant is identifiable.

```csharp title="HealthInspection.cs"
var health = app.Services.GetRequiredService<BackWaveHealth>();

if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}
```

`BackWaveHealthCheck.CheckHealthAsync` reads that in-memory state and never blocks. It returns `Healthy` with `"All Worker Groups running."` while `IsHealthy`, and otherwise `Unhealthy` with a message beginning `"Worker Group fail-stop:"` that lists each wholly-halted group and its cause. Because the host stays up, a fail-stop shows as a red `/health` endpoint an orchestrator, load balancer, or pager can act on, while the process remains alive to report it. A connection blip, which only degrades, never pages anyone because it does not flip `IsHealthy`.

The reporting methods themselves, the ones that mark a Pump halted, degraded, or recovered, are internal. A halt supersedes that Pump's degraded mark, and recovery clears only the calling Pump's mark, never a sibling's.

## Surfacing through draining

A fail-stop drains the halted group's work the same way a graceful shutdown does, and that shared mechanism is the reason a halted group needs no takeover API. **Draining** signals cancellation and stops reporting outcomes, so in-flight Leases simply lapse and another node re-claims. Inheritance is an emergent property of the Lease and its expiry, not a handoff protocol.

Draining releases in-flight work rather than blocking to finish it. The Pump does not wait for handlers to commit outcomes: a cooperative handler that honors its `CancellationToken` unwinds and its Lease lapses, while a handler that ignores cancellation runs until it finishes or the host's shutdown timeout tears the process down. The lever for granting more wind-down time is the standard .NET `HostOptions.ShutdownTimeout` (default 30 seconds), a host-level setting, and there is no BackWave-specific drain timeout.

A lapsed Lease *is* a failed **Attempt**. The recovered run retries on the same backoff schedule and attempt ceiling as any other failed Attempt, and is **Dead-Lettered** instead if the ceiling is already exhausted. This is **At-Least-Once Execution**, and it stays safe under a brief double-run because of **Effect-Once**: every outcome write is fenced to the exact `(workerId, attempt)` holding the live Lease, so a late report from a node isolated past its Lease expiry is rejected as stale. Terminal state, the dependency-latch decrement, and the **Concurrency-Limit** slot release all flow from that single fenced write.

Three cancellation sources are distinguished at the execution edge, and only the first is part of the fail-stop story:

| Cancellation source | Outcome |
|---|---|
| Host shutdown (or a fail-stop) | No outcome; the Lease lapses and the job re-runs elsewhere. Not terminal. |
| Operator cancel | The job reaches the terminal `Cancelled` state. |
| The handler's own cancellation (e.g. an `HttpClient` timeout) | A plain Failure that retries like any other failed Attempt. |

The operational view of inheritance, lease cadence, and drain timeouts lives in [Health, Fail-Stop & Draining](/docs/dashboard-operations/health-and-fail-stop); the guarantee itself is covered in [Execution Guarantee](/docs/core-concepts/execution-guarantee).

## Where to go next

- [Health, Fail-Stop & Draining](/docs/dashboard-operations/health-and-fail-stop): the operational side, covering the health check, lease inheritance, and drain timeouts.
- [The Sans-IO Node Driver](/docs/advanced/node-driver): why the pump can convert every fault into a decision the driver never has to make.
- [Extension Seams](/docs/advanced/extension-seams): where `IStoreFaultClassifier` and the other public extension points sit.
- [Transition Observer Internals](/docs/advanced/transition-observer-internals): the fail-soft observation path that contrasts with this fail-stop one.