Always-On Assertions & Fail-Stop

Assertions that run in release builds, why a violated invariant halts only the Worker Group, and the fail-closed classifier that separates a transient store fault from everything else.

BackWave borrows TigerStyle's discipline of asserting aggressively and crashing on a violated invariant, but it translates the blast radius for a library that runs as a guest inside someone else's host. Assertions are real runtime checks that survive into production release builds, never compiled out, and a violated invariant fail-stops the offending Worker Group instead of taking down the host. This page explains that translation: what an always-on assertion actually is in the source, how the Pump enacts a fail-stop, why the classifier fails closed on anything it cannot prove transient, and how the result surfaces through health and draining. It is the internal counterpart to the operational treatment in Health, Fail-Stop & Draining.

Assert in production, halt on violation#

TigerStyle's rule, assert aggressively and crash on violation, is written for a process that owns itself. BackWave is a guest library, so the rule survives but the blast radius is redrawn along two concrete lines.

First, an Always-On Assertion is an ordinary runtime check that runs in release builds, not a Debug.Assert that the compiler strips out of production. In practice these appear inline as guards that throw when a state the code believes it already validated turns out to be violated. For example, the in-memory store throws an InvalidOperationException at a commit site it annotates as an "always-on assertion: validated above". The property that would only catch a corrupted state in a debug test build instead catches it live and converts it into a controlled halt.

Second, an Invariant Violation fail-stops the Worker Group, not the host process. A fail-stop has four observable parts:

• claiming halts, permanently, for that group;
• in-flight Leases lapse, so healthy nodes inherit the work;
• the health check goes red;
• one high-severity log names the dead invariant.

The host app keeps serving traffic and the enqueue client keeps writing new work throughout, because workers are a separate failure domain from the enqueue path. A halted group never blocks jobs from being enqueued, and every sibling Worker Group keeps running.

Two alternatives were rejected on purpose. Crashing the host process is unforgivable for a library; the host owns that decision, not its dependency. Swallowing or retrying past an invariant violation is worse, because continuing to schedule jobs on corrupted assumptions is how job systems double-charge customers. The load-bearing rule for anyone extending the runtime is that wrapping an invariant violation in a retry is not resilience; it is the exact bug the discipline exists to prevent.

There is no central Assert or Guard helper in the source. Always-on assertions are simply throws that the Pump's fail-stop catch treats as invariant violations; the mechanism lives in the catch, not in a dedicated assertion API.

Fail-closed classification: transient versus everything else#

An early over-broad reading of the rule would halt a group for a dropped TCP connection. A later amendment narrows fail-stop to invariant violations, because a Transient Store Fault (a connection reset, a failover blip, a deadlock victim, a command timeout) is not a corrupted assumption; it is the network. The classifier runs in the Shell at the command-execution boundary and sorts every fault into one of two buckets.

The transient set is deliberately small. For the built-in adapters it is a DbException whose IsTransient flag is true, plus a TimeoutException. That one flag and that one exception type are the whole set. Everything else is the default bucket: an invariant violation, or any exception that cannot be positively identified as transient, halts the Pump.

This is fail-closed on the unknown. An exception BackWave cannot prove is transient is treated as a possible invariant violation and halted, rather than retried silently against a store that may be in an undefined state. The fail-stop catch is a catch-all: it does not inspect the exception to confirm it is "really" an invariant; it simply halts on anything non-transient and non-shutdown that escapes the loop.

Extending the transient set#

Providers surface faults that the generic IsTransient flag never sets, like SQLite's SQLITE_BUSY and SQLITE_LOCKED. An adapter can teach the classifier about those through the optional IStoreFaultClassifier capability. The Shell detects it at runtime (the store either implements the interface or it does not) and consults it without taking a dependency on any provider package. The capability is one method, IsTransientFault(Exception), which a storage adapter (never consumer app code) returns true from only for a fault it recognizes as transient; false defers to the host default.

An implementation must be pure inspection and must never mistake a permanent fault for a transient one. A false positive is worse than a missed one: it turns a fatal error into an endless retry, which is exactly the retry-past-an-invariant trap the amendment exists to avoid. This is the only public extension point in the classification path; the Shell's own transient probe is internal and not callable API. For where this seam sits among the others, see Extension Seams.

How the Pump enacts a fail-stop#

The per-node Pump is an internal hosted service running a claim → dispatch → report loop. It feeds events to the sans-I/O Node Driver and executes the driver's commands through the Storage Contract; all I/O, clocks, and threads live in the Shell, while the driver only decides. The fail-stop behavior lives in how that loop handles a throw.

An inner catch handles the transient case: a fault the classifier recognizes as a transient store fault logs a Warning, records the Pump as degraded, and lets the loop continue to the next tick. A skipped cycle costs only latency, never correctness, because polling is the sole correctness mechanism, and the next clean cycle clears this Pump's degraded mark.

An outer catch is the fail-stop path: any exception that escapes the loop and is not a transient store fault logs a single Critical entry naming the group, then records the halt with the exception. Normal shutdown, an OperationCanceledException raised because the stopping token fired, is caught separately and is not a fail-stop.

A dead ticker is surfaced as a fail-stop on purpose. If the poll/heartbeat timer loop throws for any reason other than cancellation, it completes the Pump's event channel with that exception; the exception then escapes the pump loop and halts the group loudly, rather than leaving it alive but silent with no polls ever arriving.

On a fail-stop, no outcomes are reported and no more heartbeats are sent. A finally block cancels every in-flight job's CancellationToken so cooperative handlers can unwind, but reports nothing; the Leases lapse and healthy nodes inherit the work. The loop catches every fault and converts it into logged output plus recorded health state for a defensive reason: a .NET BackgroundService that throws normally takes the host down, and BackWave's pump exists to guarantee that default never fires.

Contrast: the Observer pump is fail-soft#

The transition-observer dispatch pump is deliberately quieter. A non-cancellation fault stops that pump but logs at Error rather than Critical, turns nothing in health red, and self-heals when its cursor Lease lapses and another node re-claims it, which is explicitly lower severity than a Worker Group invariant halt. The two are not the same mechanism. A broken invariant in the scheduling path is meant to be louder than a hiccup in the observation path. The observer side is covered in Transition Observer Internals.

Surfacing through health#

AddBackWave auto-registers two public singletons in the BackWave.Hosting namespace: BackWaveHealth, the live state object you resolve from DI, and BackWaveHealthCheck, an IHealthCheck you wire into the standard pipeline yourself.

builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

Health is bookkept per Pump but surfaced at group altitude. Keying by (group, pump) stops one Pump's clean cycle from clearing a sibling's degraded mark, and stops one Pump's halt from reading as the whole group being down.

A group counts as wholly halted only once every one of its Pumps has stopped on an invariant violation; until then it is partially halted and still serving through its surviving pumps. HaltState is a public record carrying ExceptionType (the exception's full type name) and Message; its ToString() renders "ExceptionType: Message". Operators see the full type name rather than a friendly label, deliberately, so the dead invariant is identifiable.

var health = app.Services.GetRequiredService<BackWaveHealth>();
 
if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}

BackWaveHealthCheck.CheckHealthAsync reads that in-memory state and never blocks. It returns Healthy with "All Worker Groups running." while IsHealthy, and otherwise Unhealthy with a message beginning "Worker Group fail-stop:" that lists each wholly-halted group and its cause. Because the host stays up, a fail-stop shows as a red /health endpoint an orchestrator, load balancer, or pager can act on, while the process remains alive to report it. A connection blip, which only degrades, never pages anyone because it does not flip IsHealthy.

The reporting methods themselves, the ones that mark a Pump halted, degraded, or recovered, are internal. A halt supersedes that Pump's degraded mark, and recovery clears only the calling Pump's mark, never a sibling's.

Surfacing through draining#

A fail-stop drains the halted group's work the same way a graceful shutdown does, and that shared mechanism is the reason a halted group needs no takeover API. Draining signals cancellation and stops reporting outcomes, so in-flight Leases simply lapse and another node re-claims. Inheritance is an emergent property of the Lease and its expiry, not a handoff protocol.

Draining releases in-flight work rather than blocking to finish it. The Pump does not wait for handlers to commit outcomes: a cooperative handler that honors its CancellationToken unwinds and its Lease lapses, while a handler that ignores cancellation runs until it finishes or the host's shutdown timeout tears the process down. The lever for granting more wind-down time is the standard .NET HostOptions.ShutdownTimeout (default 30 seconds), a host-level setting, and there is no BackWave-specific drain timeout.

A lapsed Lease is a failed Attempt. The recovered run retries on the same backoff schedule and attempt ceiling as any other failed Attempt, and is Dead-Lettered instead if the ceiling is already exhausted. This is At-Least-Once Execution, and it stays safe under a brief double-run because of Effect-Once: every outcome write is fenced to the exact (workerId, attempt) holding the live Lease, so a late report from a node isolated past its Lease expiry is rejected as stale. Terminal state, the dependency-latch decrement, and the Concurrency-Limit slot release all flow from that single fenced write.

Three cancellation sources are distinguished at the execution edge, and only the first is part of the fail-stop story:

The operational view of inheritance, lease cadence, and drain timeouts lives in Health, Fail-Stop & Draining; the guarantee itself is covered in Execution Guarantee.

Where to go next#

• Health, Fail-Stop & Draining: the operational side, covering the health check, lease inheritance, and drain timeouts.
• The Sans-IO Node Driver: why the pump can convert every fault into a decision the driver never has to make.
• Extension Seams: where IStoreFaultClassifier and the other public extension points sit.
• Transition Observer Internals: the fail-soft observation path that contrasts with this fail-stop one.