Always-On Assertions & Fail-Stop

Assertions that run in release builds, why a violated invariant halts only the Worker Group, and the fail-closed classifier that separates a transient store fault from everything else.

BackWave borrows TigerStyle's discipline of asserting aggressively and crashing on a violated invariant, but it translates the blast radius for a library that runs as a guest inside someone else's host. Assertions are real runtime checks that survive into production release builds, never compiled out, and a violated invariant fail-stops the offending Worker Group instead of taking down the host. This page explains that translation: what an always-on assertion actually is in the source, how the Pump enacts a fail-stop, why the classifier fails closed on anything it cannot prove transient, and how the result surfaces through health and draining. It is the internal counterpart to the operational treatment in Health, Fail-Stop & Draining.

Assert in production, halt on violation#

TigerStyle's rule, assert aggressively and crash on violation, is written for a process that owns itself. BackWave is a guest library, so the rule survives but the blast radius is redrawn along two concrete lines.

First, an Always-On Assertion is an ordinary runtime check that runs in release builds, not a Debug.Assert that the compiler strips out of production. In practice these appear inline as guards that throw when a state the code believes it already validated turns out to be violated. For example, the in-memory store throws an InvalidOperationException at a commit site it annotates as an "always-on assertion: validated above". The property that would only catch a corrupted state in a debug test build instead catches it live and converts it into a controlled halt.

Second, an Invariant Violation fail-stops the Worker Group, not the host process. A fail-stop has four observable parts:

• claiming halts, permanently, for that group;
• in-flight Leases lapse, so healthy nodes inherit the work;
• the health check goes red;
• one high-severity log names the dead invariant.

The host app keeps serving traffic and the enqueue client keeps writing new work throughout, because workers are a separate failure domain from the enqueue path. A halted group never blocks jobs from being enqueued, and every sibling Worker Group keeps running.

Two alternatives were rejected on purpose. Crashing the host process is unforgivable for a library; the host owns that decision, not its dependency. Swallowing or retrying past an invariant violation is worse, because continuing to schedule jobs on corrupted assumptions is how job systems double-charge customers. The load-bearing rule for anyone extending the runtime is that wrapping an invariant violation in a retry is not resilience; it is the exact bug the discipline exists to prevent.

Every always-on assertion has a name. A check that fails throws InvariantViolationException, a public sealed exception whose Trigger property carries one member of the public InvariantTrigger enum. The exception message describes the failure and carries no trigger id, so the Trigger property is the stable handle. The Pump's fail-stop catch treats that exception as an invariant violation. Any other non-transient exception that escapes the loop halts the group too, but reports its trigger as unclassified.

The named triggers#

InvariantTrigger identifies each check by name. Three members are degrade-only: the group logs the violation, counts it, and keeps running. Two more, GuaranteedRowAbsent and UnexpectedAffectedRowCount, degrade on the observer dispatch path and halt everywhere else.

The enum's identity is its member name, not its numeric value, so match on the name in logs and metrics. New members can appear in later versions.

The degrade path#

A degrade-only trigger does not halt anything. Inside a Worker Group it logs Warning 2003, BackWave Worker Group '{worker_group}' tripped invariant '{invariant_trigger}': {detail}. The group keeps running, degraded., and does not touch the health check, so the group still reads Healthy. Outside a group, in the Core or an adapter, it logs Warning 1601, BackWave invariant '{invariant_trigger}' tripped: {detail}. The operation continues degraded.. Every named violation, halt or degrade, increments the counter backwave.invariant.violations, tagged with backwave.invariant.trigger (the member name) and backwave.invariant.action (Halt or Degrade). An OutcomeFenceRejected is counted only when the Pump believed the Lease was live, with a 30-second clock-skew allowance. See The Effect-Once Fence for that rule.

Fail-closed classification: transient versus everything else#

An early over-broad reading of the rule would halt a group for a dropped TCP connection. A later amendment narrows fail-stop to invariant violations, because a Transient Store Fault (a connection reset, a failover blip, a deadlock victim, a command timeout) is not a corrupted assumption; it is the network. The classifier runs in the Shell at the command-execution boundary and sorts every fault into one of two buckets.

The transient set is deliberately small. For the built-in adapters it is a DbException whose IsTransient flag is true, plus a TimeoutException. That one flag and that one exception type are the whole set. Everything else is the default bucket: an invariant violation, or any exception that cannot be positively identified as transient, halts the Pump.

This is fail-closed on the unknown. An exception BackWave cannot prove is transient is treated as a possible invariant violation and halted, rather than retried silently against a store that may be in an undefined state. The fail-stop catch is a catch-all: it does not inspect the exception to confirm it is "really" an invariant; it simply halts on anything non-transient and non-shutdown that escapes the loop.

Extending the transient set#

Providers surface faults that the generic IsTransient flag never sets, like SQLite's SQLITE_BUSY and SQLITE_LOCKED. An adapter can teach the classifier about those through the optional IStoreFaultClassifier capability. The Shell detects it at runtime (the store either implements the interface or it does not) and consults it without taking a dependency on any provider package. The capability is one method, IsTransientFault(Exception), which a storage adapter (never consumer app code) returns true from only for a fault it recognizes as transient; false defers to the host default.

An implementation must be pure inspection and must never mistake a permanent fault for a transient one. A false positive is worse than a missed one: it turns a fatal error into an endless retry, which is exactly the retry-past-an-invariant trap the amendment exists to avoid. This is the only public extension point in the classification path; the Shell's own transient probe is internal and not callable API. For where this seam sits among the others, see Extension Seams.

How the Pump enacts a fail-stop#

The per-node Pump is an internal hosted service running a claim → dispatch → report loop. It feeds events to the sans-I/O Node Driver and executes the driver's commands through the Storage Contract; all I/O, clocks, and threads live in the Shell, while the driver only decides. The fail-stop behavior lives in how that loop handles a throw.

An inner catch handles the transient case: a fault the classifier recognizes as a transient store fault logs a Warning, records the Pump as degraded, and lets the loop continue to the next tick. A skipped cycle costs only latency, never correctness, because polling is the sole correctness mechanism, and the next clean cycle clears this Pump's degraded mark.

An outer catch is the fail-stop path: any exception that escapes the loop and is not a transient store fault logs a single Critical entry (event 2001), BackWave Worker Group '{worker_group}' fail-stopped on an invariant violation ({invariant_trigger}); the group is halted., then records the halt with the exception. The trigger reads unclassified when no named check raised it. Normal shutdown, an OperationCanceledException raised because the stopping token fired, is caught separately and is not a fail-stop. An InvariantViolationException raised while the Pump drains or hands its Leases back still halts the group.

A dead ticker is surfaced as a fail-stop on purpose. If the poll/heartbeat timer loop throws for any reason other than cancellation, it completes the Pump's event channel with that exception; the exception then escapes the pump loop and halts the group loudly, rather than leaving it alive but silent with no polls ever arriving.

On a fail-stop, no outcomes are reported and no more heartbeats are sent. A finally block cancels every in-flight job's CancellationToken so cooperative handlers can unwind, but reports nothing; the Leases lapse and healthy nodes inherit the work. The loop catches every fault and converts it into logged output plus recorded health state for a defensive reason: a .NET BackgroundService that throws normally takes the host down, and BackWave's pump exists to guarantee that default never fires.

Contrast: the Observer pump is fail-soft#

The transition-observer dispatch pump is deliberately quieter. A non-cancellation fault stops that pump but logs at Error rather than Critical, turns nothing in health red, and self-heals when its cursor Lease lapses and another node re-claims it, which is explicitly lower severity than a Worker Group invariant halt. The two are not the same mechanism. A broken invariant in the scheduling path is meant to be louder than a hiccup in the observation path. The observer side is covered in Transition Observer Internals.

Surfacing through health#

AddBackWave auto-registers two public singletons in the BackWave.Hosting namespace: BackWaveHealth, the live state object you resolve from DI, and BackWaveHealthCheck, an IHealthCheck you wire into the standard pipeline yourself.

builder.Services
    .AddHealthChecks()
    .AddCheck<BackWaveHealthCheck>("backwave");
 
var app = builder.Build();
app.MapHealthChecks("/health");

Health is bookkept per Pump but surfaced at group altitude. Keying by (group, pump) stops one Pump's clean cycle from clearing a sibling's degraded mark, and stops one Pump's halt from reading as the whole group being down.

A group counts as wholly halted only once every one of its Pumps has stopped on an invariant violation; until then it is partially halted and still serving through its surviving pumps. HaltState is a public record carrying ExceptionType (the exception's full type name), Message, and Trigger (an InvariantTrigger?, null when no named check raised the halt). Its ToString() renders "ExceptionType: Message". Operators see the full type name rather than a friendly label, deliberately, so the dead invariant is identifiable.

var health = app.Services.GetRequiredService<BackWaveHealth>();
 
if (!health.IsHealthy)
{
    foreach (var (group, cause) in health.HaltedGroups)
        logger.LogCritical("Worker Group {Group} halted: {Cause}", group, cause);
}

BackWaveHealthCheck.CheckHealthAsync reads that in-memory state and never blocks. It returns one of three results.

The description names each affected group and the exception type, nothing more. Because the host stays up, a fail-stop shows as a red /health endpoint an orchestrator, load balancer, or pager can act on, while the process remains alive to report it. ASP.NET maps Degraded to 200 OK by default, so a connection blip or a partial halt does not page anyone unless your probe treats anything other than Healthy as a failure.

The reporting methods themselves, the ones that mark a Pump halted, degraded, or recovered, are internal. A halt supersedes that Pump's degraded mark, and recovery clears only the calling Pump's mark, never a sibling's.

Surfacing through draining#

A fail-stop releases the halted group's work without a takeover API. Draining on a fail-stop signals cancellation and stops reporting outcomes, so in-flight Leases lapse and another node re-claims. Inheritance after a halt is an emergent property of the Lease and its expiry, not a handoff protocol.

A clean host stop is different. The Pump stops claiming, waits for its running handlers within ShutdownBudget (default 5 seconds), reports the outcomes it has buffered, and hands its remaining Leases back. A handed-back job returns to Scheduled at "now" with its Attempt unchanged, so another node picks it up at once. When the budget runs out, the Pump hands back the Leases of the handlers still running and logs a warning. The effective budget is clamped to four fifths of the standard .NET HostOptions.ShutdownTimeout (default 30 seconds). Set ShutdownBudget to TimeSpan.Zero to skip the hand-back and let Leases lapse.

A lapsed Lease is a failed Attempt. The recovered run retries on the same backoff schedule and attempt ceiling as any other failed Attempt, and is Dead-Lettered instead if the ceiling is already exhausted. This is At-Least-Once Execution, and it stays safe under a brief double-run because of Effect-Once: every outcome write is fenced to the exact (workerId, attempt) holding the live Lease, so a late report from a node isolated past its Lease expiry is rejected as stale. Terminal state, the dependency-latch decrement, and the Concurrency-Limit slot release all flow from that single fenced write.

Three cancellation sources are distinguished at the execution edge, and only the first is part of the fail-stop story:

The operational view of inheritance, lease cadence, and drain timeouts lives in Health, Fail-Stop & Draining; the guarantee itself is covered in Execution Guarantee.

Where to go next#

• Health, Fail-Stop & Draining: the operational side, covering the health check, lease inheritance, and drain timeouts.
• The Sans-IO Node Driver: why the pump can convert every fault into a decision the driver never has to make.
• Extension Seams: where IStoreFaultClassifier and the other public extension points sit.
• Transition Observer Internals: the fail-soft observation path that contrasts with this fail-stop one.

Found a problem on this page? Report an issue