Leases & Crash Recovery

The full timeline of a node dying mid-job: leases and heartbeats, how peers detect and reclaim expired work, what the next attempt sees, and how to tune lease duration against long-running handlers.

A Worker in BackWave is alive exactly as long as its leases keep getting renewed, and by nothing else. Every running Attempt is held under a Lease — a time-boxed claim the owning worker must keep renewing by heartbeat — and when a node dies, no announcement follows: the lease simply stops being renewed, lapses, and a peer's routine maintenance sweep reclaims the work as if the crash were an ordinary timeout. That single design choice — treating a dead node, an isolated one, and a long pause as one indistinguishable event, then handling all three identically — is what this page traces, from the last heartbeat through the reclaimed job's next attempt, and into the tuning question it forces: how long should a lease be when your handlers legitimately run for a long time?

The renewal loop itself is covered in The Job Pump & DI Scope and the operational view of inheritance and draining lives in Health, Fail-Stop & Draining. This page owns the failure timeline end to end; it links both rather than repeating them.

A lease is the only liveness signal#

There is no cluster membership, no gossip protocol, and no leader watching the nodes. Peers are database-authoritative and hold no shared state about one another, so there is nothing to elect and nothing to reconcile. A worker counts as alive precisely while its leases are being renewed against the store; the moment renewals stop, it is — as far as any peer can tell — gone.

That framing is deliberate, because a distributed system cannot tell its failure modes apart. A crashed process, a node isolated from the store, and a ninety-second garbage-collection pause all look identical from the outside: renewals stop arriving. BackWave does not try to distinguish them. All three are the same event — a lease that is no longer being renewed — and all three get the same response. An isolated or stalled node is acting on a stale belief that it still owns its lease, and it is simply fenced out when it returns (see The zombie returns below). There is no split state to resolve because there was never any shared membership to disagree about.

This is also why a lease is a lease and not a lock. A lock implies ownership that persists until it is explicitly released, which is exactly the failure mode a crashed holder creates. A lease expires on its own after a bounded time, so a holder that vanishes cannot pin work indefinitely. Reclaim needs no cooperation from the departed node.

The healthy loop#

Claiming a job stamps its lease with an expiry — the moment of the claim plus the group's LeaseDuration, which defaults to 60 seconds. From there a background renewal loop heartbeats on its own independent timer, at an interval that defaults to one-third of the lease duration; at the 60-second default that is roughly every 20 seconds. Renewing three times per lease window buys margin: a lease survives a couple of missed heartbeats — roughly two renewal opportunities pass before it could lapse. A single dropped heartbeat, from a brief hiccup, never costs the worker its work.

The heartbeat round-trip carries more than a renewal. It is also the channel through which cooperative cancellation reaches a running attempt. An operator cancel does not travel on a separate push; the request is set on the job in the store, and the running worker learns of it in the reply to its next heartbeat, which then fires the handler's CancellationToken. Threads are never interrupted or killed — a handler is asked to stop and given the chance to unwind. The mechanics of that round-trip are covered in Cancel a Running Job.

The crash, minute by minute#

Take a worker holding a 60-second lease that dies mid-attempt, with peers running the default 5-second maintenance sweep. The timeline is entirely governed by the lease, not by any watcher noticing the node is gone:

Three points are worth drawing out of that table.

Detection latency is bounded by the lease duration, not by the sweep cadence. A lease cannot be reclaimed before its expiry, so no amount of frequent sweeping speeds up detection below LeaseDuration. The sweep cadence only adds latency on top of expiry: at worst one extra MaintenanceInterval passes between the lease lapsing and a sweep noticing it. Worst-case detection is therefore the lease duration plus at most one sweep interval. A sweep that runs late only delays maintenance by that much; it never affects correctness, because the expiry itself — not the sweep — is what makes the work reclaimable.

A reclaim consumes an Attempt. A lease expiry counts against the retry budget exactly as a thrown exception does. The reclaimed job re-enters the same schedule any failed attempt would: the default exponential backoff (roughly two seconds raised to the attempt number, capped at five minutes) and the same attempt ceiling, which defaults to ten. A handler that crashes the node every time it runs will therefore spend its budget and eventually Dead-Letter, rather than looping forever. That is the safety property, not a bug — a crash-loop is contained by the same ceiling that contains an ordinary failure loop. The retry side is covered in Configure Retries & Error Handling.

In-flight work is re-run from the start, not resumed. BackWave does not record or replay the steps inside a handler, so the next attempt begins the job fresh rather than continuing from wherever the dead node left off. Everything a handler needs to be safe under this re-run is the subject of the execution model, covered in The Execution Model.

The zombie returns#

Now the other case: the worker was never dead, only slow — a long garbage-collection pause, a frozen container, a stalled thread. While it was gone its lease expired, a peer reclaimed the work, and a fresh attempt ran to completion elsewhere. Then the slow node wakes, unaware any of that happened, finishes its handler, and tries to report success.

Its report is rejected. Every outcome write is fenced to the exact worker-and-attempt pair that holds the live lease, and because the reclaim advanced the attempt to a new owner, the returning zombie no longer holds it. Its outcome is discarded as stale before it can touch the job's state. What it cannot undo is anything its handler already did to the outside world: those side effects happened. That is the essence of At-Least-Once Execution — the handler body may run more than once, and the fence only guarantees the recorded outcome is written once, a property called Effect-Once. The rejection mechanics, and why a stale write is always benign, are covered in The Effect-Once Fence.

Tuning leases for long-running handlers#

The question this page exists to answer: my handler runs for twenty minutes — what lease do I need? The common wrong answer is a twenty-minute lease. You almost never want one.

The renewal loop runs independently of your handler. A handler that works for twenty minutes under the 60-second default is renewed roughly sixty times while it runs; it never needs a lease anywhere near as long as its own runtime. LeaseDuration is not a budget for handler execution — it is the crash-detection window. A longer lease means slower recovery when a node genuinely dies, because peers must wait out the full duration before they may reclaim. Tuning it up to accommodate a slow handler buys you nothing and costs you recovery latency.

Can a long-running handler block the heartbeat? A well-behaved async handler cannot, no matter how long it runs. The handler executes off the renewal loop, and the loop keeps heartbeating on its own timer the entire time the handler is working. The one real risk is a handler that performs synchronous blocking work — or many such handlers at once, up to the group's PoolSize. Blocking calls can starve the thread pool that the renewal timer also depends on, and a starved timer delays heartbeats. This is general .NET async hygiene rather than anything specific to BackWave, and the fix is to stop blocking the thread pool, never to lengthen the lease.

What lease duration actually trades against is infrastructure pause ceilings. A garbage-collection pause, a container freeze, or a live VM migration that lasts longer than the lease is indistinguishable from a crash: the lease lapses, the work is reclaimed, and the paused node's eventual outcome is fenced out when it returns. If your environment has a known worst-case pause, the lease should comfortably exceed it, so a routine pause does not trigger a needless reclaim and a wasted re-run. Set the lease to cover your realistic pause ceiling and no longer.

LeaseDuration and its heartbeat interval are per-Worker Group settings, so different groups can carry different lease profiles. A bulk-processing group that tolerates coarse recovery can run a long lease; a latency-critical group can run a short one for fast reclaim. Each group's expiry is governed by its own retry policy and lease settings regardless of which node happens to run the sweep that reclaims its work. Note that if you leave the heartbeat interval unset, it stays pinned to one-third of the lease — so raising the lease raises the heartbeat spacing proportionally unless you set the interval explicitly.

Lease inheritance and draining#

A planned shutdown differs from a crash in exactly one respect, and it is not the one you might expect. Draining does not finish in-flight attempts. When a host shuts down gracefully, the group stops claiming new work and actively signals every in-flight handler to cancel through its CancellationToken; it then reports no outcome for that cancelled work. Those leases lapse and peers inherit and re-run the jobs under At-Least-Once, precisely as they would after a crash. The Pump does not wait for in-flight handlers to commit. The only difference from a crash is that a drain actively asks handlers to stop, where a crash merely stops renewing.

Inheritance, then, is an emergent property of the lease and its expiry, not a handoff protocol: there is no takeover message, only a lease that stops being renewed and a peer that reclaims it. The lever for granting handlers more time to wind down during a drain is the standard .NET HostOptions.ShutdownTimeout (default 30 seconds), a host-level setting; there is no BackWave-specific drain timeout. The operational treatment of inheritance, lease cadence, and drain timeouts — including the option tables — lives in Health, Fail-Stop & Draining.

Where to go next#

• The Job Pump & DI Scope: the renewal loop that keeps a lease alive and the scope each attempt runs in.
• The Effect-Once Fence: why a returning zombie's late report is always rejected as stale.
• Health, Fail-Stop & Draining: the operational side of inheritance, lease cadence, and drain timeouts.
• The Execution Model: what every handler must assume, given that any attempt can be re-run from the start.