The Sans-IO Node Driver

How all node logic is a sans-I/O state machine with no awaits, timers, or threads, running identically in production and the Simulator.

Every per-node scheduling decision BackWave makes lives in one place: the Node Driver, a sans-I/O state machine shaped as a single step that turns an event into a list of commands. It decides when to claim, when to renew a Lease, when to stop claiming under Backpressure, and when to flush a batch of outcomes, all without ever awaiting, sleeping, threading, or reading a clock. This page is for the advanced reader who wants to understand how the decide/act split actually works, why it makes the whole system reproducible on a seed, and what lease, heartbeat, and backpressure mechanics the Driver runs underneath.

The decide/act split: a pure function with two callers#

The Driver is the Core of a node expressed as one pure function. You feed it an event describing what just happened, and it returns an ordered list of Command values describing what to do next. It performs no I/O of its own. In the words of the design, it "asks for effects and is told outcomes." It emits commands and receives events, but it never carries out an effect itself.

Its only mutable state is small and strictly per-node: the set of job IDs currently executing, a buffer of terminal outcomes waiting to be reported, the remaining per-Queue batches of the current weighted claim pass, a cron cache, and the timestamp of the last maintenance sweep. Everything else, every socket, timer, thread, and real clock, belongs to the Shell that surrounds it.

That Shell is the imperative pump loop. It fetches state, calls the Driver, and executes the returned commands through the Storage Contract, then feeds the results back in as the next events. It is kept deliberately too dumb to have interesting bugs: when the Driver asks it to poll again, it "only obeys" and holds no scheduling judgment of its own.

The payoff is that the very same Driver code has two callers. In production, a hosted background service pumps it against a real Storage Adapter under the wall clock. In tests, a deterministic pump drives the identical Driver synchronously against the In-Memory Store on Virtual Time, which is what lets the Simulator inject the exact same events (faults included) and replay any run from a seed. Both pumps drive the same Driver; each implements its own command switch to carry out the Driver's commands, differing in real versus virtual clock, threads versus a synchronous drain, and an unbounded channel versus an in-memory queue. Because the Core does no I/O and never reads the wall clock, identical events plus identical time yield identical commands every run. That is the Determinism Boundary in motion.

The event and command loop#

The Shell constructs one Node Driver per pump loop and maps the public WorkerGroupOptions onto its configuration. From there the cycle is mechanical: an event arrives, the Shell calls the Driver's step, then executes every returned command in order, and the results of those commands become new events, draining until the node is quiescent. Timers live entirely in the Shell. Two tickers write a poll-due and a heartbeat-due event, and the Driver never starts a timer. A Wake-Up Hint is only ever an earlier poll: it takes the same request-poll path as the timer and is coalesced through a single pending slot, so losing a hint costs latency, never correctness.

Handlers run outside the loop. When the Driver emits an execute command, the Shell routes and runs the handler on a background task, and only its bounded outcome (plus an error message) re-enters the Driver. Exception control flow never reaches the Core. In the same spirit, opaque diagnostics (Failure Detail, runtime Tag deltas, and Job Output) are stashed Shell-side keyed by job and Attempt, and never travel on an event or command, so they stay off the determinism boundary. The Driver only ever learns that an Attempt succeeded or failed.

The events the Shell feeds in are its report of what happened:

And the commands are the Driver's only outputs, inert data the Shell carries out:

The lease and heartbeat state machine#

The claim/heartbeat/abandon cycle the Driver runs is the lease lifecycle. On a poll tick, if the pool has room, the Driver starts a claim pass and asks the Shell to claim a bounded batch of due jobs, leasing them to this worker for a configured Lease duration. When the claim returns, each job's ID enters the executing set and the Driver emits one execute command per job.

Keeping those leases alive is the Heartbeat. On a heartbeat tick, if anything is executing, the Driver asks the Shell to renew the leases on everything in flight and reports back per-job flags. The Driver then reacts to each result:

• A lost lease (renew declined) means another node may already own the work, so the Driver drops the job from its executing set and abandons it. Abandoning is benign under at-least-once delivery. A lapsed lease simply lets another node inherit the job.
• A renewed lease that also carries a cancel request makes the Driver signal cancellation, firing the handler's CancellationToken. Cancellation is cooperative; threads are never killed.

When a handler finishes, the Shell reports the terminal outcome and the Driver removes the job from executing and buffers the outcome for a coalesced report. Every reported outcome is fenced at the store by worker and Attempt: an applied result mutates state, while a stale-lease result (the write a node isolated past its lease expiry attempts) mutates nothing. That fence is the Effect-Once guarantee, and it is why a late write from a recovered node is harmless. An applied outcome also triggers an immediate re-poll, because releasing a terminal state may have made a Dependency due this very instant, and that decision lives in the Driver so production and the Simulator drain due work identically.

Finally, the Driver's maintenance path expires leases scoped to its own group's Queues, so a stalled or crashed node's jobs get reclaimed even though that node emits no events at all.

Backpressure lives in the Driver, not the Shell#

Backpressure is a capacity cap, not a time-based throttle, and it is enforced inside the Core when a claim pass begins. The Driver computes its free capacity and, if there is none, emits no claim command at all. The node simply stops claiming until something finishes.

Counting buffered-but-unreported outcomes against capacity matters: a completed job whose outcome has not yet flushed still holds its store Lease, so it occupies a Worker slot until the report lands. Counting only live executions would let the node over-claim past its pool size.

This is node-local flow control, and it is distinct from the cluster-wide, per-Queue Concurrency Limit enforced at claim time in the store. A node can be backpressured while the Concurrency Limit still has free slots, or limit-saturated while its own pool sits idle.

How the free capacity is spent depends on the Dispatch Policy. A strict policy claims the whole free capacity in one priority-ordered batch, a single round trip. A weighted policy runs a deterministic (no randomness) weighted round-robin over the free capacity, sizing a per-Queue batch and issuing one claim per Queue, chaining the remaining batches as each claim completes. Each weighted batch lists its own Queue first and the rest as fallthrough, so an empty allocated Queue reflows its slots to Queues that do have due work in the same pass. The scheme is work-conserving, and a worker never idles while any served Queue has due work. Weighted credit advances at issue rather than at allocation, so a batch left un-issued keeps its credit instead of running a deficit.

The production pump keeps a coarse belt-and-suspenders guard (a re-poll no-ops when the in-flight count already meets the pool size), but the authoritative admission bound is always the Driver's free-capacity math.

Outcome coalescing and its flush triggers#

Terminal outcomes are buffered in completion order and flushed together as one report command, which keeps the pump a single writer and raises throughput-per-write without opening more connections. A single buffered outcome flushes as a batch of one, and every row in the batch is fenced by its own worker and Attempt.

Three conditions flush the buffer:

1. It reaches the configured max outcome batch size.
2. The node just went idle (its executing count hit zero). This drain-tail means a lone outcome never waits a whole poll interval before releasing its dependents.
3. A poll or heartbeat tick lands, each of which flushes any partial batch first: on a poll so released Dependencies become claimable on the flush's own re-poll, and on a heartbeat so a partial batch under sustained load still gets written.

There is an ordering subtlety on a poll: the claim pass is computed before the buffer drains, because a buffered outcome still holds its Lease and must keep counting against the pool. The freed slots reopen only when the flush's applied outcomes trigger their re-poll. And if a node crashes, its outcome buffer is simply discarded along with the Driver, those leases lapse, and the jobs are reclaimed, which is At-Least-Once working exactly as designed. The max outcome batch defaults to the max claim batch, so one claim batch's worth of outcomes coalesces into one fenced write.

Cadence: claims every poll, maintenance on a slower beat#

Every poll tick runs a claim pass (the fast path a Wake-Up Hint or an applied-outcome re-poll takes), but maintenance runs only once per maintenance interval. When maintenance is due, the Driver expires lapsed leases in its group's Queues, loads recurring schedules so the Core can plan mints, and, if a RetentionPolicy is configured, runs two purge sweeps: one for succeeded and cancelled jobs, and one for Dead-Lettered and Quarantined jobs, each with its own keep-then-purge horizon.

Schedule minting is a two-step event dance. Loading schedules triggers the Core's mint planner, which yields a mint command; the store applies it and answers back, and the Driver re-polls only if the mint produced work due now. Retention purges are bounded and self-repeating: when a sweep reports a full batch, the Driver re-issues the same sweep, so cleanup drains to zero without ever becoming a lock storm.

Crucially, the last-maintenance timestamp is Driver state, but the clock is not. The Driver compares the Now supplied on each poll against its stored last-maintenance instant. It never reads a clock itself, which is the whole reason this cadence logic replays identically in the Simulator.

The public knob panel#

None of the Driver, its events, or its commands are types you touch directly; they are internal. The public surface that shapes the Driver's behavior is the WorkerGroupOptions record you register, one per group, whose pump drives one Driver instance:

bw.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "emails",
    Policy = new DispatchPolicy.Strict("emails"),
    PoolSize = 16,            // node-local backpressure bound
    LeaseDuration = TimeSpan.FromSeconds(60),
    // HeartbeatInterval defaults to LeaseDuration / 3
    MaxClaimBatch = 32,       // max jobs leased per poll
    PollInterval = TimeSpan.FromSeconds(1),
    MaintenanceInterval = TimeSpan.FromSeconds(5),
});

PoolSize is the backpressure bound, LeaseDuration and HeartbeatInterval tune the lease-renewal cadence, MaxClaimBatch and MaxOutcomeBatch tune claim and outcome batching, MaintenanceInterval sets the slower maintenance beat, and Pumps sets how many independent Driver loops the group runs. For choosing those numbers under load, see Scaling and Throughput Tuning.

Where to go next#

• The Determinism Boundary: what is simulable below the Storage Contract, and why the Driver's purity is the whole point.
• The Job Pump and DI Scope: the Shell side of this loop, per-Attempt scopes, and handler lifetime.
• Throughput and Pump Fan-Out: Pumps versus Workers, fan-out, and the poll-interval tradeoffs.
• How BackWave Is Tested: how the second caller drives this exact Driver to replay any run from a seed.