The Job Pump & DI Scope

The Shell-side claim, dispatch, report loop, the per-Attempt DI scope, and why UseJobs registers every handler as scoped.

The Pump is BackWave's Shell-side event loop: the imperative claim, dispatch, report cycle that drives one Worker Group's work forward. It owns all I/O, timers, and threads, feeding events into a single sans-I/O Node Driver that decides what to do and returns commands the Pump executes against the store. Each running Attempt gets its own dependency-injection scope, so a handler and its scoped dependencies (a DbContext, say) resolve once per run and dispose when the Attempt ends. UseJobs registers every [Job] handler as scoped precisely so that scope has something to resolve. This page explains that loop, the per-Attempt scope, and why a Pump and a Worker are two independent knobs rather than one.

What the Pump is and how the loop turns#

A Worker Group runs its Pumps count of independent Pumps, and each Pump is fully self-contained: its own Node Driver, its own PoolSize pool of Workers, its own claim stream, and a unique worker identity. In production each Pump is an internal hosted BackgroundService; nothing about the loop is public API, but its behaviour is what every throughput and latency dial ultimately steers.

On start, a Pump constructs one Node Driver out of the group's options and a single-reader channel of events. Single-reader is what makes the store writer single-threaded within a Pump, since only one round-trip is ever in flight. Two independent tickers feed that channel: a poll ticker at PollInterval and a heartbeat ticker at HeartbeatInterval (defaulting to LeaseDuration / 3). Each tick simply writes an event; the loop reads one event at a time, hands it to the Driver, and executes the ordered commands the Driver returns against the Storage Contract, whether that is a claim, a report of outcomes, a heartbeat, or the expiry of lapsed Leases. Each command's result is turned back into a follow-up event on the same channel.

The cycle reads as a small state machine the Driver drives and the Pump obeys:

1. A poll comes due, and the Driver emits a claim batch (plus any maintenance work).
2. The store claim returns, and the Driver emits one execute command per claimed job.
3. Each execution finishes as a succeeded, failed, cancelled, or unroutable event, which the Driver buffers.
4. The Driver eventually emits a batched outcome report, the store write lands, and if an outcome applied the Driver re-polls, because a released Dependency may be due that instant.

The Driver is the decision-maker; the Pump only obeys. When the Driver asks for a poll, the Pump just writes a poll event. That separation is what keeps production and the deterministic test pump byte-for-byte identical in their scheduling decisions; the split is covered in The Sans-IO Node Driver. Handler executions run off the loop: each claimed job is launched on its own task and tracked by job id in an in-flight set, so the event loop stays free to claim, heartbeat, and report while jobs run.

The per-Attempt DI scope#

Before a claimed job runs, the Pump routes it through the registry. An unroutable job, one with no handler registered for its Wire Name or a payload that no longer decodes, becomes an unroutable outcome and is Quarantined; a routed job carries its JobRegistration and decoded payload forward. For a routed job the Pump builds a JobContext (the job id, the Attempt number, and a resolver for reading ancestor Job Output) and launches the handler on its own task.

Inside that task the Pump opens a fresh DI scope that wraps exactly one handler invocation, then asks the JobRegistration to execute against that scope's provider:

using var scope = scopeFactory.CreateScope();
await routed.Registration.Execute(scope.ServiceProvider, payload, context, cancellationToken);

JobRegistration.Execute resolves the handler from the provider it is handed. It pulls IJobHandler<TJob> from that scoped provider and calls HandleAsync. Because the provider is the scoped one, the handler and any scoped or transient dependencies in its graph are created inside the scope, and the using disposes them when the Attempt ends, whether on success, on throw, or on cancellation. That is the whole point: a handler can take a scoped DbContext for, say, an idempotent dedup write, and it is disposed per run instead of being captured by the root container for the process lifetime. That capture is the captive-dependency leak the scoping exists to prevent.

The scope's lifetime is deliberately narrow. It wraps only the handler call; the outcome event that follows is written and applied on the event loop, outside the scope. So the scope never touches the Core and never affects determinism; it lives entirely in the imperative Shell. The JobContext, by contrast, is hoisted above the try so a handler that buffered Tags or Job Output and then threw still flushes them. This is the same per-delivery-scope pattern the transition-observer pump uses, applied to a different at-least-once loop; see Transition Observer Internals.

This scope is purely a production behaviour. The deterministic test pump runs handlers against the root provider directly and opens no per-Attempt scope, which is exactly why the scope never crosses the determinism boundary; the deterministic tests and the Simulator never observe it.

How UseJobs registers a scoped handler per [Job]#

UseJobs takes a JobModule, the DI-free bundle the source generator emits as BackWaveJobs.Module, and wires its contents into the container. The module carries three things: the JobRegistration set (the Job Manifest, one entry per [Job]), a handler mapping per [Job], and the containing types that declare method-sugar jobs.

The store and the registry go in as singletons, but each handler mapping is registered scoped. That is the registration the per-Attempt scope resolves:

Scoped is the deliberate default. A per-Attempt scope resolves the handler exactly once, so transient would be a wash, and singleton re-creates the captive-dependency foot-gun for anything disposable in the graph. The Pump does not resolve handlers from the root container; it resolves them through the scope factory, so every resolution lands inside a per-Attempt scope. The Pump honours whatever lifetime is ultimately registered, so a host that wants a handler at a different lifetime can register it itself and the scoped default steps aside.

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)          // one scoped handler per [Job]
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
    }));

A handler then takes its scoped dependencies by constructor, and the scope handles their lifetime:

public sealed class SendWelcomeEmailHandler(AppDbContext db) : IJobHandler<SendWelcomeEmail>
{
    public async Task HandleAsync(
        SendWelcomeEmail job, JobContext context, CancellationToken cancellationToken)
    {
        // `db` is resolved from this Attempt's DI scope and disposed when the Attempt ends.
        await db.MarkEmailSentAsync(job.UserId, cancellationToken);
    }
}

For the registration surface itself, see Registering Jobs & Handlers.

Pump versus Worker: two parallelism axes#

A Pump and a Worker are easy to conflate and worth keeping apart. A Pump is the fetch loop; a Worker is an execution slot the loop fills. They scale different bottlenecks.

A group's store I/O is serial within one Pump, with one round-trip in flight at a time, so a single Pump's throughput is bounded by round-trip latency, not by Workers or CPU. Raising PoolSize lets more jobs execute at once; it does not raise the rate at which one Pump can claim and report. Adding Pumps is the lever for that: N Pumps claim disjoint rows under the store's row-locking claim with no added coordination, each under its own worker identity. PoolSize is a flat constant, deliberately not CPU-scaled. The async loop has no thread-per-job to starve, and a machine-independent admission bound keeps the connection footprint predictable.

The costs are asymmetric. An idle PoolSize slot costs nothing, but each Pump draws database connections whether busy or not, so budget roughly 2 to 3 per Pump as a sizing rule of thumb, not a measured limit. Leave Pumps at 1 unless a node is throughput-bound and has connection headroom. A single process may host several Worker Groups, each with its own queues, Dispatch Policy, pool, and Pump count. The full fan-out mechanics and the store-I/O ceiling live in Throughput & Pump Fan-Out.

Backpressure caps claims at the pool#

Backpressure is node-local flow control: a Pump stops claiming when its Worker pool has no free slot. It is enforced authoritatively in the Node Driver's claim pass, which sizes each claim at PoolSize minus in-flight executions minus buffered-but-unreported outcomes, bounded by MaxClaimBatch. A completed job whose outcome has not yet flushed still holds its Lease, so it still counts against the pool, which is why buffered outcomes stay in the subtraction until their write lands. When free capacity reaches zero, the Pump claims nothing until a slot frees. A coarse second guard sits in the Shell: the Pump declines to queue a poll at all when the in-flight set is already at PoolSize, so a burst of Wake-Up Hints against a full pool does not spin.

Backpressure is a capacity cap, distinct from the cluster-wide per-Queue Concurrency Limit, which caps how many jobs of a queue run across the whole cluster. Claims can never overshoot PoolSize.

The Pump-relevant options#

Every dial below is an init-only property on the public record WorkerGroupOptions, passed once per group to AddWorkerGroup.

To tune the two axes independently, set them separately on one group:

bw.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "emails",
    Policy = new DispatchPolicy.Strict("emails"),
    PoolSize = 16, // up to 16 jobs run concurrently per Pump
    Pumps = 4,     // 4 independent claim->execute->report loops
});

Polling, maintenance, and fail-stop#

Polling is the sole correctness mechanism for acquiring work. A single pending-poll slot coalesces the poll ticker with every Wake-Up Hint, so a burst of hints while a poll is already queued collapses to one extra claim pass, and a hint arriving mid-cycle still re-arms the next poll. Wake-Up Hints are optional and latency-only: if the store exposes them the Pump subscribes for its served queues, and if that channel dies latency degrades to PollInterval and nothing else. Maintenance work, meaning lease expiry, schedule load and mint, and retention purge, runs on the slower MaintenanceInterval cadence rather than on every poll, keeping claim polls a cheap fast path.

A violated invariant, or any fault the Shell cannot classify as a Transient Store Fault, halts this Pump only: its Leases lapse, healthy nodes inherit the work, and the health check goes red, but the host process never crashes. A transient store fault instead logs a warning, marks the Pump degraded, and retries on the next tick. Health is bookkept per Pump and surfaced at group altitude, so a sibling Pump's clean cycle never clears another's mark and the group reads wholly halted only once every Pump is down. The full treatment is in Always-On Assertions & Fail-Stop.

Where to go next#

• Throughput & Pump Fan-Out: the store-I/O ceiling of one Pump and how fan-out multiplies it.
• The Sans-IO Node Driver: the state machine the Pump obeys and where Backpressure is computed.
• Registering Jobs & Handlers: the registry surface UseJobs wires up.
• Always-On Assertions & Fail-Stop: how a single Pump halts without taking the host down.