# The Job Pump & DI Scope

The **Pump** is BackWave's Shell-side event loop: the imperative claim, dispatch, report cycle that drives one **Worker Group**'s work forward. It owns all I/O, timers, and threads, feeding events into a single sans-I/O [Node Driver](/docs/advanced/node-driver) that decides what to do and returns commands the Pump executes against the store. Each running **Attempt** gets its own dependency-injection scope, so a handler and its scoped dependencies (a `DbContext`, say) resolve once per run and dispose when the Attempt ends. `UseJobs` registers every `[Job]` handler as scoped precisely so that scope has something to resolve. This page explains that loop, the per-Attempt scope, and why a Pump and a **Worker** are two independent knobs rather than one.

## What the Pump is and how the loop turns

A Worker Group runs its `Pumps` count of independent Pumps, and each Pump is fully self-contained: its own Node Driver, its own `PoolSize` pool of Workers, its own claim stream, and a unique worker identity. In production each Pump is an internal hosted `BackgroundService`; nothing about the loop is public API, but its behaviour is what every throughput and latency dial ultimately steers.

On start, a Pump constructs one Node Driver out of the group's options and a single-reader channel of events. Single-reader is what makes the store writer single-threaded within a Pump, since only one round-trip is ever in flight. Two independent tickers feed that channel: a poll ticker at `PollInterval` and a heartbeat ticker at `HeartbeatInterval` (defaulting to `LeaseDuration / 3`). Each tick simply writes an event; the loop reads one event at a time, hands it to the Driver, and executes the ordered commands the Driver returns against the **Storage Contract**, whether that is a claim, a report of outcomes, a heartbeat, or the expiry of lapsed **Leases**. Each command's result is turned back into a follow-up event on the same channel.

The cycle reads as a small state machine the Driver drives and the Pump obeys:

1. A poll comes due, and the Driver emits a claim batch (plus any maintenance work).
2. The store claim returns, and the Driver emits one execute command per claimed job.
3. Each execution finishes as a succeeded, failed, cancelled, or unroutable event, which the Driver buffers.
4. The Driver eventually emits a batched outcome report, the store write lands, and if an outcome applied the Driver re-polls, because a released **Dependency** may be due that instant.

The Driver is the decision-maker; the Pump only obeys. When the Driver asks for a poll, the Pump just writes a poll event. That separation is what keeps production and the deterministic test pump byte-for-byte identical in their scheduling decisions; the split is covered in [The Sans-IO Node Driver](/docs/advanced/node-driver). Handler executions run *off* the loop: each claimed job is launched on its own task and tracked by job id in an in-flight set, so the event loop stays free to claim, heartbeat, and report while jobs run.

## The per-Attempt DI scope

Before a claimed job runs, the Pump routes it through the registry. An unroutable job, one with no handler registered for its **Wire Name** or a payload that no longer decodes, becomes an unroutable outcome and is **Quarantined**; a routed job carries its `JobRegistration` and decoded payload forward. For a routed job the Pump builds a `JobContext` (the job id, the Attempt number, and a resolver for reading ancestor **Job Output**) and launches the handler on its own task.

Inside that task the Pump opens a fresh DI scope that wraps exactly one handler invocation, then asks the `JobRegistration` to execute against that scope's provider:

```csharp title="Per-Attempt scope (conceptual)"
using var scope = scopeFactory.CreateScope();
await routed.Registration.Execute(scope.ServiceProvider, payload, context, cancellationToken);
```

`JobRegistration.Execute` resolves the handler *from the provider it is handed*. It pulls `IJobHandler<TJob>` from that scoped provider and calls `HandleAsync`. Because the provider is the scoped one, the handler and any scoped or transient dependencies in its graph are created inside the scope, and the `using` disposes them when the Attempt ends, whether on success, on throw, or on cancellation. That is the whole point: a handler can take a scoped `DbContext` for, say, an idempotent dedup write, and it is disposed per run instead of being captured by the root container for the process lifetime. That capture is the captive-dependency leak the scoping exists to prevent.

The scope's lifetime is deliberately narrow. It wraps only the handler call; the outcome event that follows is written and applied *on the event loop, outside the scope*. So the scope never touches the Core and never affects determinism; it lives entirely in the imperative Shell. The `JobContext`, by contrast, is hoisted above the try so a handler that buffered **Tags** or Job Output and then threw still flushes them. This is the same per-delivery-scope pattern the transition-observer pump uses, applied to a different at-least-once loop; see [Transition Observer Internals](/docs/advanced/transition-observer-internals).

This scope is purely a *production* behaviour. The deterministic test pump runs handlers against the root provider directly and opens no per-Attempt scope, which is exactly why the scope never crosses the [determinism boundary](/docs/advanced/determinism-boundary); the deterministic tests and the Simulator never observe it.

## How UseJobs registers a scoped handler per [Job]

`UseJobs` takes a `JobModule`, the DI-free bundle the source generator emits as `BackWaveJobs.Module`, and wires its contents into the container. The module carries three things: the `JobRegistration` set (the **Job Manifest**, one entry per `[Job]`), a handler mapping per `[Job]`, and the containing types that declare method-sugar jobs.

The store and the registry go in as singletons, but each handler mapping is registered *scoped*. That is the registration the per-Attempt scope resolves:

| Declaration shape | Registration | Why |
|---|---|---|
| Class handler (`IJobHandler<TJob>` implementation) | Scoped | Resolved once per Attempt; a host may re-register it for a different lifetime |
| Class declaring method-sugar `[Job]`s | Scoped, added only if absent | A host that pre-registered the type its own way (the rare deliberate stateful singleton) wins |

Scoped is the deliberate default. A per-Attempt scope resolves the handler exactly once, so transient would be a wash, and singleton re-creates the captive-dependency foot-gun for anything disposable in the graph. The Pump does not resolve handlers from the root container; it resolves them through the scope factory, so every resolution lands inside a per-Attempt scope. The Pump honours whatever lifetime is ultimately registered, so a host that wants a handler at a different lifetime can register it itself and the scoped default steps aside.

```csharp title="Program.cs" {3}
builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)          // one scoped handler per [Job]
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
    }));
```

A handler then takes its scoped dependencies by constructor, and the scope handles their lifetime:

```csharp title="SendWelcomeEmailHandler.cs"
public sealed class SendWelcomeEmailHandler(AppDbContext db) : IJobHandler<SendWelcomeEmail>
{
    public async Task HandleAsync(
        SendWelcomeEmail job, JobContext context, CancellationToken cancellationToken)
    {
        // `db` is resolved from this Attempt's DI scope and disposed when the Attempt ends.
        await db.MarkEmailSentAsync(job.UserId, cancellationToken);
    }
}
```

For the registration surface itself, see [Registering Jobs & Handlers](/docs/core-concepts/jobs-and-handlers).

## Pump versus Worker: two parallelism axes

A Pump and a Worker are easy to conflate and worth keeping apart. A Pump is the fetch loop; a Worker is an execution slot the loop fills. They scale different bottlenecks.

| Concept | What it is | Knob (default) | Scaling effect |
|---|---|---|---|
| **Pump** | The Shell-side claim, dispatch, report loop (one Node Driver, one claim stream, serial store I/O) | `Pumps` (`1`) | More parallel store round-trips; each Pump draws ~2-3 DB connections |
| **Worker** | One execution slot in a Pump's pool; execution concurrency | `PoolSize` (`20`) | More jobs run at once per Pump; idle slots cost nothing |

A group's store I/O is serial within one Pump, with one round-trip in flight at a time, so a single Pump's throughput is bounded by round-trip latency, not by Workers or CPU. Raising `PoolSize` lets more jobs *execute* at once; it does not raise the rate at which one Pump can *claim and report*. Adding Pumps is the lever for that: N Pumps claim disjoint rows under the store's row-locking claim with no added coordination, each under its own worker identity. `PoolSize` is a flat constant, deliberately not CPU-scaled. The async loop has no thread-per-job to starve, and a machine-independent admission bound keeps the connection footprint predictable.

The costs are asymmetric. An idle `PoolSize` slot costs nothing, but each Pump draws database connections whether busy or not, so budget roughly 2 to 3 per Pump as a sizing rule of thumb, not a measured limit. Leave `Pumps` at `1` unless a node is throughput-bound and has connection headroom. A single process may host several Worker Groups, each with its own queues, **Dispatch Policy**, pool, and Pump count. The full fan-out mechanics and the store-I/O ceiling live in [Throughput & Pump Fan-Out](/docs/advanced/throughput-model).

### Backpressure caps claims at the pool

**Backpressure** is node-local flow control: a Pump stops claiming when its Worker pool has no free slot. It is enforced authoritatively in the Node Driver's claim pass, which sizes each claim at `PoolSize` minus in-flight executions minus buffered-but-unreported outcomes, bounded by `MaxClaimBatch`. A completed job whose outcome has not yet flushed still holds its Lease, so it still counts against the pool, which is why buffered outcomes stay in the subtraction until their write lands. When free capacity reaches zero, the Pump claims nothing until a slot frees. A coarse second guard sits in the Shell: the Pump declines to queue a poll at all when the in-flight set is already at `PoolSize`, so a burst of **Wake-Up Hints** against a full pool does not spin.

Backpressure is a capacity cap, distinct from the cluster-wide per-Queue **Concurrency Limit**, which caps how many jobs of a queue run across the whole cluster. Claims can never overshoot `PoolSize`.

## The Pump-relevant options

Every dial below is an init-only property on the public record `WorkerGroupOptions`, passed once per group to `AddWorkerGroup`.

| Option | Type | Default | Role |
|---|---|---|---|
| `Name` | `string` | required | Unique group name; appears in health, metrics, logs, and each Pump's worker identity |
| `Policy` | `DispatchPolicy` | required | Which queues the group serves and how it shares effort (`Strict` or weighted) |
| `PoolSize` | `int` | `20` | Worker slots per Pump; max jobs run concurrently per node; claim polling pauses while full |
| `Pumps` | `int` | `1` | Independent Pump loops the group runs; fetch-loop parallelism (~2-3 DB connections each) |
| `MaxClaimBatch` | `int` | `32` | Max jobs claimed per poll (effectively capped at `PoolSize`) |
| `MaxOutcomeBatch` | `int?` | `null` → `MaxClaimBatch` | Terminal outcomes buffered before one batched fenced write |
| `PollInterval` | `TimeSpan` | `1 second` | How often the group polls for due work |
| `MaxPollInterval` | `TimeSpan` | `TimeSpan.Zero` → off | Ceiling on the idle poll delay. Above `PollInterval` it enables idle backoff |
| `MaintenanceInterval` | `TimeSpan` | `5 seconds` | Cadence for lease expiry, schedule mint, and retention, separate from claim polling |
| `LeaseDuration` | `TimeSpan` | `60 seconds` | How long a claimed job's Lease is held before it lapses |
| `HeartbeatInterval` | `TimeSpan?` | `null` → `LeaseDuration / 3` | How often in-flight Leases are renewed |
| `RetryPolicy` | `RetryPolicy` | Default | Backoff schedule and attempt ceiling before a job is **Dead-Lettered** |
| `Retention` | `RetentionPolicy?` | Default | Keep-then-purge policy; `null` disables retention sweeping |

To tune the two axes independently, set them separately on one group:

```csharp title="Program.cs" {5-6}
bw.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "emails",
    Policy = new DispatchPolicy.Strict("emails"),
    PoolSize = 16, // up to 16 jobs run concurrently per Pump
    Pumps = 4,     // 4 independent claim->execute->report loops
});
```

## Polling, maintenance, and fail-stop

Polling is the sole correctness mechanism for acquiring work. A single pending-poll slot coalesces the poll ticker with every Wake-Up Hint, so a burst of hints while a poll is already queued collapses to one extra claim pass, and a hint arriving mid-cycle still re-arms the next poll. Wake-Up Hints are optional and latency-only: if the store exposes them the Pump subscribes for its served queues, and if that channel dies latency degrades to `PollInterval` and nothing else. Maintenance work, meaning lease expiry, schedule load and mint, and retention purge, runs on the slower `MaintenanceInterval` cadence rather than on every poll, keeping claim polls a cheap fast path.

`MaxPollInterval` makes the idle poll delay adaptive instead of fixed, and only when it is set strictly above `PollInterval`. Each idle poll doubles the next delay toward that ceiling. The Pump's claim pass also returns an advisory `NextDue` from the store, the earliest instant at which an empty claim can return work. The Pump then sleeps until that instant, between the floor and the ceiling. The delay snaps back to `PollInterval` when a claim returns work. It also snaps back when `NextDue` is Now. That happens when due work exists but a **Concurrency Limit**, the batch cap, or a full pool held it back. A Wake-Up Hint brings the next poll forward, but it does not reset the floor. A hint on an idle-backed-off Pump costs one claim pass and no more. An adapter that does not compute `NextDue` reports `null`, and the Pump falls back to plain doubling.

A violated invariant, or any fault the Shell cannot classify as a **Transient Store Fault**, halts *this Pump only*: its Leases lapse, healthy nodes inherit the work, and the health check goes red, but the host process never crashes. A transient store fault instead logs a warning, marks the Pump degraded, and retries on the next tick. Health is bookkept per Pump and surfaced at group altitude, so a sibling Pump's clean cycle never clears another's mark and the group reads wholly halted only once every Pump is down. The full treatment is in [Always-On Assertions & Fail-Stop](/docs/advanced/fail-stop).

## Where to go next

- [Throughput & Pump Fan-Out](/docs/advanced/throughput-model): the store-I/O ceiling of one Pump and how fan-out multiplies it.
- [The Sans-IO Node Driver](/docs/advanced/node-driver): the state machine the Pump obeys and where Backpressure is computed.
- [Registering Jobs & Handlers](/docs/core-concepts/jobs-and-handlers): the registry surface `UseJobs` wires up.
- [Always-On Assertions & Fail-Stop](/docs/advanced/fail-stop): how a single Pump halts without taking the host down.