# Throughput & Pump Fan-Out

Single-node throughput in BackWave is governed by two independent concurrency gates that live as properties on one public record, `WorkerGroupOptions`. `PoolSize` sets execution slots; `Pumps` sets fetch-loop parallelism. They relieve different bottlenecks, carry different costs, and multiply: a group's peak concurrent executions on a node is `Pumps × PoolSize`. This page is the mechanism view of how a pump moves work through the store and why adding pumps is the lever that raises the ceiling. For the symptom-to-dial tuning guide, see [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning); this page explains why those dials behave the way they do.

## The two independent gates

A **Worker** is one execution slot in a group's pool, a logical slot rather than a thread; `PoolSize` is how many of them a group runs concurrently per pump per node. A **Pump** is one Shell-side event loop that runs a group's claim, dispatch, and report cycle; `Pumps` is how many of those loops the group runs in a single process. The two are orthogonal axes. `PoolSize` is *execution* concurrency (how many jobs run at once); `Pumps` is *fetch-loop* parallelism (how many independent claim-to-report loops turn at once).

They combine multiplicatively. Because each pump gets its own `PoolSize` pool, a group's peak concurrent executions on a node is `Pumps × PoolSize`, not `PoolSize` alone. A group configured with `PoolSize = 16` and `Pumps = 4` runs up to 64 jobs at once on that node, spread across 4 store-I/O loops.

| Aspect | `PoolSize` (Workers) | `Pumps` |
|---|---|---|
| Axis | Execution concurrency (jobs running at once) | Fetch-loop parallelism (independent claim-to-report loops) |
| Domain term | Worker, one pool slot | Pump, one Shell event loop plus its Driver |
| Default | `20` | `1` |
| Bottleneck it relieves | Slow or IO-bound handlers; jobs queue while CPU and DB idle | Serial store round-trip ceiling of a single pump |
| Cost | Free; idle slots cost nothing | Draws database connections whether busy or not |
| Combined effect | Peak concurrent jobs per group per node = `Pumps × PoolSize` | Multiplies store-I/O parallelism within one process |

The `PoolSize` default is a flat constant, deliberately not scaled to the machine's CPU count. The async claim loop has no thread-per-job to starve, and a machine-independent admission bound keeps the connection footprint predictable, so you tune the value per deployment. `Pumps` costs connections instead: every pump draws database connections whether it is busy or not. The `WorkerGroupOptions` sizing guidance budgets roughly 2 to 3 connections per pump, so a `Pumps = 4` group draws on the order of 8 to 12 at steady state. That is guidance for sizing your connection pool, not a limit the library enforces.

## The throughput-relevant properties

Every single-node throughput dial is an init-only property on the public sealed record `WorkerGroupOptions`, passed once per group to `AddWorkerGroup`.

| Property | Type | Default | Controls / notes |
|---|---|---|---|
| `PoolSize` | `int` | `20` | Max concurrent executions per pump per node. Execution concurrency; not CPU-scaled. |
| `Pumps` | `int` | `1` | Independent claim loops per process. Must be at least `1`. Each draws ~2-3 DB connections. |
| `MaxClaimBatch` | `int` | `32` | Max jobs claimed per poll. Effectively capped at `PoolSize` in practice. |
| `MaxOutcomeBatch` | `int?` | `null` → `MaxClaimBatch` | Max completed outcomes buffered before one batched write. |
| `PollInterval` | `TimeSpan` | `1 second` | Claim cadence; shorter means lower latency and more store queries. Also the transient-fault retry cadence. |
| `MaxPollInterval` | `TimeSpan` | `TimeSpan.Zero` → off | Ceiling on the idle claim cadence. Above `PollInterval`, an idle pump backs off toward it. |
| `MaintenanceInterval` | `TimeSpan` | `5 seconds` | Cadence of lease expiry, schedule mint, and retention purge, separate from claim polling. |
| `HeartbeatInterval` | `TimeSpan?` | `null` → `LeaseDuration / 3` | Lease-renewal cadence; also a partial-outcome-buffer flush trigger. |

```csharp title="Program.cs"
builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "emails",
        Policy = new DispatchPolicy.Strict("emails"),
        PoolSize = 16,   // up to 16 jobs run concurrently per pump
        Pumps = 4,       // 4 independent claim loops -> up to 64 concurrent, ~8-12 DB connections
    }));
```

## How a single pump loops

Each pump is one Shell-side loop that builds a [Node Driver](/docs/advanced/node-driver) (the sans-IO state machine) and feeds it events one at a time, executing the Driver's resulting commands against the store. The loop is single-reader: it processes one event and awaits its store round-trip before moving to the next. Claiming, heartbeating, reporting outcomes, minting scheduled work, expiring lapsed leases, and purging retention all share that one loop and run one at a time on it.

This shape is the source of the serial store-I/O ceiling. Because only one round-trip is in flight at a time, a single pump's throughput is bounded by store round-trip latency, roughly `1 / (store round-trip latency)` for its claim-then-report cycle, no matter how large `PoolSize` is or how much CPU the box has. Raising `PoolSize` lets more jobs *execute* concurrently; it does not raise the rate at which one pump can *claim and report*.

Two internal timers drive the loop: a poll ticker at the `PollInterval` cadence and a heartbeat ticker at the `HeartbeatInterval` cadence for [Lease](/docs/core-concepts/job-lifecycle) renewal. Both feed events into the same loop. A single pending-poll slot coalesces the poll ticker with **Wake-Up Hint** notifications, so a burst of hints that arrives while a poll is already queued collapses to one extra claim pass, and a hint arriving mid-cycle still re-arms the next poll. The heartbeat ticker enqueues its renewal event directly onto the loop and is not routed through that slot.

Handler execution runs *off* the loop. Each claimed job runs on its own task under a per-Attempt DI scope, so slow handlers never block the claim and report loop; only the completed outcome comes back onto the loop to be buffered and written. For the per-Attempt scope mechanics, see [The Job Pump & DI Scope](/docs/advanced/job-pump). Wake-Up Hints are subscribed only when the store exposes them and only for the queues this group's **Dispatch Policy** serves.

## Backpressure and the claim-batch math

**Backpressure** is node-local flow control: a pump stops claiming when its Worker pool has no free slot. It is enforced authoritatively in the Node Driver, which computes free capacity before each claim pass and claims nothing when the pool is full, with a coarse pre-check in the Shell pump before it even enqueues a poll.

Free capacity is the pool size minus what already holds a slot. `MaxClaimBatch` (`32` by default) bounds how many jobs a pass claims, but the pool bound binds first: a group never claims more than it can run, so `MaxClaimBatch` is effectively capped at `PoolSize`. With the defaults, the pool bound of `20` is what binds. A subtle case: a job that has finished executing but whose outcome has not yet flushed to the store still holds its slot until the batched write lands, because a completed-but-unreported job still holds its store Lease. So the effective free capacity is `PoolSize` minus executing jobs minus buffered-but-unreported outcomes, not just `PoolSize` minus executing jobs. The claim pass is computed before the outcome buffer drains on a poll, so buffered outcomes keep counting against the pool until their flush reopens the slots, which prevents over-admitting past `PoolSize`.

Backpressure is a distinct gate from the cluster-wide per-Queue **Concurrency Limit**. Backpressure caps claims on one node's pool; the Concurrency Limit caps how many jobs of a queue execute simultaneously across the whole cluster, regardless of how many pumps or nodes you run. If a queue is Concurrency-Limit-saturated, more pumps will not raise its throughput; you raise the limit, which is a queue setting rather than a `WorkerGroupOptions` dial.

The Dispatch Policy shapes how those claims turn into store round-trips. A `Strict` policy claims the whole free capacity in one priority-ordered batch, an `O(1)` round-trip cost. A weighted policy runs a smooth weighted round-robin and issues one claim per queue, an `O(Q)` cost per pass. Both are work-conserving.

## Outcome batching and flush triggers

Completed outcomes are coalesced into a **Batch** and applied as a single fenced store write. `MaxOutcomeBatch` bounds how many outcomes buffer before that write; when it is `null` (the default) it falls back to `MaxClaimBatch`, so a claim batch's worth of outcomes coalesces into one write. The purpose is to keep the writer single-threaded, so throughput rises without opening more database connections. Each row in the batch is fenced independently by the Effect-Once fence, so a late write from a recovered node stays harmless; see [Execution Guarantee](/docs/core-concepts/execution-guarantee).

Batching never adds latency to a lone result, because the buffer flushes on the earliest of several triggers.

| Trigger | Condition | Effect |
|---|---|---|
| Buffer full | Buffer reaches `MaxOutcomeBatch` | Drains as one batched write |
| Drain-tail (idle) | Nothing else is executing after an outcome is buffered | A lone result flushes immediately, never waiting a poll interval |
| Poll tick | A poll fires with a non-empty buffer | Partial batch lands so released **Dependency** work is claimable on the re-poll |
| Heartbeat tick | A heartbeat fires with a non-empty buffer | Flushes a partial batch under sustained load |

Each applied outcome triggers a re-poll, because a released Dependency may become due that instant, so throughput of dependency chains does not wait for the next timer tick. The batch rides Shell-stashed side channels (failure detail, runtime Tag deltas, Job Output) keyed by job and attempt; none of that crosses into the Core. For how outcomes surface to callers, see [React to Job Outcomes](/docs/guides/react-to-job-outcomes).

## Polling, maintenance, and latency

Bounded, batched polling is the sole correctness mechanism for acquiring work. A Wake-Up Hint only ever brings a poll forward; it never replaces polling. If the hint channel dies, latency degrades to `PollInterval` and nothing else changes. `PollInterval` is also the retry and backoff cadence for transient store faults, where a skipped cycle costs latency and nothing more.

`MaintenanceInterval` governs background maintenance, expiring lapsed leases, loading and minting recurring schedules, and purging retained history, separately from claim polling. Every poll claims, but maintenance runs only once per `MaintenanceInterval`, which keeps claim polls a cheap fast path. Set it slower than `PollInterval`. A missed maintenance sweep only delays maintenance by that much; it never affects correctness.

Lowering `PollInterval` is the dial for enqueue-to-pickup latency, at the cost of more frequent store queries. It does not raise steady-state throughput; that is `PoolSize` and `Pumps`. For how the virtual clock lets you exercise these intervals deterministically in tests, see [Virtual-Time Testing](/docs/testing/behavior-over-time).

`MaxPollInterval` is the dial for the opposite cost: the queries an idle fleet issues when there is nothing to claim. Set it above `PollInterval` and an idle pump doubles its delay toward that ceiling. When the store reports the instant the next scheduled job is due, the pump sleeps until then. The first claim that returns work, or any due-now work a limit held back, restores `PollInterval`. Because the ceiling bounds the delay, it is also the worst-case pickup latency on a store with no hint channel. It changes idle query load only. A busy pump never backs off, and the loaded-throughput numbers are unchanged.

## How pumps fan out

At registration, a group registers its `Pumps` count of pump instances in one process; the default of `1` is byte-for-byte the prior single-pump behaviour. Each pump gets a unique worker identity, so the store hands each pump a disjoint set of rows under its row-locking claim with no extra coordination. N pumps therefore multiply a group's store-I/O parallelism within one process, the direct answer to the single-pump serial ceiling, at the cost of their connections.

Health rolls up per pump. A pump that fail-stops reports halted under its own worker identity, and a sibling pump's clean cycle clears only its own mark, so the group reads wholly halted only once every pump is down. See [Always-On Assertions & Fail-Stop](/docs/advanced/fail-stop) and [Health, Fail-Stop & Draining](/docs/dashboard-operations/health-and-fail-stop). Overlapping a single pump's own round-trips to lift its ceiling without adding pumps is not a lever that exists today; to get more store-I/O parallelism, add pumps or nodes.

## Validation

`AddWorkerGroup` validates exactly two things. A duplicate group `Name` throws `InvalidOperationException` with the message `Worker Group '{name}' is configured twice.`, and a `Pumps` value below `1` throws `InvalidOperationException` with `Worker Group '{name}' must run at least one Pump (Pumps = {n}).`. `PoolSize`, `MaxClaimBatch`, `MaxOutcomeBatch`, and the `TimeSpan` intervals are not validated and have no enforced upper bound, so set them deliberately. `Name` and `Policy` are required members; a process may host several groups, each with its own independent pool, pumps, polling, batch, lease, and retention settings.

## Where to go next

- [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning): the symptom-to-dial tuning guide and when to add nodes.
- [The Job Pump & DI Scope](/docs/advanced/job-pump): the per-Attempt scope that runs handlers off the claim loop.
- [The Sans-IO Node Driver](/docs/advanced/node-driver): the state machine that computes each claim pass and Lease renewal.
- [Performance & Benchmarks](/docs/advanced/performance-and-benchmarks): the benchmark suite behind the real throughput figures.