Throughput & Pump Fan-Out

Two orthogonal concurrency gates on WorkerGroupOptions, the serial store-I/O ceiling of a single pump, outcome batching, and how pump fan-out multiplies throughput.

Single-node throughput in BackWave is governed by two independent concurrency gates that live as properties on one public record, WorkerGroupOptions. PoolSize sets execution slots; Pumps sets fetch-loop parallelism. They relieve different bottlenecks, carry different costs, and multiply: a group's peak concurrent executions on a node is Pumps × PoolSize. This page is the mechanism view of how a pump moves work through the store and why adding pumps is the lever that raises the ceiling. For the symptom-to-dial tuning guide, see Scaling & Throughput Tuning; this page explains why those dials behave the way they do.

The two independent gates#

A Worker is one execution slot in a group's pool, a logical slot rather than a thread; PoolSize is how many of them a group runs concurrently per pump per node. A Pump is one Shell-side event loop that runs a group's claim, dispatch, and report cycle; Pumps is how many of those loops the group runs in a single process. The two are orthogonal axes. PoolSize is execution concurrency (how many jobs run at once); Pumps is fetch-loop parallelism (how many independent claim-to-report loops turn at once).

They combine multiplicatively. Because each pump gets its own PoolSize pool, a group's peak concurrent executions on a node is Pumps × PoolSize, not PoolSize alone. A group configured with PoolSize = 16 and Pumps = 4 runs up to 64 jobs at once on that node, spread across 4 store-I/O loops.

The PoolSize default is a flat constant, deliberately not scaled to the machine's CPU count. The async claim loop has no thread-per-job to starve, and a machine-independent admission bound keeps the connection footprint predictable, so you tune the value per deployment. Pumps costs connections instead: every pump draws database connections whether it is busy or not. The WorkerGroupOptions sizing guidance budgets roughly 2 to 3 connections per pump, so a Pumps = 4 group draws on the order of 8 to 12 at steady state. That is guidance for sizing your connection pool, not a limit the library enforces.

The throughput-relevant properties#

Every single-node throughput dial is an init-only property on the public sealed record WorkerGroupOptions, passed once per group to AddWorkerGroup.

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "emails",
        Policy = new DispatchPolicy.Strict("emails"),
        PoolSize = 16,   // up to 16 jobs run concurrently per pump
        Pumps = 4,       // 4 independent claim loops -> up to 64 concurrent, ~8-12 DB connections
    }));

How a single pump loops#

Each pump is one Shell-side loop that builds a Node Driver (the sans-IO state machine) and feeds it events one at a time, executing the Driver's resulting commands against the store. The loop is single-reader: it processes one event and awaits its store round-trip before moving to the next. Claiming, heartbeating, reporting outcomes, minting scheduled work, expiring lapsed leases, and purging retention all share that one loop and run one at a time on it.

This shape is the source of the serial store-I/O ceiling. Because only one round-trip is in flight at a time, a single pump's throughput is bounded by store round-trip latency, roughly 1 / (store round-trip latency) for its claim-then-report cycle, no matter how large PoolSize is or how much CPU the box has. Raising PoolSize lets more jobs execute concurrently; it does not raise the rate at which one pump can claim and report.

Two internal timers drive the loop: a poll ticker at the PollInterval cadence and a heartbeat ticker at the HeartbeatInterval cadence for Lease renewal. Both feed events into the same loop. A single pending-poll slot coalesces the poll ticker with Wake-Up Hint notifications, so a burst of hints that arrives while a poll is already queued collapses to one extra claim pass, and a hint arriving mid-cycle still re-arms the next poll. The heartbeat ticker enqueues its renewal event directly onto the loop and is not routed through that slot.

Handler execution runs off the loop. Each claimed job runs on its own task under a per-Attempt DI scope, so slow handlers never block the claim and report loop; only the completed outcome comes back onto the loop to be buffered and written. For the per-Attempt scope mechanics, see The Job Pump & DI Scope. Wake-Up Hints are subscribed only when the store exposes them and only for the queues this group's Dispatch Policy serves.

Backpressure and the claim-batch math#

Backpressure is node-local flow control: a pump stops claiming when its Worker pool has no free slot. It is enforced authoritatively in the Node Driver, which computes free capacity before each claim pass and claims nothing when the pool is full, with a coarse pre-check in the Shell pump before it even enqueues a poll.

Free capacity is the pool size minus what already holds a slot. MaxClaimBatch (32 by default) bounds how many jobs a pass claims, but the pool bound binds first: a group never claims more than it can run, so MaxClaimBatch is effectively capped at PoolSize. With the defaults, the pool bound of 20 is what binds. A subtle case: a job that has finished executing but whose outcome has not yet flushed to the store still holds its slot until the batched write lands, because a completed-but-unreported job still holds its store Lease. So the effective free capacity is PoolSize minus executing jobs minus buffered-but-unreported outcomes, not just PoolSize minus executing jobs. The claim pass is computed before the outcome buffer drains on a poll, so buffered outcomes keep counting against the pool until their flush reopens the slots, which prevents over-admitting past PoolSize.

Backpressure is a distinct gate from the cluster-wide per-Queue Concurrency Limit. Backpressure caps claims on one node's pool; the Concurrency Limit caps how many jobs of a queue execute simultaneously across the whole cluster, regardless of how many pumps or nodes you run. If a queue is Concurrency-Limit-saturated, more pumps will not raise its throughput; you raise the limit, which is a queue setting rather than a WorkerGroupOptions dial.

The Dispatch Policy shapes how those claims turn into store round-trips. A Strict policy claims the whole free capacity in one priority-ordered batch, an O(1) round-trip cost. A weighted policy runs a smooth weighted round-robin and issues one claim per queue, an O(Q) cost per pass. Both are work-conserving.

Outcome batching and flush triggers#

Completed outcomes are coalesced into a Batch and applied as a single fenced store write. MaxOutcomeBatch bounds how many outcomes buffer before that write; when it is null (the default) it falls back to MaxClaimBatch, so a claim batch's worth of outcomes coalesces into one write. The purpose is to keep the writer single-threaded, so throughput rises without opening more database connections. Each row in the batch is fenced independently by the Effect-Once fence, so a late write from a recovered node stays harmless; see Execution Guarantee.

Batching never adds latency to a lone result, because the buffer flushes on the earliest of several triggers.

Each applied outcome triggers a re-poll, because a released Dependency may become due that instant, so throughput of dependency chains does not wait for the next timer tick. The batch rides Shell-stashed side channels (failure detail, runtime Tag deltas, Job Output) keyed by job and attempt; none of that crosses into the Core. For how outcomes surface to callers, see React to Job Outcomes.

Polling, maintenance, and latency#

Bounded, batched polling is the sole correctness mechanism for acquiring work. A Wake-Up Hint only ever brings a poll forward; it never replaces polling. If the hint channel dies, latency degrades to PollInterval and nothing else changes. PollInterval is also the retry and backoff cadence for transient store faults, where a skipped cycle costs latency and nothing more.

MaintenanceInterval governs background maintenance, expiring lapsed leases, loading and minting recurring schedules, and purging retained history, separately from claim polling. Every poll claims, but maintenance runs only once per MaintenanceInterval, which keeps claim polls a cheap fast path. Set it slower than PollInterval. A missed maintenance sweep only delays maintenance by that much; it never affects correctness.

Lowering PollInterval is the dial for enqueue-to-pickup latency, at the cost of more frequent store queries. It does not raise steady-state throughput; that is PoolSize and Pumps. For how the virtual clock lets you exercise these intervals deterministically in tests, see Virtual-Time Testing.

How pumps fan out#

At registration, a group registers its Pumps count of pump instances in one process; the default of 1 is byte-for-byte the prior single-pump behaviour. Each pump gets a unique worker identity, so the store hands each pump a disjoint set of rows under its row-locking claim with no extra coordination. N pumps therefore multiply a group's store-I/O parallelism within one process, the direct answer to the single-pump serial ceiling, at the cost of their connections.

Health rolls up per pump. A pump that fail-stops reports halted under its own worker identity, and a sibling pump's clean cycle clears only its own mark, so the group reads wholly halted only once every pump is down. See Always-On Assertions & Fail-Stop and Health, Fail-Stop & Draining. Overlapping a single pump's own round-trips to lift its ceiling without adding pumps is not a lever that exists today; to get more store-I/O parallelism, add pumps or nodes.

Validation#

AddWorkerGroup validates exactly two things. A duplicate group Name throws InvalidOperationException with the message Worker Group '{name}' is configured twice., and a Pumps value below 1 throws InvalidOperationException with Worker Group '{name}' must run at least one Pump (Pumps = {n}).. PoolSize, MaxClaimBatch, MaxOutcomeBatch, and the TimeSpan intervals are not validated and have no enforced upper bound, so set them deliberately. Name and Policy are required members; a process may host several groups, each with its own independent pool, pumps, polling, batch, lease, and retention settings.

Where to go next#

• Scaling & Throughput Tuning: the symptom-to-dial tuning guide and when to add nodes.
• The Job Pump & DI Scope: the per-Attempt scope that runs handlers off the claim loop.
• The Sans-IO Node Driver: the state machine that computes each claim pass and Lease renewal.
• Performance & Benchmarks: the benchmark suite behind the real throughput figures.