Worker Groups & Dispatch

How a process consumes work: Worker Groups, Workers, Pumps, Dispatch Policies, and node-local backpressure.

A Worker Group is a registered set of Workers in your process that declares which Queues it serves, how it shares effort across them, and how hard it runs. It is the unit by which you allocate execution capacity and isolate workloads. Each group polls its own Queues, claims jobs in Due Time order, runs them through its own pool, and reports outcomes, all on its own polling cadence with its own lease timing, Retry Policy, and retention. A process can host several groups, so a noisy or latency-sensitive Queue can have dedicated capacity that nothing else competes for. At least one group is required for any job to run.

Registering a worker group#

You add a group while configuring BackWave, through the builder's AddWorkerGroup method. It takes a WorkerGroupOptions record. Only two fields are required: a Name that is unique within the registration, and a Policy that says which Queues the group serves.

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
    }));

Call AddWorkerGroup more than once, with distinct Names, to register several groups in one process. A duplicate Name throws at startup, so each group is guaranteed its own identity in health, metrics, and logs.

Which queues a group serves#

The Queues a group serves and the way it shares effort across them are the same single field: Policy. There is no separate queues property. The DispatchPolicy you assign carries the list of Queues, and BackWave uses it to decide which Queue to claim from next. Within whichever Queue it picks, jobs are claimed in Due Time order.

Priority lives in the policy, never on the job. Jobs carry no priority of their own. BackWave ships two policies, and both are work-conserving: a Worker never sits idle while any served Queue has due work.

// Strict priority: "critical" is always tried before "default".
Policy = new DispatchPolicy.Strict("critical", "default")
 
// Weighted fair-share: roughly 6:3:1 across three queues.
Policy = new DispatchPolicy.Weighted(new[] { ("a", 6), ("b", 3), ("c", 1) })

DispatchPolicy.Strict serves its Queues in fixed priority order. The first Queue is always tried before the second, and a later Queue is reached only when every earlier one has no due work. A continuously busy high-priority Queue can therefore starve the tail, which is the deliberate trade-off you accept when you choose strict ordering.

DispatchPolicy.Weighted shares claim opportunity across Queues in proportion to their weights. A weighting like 6:3:1 is honoured smoothly and deterministically, with no random clumping. The higher-weight Queue serves more often while every Queue keeps making progress, and a Queue with no due work yields its turn. Every weight must be at least 1, and a Weighted policy needs at least one Queue, or it throws when you build it.

The full semantics of both policies, including starvation behaviour and weight tuning, live in Dispatch Policies.

Concurrency: workers and the pool#

PoolSize is the group's Worker count: the most jobs it runs at once on one node. It defaults to 20. Polling pauses while the pool is full and resumes as jobs finish, so the pool size is a hard ceiling on concurrent execution per group, per node.

That default is a flat constant. It is not scaled to the core count of the machine, so a laptop and a 64-core server both start at 20. The reasoning is straightforward. BackWave executes jobs asynchronously, so there is no thread-per-job that could starve a thread pool, which is the failure mode CPU-scaled defaults exist to bound. A machine-independent admission bound keeps the worst case predictable and protects against silently exhausting a database connection pool on a large host. Idle pool slots cost nothing, so a generous flat ceiling is a safe starting point. Tuning is yours: override PoolSize per deployment, and compute it from the core count yourself if you want core-relative sizing.

Workers and Pumps are different things#

A Worker is an execution slot, counted by PoolSize. A Pump is fetch-loop parallelism, counted by Pumps, which defaults to 1 and must be at least 1. Each Pump runs its own independent claim-and-execute loop with its own pool, claiming a disjoint set of jobs from the store.

The two scale differently. Idle Workers are free, but each Pump draws database connections whether it is busy or not, roughly 2 to 3 per Pump. A group with Pumps = 4 holds around 8 to 12 connections at steady state. Leave Pumps at 1 unless a node is genuinely throughput-bound and has connection headroom to spare. Raising it is a throughput lever for the case where a single Pump is limited by store round-trip latency rather than by concurrency or CPU. The throughput model behind that lever is covered in Throughput & Pump Fan-Out.

Isolating workloads with multiple groups#

Each Worker Group is self-contained. It has its own pool, poll cadence, lease timing, Retry Policy, retention, and set of served Queues. Putting a Queue in its own group gives its jobs dedicated capacity and settings, so a flood on one Queue cannot drain the Workers another Queue depends on. Hosting several groups in one process is the supported way to keep workloads apart.

PoolSize is node-local backpressure: it bounds concurrency for one group on one node. It is distinct from a Queue's Concurrency Limit, which is a separate cluster-wide cap enforced when jobs are claimed. See Queues for that limit and how Due Time ordering works inside a Queue.

When a group hits a hard error#

BackWave checks its own internal invariants in release builds, not only in debug. If one of those invariants is violated, the affected group fail-stops: it stops claiming, its in-flight leases lapse so healthy nodes inherit the work, a health check goes red, and a single high-severity log names the failure. The failure is contained. The host application keeps serving traffic, because BackWave is a guest library and never crashes the host process, and the enqueue client keeps working, because enqueuing is a separate failure domain from the Workers. When a group runs multiple Pumps, only the faulting Pump halts while its siblings keep claiming.

Transient store faults, like a connection reset or a brief failover, are not treated as invariant violations. The group is marked degraded rather than halted, logs a warning, and retries on the next poll. The fail-stop is reserved for a genuinely corrupted assumption, where stopping cleanly is safer than quietly mis-processing jobs. The mechanics, including degraded-versus-halted handling and the health surfaces, are covered in Always-On Assertions & Fail-Stop.

Settings and defaults#

WorkerGroupOptions exposes the full set of knobs. Name and Policy are required; everything else has a default you can leave alone.

A few settings throw rather than coerce. A duplicate group Name and a Pumps value below 1 throw at startup. A Weighted policy with no Queues, or any weight below 1, throws when the policy is built.

Where to go next#

• Queues: what a Queue is, the cluster-wide Concurrency Limit, and Due Time ordering within a Queue.
• Scheduling: how work becomes due and how recurring schedules feed a Queue.
• Dispatch Policies: the full Strict and Weighted semantics, starvation, and weight tuning.
• Throughput & Pump Fan-Out: when raising Pumps helps, and the round-trip-latency ceiling behind it.
• Always-On Assertions & Fail-Stop: the fail-stop mechanics, degraded versus halted, and the health checks.