# Worker Groups & Dispatch

A Worker Group is a registered set of Workers in your process that declares which Queues it serves, how it shares effort across them, and how hard it runs. It is the unit by which you allocate execution capacity and isolate workloads. Each group polls its own Queues, claims jobs in Due Time order, runs them through its own pool, and reports outcomes, all on its own polling cadence with its own lease timing, Retry Policy, and retention. A process can host several groups, so a noisy or latency-sensitive Queue can have dedicated capacity that nothing else competes for. At least one group is required for any job to run.

## Registering a worker group

You add a group while configuring BackWave, through the builder's `AddWorkerGroup` method. It takes a `WorkerGroupOptions` record. Only two fields are required: a `Name` that is unique within the registration, and a `Policy` that says which Queues the group serves.

```csharp title="Program.cs"
builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("emails", "reports"),
    }));
```

Call `AddWorkerGroup` more than once, with distinct `Name`s, to register several groups in one process. A duplicate `Name` throws at startup, so each group is guaranteed its own identity in health, metrics, and logs.

## Which queues a group serves

The Queues a group serves and the way it shares effort across them are the same single field: `Policy`. There is no separate queues property. The `DispatchPolicy` you assign carries the list of Queues, and BackWave uses it to decide which Queue to claim from next. Within whichever Queue it picks, jobs are claimed in Due Time order.

Priority lives in the policy, never on the job. Jobs carry no priority of their own. BackWave ships two policies, and both are work-conserving: a Worker never sits idle while any served Queue has due work.

```csharp title="Program.cs"
// Strict priority: "critical" is always tried before "default".
Policy = new DispatchPolicy.Strict("critical", "default")

// Weighted fair-share: roughly 6:3:1 across three queues.
Policy = new DispatchPolicy.Weighted(new[] { ("a", 6), ("b", 3), ("c", 1) })
```

`DispatchPolicy.Strict` serves its Queues in fixed priority order. The first Queue is always tried before the second, and a later Queue is reached only when every earlier one has no due work. A continuously busy high-priority Queue can therefore starve the tail, which is the deliberate trade-off you accept when you choose strict ordering.

`DispatchPolicy.Weighted` shares claim opportunity across Queues in proportion to their weights. A weighting like 6:3:1 is honoured smoothly and deterministically, with no random clumping. The higher-weight Queue serves more often while every Queue keeps making progress, and a Queue with no due work yields its turn. Every weight must be at least 1, and a Weighted policy needs at least one Queue, or it throws when you build it.

The full semantics of both policies, including starvation behaviour and weight tuning, live in [Dispatch Policies](/docs/reference/dispatch-policies).

## Concurrency: workers and the pool

`PoolSize` is the group's Worker count: the most jobs it runs at once on one node. It defaults to 20. Polling pauses while the pool is full and resumes as jobs finish, so the pool size is a hard ceiling on concurrent execution per group, per node.

That default is a flat constant. It is not scaled to the core count of the machine, so a laptop and a 64-core server both start at 20. The reasoning is straightforward. BackWave executes jobs asynchronously, so there is no thread-per-job that could starve a thread pool, which is the failure mode CPU-scaled defaults exist to bound. A machine-independent admission bound keeps the worst case predictable and protects against silently exhausting a database connection pool on a large host. Idle pool slots cost nothing, so a generous flat ceiling is a safe starting point. Tuning is yours: override `PoolSize` per deployment, and compute it from the core count yourself if you want core-relative sizing.

### Workers and Pumps are different things

A Worker is an execution slot, counted by `PoolSize`. A Pump is fetch-loop parallelism, counted by `Pumps`, which defaults to 1 and must be at least 1. Each Pump runs its own independent claim-and-execute loop with its own pool, claiming a disjoint set of jobs from the store.

The two scale differently. Idle Workers are free, but each Pump draws database connections whether it is busy or not, roughly 2 to 3 per Pump. A group with `Pumps = 4` holds around 8 to 12 connections at steady state. Leave `Pumps` at 1 unless a node is genuinely throughput-bound and has connection headroom to spare. Raising it is a throughput lever for the case where a single Pump is limited by store round-trip latency rather than by concurrency or CPU. The throughput model behind that lever is covered in [Throughput & Pump Fan-Out](/docs/advanced/throughput-model).

## Isolating workloads with multiple groups

Each Worker Group is self-contained. It has its own pool, poll cadence, lease timing, Retry Policy, retention, and set of served Queues. Putting a Queue in its own group gives its jobs dedicated capacity and settings, so a flood on one Queue cannot drain the Workers another Queue depends on. Hosting several groups in one process is the supported way to keep workloads apart.

`PoolSize` is node-local backpressure: it bounds concurrency for one group on one node. It is distinct from a Queue's Concurrency Limit, which is a separate cluster-wide cap enforced when jobs are claimed. See [Queues](/docs/core-concepts/queues) for that limit and how Due Time ordering works inside a Queue.

## When a group hits a hard error

BackWave checks its own internal invariants in release builds, not only in debug. If one of those invariants is violated, the affected group fail-stops: it stops claiming, its in-flight leases lapse so healthy nodes inherit the work, a health check goes red, and a single high-severity log names the failure. The failure is contained. The host application keeps serving traffic, because BackWave is a guest library and never crashes the host process, and the enqueue client keeps working, because enqueuing is a separate failure domain from the Workers. When a group runs multiple Pumps, only the faulting Pump halts while its siblings keep claiming.

Transient store faults, like a connection reset or a brief failover, are not treated as invariant violations. The group is marked degraded rather than halted, logs a warning, and retries on the next poll. The fail-stop is reserved for a genuinely corrupted assumption, where stopping cleanly is safer than quietly mis-processing jobs. The mechanics, including degraded-versus-halted handling and the health surfaces, are covered in [Always-On Assertions & Fail-Stop](/docs/advanced/fail-stop).

## Settings and defaults

`WorkerGroupOptions` exposes the full set of knobs. `Name` and `Policy` are required; everything else has a default you can leave alone.

| Setting | Default | What it controls |
|---|---|---|
| `Name` | required | Unique group name within one registration; appears in health, metrics, and logs. A duplicate throws. |
| `Policy` | required | Which Queues the group serves and how it shares effort across them. |
| `PoolSize` | 20 | Worker count: max jobs run at once per node. Flat constant, not core-scaled. Idle slots are free. |
| `Pumps` | 1 | Independent fetch loops in one process. Must be at least 1. Budget about 2 to 3 DB connections each. |
| `MaxClaimBatch` | 32 | Max jobs claimed in a single poll. Effectively capped at `PoolSize`. |
| `MaxOutcomeBatch` | `null`, falls back to `MaxClaimBatch` | Max completed outcomes buffered before one batched write. |
| `PollInterval` | 1 second | How often the group polls for work. Shorter means lower latency and more store queries. |
| `MaxPollInterval` | `TimeSpan.Zero`, meaning off | Longest wait between polls while the group is idle. Above `PollInterval` it enables idle backoff. |
| `MaintenanceInterval` | 5 seconds | How often background maintenance runs. Keep it slower than `PollInterval`. |
| `LeaseDuration` | 60 seconds | How long a claimed job's lease is held before it lapses. |
| `HeartbeatInterval` | `null`, falls back to one-third of `LeaseDuration` | How often the group renews in-flight leases. |
| `RetryPolicy` | `RetryPolicy.Default` | Backoff schedule and attempt ceiling before a job is dead-lettered. A `[Retry]` Job overrides it. |
| `Retention` | `RetentionPolicy.Default`, or `null` to disable | How terminal jobs are kept and purged. |

A few settings throw rather than coerce. A duplicate group `Name` and a `Pumps` value below 1 throw at startup. A Weighted policy with no Queues, or any weight below 1, throws when the policy is built.

`MaxPollInterval` is the one setting that changes how often the group queries the store. Left at zero, the group polls at the fixed `PollInterval`, which is the behavior of every earlier version. Set it above `PollInterval` and an idle group stretches its delay toward that ceiling. When the store reports the instant the next scheduled job comes due, the group sleeps until then. It drops back to `PollInterval` as soon as a poll claims work, or the store reports work due now that a limit held back. The ceiling is the worst-case pickup latency, so a mostly idle fleet trades a little latency for far fewer queries. [Configuration](/docs/reference/configuration#idle-poll-backoff) covers the rules in full.

## Where to go next

- [Queues](/docs/core-concepts/queues): what a Queue is, the cluster-wide Concurrency Limit, and Due Time ordering within a Queue.
- [Scheduling](/docs/core-concepts/scheduling): how work becomes due and how recurring schedules feed a Queue.
- [Dispatch Policies](/docs/reference/dispatch-policies): the full Strict and Weighted semantics, starvation, and weight tuning.
- [Throughput & Pump Fan-Out](/docs/advanced/throughput-model): when raising `Pumps` helps, and the round-trip-latency ceiling behind it.
- [Always-On Assertions & Fail-Stop](/docs/advanced/fail-stop): the fail-stop mechanics, degraded versus halted, and the health checks.