Prioritize Work with Queues

Set up a critical lane and a bulk lane with named queues, choose strict versus weighted dispatch, isolate slow work in its own Worker Group, and cap a queue cluster-wide.

Set up a critical lane and a bulk lane with named queues, choose strict versus weighted dispatch, isolate slow work in its own Worker Group, and cap a queue cluster-wide.

BackWave has no per-job priority number. Priority is expressed structurally: you route jobs onto named Queues and tell each Worker Group how to divide its attention across them. This guide walks the common shape end to end — a latency-sensitive lane that must never wait behind bulk work — and shows when to reach for a Dispatch Policy, when to split off a second Worker Group, and when to cap a queue with a cluster-wide Concurrency Limit.

If you have not read Queues and Worker Groups & Dispatch yet, skim them first. They define the model this guide leans on: Queue, Worker Group, Pump, Dispatch Policy, and Concurrency Limit. The running example throughout is a critical lane (password-reset emails, low volume, latency-sensitive) competing with a bulk lane (search-index rebuilds, slow, high volume).

Priority is routing, not a number#

Every job belongs to exactly one Queue. You declare it on the [Job] attribute through the Queue property, and unless an enqueue call says otherwise, every job of that type lands there. When Queue is omitted it defaults to "default".

[Job("password-reset", Queue = "critical")]
public Task SendPasswordReset(Guid userId, CancellationToken cancellationToken)
    => /* send the email */;

The declared Queue is a default, not a fixture. Any single enqueue can override it by passing queue, which routes that one job onto a different lane while leaving the type's default untouched.

// normally "critical" per the attribute; this one call diverts to "bulk"
await client.EnqueueAsync(new SendPasswordReset(userId), dueTime: DateTimeOffset.UtcNow, queue: "bulk");

What you will not find is a priority field on the job. There is no number to bump, no HighPriority = true. Priority is not a property of the work; it lives in the consumer's Dispatch Policy — the rule a Worker Group follows when it decides which Queue to pull from next. Two jobs are ordered relative to each other only because they sit on different Queues and a policy ranks those Queues. This is deliberate: it keeps ordering a property of your topology, which you can reason about and change in one place, rather than a per-message attribute scattered across every call site.

If you are arriving from Hangfire, the shift is small. Hangfire queues are also drained in an order the server defines, so the mental model — name your lanes, then tell the consumer how to walk them — already fits. What changes is that in BackWave the walking rule is an explicit, first-class object you construct, described next.

Strict priority with DispatchPolicy.Strict#

A DispatchPolicy.Strict (from the BackWave.Core namespace) lists Queues in priority order. The Worker Group always tries an earlier Queue before a later one, and reaches a later Queue only when every earlier Queue has no due work. The constructor takes the names either as a list or as loose arguments, so the common case reads naturally.

Policy = new DispatchPolicy.Strict("critical", "bulk"),

With this policy the group empties critical completely before it touches bulk, on every pass. A password reset enqueued while a thousand index-rebuild jobs are waiting still runs next.

Be clear-eyed about the trade. Strict priority accepts tail starvation: if critical is continuously busy, bulk can wait indefinitely, and that is by design, not a bug to tune around. It is the right choice precisely when the high-priority lane is low-volume by construction — password resets arrive in a trickle, so bulk gets the vast majority of the group's time in practice and only ever yields during the brief moments a reset needs to jump the line. When the head lane can realistically saturate, you want fair sharing instead.

Fair sharing with DispatchPolicy.Weighted#

A DispatchPolicy.Weighted divides attention across Queues in proportion to integer weights rather than draining strictly in order. It takes a list of (queue, weight) pairs — there is no loose-argument form, so pass the tuples as a collection.

Policy = new DispatchPolicy.Weighted([("high", 6), ("low", 1)]),

The selection is a smooth weighted round-robin: a 6:1 split is honored exactly and deterministically, interleaving turns evenly rather than running six high jobs in a random clump and then one low. There is no randomness to average out over time; the ratio holds turn by turn.

Weights are shares, not reservations, and the policy is work-conserving: a worker never idles while any served Queue has due work. If low is momentarily empty, its turn flows to high instead of being held open — you get the full ratio only while both lanes have work, and all available capacity otherwise. Weights are positive integers; the larger relative to its peers, the more turns a Queue earns.

Choosing between the two policies comes down to whether starving the tail is acceptable:

Both policies live in BackWave.Core; see Dispatch Policies for the full reference.

Isolate slow work in its own Worker Group#

A Dispatch Policy decides which Queue a group pulls from next, but every job the group claims still runs in the same shared pool of execution slots. When bulk jobs are long-running, that is the limit dispatch tuning cannot fix: a burst of slow index rebuilds can occupy every slot for minutes, and a password reset the policy would happily prioritize has nowhere to run until a slot frees. No weighting changes that, because the contention is over execution capacity, not fetch order.

The fix is to give the slow work its own Worker Group. A single process can host several groups, each with its own Dispatch Policy and its own pool, and each claiming independently. Bind one group to bulk alone and cap its resources; leave the other to serve critical on its own dial.

using BackWave.Core;
using BackWave.Hosting;
 
backwave.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "critical",
    Policy = new DispatchPolicy.Strict("critical"),
});
 
backwave.AddWorkerGroup(new WorkerGroupOptions
{
    Name = "bulk",
    Policy = new DispatchPolicy.Strict("bulk"),
    PoolSize = 4,
});

Now a flood of bulk work can occupy at most its own group's slots; the critical group's capacity is untouched, and resets keep running. Two knobs govern how much a group can run at once:

The distinction matters when you size a group. PoolSize is execution concurrency and Pumps is fetch-loop parallelism: each Pump runs its own pool, so a group's node-local ceiling is effectively PoolSize × Pumps, multiplied again across the nodes running the group. A group with PoolSize = 4 and the default single Pump runs at most four bulk jobs at once per node. Splitting a lane into its own group also narrows the blast radius: a group that fail-stops takes down only its own lane, so a defect in bulk handlers cannot halt critical.

Sizing these numbers — matching PoolSize to your workload, adding Pumps, budgeting store connections — is its own topic. See Scaling & Throughput Tuning rather than guessing here.

Cap a queue cluster-wide with a Concurrency Limit#

PoolSize bounds execution on one node, but it says nothing about the cluster as a whole: run three nodes with a pool of four each and a lane can execute twelve jobs at once. When the constraint is global — an upstream API that tolerates only N concurrent calls, a rate-limited partner, a database that must not see more than a handful of a certain job at a time — a per-node pool is the wrong tool. That is what a Concurrency Limit is for.

A Concurrency Limit is a per-Queue, cluster-wide cap on how many of its jobs run simultaneously, enforced at claim time: once a Queue is at its cap, no worker anywhere claims more of its work until a running job finishes. It is a different gate from PoolSize and Backpressure — those are each node's own pool capacity, while the Concurrency Limit is one shared counter across the whole cluster. The two kinds of cap are independent; a job must clear both to run. A slot is counted as used while a job is leased and frees the instant the job reaches a terminal state or its lease expires, so a crashed node never leaks a slot — the cap self-heals by construction rather than needing a cleanup sweep.

A Concurrency Limit is configured through the storage layer today; it does not yet have a dedicated hosting or operator surface. If you need to cap a Queue, coordinate with your BackWave setup — treat the cap as an operational configuration on the store rather than something you wire in the AddBackWave builder, and expect a first-class surface in a future release.

Reading the cap back is first-class. The Monitor's GetQueueSettingsAsync returns each Queue's operational settings — its name, whether it is paused, and its configured Concurrency Limit, which is null when the Queue is uncapped. Pair it with GetQueueDepthsAsync for how much work is waiting and in flight per Queue. The dashboard Queues view renders the same numbers live, showing in-use against the cap (and infinity when there is none), so you can watch a capped lane hold at its ceiling under load.

Operating the lanes#

Once the lanes are live you operate them without redeploying. Pausing a Queue is an Operator Action: BackWaveOperator.PauseQueueAsync stops the whole cluster from claiming new work off that Queue until ResumeQueueAsync lifts it, and jobs already running are left alone to finish. Draining bulk before a migration, or holding a lane while an upstream dependency recovers, is a pause and a later resume rather than a config change. See operator actions for the full set.

For visibility, the Monitor's queue-depth read shows what is waiting and running per Queue, which is how you confirm a Strict lane is draining, a Weighted split is holding its ratio, or a capped Queue is sitting at its ceiling. The Monitor API covers the read surface in full.

Where to go next#

• Queues: the model behind named lanes and per-job routing.
• Worker Groups & Dispatch: how a group claims, dispatches, and runs work.
• Dispatch Policies: the full reference for Strict and Weighted.
• Scaling & Throughput Tuning: sizing PoolSize, Pumps, and connections.
• Configuration: every option involved in registering BackWave.
• Operator actions: pause, resume, and the rest of the runtime controls.