The Effect-Once Fence

How state transitions stay exactly-once while handler execution is at-least-once: the worker-and-attempt fence at the storage boundary, what happens to a stale worker's late report, and what the fence does and does not promise your handler.

BackWave makes two promises that only sound like they cancel out. The first is At-Least-Once Execution: your handler body may run more than once for a single job, and nothing in the system will change that. The second is that a job's recorded state transitions happen exactly once — the Effect-Once guarantee. Both hold at the same time because they describe different things: the work your code performs, versus the outcome the store commits. The bridge between them is a fence at the storage boundary, keyed on the Worker that claimed the current Attempt and on the Attempt number itself. When two workers each believe they own the same job, that fence is the reason only one of them can ever write the result. This page walks the mechanism — the race it exists to win, where it lives, and, most of all, the line it draws between what BackWave de-duplicates for you and what your handler still has to tolerate.

Two guarantees, one system#

Exactly-once handler execution — running your job body a single time, ever, no matter what fails — is not on offer, and not from BackWave alone. There are only two roads to it, and both are dead ends. One is at-most-once: accept that a crash at the wrong moment simply loses the job. The other is durable execution, where the engine owns your side effects inside a transaction it controls — a different product category, a workflow engine, not a background job queue. BackWave is deliberately neither. So the honest position is the one it takes: a system that advertises exactly-once handler execution is moving the idempotency burden onto a narrow set of engine-controlled operations, not removing it from the world. The moment your handler calls out to a payment API or an SMTP server, that call lives outside any transaction the queue can roll back.

What BackWave does instead is split the promise cleanly along the boundary it can actually enforce. At-least-once covers the body; Effect-Once covers everything the store records.

So the practical reading, before any mechanism: design your handler for the possibility that its body runs twice, and rely on the store — not on hope that the body ran once — for anything that has to be true exactly once.

The race: a slow worker and an expired lease#

Picture worker A. It claims the job, starting the first Attempt, and holds a Lease — a time-boxed grant of ownership that expires on its own rather than being held indefinitely. Then A stalls. The cause does not matter much: a garbage-collection pause, lost connectivity, a long blocking call to a slow dependency. What matters is that A goes quiet while still, from its own point of view, holding the Lease and running the handler.

This is a Node Isolation episode. An isolated node keeps executing its handler and keeps believing it owns the Lease, but it cannot fork the authoritative state of the job, because the database is the single authority and the node carries no state that can diverge from it. A is acting on a stale lease belief, nothing more.

Meanwhile the Lease has a finite life, and A's is now past it. A lapsed Lease counts as a failed Attempt, so the store is free to hand the job out again. Worker B claims it, starting the next Attempt, and begins its own execution. Now there are two live executions of one job, both sincere: A, which never learned it lost the Lease, and B, which legitimately holds it. Eventually A wakes up and tries to report its outcome for work it genuinely finished. The full lease-death timeline — expiry, reclaim, and inheritance — is its own subject, covered in Leases & Crash Recovery.

At this instant the system has two executions in flight and one late report about to arrive for a job the world has already moved past.

The fence at the storage boundary#

Every claim stamps the job with the claiming Worker's identity and the Attempt number, and each Attempt bumps that number forward. Every outcome report a worker submits carries the same pair back to the store. The store commits an outcome only when that pair still matches the job's live Lease: the same Worker, the same Attempt, and the Lease not yet expired. Owner identity alone is not enough — the Attempt number has to match too.

That single rule settles the race. When B re-claimed the job, the Attempt number advanced. A's late report still carries the earlier Attempt number, so it matches nothing the store currently recognizes. The report is rejected as stale: the outcome is not applied, the transition A wanted to make never happens, and any buffered annotations A meant to attach along with it — tag changes, job output — are discarded with the rejected write, so a stale node never leaves partial or conflicting state behind.

Where this check lives is the whole point. It runs inside the store's outcome commit, the one operation every node must pass through to record a result. There is no leader to elect and no cross-node coordination to arrange, because the database is the single shared authority and there is no node-local state to reconcile against it. The fence is just a local consequence of a write that everyone already funnels through the same place. For why the store is the one shared authority that makes this possible, see The Determinism Boundary.

Because every downstream effect is itself a store-applied consequence of that one outcome write, fencing the write buys the rest for free. The job's terminal state, the decrement that unlatches jobs waiting on this one, and the release of a Concurrency-Limit slot all flow from the same commit. Fence the outcome and each of those inherits the same exactly-once property, uniformly, no matter which database backs the store. Your handler does not participate in this and cannot weaken it; it is enforced beneath you.

What the fence does not stop#

Read the previous section again for what it actually rejected: A's report. It did nothing about A's work. By the time A tried to commit, its side effects had already happened: the email sent, the charge posted. The fence guarantees the recorded outcome is written once; it cannot reach out into the world and un-send an email.

That is the entire shape of the guarantee in one line: the record is exactly-once, the work is at-least-once. Two nodes executing the same job at the same moment is legal and correct under at-least-once — the fence does not prevent the concurrent runs, it only ensures that the two runs cannot both write conflicting results. So the duplicate you must design for is not a bookkeeping glitch the store will quietly absorb. It is a real second execution of your handler body, with real external effects, that already completed before anything got fenced.

Designing under the fence#

Everything above is context for one question: what does it mean for your handler? Each of the following is a way to make a second body execution harmless, since the fence guarantees the record but never the work.

Derive an idempotency key from the job context. BackWave hands your handler a JobContext carrying JobContext.JobId and JobContext.Attempt. JobId is stable for the life of the job across every Attempt; Attempt starts at 1 on the first try and climbs with each retry. Key your side effect on JobId — the identity of the unit of work no matter how many times its body runs — and record that key alongside the effect, so a repeat execution can look it up and see the work is already done. Attempt is there too when you want to distinguish tries, but it is JobId that gives you one identity across the duplicates.

Prefer naturally idempotent work. Some operations are already safe to repeat: an upsert keyed on a stable id, a "set status to X", a write to a fixed path. Run them twice and the second run is a no-op. Where you can shape the work this way, you need no key at all — the fence protects the record and the operation protects itself.

Use Transactional Enqueue for the enqueue-side version of the problem. The same hazard exists at the other end of a job's life: you want to write a business row and enqueue the job that acts on it without a window where one commits and the other does not. Where the store supports it, Transactional Enqueue lets you enqueue on the very same database transaction as your business write, so both commit or neither does — which is why you never need a separate relay table to bridge the two. Transactional Enqueue with EF Core walks the pattern.

Recognize that checking your own database is the same move. Guarding a side effect behind a marker you keep in your own store — "have I already done this?" — is the fence idea one level up. You are making an external effect conditional on a durable record, exactly as BackWave makes the outcome conditional on a live Lease. The store fences the job's outcome; you fence the job's effects.

Honor your CancellationToken, because cancel and the fence combine. Cancellation in BackWave is cooperative: an operator cancel sets a flag that the worker learns on its next heartbeat, which then trips the handler's CancellationToken. A handler that honors the token unwinds promptly. A handler that ignores it can run all the way to a "successful" finish — and then discover its outcome report fenced out as stale, exactly as A's was, because the Lease lapsed while it was busy ignoring the signal. Honoring the token is therefore not only about responsiveness; it is how you avoid grinding through work whose result the store will refuse to record. Cancel a Running Job covers the handler side.

Where to go next#

• Execution Guarantee: the promise this page explains the mechanism for.
• Leases & Crash Recovery: the lease-expiry and reclaim half of the race, in full.
• The Determinism Boundary: why the store is the one shared authority that lets the fence work without a leader.
• Retries & Error Handling: writing handlers that stay correct across the Attempts the fence permits.