Execution Guarantee

At-least-once execution with effect-once at the storage boundary: a handler may run more than once, so idempotency is your responsibility.

Two windows make exactly-once handler execution impossible. A Worker can run your handler to completion and then crash, or be cut off from the store, before the outcome is recorded. Its Lease then expires, expiry counts as an Attempt the same as a thrown exception, the job becomes claimable again, and another node runs the handler a second time. There is no instant where "the work ran" and "the outcome is durably recorded" are the same atomic fact, so no system can promise a job body executes exactly once.

BackWave does not pretend otherwise. It gives you two precise guarantees instead. The handler body is At-Least-Once Execution: it may run more than once, and idempotency is your responsibility. The recorded outcome and the state changes that flow from it are Effect-Once: they apply exactly once at the Storage Contract boundary, no matter how many times the body ran.

At-Least-Once Execution#

At-least-once is BackWave's delivery contract. A job's handler body may run more than once for the same job. This is encoded directly in the handler interface: IJobHandler<TJob>.HandleAsync runs one Attempt, where returning normally succeeds and throwing fails and may retry.

Re-execution is a normal, designed-for event, not a rare failure. A Lease is time-bounded and heartbeat-renewed, and its expiry returns the job to a claimable state. A Worker that does the work but dies before reporting the outcome will have that job picked up and run again by another node. Plan for it.

At-least-once is the field standard, not a BackWave compromise. Hangfire performs each job "at least once" and tells you to make methods idempotent. Sidekiq is explicit that it is "at least once, not exactly once." Celery's acks_late, River's "make every operation idempotent," and RabbitMQ-with-acks all land in the same place. Even Temporal, which offers exactly-once at its workflow layer, delivers its activities at-least-once.

Effect-Once at the Storage Contract boundary#

At-least-once handles the body. Effect-Once handles everything the store records. The recorded outcome of an Attempt and every state transition that flows from it, the terminal state, the Dependency latch decrement, the Concurrency Limit slot release, all apply exactly once, caused only by the node holding the live Lease for that exact Attempt.

So two facts hold at the same time. The handler body may have run three times across three nodes. The outcome write and its downstream transitions still apply once. The boundary is geographic. At-least-once holds where the handler runs, above the storage boundary, in your code; exactly-once holds where the store applies the outcome, at the Storage Contract boundary.

This matters for what Effect-Once does not cover. It is a Lease fence, not deduplication of your handler's own writes. The email your handler sends and the row it inserts are not fenced. They can happen twice. Only BackWave's own outcome and transition write is fenced. That gap is exactly why handler idempotency is required.

The (workerId, attempt) fence#

The mechanism is a single fence on the outcome write. When a worker reports an outcome, it passes its own worker id and the attempt number it ran, and the store applies the outcome only when that pair still holds the live Lease for exactly this Attempt. A wrong worker, a wrong attempt number, or an expired Lease is rejected as a stale lease, and nothing changes. That rejection is what makes a late report from an isolated-then-recovered worker harmless.

The cooperative half of the fence runs on the heartbeat. For any job a worker no longer holds, its next heartbeat tells it so, which is its cue to stop applying that job's effects rather than push toward a write the fence would reject anyway.

The fence is a single chokepoint, and that is what makes Effect-Once cheap. The functional Core makes every downstream effect a store-applied consequence of the outcome write. The Shell never fires the latch decrement or the slot release on its own. So fencing the one write gives downstream-once for free, because the effects cannot fire twice when the outcome that triggers them cannot apply twice.

Two nodes executing the same job concurrently is legal and correct, at-least-once working as designed. Only a stale store write that mutated state would be a bug, and the fence is precisely what prevents it.

It is verified, not asserted#

Effect-Once is a mechanically simulated invariant. Node Isolation, a node cut off from the store that keeps executing under a stale Lease belief and then heals, manufactures the exact stale-write interleaving the fence exists to reject. The Simulator's outcome-provenance oracle proves Effect-Once holds across those interleavings, and a sabotage self-test removes the fence and must trip the oracle. A conformance suite then pins the behavior on every adapter, so Postgres, SQL Server, SQLite, EF, and the In-Memory Store all enforce the same fence.

Writing idempotent handlers#

Because the body may run more than once, your handler must be safe to run more than once. That is your responsibility, and BackWave gives you the Attempt number to key on. JobContext.Attempt starts at 1 and is visible inside the handler.

Three patterns cover most work.

  • Natural idempotency. Make the write one whose repeat is a no-op, like setting a state, upserting by key, or any PUT-style write. Running it twice lands on the same result.
  • Dedupe keys. Record a unique key for the unit of work, the JobId or a business key, and skip the work when the key is already present.
  • Conditional writes. Push the guard into the store, with an INSERT ... ON CONFLICT DO NOTHING or an UPDATE ... WHERE on a not-yet-done condition.

The last two want a database, and BackWave wires for it. Each Attempt opens a fresh DI scope, so a handler can inject a scoped DbContext for the dedup or unit-of-work write that makes it idempotent. The scope wraps exactly one Attempt's invocation.

The enqueue side is exactly-once when co-committed#

Execution is at-least-once. The enqueue can be exactly-once. With Transactional Enqueue, a job is committed inside your application's own database transaction, so the business write and the job commit or roll back atomically.

Program.cs
using var scope = sp.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<SampleDbContext>();
 
var rowId = Guid.NewGuid();
await using var transaction = await db.Database.BeginTransactionAsync();
 
db.BusinessRows.Add(new SampleBusinessRow { Id = rowId, Note = fail ? "to be rolled back" : "committed" });
await db.SaveChangesAsync();
 
// The job rides on the EF transaction: it commits or rolls back atomically with the row.
var jobId = await client.EnqueueAsync(new TxFinalize(rowId), db, dueTime: DateTimeOffset.UtcNow);
 
if (fail)
{
    await transaction.RollbackAsync();
    return Results.Ok(new { rowId, jobId, committed = false, note = "Both the business row and the job were rolled back." });
}
 
await transaction.CommitAsync();
return Results.Ok(new { rowId, jobId, committed = true, note = "The business row and the job committed atomically." });

The result is one durable job row, no duplicate enqueue and no lost enqueue. Execution of that one job is still at-least-once. The Transactional Enqueue page owns the co-commit mechanics and the SupportsTransactionalEnqueue capability.

The same discipline applies to observers#

The at-least-once contract is consistent across BackWave's other egress. The Transition Observer that fires when a job reaches a declared state is delivered at-least-once and is not Effect-Once. Its fire is a new side effect outside the (workerId, attempt) fence, so duplicates are legal by design and the subscriber's reaction needs the same idempotency discipline a handler carries.

Where to go next#

  • Job Lifecycle: the state machine, how a Lease expiry counts as an Attempt, and the terminal transitions Effect-Once protects.
  • Jobs and Handlers: IJobHandler<T>, the [Job] attribute, and JobContext.Attempt, where you write the idempotent code.
  • Execution Model: the Core and Shell split and where the storage boundary sits.
  • Dependencies: the Dependency latch decrement that Effect-Once makes exactly-once.
  • Transactional Enqueue: the exactly-once enqueue side in full.
  • Transition Observers: the second place at-least-once and idempotency apply.
Previous← Job LifecycleNextExecution Model →