# Execution Guarantee

Two windows make exactly-once handler execution impossible. A Worker can run
your handler to completion and then crash, or be cut off from the store, before
the outcome is recorded. Its Lease then expires, expiry counts as an Attempt the
same as a thrown exception, the job becomes claimable again, and another node
runs the handler a second time. There is no instant where "the work ran" and
"the outcome is durably recorded" are the same atomic fact, so no system can
promise a job body executes exactly once.

BackWave does not pretend otherwise. It gives you two precise guarantees instead.
The handler body is **At-Least-Once Execution**: it may run more than once, and
idempotency is your responsibility. The recorded outcome and the state changes
that flow from it are **Effect-Once**: they apply exactly once at the
Storage Contract boundary, no matter how many times the body ran.

## At-Least-Once Execution

At-least-once is BackWave's delivery contract. A job's handler body may run more
than once for the same job. This is encoded directly in the handler interface:
`IJobHandler<TJob>.HandleAsync` runs one Attempt, where returning normally
succeeds and throwing fails and may retry.

Re-execution is a normal, designed-for event, not a rare failure. A Lease is
time-bounded and heartbeat-renewed, and its expiry returns the job to a
claimable state. A Worker that does the work but dies before reporting the
outcome will have that job picked up and run again by another node. Plan for it.

At-least-once is the field standard, not a BackWave compromise. Hangfire performs
each job "at least once" and tells you to make methods idempotent. Sidekiq is
explicit that it is "at least once, not exactly once." Celery's `acks_late`,
River's "make every operation idempotent," and RabbitMQ-with-acks all land in the
same place. Even Temporal, which offers exactly-once at its workflow layer,
delivers its activities at-least-once.

## Effect-Once at the Storage Contract boundary

At-least-once handles the body. Effect-Once handles everything the store records.
The recorded outcome of an Attempt and every state transition that flows from it,
the terminal state, the Dependency latch decrement, the Concurrency Limit slot
release, all apply exactly once, caused only by the node holding the live Lease
for that exact Attempt.

So two facts hold at the same time. The handler body may have run three times
across three nodes. The outcome write and its downstream transitions still apply
once. The boundary is geographic. At-least-once holds where the handler runs,
above the storage boundary, in your code; exactly-once holds where the store
applies the outcome, at the Storage Contract boundary.

This matters for what Effect-Once does not cover. It is a Lease fence, not
deduplication of your handler's own writes. The email your handler sends and the
row it inserts are not fenced. They can happen twice. Only BackWave's own outcome
and transition write is fenced. That gap is exactly why handler idempotency is
required.

### The (workerId, attempt) fence

The mechanism is a single fence on the outcome write. When a worker reports an
outcome, it passes its own worker id and the attempt number it ran, and the store
applies the outcome only when that pair still holds the live Lease for exactly
this Attempt. A wrong worker, a wrong attempt number, or an expired Lease is
rejected as a stale lease, and nothing changes. That rejection is what makes a
late report from an isolated-then-recovered worker harmless.

The cooperative half of the fence runs on the heartbeat. For any job a worker no
longer holds, its next heartbeat tells it so, which is its cue to stop applying
that job's effects rather than push toward a write the fence would reject anyway.

The fence is a single chokepoint, and that is what makes Effect-Once cheap. The
functional Core makes every downstream effect a store-applied consequence of the
outcome write. The Shell never fires the latch decrement or the slot release on
its own. So fencing the one write gives downstream-once for free, because the
effects cannot fire twice when the outcome that triggers them cannot apply twice.

Two nodes executing the same job concurrently is legal and correct, at-least-once
working as designed. Only a stale store write that mutated state would be a bug,
and the fence is precisely what prevents it.

### It is verified, not asserted

Effect-Once is a mechanically simulated invariant. Node Isolation, a node cut off
from the store that keeps executing under a stale Lease belief and then heals,
manufactures the exact stale-write interleaving the fence exists to reject. The
Simulator's outcome-provenance oracle proves Effect-Once holds across those
interleavings, and a sabotage self-test removes the fence and must trip the
oracle. A conformance suite then pins the behavior on every adapter, so Postgres,
SQL Server, SQLite, EF, and the In-Memory Store all enforce the same fence.

## Writing idempotent handlers

Because the body may run more than once, your handler must be safe to run more
than once. That is your responsibility, and BackWave gives you the Attempt number
to key on. `JobContext.Attempt` starts at 1 and is visible inside the handler.

Three patterns cover most work.

- **Natural idempotency.** Make the write one whose repeat is a no-op, like
  setting a state, upserting by key, or any PUT-style write. Running it twice
  lands on the same result.
- **Dedupe keys.** Record a unique key for the unit of work, the `JobId` or a
  business key, and skip the work when the key is already present.
- **Conditional writes.** Push the guard into the store, with an
  `INSERT ... ON CONFLICT DO NOTHING` or an `UPDATE ... WHERE` on a not-yet-done
  condition.

The last two want a database, and BackWave wires for it. Each Attempt opens a
fresh DI scope, so a handler can inject a scoped `DbContext` for the dedup or
unit-of-work write that makes it idempotent. The scope wraps exactly one
Attempt's invocation.

## The enqueue side is exactly-once when co-committed

Execution is at-least-once. The enqueue can be exactly-once. With
Transactional Enqueue, a job is committed inside your application's own database
transaction, so the business write and the job commit or roll back atomically.

```csharp title="Program.cs"
using var scope = sp.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<SampleDbContext>();

var rowId = Guid.NewGuid();
await using var transaction = await db.Database.BeginTransactionAsync();

db.BusinessRows.Add(new SampleBusinessRow { Id = rowId, Note = fail ? "to be rolled back" : "committed" });
await db.SaveChangesAsync();

// The job rides on the EF transaction: it commits or rolls back atomically with the row.
var jobId = await client.EnqueueAsync(new TxFinalize(rowId), db, dueTime: DateTimeOffset.UtcNow);

if (fail)
{
    await transaction.RollbackAsync();
    return Results.Ok(new { rowId, jobId, committed = false, note = "Both the business row and the job were rolled back." });
}

await transaction.CommitAsync();
return Results.Ok(new { rowId, jobId, committed = true, note = "The business row and the job committed atomically." });
```

The result is one durable job row, no duplicate enqueue and no lost enqueue.
Execution of that one job is still at-least-once. The
[Transactional Enqueue](/docs/guides/transactional-enqueue-with-ef-core) page owns the
co-commit mechanics and the `SupportsTransactionalEnqueue` capability.

## The same discipline applies to observers

The at-least-once contract is consistent across BackWave's other egress. The
Transition Observer that fires when a job reaches a declared state is delivered
at-least-once and is not Effect-Once. Its fire is a new side effect outside the
(workerId, attempt) fence, so duplicates are legal by design and the
subscriber's reaction needs the same idempotency discipline a handler carries.

## Where to go next

- [Job Lifecycle](/docs/core-concepts/job-lifecycle): the state machine, how a
  Lease expiry counts as an Attempt, and the terminal transitions Effect-Once
  protects.
- [Jobs and Handlers](/docs/core-concepts/jobs-and-handlers): `IJobHandler<T>`,
  the `[Job]` attribute, and `JobContext.Attempt`, where you write the idempotent
  code.
- [Execution Model](/docs/core-concepts/execution-model): the Core and Shell split
  and where the storage boundary sits.
- [Dependencies](/docs/core-concepts/dependencies): the Dependency latch decrement
  that Effect-Once makes exactly-once.
- [Transactional Enqueue](/docs/guides/transactional-enqueue-with-ef-core): the exactly-once
  enqueue side in full.
- [Transition Observers](/docs/advanced/transition-observer-internals): the second
  place at-least-once and idempotency apply.