Transition Observer Internals

Why lifecycle reactions are delivered by a lease-claimed walk of the Transition Log, with a per-observer cursor, at-least-once semantics, and a dead-letter ceiling.

A Transition Observer is host-supplied, egress-only code BackWave invokes when a job reaches a state you subscribe to. It is the sanctioned "do X when Y happens to a job" without polling. Internally it is not a callback list wired into the state machine but a durable, leaderless delivery subsystem that walks the append-only Transition Log as its source of truth, hands each matching transition to your callback at-least-once, and advances a per-observer cursor over what it has resolved. This page explains the mechanism end to end, from the sans-IO dispatch Core through the claim-and-lease fence that keeps delivery single in the happy path, and why the design deliberately refuses to be a filter or interceptor pipeline.

The Transition Log is the source of truth#

Observers are driven by a Lease-claimed walk of the Transition Log, not a live event bus. The store records every state change as it applies it, and the dispatch subsystem reads those recorded rows back out. There is no separate event stream to subscribe to. The recorded history is the stream.

Because delivery reads recorded history, it requires a Job History Policy of at least Transitions. With history Off there are no transition rows to read, so an observer could never fire. That constraint is enforced at container composition, not at first tick. Registering an observer under a history policy of Off throws an InvalidOperationException that names the offending observer and tells you to raise the policy, so the misconfiguration surfaces at startup rather than as silent non-delivery in production.

The subsystem also costs nothing when unused. If no observers are declared, no hosted service is registered, nothing polls, and the dashboard's observer surface is simply empty. You pay for the pump only once you subscribe.

Each recorded delivery row carries the transition facts eagerly, so a callback rarely needs a second store read. Alongside the facts the row keeps its own durable bookkeeping: the global log position the cursor advances over, the per-job ordinal, and a delivery-attempt counter that is distinct from the job's own Attempt.

The dispatch Core is a sans-IO state machine#

The heart of the subsystem is an internal Step(event) -> commands state machine, a sibling to the Node Driver. It is sans-IO in the same sense: it never awaits, times, or threads, and the surrounding Shell owns every clock and store call. The Core owns delivery ordering per cursor, attempt counting, backoff, the dead-letter ceiling, and bounded head-of-line advance, and nothing else. Keeping it a separate module from the Node Driver is deliberate, so job-claim logic and delivery logic never entangle and each stays testable in isolation.

Its only state is a per-node in-flight set: the observers this node has a claim outstanding for but has not yet reported on. That guard is what stops a single node from issuing a second concurrent claim for the same observer while a round-trip is in the air. When a poll comes due, the Core emits a claim for every observer not already in flight. An empty claim, meaning the lease is held elsewhere or nothing is due, releases the guard. A claim with rows drives an invoke, then a report.

One ordering detail keeps a failed report from stranding a node. The Core releases the in-flight guard at the invoke-to-report decision point, before the report is confirmed. A store fault on the report can therefore never wedge the node. A lost report simply leaves the cursor un-advanced, and the next poll re-claims and redelivers. If a claim or report round-trip faults at the Shell edge, the Core is told, releases the guard, and re-claims on the next poll, so one faulted call can never strand an observer.

The decision the Core makes for each invocation is small. A success becomes Delivered and the cursor may advance past the row. A failure is handed to the delivery retry policy, which either returns the next backoff instant (retry, cursor holds) or, once the ceiling is exhausted, dead-letters the row so one poison delivery cannot wedge every later notification behind it.

The dispatch pump is one per process#

The production Shell that drives the Core is a single background service, one per process. This is the one-per-host shape, not one-per-worker-group. A single pump owns one dispatch Core over every registered observer.

Delivery stays leaderless and database-authoritative. The per-observer cursor Lease serializes claims across nodes, so any number of these pumps may run at once. More pumps only mean faster delivery, never double delivery in the happy path. There is no elected leader to lose.

A timer raises a poll each PollInterval, and the loop walks the Core's commands: claim, invoke, report, and recurse when a drained batch signals more rows may be waiting. Claims and reports stay serialized on the loop as bounded store calls. The invocation itself is dispatched off the loop on a Task.Run, exactly the way job handlers run, and posts its result back when it completes. Because the Core releases its per-observer guard only when that result lands, the concurrency is across observers: one slow or hung observer never starves another. Within a single batch, invocations stay sequential in log order, because the cursor's contiguity depends on it.

Each observer is resolved fresh from a DI scope per delivery, opened inside the off-loop task, so it can take scoped dependencies such as a DbContext for its own dedup write. The pump's worker identity is auto-derived and fresh on every start, in a claim space disjoint from job Leases, so a crashed process's identity never resurfaces. Its cursor lease simply expires and another node re-claims.

The pump is fail-soft. A non-cancellation fault stops this one pump and is logged at error level, but it never fail-stops the host. The cursor lease lapses and another node picks the work up. Unlike a worker group, there is no health to turn red. See Always-On Assertions & Fail-Stop for how that contrasts with the job-execution path.

The claim, lease, and cursor mechanism#

Delivery rides the same Storage Contract claim-lease-heartbeat mechanism that job claiming uses, in a disjoint worker-id space. The contract has three moving parts, described here as read from the deterministic In-Memory Store; production Storage Adapters must honor the same semantics.

A claim asks the store for a bounded batch of one observer's undelivered rows under a lease. If a live lease is already held by a different worker, the store returns nothing, because that node is delivering, so this one backs off. That leaderless back-off is exactly what yields single delivery in the happy path. The store walks the log in append order from just past the cursor and hands back only rows matching the subscription's states, optional wire name, and optional queue. A non-matching row never blocks and is never delivered, and it never counts against a batch.

Reporting is fenced the same way the outcome write is fenced in the Execution Guarantee. Only the worker still holding the live claim lease may resolve deliveries and advance the cursor. A stale survivor of a lapsed claim reports into the void, and at-least-once stays intact because its rows were never resolved and will redeliver.

The cursor is a per-observer monotonic position over the global log, keyed by the observer's stable Id. It advances over the contiguous prefix of resolved rows, meaning every matching row up to that point is either Delivered or Dead-Lettered. It sweeps over non-matching rows freely and stops at the first matching row still pending. A row inside its backoff window stops the scan too, so nothing past it is claimed and deliveries stay in order per observer. This single moving position is where head-of-line ordering comes from. The initial cursor is -1, meaning nothing has been delivered yet, and an unknown observer reads back -1 as well.

At-least-once delivery and dead-lettering#

Observer delivery is at-least-once and, unlike a job's recorded outcome, explicitly not Effect-Once. The fire is a new side effect outside the job's (workerId, attempt) fence, so making the reaction idempotent is the subscriber's responsibility, the same contract a job handler already carries. A crash between the transition and the callback completing is covered by redelivery: the claim lease lapses, another node re-claims, and the un-resolved rows fire again.

A failed delivery is itself at-least-once work, so it reuses the same retry policy the Node Driver uses for a job Attempt. The delivery is held and retried with backoff until its ceiling, then dead-lettered. A hung callback is contained by proceed-not-await: each delivery races a DeliveryTimeout, and when the deadline wins, the delivery is recorded failed and the loop moves on without awaiting the orphaned callback. The orphan's eventual exception is still observed in the background and logged loudly as the subscriber's bug, so a token-ignoring callback that later throws can never surface as an unobserved task exception. Any throw, including a cooperative cancellation on the deadline token, is turned into a contained failure and never escapes to fail-stop the pump.

Worst-case latency for a fully-hung observer is bounded by roughly MaxBatch times DeliveryTimeout for that observer alone, after which it dead-letters and self-heals. Other observers are unaffected, because the concurrency is across observers. Treat that figure as a design bound, not a measured benchmark.

Dead-lettered deliveries are surfaced the way dead-lettered jobs are, and they are metadata-only: position, job id, ordinal, state, attempt counts, and timestamps, never payload or failure detail.

What an observer receives#

The callback is ITransitionObserver.OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken), returning ValueTask. Completing without throwing marks the delivery delivered. The token is cancelled when the dispatch timeout elapses or the host is shutting down, and honoring it is what keeps a slow reaction contained rather than left hanging.

public sealed class DeadLetterAlertObserver : ITransitionObserver
{
    public async ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken)
    {
        // At-least-once: this may run more than once for the same transition, so make it idempotent
        // (dedupe on (context.JobId, context.State)) and honor the token so a slow reaction is contained.
        var message = $"Job {context.JobId} ({context.WireName}) reached {context.State} on attempt {context.Attempt}.";
        // ...post to your alerting sink...
    }
}

ObserverContext carries the transition facts eagerly, so a useful message needs no extra store call.

The job payload is exposed through a lazy accessor. The store read is deferred to first touch and memoized, so a delivery that never reaches for the payload pays no read cost. Because a lazy read can race the retention sweep, the result reports whether the bytes are available, and the subscriber must check before reading them.

var payload = await context.Payload.GetAsync(cancellationToken);
if (payload.Available)
{
    ReadOnlyMemory<byte> bytes = payload.Bytes;
    // ...deserialize with your own serializer...
}
// else: the job was already purged under retention, so handle the absent case.

Registering and subscribing#

Observers are registered in one cohesive block through AddObservers. The block configures both the observer list and, optionally, the pump, so pump knobs and the observer set can never drift apart. The Id you give each observer keys its durable cursor: keep it stable and unique within the block. Reusing an Id across a restart resumes delivery where it left off, while changing it starts a fresh cursor from the beginning of the log. Registering the same Id twice throws.

bw.AddObservers(obs => obs
    // Audit everything.
    .Add<AuditObserver>("audit", ObserverSubscription.AllTransitions)
    // React only to PaymentJob dead-letters.
    .Add<DeadLetterAlertObserver>(
        "payment-dead-letters",
        new ObserverSubscription(new[] { JobState.DeadLettered }) { WireName = "PaymentJob" })
    .ConfigurePump(pump =>
    {
        pump.PollInterval = TimeSpan.FromSeconds(1);
        pump.DeliveryTimeout = TimeSpan.FromSeconds(30);
    }));

An ObserverSubscription declares which state transitions match, optionally narrowed by wire name and queue. ObserverSubscription.AllTransitions subscribes to every state, and you narrow it with a record with expression. Everything on the pump is optional, with sensible defaults.

Observing delivery health#

The read surface lives on BackWaveMonitor. GetObserverCursorAsync(observerId) returns the delivered-through global timeline position, or -1 when the observer has nothing delivered yet, including an unknown Id. Use it to gauge how far behind an observer is. ListObserverDeadLettersAsync(observerId) returns that observer's dead-lettered deliveries oldest first, empty for a healthy or unknown observer. Both are metadata-only, never payload or failure detail.

var cursor = await monitor.GetObserverCursorAsync("payment-dead-letters");
// -1 means nothing delivered yet, or an unknown id.
 
var deadLetters = await monitor.ListObserverDeadLettersAsync("payment-dead-letters");
foreach (var dl in deadLetters)
{
    // Metadata only: dl.JobId, dl.State, dl.DeliveryAttempts, dl.DeadLetteredAt, never payload.
}

The free dashboard renders an observer deliveries page from these same reads, and the registration list has exactly one home that both the pump and the dashboard read, so the observer set never forks.

Why not a filter or interceptor pipeline#

An observer only reacts. It can never veto, redirect, or rewrite a transition. It is strictly downstream of the Core's already-recorded decision, observing the transitions the Core produced rather than the events the Core consumes, which is why it is the push twin of the Monitor's pull. An in-path hook that could veto or rewrite a transition is permanently rejected by design, and the observer subsystem is the sanctioned way to react to the lifecycle.

Structurally, this downstream-only stance is what lets delivery be a durable log walk instead of an in-line call. The transition is already committed to the log before any observer runs, so a slow, failing, or crashed observer cannot stall or corrupt the job's own progress. It can only affect its own cursor.

The following surfaces are public and yours to call. The dispatch Core and pump are internal mechanism, described here but not part of the shipped API.

Where to go next#

• React to Job Outcomes: the task-focused guide to writing and registering an observer.
• The Sans-IO Node Driver: the sibling Core the dispatch Core mirrors.
• Execution Guarantee: the at-least-once and Effect-Once contract observers deliberately sit outside.
• Storage overview: the Storage Contract that the claim, lease, and cursor mechanics belong to.