React to Job Outcomes

Register a Transition Observer to push a notification when a job reaches a state, for example posting to Slack on dead-letter.

Register a Transition Observer to push a notification when a job reaches a state, for example posting to Slack on dead-letter.

This guide covers the supported way to run "do X when Y happens to a Job": a Transition Observer. You declare which transitions you care about, BackWave invokes your callback after each one is recorded, and you turn that into a Slack message, a webhook, an audit row, or whatever else belongs outside the Job's own execution path. By the end you will know what an Observer can and cannot do, how to register one, what arrives in the callback, and which sharp edges to plan for.

An Observer is the push twin of the Monitor API. The Monitor lets you poll a Job's history when you want it. An Observer lets BackWave push you a callback the moment a transition lands. The running example throughout is the common one: notify Slack when a payment Job reaches Dead-Lettered.

What an Observer is, and is not#

A Transition Observer is host-supplied, egress-only code that BackWave invokes after a Job reaches a state you subscribed to. It reacts to a Transition, which is the recorded result of a state change, and it reacts only. It cannot veto the transition, rewrite it, or hold it up. If you are looking for a place to intercept a Job and change its course, this is not it, and there is no such seam by design. An Observer observes; it never intercepts.

Three properties follow from that, and each one shapes how you write the callback.

• It is driven by recorded history. Observers walk the Transition Log, the append-only per-Job record of state changes. If a transition was never written down, there is nothing to observe, which is why Observers require Job History to be turned on. See History has to be on below.
• Delivery is At-Least-Once, not exactly once. On the happy path one node delivers each transition once. A crash mid-delivery causes a redelivery, so your callback may run more than once for the same transition. It has to be idempotent, the same contract a Job Handler already carries.
• It runs out of band. The callback fires from a separate background dispatcher that reads the committed log. It is not in the Job's execution path, so a throw, a timeout, or a hang in your code is contained and never slows or stops the Worker that runs Jobs.

Register nothing and there is no cost: with no Observers configured, no dispatcher polls and nothing runs.

The interface you implement#

An Observer is one method. Implement ITransitionObserver from the BackWave.Observers namespace.

using BackWave.Observers;
 
public sealed class PaymentDeadLetterAlert(ISlackClient slack) : ITransitionObserver
{
    public async ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken)
    {
        // This may run more than once for the same transition.
        // Key any dedupe on (JobId, Attempt), which is stable per transition.
        var text =
            $"PaymentJob {context.JobId} dead-lettered on attempt {context.Attempt} " +
            $"at {context.Timestamp:u}. {context.FailureDetail ?? "(failure detail not captured)"}";
 
        await slack.PostAsync("#payments-alerts", text, cancellationToken);
    }
}

Returning normally marks the delivery as delivered, and BackWave advances its cursor past that transition. Throwing marks the delivery failed, which retries it. The constructor can take scoped dependencies, an ISlackClient, a DbContext, anything in the container, because BackWave resolves the Observer fresh from a DI scope for each delivery. The cancellationToken is signaled when the delivery timeout elapses or the host is shutting down, so honor it: a callback that ignores it is what turns a slow reaction into a leaked one.

What the callback receives#

ObserverContext is a record carrying everything about the transition up front, plus a lazy accessor for the payload.

Two of these carry caveats. FailureDetail is the diagnostic text captured at the edge when an Attempt fails, the same Failure Detail you see in the dashboard. It is null when capture is disabled, even on a failing transition, so never assume it is present. And Payload is lazy: the bytes are not read from the store until you ask for them.

Reading the payload#

Call GetAsync on the accessor only when you actually need the payload. A delivery that never reaches for it pays no read cost, and the first read is memoized for the rest of the callback.

var payload = await context.Payload.GetAsync(cancellationToken);
if (payload.Available)
{
    ReadOnlyMemory<byte> bytes = payload.Bytes; // deserialize as needed
    // ...
}
// else: the Job's payload was already purged under retention; handle gracefully

GetAsync returns an ObserverPayload with an Available flag and a Bytes buffer. Always check Available before touching Bytes. A lazy read races the retention sweep, so a Job whose payload has already been purged reports unavailable rather than handing you fabricated bytes. The accessor is also single-threaded: one delivery is invoked at a time, so do not read it concurrently. See Retention & Purge for when payloads disappear.

Subscribing to the right transitions#

A subscription is what tells BackWave which transitions to hand your Observer. You build an ObserverSubscription from a list of states, optionally narrowed by Wire Name and Queue.

// One state, one Job type: the Slack-on-dead-letter case.
new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" }
 
// Audit everything for one type, narrowed with 'with'.
ObserverSubscription.AllTransitions with { WireName = "PaymentJob" }

WireName and Queue are both nullable. Leave one null and it matches every type or every Queue. ObserverSubscription.AllTransitions is the catch-all set of states, and the with { ... } idiom is the documented way to narrow it. The states you can name come from the JobState enum:

• Scheduled, AwaitingParent, Leased are the non-terminal states. They are valid subscription targets, but the recommended pattern is to subscribe to terminal states.
• Succeeded, Cancelled, DeadLettered, Quarantined are the four terminal states.

Pick the state that matches the outcome you care about. DeadLettered means the Job ran and kept failing until it exhausted its Attempt ceiling. Quarantined means the Job could not be routed or decoded, because no Handler was registered for its Wire Name or its payload no longer deserializes. They are different problems with different fixes, so subscribe to whichever one your notification is actually about. Job States and Job Lifecycle spell out the full set.

Registering the Observer#

Observers register inside the same AddBackWave call as the rest of your setup, in one AddObservers block. The block holds the Observer list and the pump tuning together.

builder.Services.AddBackWave(backwave => backwave
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "payments",
        Policy = new DispatchPolicy.Strict(["payments"]),
    })
    .AddObservers(obs => obs
        .Add<PaymentDeadLetterAlert>(
            id: "payment-dead-letter-slack",
            subscription: new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" })));

Add<TObserver> takes a stable id and a subscription, and returns the builder so you can chain more. A single background dispatcher per process drives every Observer you register.

The id is load-bearing. It keys the durable cursor that tracks how far delivery has progressed for that Observer, so keep it stable across restarts and deploys. A running Observer resumes exactly where it left off. Change the id only when you deliberately want to reset the cursor, and be aware that adding two Observers with the same id throws InvalidOperationException with Transition Observer '{id}' is configured twice. at startup.

One gotcha: calling AddObservers more than once does not merge the blocks, it replaces the previous one. Only the last call survives. Register all of your Observers in a single AddObservers block.

History has to be on#

Observers read the Transition Log, so the log has to exist. The Job History Policy controls that, and it has three settings:

• Off records no transitions at all.
• Transitions records the state changes but not the failure text.
• TransitionsAndFailureDetail records both, and is the default.

Observers require at least Transitions. If you register an Observer while the policy is Off, BackWave fails fast at startup with an InvalidOperationException rather than silently delivering nothing: Transition Observer '{Id}' requires Job History Policy of at least Transitions, but it is Off: with history Off no transition rows are recorded, so there is nothing to observe. The default store configuration records full history, so the Slack example above works out of the box.

The policy lives in two places that must agree. The store options record what is actually written, and UseHistoryPolicy on the builder is what the startup deliverability check reads. There is also an environment kill-switch, BACKWAVE_DISABLE_FAILURE_DETAIL, that downgrades the top setting to Transitions on PII-sensitive hosts. When it is set, FailureDetail arrives null even on a failing transition, which is the other reason to treat that property as optional. Configuration covers the policy and the kill-switch in full.

Tuning the pump#

The dispatcher that delivers transitions is the pump, and ConfigurePump tunes it. Every setting has a default, so this whole step is optional. Reach for it when your egress wants a tighter deadline or a smaller batch.

.AddObservers(obs => obs
    .Add<PaymentDeadLetterAlert>("payment-dead-letter-slack",
        new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" })
    .ConfigurePump(pump =>
    {
        pump.DeliveryTimeout = TimeSpan.FromSeconds(10); // a Slack POST should be quick
        pump.MaxBatch = 16;
        // pump.DeliveryRetryPolicy = new RetryPolicy { MaxAttempts = 5 };
    }))

DeliveryRetryPolicy is the same RetryPolicy type a Job Attempt uses. Its default allows 10 attempts and backs off 2^attempt seconds, capped at 5 minutes, exactly as described in Retries & Error Handling. For fast egress like a Slack POST, lowering DeliveryTimeout is the highest-value knob, and the next section explains why.

Failure modes and idempotency#

Because delivery is At-Least-Once, the fire is a fresh side effect that sits outside the Job's Effect-Once fence. The same transition can reach your callback more than once, and making that safe is your job, not BackWave's. The natural dedup key is (JobId, Attempt), or (JobId, State, Attempt) if you subscribe to several states. Both are stable per transition, so write your dedup check against them before doing anything irreversible.

A few more behaviors to keep in mind.

• A callback is bounded by DeliveryTimeout. Run past it and the delivery is recorded failed, retried on the DeliveryRetryPolicy backoff, and eventually Dead-Lettered if it keeps failing. Keep the callback fast and non-blocking; do heavy work elsewhere.
• A throw, a timeout, or a hang never fail-stops anything. It is contained and turned into a failed delivery. The Workers that run Jobs are untouched, and the pump itself survives. A callback that ignores its cancellationToken and runs past the deadline is left to finish in the background, its eventual exception is observed so it cannot crash the process, and the leak is logged loudly as your bug.
• A poison delivery dead-letters after the ceiling. A delivery that exhausts DeliveryRetryPolicy.MaxAttempts is Dead-Lettered and the cursor advances past it, so one bad row never blocks later notifications forever. The worst-case latency a single fully-hung Observer can add to its own stream is bounded by its batch size times its delivery timeout, after which it self-heals. Other Observers are unaffected.
• Multi-node is automatic. Run as many processes as you like. A per-Observer durable cursor serializes claims so each transition is delivered once on the happy path. More processes mean faster delivery, never duplicates by design.

The through-line is the same one Job Handlers live under: write the callback so that running it twice produces the same end state as running it once, and honor the cancellation token so a slow reaction stays contained. Execution Guarantee is the canonical statement of that contract.

Where to go next#

• Observers API: the type-by-type reference for everything in this guide.
• Job States: the full set of states, terminal and not, you can subscribe to.
• Configure Retries & Error Handling: the RetryPolicy shape that DeliveryRetryPolicy reuses.
• Execution Guarantee: why delivery is At-Least-Once and why your callback must be idempotent.
• Retention & Purge: why a payload read can come back unavailable.
• Read Another Job's Output: the success-side data channel, the twin of Failure Detail.