Observers API

The Transition Observer authoring surface: the observer interface and the context it receives, the subscription and builder that register it, and the pump options that tune delivery.

A Transition Observer is host-supplied, egress-only code BackWave invokes when a job reaches a state you declared an interest in. It is the sanctioned way to react to the lifecycle, for example posting to Slack when a job is Dead-Lettered. An observer only reacts. It reads the recorded transition and runs a side effect; it can never veto, redirect, or rewrite the transition, and a slow or throwing observer never stops a worker. This page is the reference for the authoring surface: the ITransitionObserver interface, the ObserverContext it receives, the ObserverSubscription that decides which transitions reach it, the builder that registers it, and the pump options that tune delivery.

The observer types live in the BackWave.Observers namespace, in the core BackWave assembly. The builder and pump options live in BackWave.Hosting. Registration flows through BackWaveBuilder.AddObservers.

Delivery is at-least-once and is not Effect-Once. The same transition may reach an observer more than once, so an observer must be idempotent, the same contract a handler carries. Observers require a Job History Policy of at least Transitions, because with no transition log there is nothing to observe, and they cost nothing when none are registered.

The observer interface#

ITransitionObserver is the one interface you implement. It has a single member.

public interface ITransitionObserver
{
    ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken);
}

OnTransitionAsync returns a ValueTask, not a Task. BackWave calls it once per delivered transition, passing the ObserverContext for that transition and a CancellationToken. The contract of the method is narrow and worth stating in full:

An observer runs at handler trust and is bounded by a timeout. A callback that ignores its token past the deadline is left to finish in the background rather than being aborted, but the delivery is already recorded failed by then.

public sealed class SlackDeadLetterObserver(ISlackClient slack) : ITransitionObserver
{
    public async ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken)
    {
        // Idempotent by construction: keyed on job id + state.
        await slack.PostAsync(
            $"Job {context.JobId} ({context.WireName}) dead-lettered on attempt {context.Attempt}.",
            cancellationToken);
    }
}

The observer context#

ObserverContext is the sealed record handed to OnTransitionAsync. Every transition fact is carried eagerly, so reading them costs no store round trip. The payload alone is lazy, fetched only if you ask for it.

JobId through Timestamp are required init-only members. FailureDetail and Payload are optional init-only members.

When FailureDetail is null#

FailureDetail is doubly gated, and it is null even for a failing transition when capture is off:

• It is null when the Job History Policy is only Transitions rather than TransitionsAndFailureDetail.
• It is null when the BACKWAVE_DISABLE_FAILURE_DETAIL environment kill-switch downgrades the top rung.

The default Job History Policy is TransitionsAndFailureDetail, so failure detail is captured out of the box and the guard only bites when a host has explicitly lowered the policy. The Retention and Purge page owns the policy ladder and the kill-switch.

Reading the payload#

The job payload is not carried on the context. ObserverContext.Payload is an accessor that reads the bytes lazily, so an observer that only reacts to transition facts never pays for a payload read. Two public types make up the accessor surface, both in BackWave.Observers.

ObserverPayload is the result of a read. It tells you whether the read succeeded rather than handing back bytes that were never there, so check Available before touching Bytes.

public readonly record struct ObserverPayload(bool Available, ReadOnlyMemory<byte> Bytes)
{
    public static ObserverPayload Present(ReadOnlyMemory<byte> bytes);
    public static ObserverPayload NotAvailable { get; }
}

ObserverPayloadAccessor performs the deferred read.

public sealed class ObserverPayloadAccessor
{
    public static ObserverPayloadAccessor Unavailable { get; }
    public ValueTask<ObserverPayload> GetAsync(CancellationToken cancellationToken = default);
}

GetAsync reads the payload from the store on first call and memoizes the result at most once; later calls return the cached value without a second read. The accessor is not safe for concurrent reads, which is not a constraint in practice because a callback handles one delivery at a time. Unavailable is a static accessor with no source that always reports NotAvailable; it is the default value of ObserverContext.Payload for a context built without a payload source.

A payload can be unavailable even when the transition is real. If the job row was already purged under retention by the time the observer runs, the read reports NotAvailable.

ObserverPayload payload = await context.Payload.GetAsync(cancellationToken);
if (payload.Available)
{
    ReadOnlyMemory<byte> bytes = payload.Bytes;
    // deserialize and use the payload
}

Subscriptions#

An ObserverSubscription declares which transitions reach an observer. It matches on the state the job reached, and optionally narrows to a single wire name or a single queue.

public sealed record ObserverSubscription(IReadOnlyList<JobState> States)
{
    public static ObserverSubscription AllTransitions { get; }
    public string? WireName { get; init; }
    public string? Queue { get; init; }
    public bool Matches(JobState state, string wireName, string queue);
}

The wire-name and queue comparisons use StringComparison.Ordinal. Narrow AllTransitions with a with expression rather than building a subscription from scratch.

// Every transition of every job.
ObserverSubscription everything = ObserverSubscription.AllTransitions;
 
// Every transition, but only for PaymentJob.
ObserverSubscription payments = ObserverSubscription.AllTransitions with { WireName = "PaymentJob" };
 
// Only dead-letters, on any job type, in the billing queue.
var deadLetters = new ObserverSubscription(new[] { JobState.DeadLettered }) { Queue = "billing" };

The seven JobState members are Scheduled, AwaitingParent, Leased, Succeeded, Cancelled, DeadLettered, and Quarantined; AllTransitions covers all seven. The Job Lifecycle page defines each state.

Registering observers#

Observers are registered in one block through AddObservers on the BackWaveBuilder.

public BackWaveBuilder AddObservers(Action<ObserverBuilder> configure)

The callback receives an ObserverBuilder and configures both the observer list and, optionally, the pump. AddObservers returns the same builder so it chains. Calling it more than once replaces the previous block rather than adding to it, so register every observer in a single call.

bw.AddObservers(obs => obs
    .Add<AuditObserver>("audit", ObserverSubscription.AllTransitions));

ObserverBuilder has two public methods, both chainable.

Add#

public ObserverBuilder Add<TObserver>(string id, ObserverSubscription subscription)
    where TObserver : class, ITransitionObserver;

The id is the durable identity. Reusing the same id with the same subscription across a restart resumes delivery where it left off; changing it starts a fresh cursor. Registering two observers with the same id in one block throws InvalidOperationException with the message Transition Observer '{id}' is configured twice.

The observer type is registered scoped. BackWave resolves a fresh instance from a dependency-injection scope per delivery, so an observer may take scoped dependencies such as a database context.

bw.AddObservers(obs => obs
    .Add<AuditObserver>("audit", ObserverSubscription.AllTransitions)
    .Add<SlackDeadLetterObserver>(
        "slack-dead-letter",
        new ObserverSubscription(new[] { JobState.DeadLettered })));

ConfigurePump#

public ObserverBuilder ConfigurePump(Action<ObserverPumpOptions> configure)

ConfigurePump tunes batch size, lease duration, retry policy, poll interval, and per-delivery timeout. It is optional; sensible defaults apply when it is not called. Only the last call wins.

The Job History Policy guard#

An observer registered while the Job History Policy is Off is a misconfiguration, because with history off no transition rows are recorded and there is nothing to observe. This fails at container composition, not at the first poll, throwing InvalidOperationException with the message:

Transition Observer '{Id}' requires Job History Policy of at least Transitions, but it is Off: with history Off no transition rows are recorded, so there is nothing to observe. Raise the policy to Transitions or TransitionsAndFailureDetail to enable observers.

When no observers are added, nothing is registered: no pump polls and the dashboard observer surface is empty. That is the zero-cost-when-unused guarantee. A single background dispatcher per process drives every registered observer.

The observer pump options#

ObserverPumpOptions tunes how the background dispatcher delivers transitions. Every property is mutable and has a default, so configuring the pump is optional.

DeliveryRetryPolicy is a BackWave.Core.RetryPolicy.

A callback that keeps running past DeliveryTimeout and ignores its token is left to finish in the background; its exception is still observed rather than lost. The failed delivery is retried with backoff and eventually dead-lettered. The worst-case latency for one fully-hung observer is MaxBatch × DeliveryTimeout for that observer alone; it is bounded and self-healing once the poison delivery dead-letters.

bw.AddObservers(obs => obs
    .Add<AuditObserver>("audit", ObserverSubscription.AllTransitions)
    .ConfigurePump(pump =>
    {
        pump.MaxBatch = 64;
        pump.DeliveryTimeout = TimeSpan.FromSeconds(10);
    }));

Delivery retries#

DeliveryRetryPolicy defaults to RetryPolicy.Default, the same retry type BackWave uses for jobs.

DefaultBackoff(attempt) is 2^attempt seconds, capped at five minutes (300 seconds). The first attempt is numbered 1. When a delivery has used up its attempt ceiling, the next-attempt calculation returns nothing and the delivery is dead-lettered instead of retried.

Delivery has its own retry counter, distinct from the job's Attempt. A dead-lettered delivery does not dead-letter the job; it means BackWave gave up redelivering that one transition to that one observer, and its cursor advances past the poison row so later transitions still flow. Every delivery ends in one of three dispositions: Delivered, Retry, or DeadLettered.

Delivery ordering and isolation#

The dispatcher gives observers a few guarantees worth relying on when you write one:

• Sequential within a batch. Deliveries in a single batch are invoked one at a time in transition-log order. An observer sees a job's transitions in the order they happened.
• Concurrent across observers. Two different observers make progress in parallel. One slow or hung observer never starves another.
• Failure is soft. A throw or a timeout from one callback is contained. The delivery is recorded failed and retried; the worker and the rest of the pump keep running.
• Leaderless. Delivery is driven by a durable per-observer cursor and lease, so running more processes only makes delivery faster, never double-counts, and survives the crash of the node that recorded the transition.

Where to go next#

• Job Lifecycle: the seven states, the transition log observers walk, and where the Transition Observer sits in the lifecycle.
• React to Job Outcomes: a worked walkthrough of registering an observer to push a notification when a job reaches a state.
• Retention and Purge: the Job History Policy ladder that gates whether transitions and failure detail are recorded at all.
• Client API: enqueueing the jobs whose transitions observers react to.
• Limits & Defaults: the pump defaults and other defaults in one place.

Found a problem on this page? Report an issue