# Observers API A [Transition Observer](/docs/core-concepts/job-lifecycle#the-transition-observer) is host-supplied, egress-only code BackWave invokes when a job reaches a state you declared an interest in. It is the sanctioned way to react to the lifecycle, for example posting to Slack when a job is Dead-Lettered. An observer only reacts. It reads the recorded transition and runs a side effect; it can never veto, redirect, or rewrite the transition, and a slow or throwing observer never stops a worker. This page is the reference for the authoring surface: the `ITransitionObserver` interface, the `ObserverContext` it receives, the `ObserverSubscription` that decides which transitions reach it, the builder that registers it, and the pump options that tune delivery. The observer types live in the `BackWave.Observers` namespace, in the core `BackWave` assembly. The builder and pump options live in `BackWave.Hosting`. Registration flows through `BackWaveBuilder.AddObservers`. Delivery is **at-least-once** and is not Effect-Once. The same transition may reach an observer more than once, so an observer must be idempotent, the same contract a handler carries. Observers require a Job History Policy of at least `Transitions`, because with no transition log there is nothing to observe, and they cost nothing when none are registered. ## The observer interface `ITransitionObserver` is the one interface you implement. It has a single member. ```csharp title="ITransitionObserver.cs" public interface ITransitionObserver { ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken); } ``` `OnTransitionAsync` returns a `ValueTask`, not a `Task`. BackWave calls it once per delivered transition, passing the `ObserverContext` for that transition and a `CancellationToken`. The contract of the method is narrow and worth stating in full: | Rule | Behavior | |---|---| | Idempotent | The method may be invoked more than once for the same transition, because delivery is at-least-once. Make the reaction safe to repeat. | | Honor the token | The `cancellationToken` is cancelled when the dispatch timeout elapses or the host is shutting down. Observe it and return promptly. | | Completion means delivered | Returning without throwing marks the delivery delivered and advances the cursor. | | Failure is contained | A throw, a timeout, or a hang is contained. It never stops the worker; the delivery is marked for retry. | | Observe only | The method can react to a transition but can never veto, redirect, or rewrite it. | An observer runs at handler trust and is bounded by a timeout. A callback that ignores its token past the deadline is left to finish in the background rather than being aborted, but the delivery is already recorded failed by then. ```csharp title="SlackDeadLetterObserver.cs" public sealed class SlackDeadLetterObserver(ISlackClient slack) : ITransitionObserver { public async ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken) { // Idempotent by construction: keyed on job id + state. await slack.PostAsync( $"Job {context.JobId} ({context.WireName}) dead-lettered on attempt {context.Attempt}.", cancellationToken); } } ``` ## The observer context `ObserverContext` is the sealed record handed to `OnTransitionAsync`. Every transition fact is carried eagerly, so reading them costs no store round trip. The payload alone is lazy, fetched only if you ask for it. | Member | Type | Notes | |---|---|---| | `JobId` | `Guid` | The id of the job whose transition this is. | | `WireName` | `string` | The job's wire name, the type name used to route it to a handler. | | `Queue` | `string` | The queue the job lives in. | | `State` | `JobState` | The state the job reached, the one your subscription matched. | | `Attempt` | `int` | The job's attempt number at this transition. | | `Timestamp` | `DateTimeOffset` | When the transition was recorded. | | `FailureDetail` | `string?` | The captured failure detail, or `null`. See the gating rules below. | | `Payload` | `ObserverPayloadAccessor` | Lazy accessor for the job's payload bytes. Defaults to an accessor that always reports unavailable. | `JobId` through `Timestamp` are `required` init-only members. `FailureDetail` and `Payload` are optional init-only members. ### When FailureDetail is null `FailureDetail` is doubly gated, and it is `null` even for a failing transition when capture is off: - It is `null` when the Job History Policy is only `Transitions` rather than `TransitionsAndFailureDetail`. - It is `null` when the `BACKWAVE_DISABLE_FAILURE_DETAIL` environment kill-switch downgrades the top rung. The default Job History Policy is `TransitionsAndFailureDetail`, so failure detail is captured out of the box and the guard only bites when a host has explicitly lowered the policy. The [Retention and Purge](/docs/dashboard-operations/retention-and-purge#configuring-the-policy-in-two-places) page owns the policy ladder and the kill-switch. ## Reading the payload The job payload is not carried on the context. `ObserverContext.Payload` is an accessor that reads the bytes lazily, so an observer that only reacts to transition facts never pays for a payload read. Two public types make up the accessor surface, both in `BackWave.Observers`. `ObserverPayload` is the result of a read. It tells you whether the read succeeded rather than handing back bytes that were never there, so check `Available` before touching `Bytes`. ```csharp title="ObserverPayload.cs" public readonly record struct ObserverPayload(bool Available, ReadOnlyMemory Bytes) { public static ObserverPayload Present(ReadOnlyMemory bytes); public static ObserverPayload NotAvailable { get; } } ``` | Member | Behavior | |---|---| | `Available` | Whether the payload was read. `false` means it could not be retrieved. | | `Bytes` | The payload bytes when available; empty otherwise. | | `Present(bytes)` | Builds an available result carrying `bytes`. | | `NotAvailable` | A static unavailable result, empty bytes. | `ObserverPayloadAccessor` performs the deferred read. ```csharp title="ObserverPayloadAccessor.cs" public sealed class ObserverPayloadAccessor { public static ObserverPayloadAccessor Unavailable { get; } public ValueTask GetAsync(CancellationToken cancellationToken = default); } ``` `GetAsync` reads the payload from the store on first call and memoizes the result at most once; later calls return the cached value without a second read. The accessor is not safe for concurrent reads, which is not a constraint in practice because a callback handles one delivery at a time. `Unavailable` is a static accessor with no source that always reports `NotAvailable`; it is the default value of `ObserverContext.Payload` for a context built without a payload source. A payload can be unavailable even when the transition is real. If the job row was already purged under retention by the time the observer runs, the read reports `NotAvailable`. ```csharp title="ReadPayload.cs" ObserverPayload payload = await context.Payload.GetAsync(cancellationToken); if (payload.Available) { ReadOnlyMemory bytes = payload.Bytes; // deserialize and use the payload } ``` ## Subscriptions An `ObserverSubscription` declares which transitions reach an observer. It matches on the state the job reached, and optionally narrows to a single wire name or a single queue. ```csharp title="ObserverSubscription.cs" public sealed record ObserverSubscription(IReadOnlyList States) { public static ObserverSubscription AllTransitions { get; } public string? WireName { get; init; } public string? Queue { get; init; } public bool Matches(JobState state, string wireName, string queue); } ``` | Member | Behavior | |---|---| | `States` | The positional required member. A transition into any one of these states matches. | | `AllTransitions` | A static subscription to a transition into any `JobState`, the "audit everything" shape. | | `WireName` | Optional narrowing filter. `null` matches every job type. | | `Queue` | Optional narrowing filter. `null` matches every queue. | | `Matches(state, wireName, queue)` | True when `States` contains `state` and each of `WireName` and `Queue` is either `null` or an ordinal-equal match. | The wire-name and queue comparisons use `StringComparison.Ordinal`. Narrow `AllTransitions` with a `with` expression rather than building a subscription from scratch. ```csharp title="Subscriptions.cs" // Every transition of every job. ObserverSubscription everything = ObserverSubscription.AllTransitions; // Every transition, but only for PaymentJob. ObserverSubscription payments = ObserverSubscription.AllTransitions with { WireName = "PaymentJob" }; // Only dead-letters, on any job type, in the billing queue. var deadLetters = new ObserverSubscription(new[] { JobState.DeadLettered }) { Queue = "billing" }; ``` The seven `JobState` members are `Scheduled`, `AwaitingParent`, `Leased`, `Succeeded`, `Cancelled`, `DeadLettered`, and `Quarantined`; `AllTransitions` covers all seven. The [Job Lifecycle](/docs/core-concepts/job-lifecycle#the-seven-states) page defines each state. ## Registering observers Observers are registered in one block through `AddObservers` on the `BackWaveBuilder`. ```csharp title="AddObservers.cs" public BackWaveBuilder AddObservers(Action configure) ``` The callback receives an `ObserverBuilder` and configures both the observer list and, optionally, the pump. `AddObservers` returns the same builder so it chains. Calling it more than once replaces the previous block rather than adding to it, so register every observer in a single call. ```csharp title="Register.cs" bw.AddObservers(obs => obs .Add("audit", ObserverSubscription.AllTransitions)); ``` `ObserverBuilder` has two public methods, both chainable. ### Add ```csharp title="ObserverBuilder.Add.cs" public ObserverBuilder Add(string id, ObserverSubscription subscription) where TObserver : class, ITransitionObserver; ``` | Parameter | Behavior | |---|---| | `id` | A stable identifier for this observer. The durable delivery cursor is keyed by it, so it must be unique within the block and stable across restarts. | | `subscription` | The set of transitions this observer receives. | The `id` is the durable identity. Reusing the same `id` with the same subscription across a restart resumes delivery where it left off; changing it starts a fresh cursor. Registering two observers with the same `id` in one block throws `InvalidOperationException` with the message `Transition Observer '{id}' is configured twice.` The observer type is registered **scoped**. BackWave resolves a fresh instance from a dependency-injection scope per delivery, so an observer may take scoped dependencies such as a database context. ```csharp title="MultipleObservers.cs" bw.AddObservers(obs => obs .Add("audit", ObserverSubscription.AllTransitions) .Add( "slack-dead-letter", new ObserverSubscription(new[] { JobState.DeadLettered }))); ``` ### ConfigurePump ```csharp title="ObserverBuilder.ConfigurePump.cs" public ObserverBuilder ConfigurePump(Action configure) ``` `ConfigurePump` tunes batch size, lease duration, retry policy, poll interval, and per-delivery timeout. It is optional; sensible defaults apply when it is not called. Only the last call wins. ### The Job History Policy guard An observer registered while the Job History Policy is `Off` is a misconfiguration, because with history off no transition rows are recorded and there is nothing to observe. This fails at container composition, not at the first poll, throwing `InvalidOperationException` with the message: > Transition Observer '\{Id\}' requires Job History Policy of at least Transitions, but it is Off: with > history Off no transition rows are recorded, so there is nothing to observe. Raise the policy to > Transitions or TransitionsAndFailureDetail to enable observers. When no observers are added, nothing is registered: no pump polls and the dashboard observer surface is empty. That is the zero-cost-when-unused guarantee. A single background dispatcher per process drives every registered observer. ## The observer pump options `ObserverPumpOptions` tunes how the background dispatcher delivers transitions. Every property is mutable and has a default, so configuring the pump is optional. | Property | Type | Default | Meaning | |---|---|---|---| | `MaxBatch` | `int` | `32` | The maximum number of transitions claimed for each observer per poll. Bounds how much one observer processes at a time. | | `LeaseDuration` | `TimeSpan` | 60 seconds | How long a claim is held before another node may re-claim it, and so the redelivery window after a crash mid-delivery. | | `DeliveryRetryPolicy` | `RetryPolicy` | `RetryPolicy.Default` | The backoff schedule and attempt ceiling for a failed delivery, after which the delivery is dead-lettered. | | `PollInterval` | `TimeSpan` | 1 second | How often the pump polls each observer for its next batch. Shorter means lower latency and more store queries. | | `DeliveryTimeout` | `TimeSpan` | 30 seconds | How long a single observer callback may run before the pump records the delivery failed and moves on. | `DeliveryRetryPolicy` is a `BackWave.Core.RetryPolicy`. A callback that keeps running past `DeliveryTimeout` and ignores its token is left to finish in the background; its exception is still observed rather than lost. The failed delivery is retried with backoff and eventually dead-lettered. The worst-case latency for one fully-hung observer is `MaxBatch × DeliveryTimeout` for that observer alone; it is bounded and self-healing once the poison delivery dead-letters. ```csharp title="ConfigurePump.cs" bw.AddObservers(obs => obs .Add("audit", ObserverSubscription.AllTransitions) .ConfigurePump(pump => { pump.MaxBatch = 64; pump.DeliveryTimeout = TimeSpan.FromSeconds(10); })); ``` ### Delivery retries `DeliveryRetryPolicy` defaults to `RetryPolicy.Default`, the same retry type BackWave uses for jobs. | Member | Default | Behavior | |---|---|---| | `MaxAttempts` | `10` | The maximum number of delivery attempts before the delivery dead-letters. | | `Backoff` | `DefaultBackoff` | Maps an attempt number to a wait before the next attempt. | `DefaultBackoff(attempt)` is `2^attempt` seconds, capped at five minutes (300 seconds). The first attempt is numbered 1. When a delivery has used up its attempt ceiling, the next-attempt calculation returns nothing and the delivery is dead-lettered instead of retried. Delivery has its own retry counter, distinct from the job's `Attempt`. A dead-lettered delivery does not dead-letter the job; it means BackWave gave up redelivering that one transition to that one observer, and its cursor advances past the poison row so later transitions still flow. Every delivery ends in one of three dispositions: `Delivered`, `Retry`, or `DeadLettered`. ## Delivery ordering and isolation The dispatcher gives observers a few guarantees worth relying on when you write one: - **Sequential within a batch.** Deliveries in a single batch are invoked one at a time in transition-log order. An observer sees a job's transitions in the order they happened. - **Concurrent across observers.** Two different observers make progress in parallel. One slow or hung observer never starves another. - **Failure is soft.** A throw or a timeout from one callback is contained. The delivery is recorded failed and retried; the worker and the rest of the pump keep running. - **Leaderless.** Delivery is driven by a durable per-observer cursor and lease, so running more processes only makes delivery faster, never double-counts, and survives the crash of the node that recorded the transition. ## Where to go next - [Job Lifecycle](/docs/core-concepts/job-lifecycle#the-transition-observer): the seven states, the transition log observers walk, and where the Transition Observer sits in the lifecycle. - [React to Job Outcomes](/docs/guides/react-to-job-outcomes): a worked walkthrough of registering an observer to push a notification when a job reaches a state. - [Retention and Purge](/docs/dashboard-operations/retention-and-purge): the Job History Policy ladder that gates whether transitions and failure detail are recorded at all. - [Client API](/docs/reference/client-api): enqueueing the jobs whose transitions observers react to. - [Limits & Defaults](/docs/reference/limits-and-defaults): the pump defaults and other defaults in one place.