# React to Job Outcomes

Register a Transition Observer to push a notification when a job reaches a state, for example posting to Slack on dead-letter.

This guide covers the supported way to run "do X when Y happens to a Job": a Transition Observer. You declare which transitions you care about, BackWave invokes your callback after each one is recorded, and you turn that into a Slack message, a webhook, an audit row, or whatever else belongs outside the Job's own execution path. By the end you will know what an Observer can and cannot do, how to register one, what arrives in the callback, and which sharp edges to plan for.

An Observer is the push twin of the [Monitor API](/docs/dashboard-operations/monitor-api). The Monitor lets you poll a Job's history when you want it. An Observer lets BackWave push you a callback the moment a transition lands. The running example throughout is the common one: notify Slack when a payment Job reaches Dead-Lettered.

## What an Observer is, and is not

A Transition Observer is host-supplied, egress-only code that BackWave invokes after a Job reaches a state you subscribed to. It reacts to a Transition, which is the recorded result of a state change, and it reacts only. It cannot veto the transition, rewrite it, or hold it up. If you are looking for a place to intercept a Job and change its course, this is not it, and there is no such seam by design. An Observer observes; it never intercepts.

Three properties follow from that, and each one shapes how you write the callback.

- **It is driven by recorded history.** Observers walk the Transition Log, the append-only per-Job record of state changes. If a transition was never written down, there is nothing to observe, which is why Observers require [Job History](/docs/introduction/configuration) to be turned on. See [History has to be on](#history-has-to-be-on) below.
- **Delivery is At-Least-Once, not exactly once.** On the happy path one node delivers each transition once. A crash mid-delivery causes a redelivery, so your callback may run more than once for the same transition. It has to be idempotent, the same contract a [Job Handler](/docs/core-concepts/execution-guarantee) already carries.
- **It runs out of band.** The callback fires from a separate background dispatcher that reads the committed log. It is not in the Job's execution path, so a throw, a timeout, or a hang in your code is contained and never slows or stops the Worker that runs Jobs.

Register nothing and there is no cost: with no Observers configured, no dispatcher polls and nothing runs.

## The interface you implement

An Observer is one method. Implement `ITransitionObserver` from the `BackWave.Observers` namespace.

```csharp title="PaymentDeadLetterAlert.cs"
using BackWave.Observers;

public sealed class PaymentDeadLetterAlert(ISlackClient slack) : ITransitionObserver
{
    public async ValueTask OnTransitionAsync(ObserverContext context, CancellationToken cancellationToken)
    {
        // This may run more than once for the same transition.
        // Key any dedupe on (JobId, Attempt), which is stable per transition.
        var text =
            $"PaymentJob {context.JobId} dead-lettered on attempt {context.Attempt} " +
            $"at {context.Timestamp:u}. {context.FailureDetail ?? "(failure detail not captured)"}";

        await slack.PostAsync("#payments-alerts", text, cancellationToken);
    }
}
```

Returning normally marks the delivery as delivered, and BackWave advances its cursor past that transition. Throwing marks the delivery failed, which retries it. The constructor can take scoped dependencies, an `ISlackClient`, a `DbContext`, anything in the container, because BackWave resolves the Observer fresh from a DI scope for each delivery. The `cancellationToken` is signaled when the delivery timeout elapses or the host is shutting down, so honor it: a callback that ignores it is what turns a slow reaction into a leaked one.

## What the callback receives

`ObserverContext` is a record carrying everything about the transition up front, plus a lazy accessor for the payload.

| Property | Type | What it is |
| --- | --- | --- |
| `JobId` | `Guid` | The transitioning Job. |
| `WireName` | `string` | The Job type's stable storage identity, not the CLR class name. |
| `Queue` | `string` | The Job's Queue. |
| `State` | `JobState` | The state reached, the one your subscription matched. |
| `Attempt` | `int` | The Job's Attempt number at this transition. |
| `Timestamp` | `DateTimeOffset` | When the transition was recorded. |
| `FailureDetail` | `string?` | Exception type, message, and stack on a failing transition, or null when failure-detail capture is off. |
| `Payload` | `ObserverPayloadAccessor` | A lazy accessor for the Job payload bytes. |

Two of these carry caveats. `FailureDetail` is the diagnostic text captured at the edge when an Attempt fails, the same Failure Detail you see in the dashboard. It is null when capture is disabled, even on a failing transition, so never assume it is present. And `Payload` is lazy: the bytes are not read from the store until you ask for them.

### Reading the payload

Call `GetAsync` on the accessor only when you actually need the payload. A delivery that never reaches for it pays no read cost, and the first read is memoized for the rest of the callback.

```csharp title="PaymentDeadLetterAlert.cs"
var payload = await context.Payload.GetAsync(cancellationToken);
if (payload.Available)
{
    ReadOnlyMemory<byte> bytes = payload.Bytes; // deserialize as needed
    // ...
}
// else: the Job's payload was already purged under retention; handle gracefully
```

`GetAsync` returns an `ObserverPayload` with an `Available` flag and a `Bytes` buffer. Always check `Available` before touching `Bytes`. A lazy read races the retention sweep, so a Job whose payload has already been purged reports unavailable rather than handing you fabricated bytes. The accessor is also single-threaded: one delivery is invoked at a time, so do not read it concurrently. See [Retention & Purge](/docs/dashboard-operations/retention-and-purge) for when payloads disappear.

## Subscribing to the right transitions

A subscription is what tells BackWave which transitions to hand your Observer. You build an `ObserverSubscription` from a list of states, optionally narrowed by Wire Name and Queue.

```csharp title="Program.cs"
// One state, one Job type: the Slack-on-dead-letter case.
new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" }

// Audit everything for one type, narrowed with 'with'.
ObserverSubscription.AllTransitions with { WireName = "PaymentJob" }
```

`WireName` and `Queue` are both nullable. Leave one null and it matches every type or every Queue. `ObserverSubscription.AllTransitions` is the catch-all set of states, and the `with { ... }` idiom is the documented way to narrow it. The states you can name come from the `JobState` enum:

- `Scheduled`, `AwaitingParent`, `Leased` are the non-terminal states. They are valid subscription targets, but the recommended pattern is to subscribe to terminal states.
- `Succeeded`, `Cancelled`, `DeadLettered`, `Quarantined` are the four terminal states.

Pick the state that matches the outcome you care about. **`DeadLettered`** means the Job ran and kept failing until it exhausted its Attempt ceiling. **`Quarantined`** means the Job could not be routed or decoded, because no Handler was registered for its Wire Name or its payload no longer deserializes. They are different problems with different fixes, so subscribe to whichever one your notification is actually about. [Job States](/docs/reference/job-states) and [Job Lifecycle](/docs/core-concepts/job-lifecycle) spell out the full set.

## Registering the Observer

Observers register inside the same `AddBackWave` call as the rest of your setup, in one `AddObservers` block. The block holds the Observer list and the pump tuning together.

```csharp title="Program.cs" {9-12}
builder.Services.AddBackWave(backwave => backwave
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "payments",
        Policy = new DispatchPolicy.Strict(["payments"]),
    })
    .AddObservers(obs => obs
        .Add<PaymentDeadLetterAlert>(
            id: "payment-dead-letter-slack",
            subscription: new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" })));
```

`Add<TObserver>` takes a stable `id` and a subscription, and returns the builder so you can chain more. A single background dispatcher per process drives every Observer you register.

The `id` is load-bearing. It keys the durable cursor that tracks how far delivery has progressed for that Observer, so keep it stable across restarts and deploys. A running Observer resumes exactly where it left off. Change the `id` only when you deliberately want to reset the cursor, and be aware that adding two Observers with the same `id` throws `InvalidOperationException` with `Transition Observer '{id}' is configured twice.` at startup.

One gotcha: calling `AddObservers` more than once does not merge the blocks, it replaces the previous one. Only the last call survives. Register all of your Observers in a single `AddObservers` block.

## History has to be on

Observers read the Transition Log, so the log has to exist. The Job History Policy controls that, and it has three settings:

- `Off` records no transitions at all.
- `Transitions` records the state changes but not the failure text.
- `TransitionsAndFailureDetail` records both, and is the default.

Observers require at least `Transitions`. If you register an Observer while the policy is `Off`, BackWave fails fast at startup with an `InvalidOperationException` rather than silently delivering nothing: `Transition Observer '{Id}' requires Job History Policy of at least Transitions, but it is Off: with history Off no transition rows are recorded, so there is nothing to observe.` The default store configuration records full history, so the Slack example above works out of the box.

The policy lives in two places that must agree. The store options record what is actually written, and `UseHistoryPolicy` on the builder is what the startup deliverability check reads. There is also an environment kill-switch, `BACKWAVE_DISABLE_FAILURE_DETAIL`, that downgrades the top setting to `Transitions` on PII-sensitive hosts. When it is set, `FailureDetail` arrives null even on a failing transition, which is the other reason to treat that property as optional. [Configuration](/docs/introduction/configuration) covers the policy and the kill-switch in full.

## Tuning the pump

The dispatcher that delivers transitions is the pump, and `ConfigurePump` tunes it. Every setting has a default, so this whole step is optional. Reach for it when your egress wants a tighter deadline or a smaller batch.

```csharp title="Program.cs"
.AddObservers(obs => obs
    .Add<PaymentDeadLetterAlert>("payment-dead-letter-slack",
        new ObserverSubscription([JobState.DeadLettered]) { WireName = "PaymentJob" })
    .ConfigurePump(pump =>
    {
        pump.DeliveryTimeout = TimeSpan.FromSeconds(10); // a Slack POST should be quick
        pump.MaxBatch = 16;
        // pump.DeliveryRetryPolicy = new RetryPolicy { MaxAttempts = 5 };
    }))
```

| Setting | Default | What it does |
| --- | --- | --- |
| `MaxBatch` | `32` | Max transitions claimed per Observer per poll. |
| `LeaseDuration` | `60 s` | How long a claim is held; the redelivery window after a crash mid-delivery. |
| `DeliveryRetryPolicy` | `RetryPolicy.Default` | Backoff and ceiling for failed deliveries. |
| `PollInterval` | `1 s` | How often the pump checks each Observer for a new batch. |
| `DeliveryTimeout` | `30 s` | How long one callback may run before the delivery is recorded failed and the pump moves on. |

`DeliveryRetryPolicy` is the same `RetryPolicy` type a Job Attempt uses. Its default allows 10 attempts and backs off `2^attempt` seconds, capped at 5 minutes, exactly as described in [Retries & Error Handling](/docs/guides/configure-retries-and-error-handling). For fast egress like a Slack POST, lowering `DeliveryTimeout` is the highest-value knob, and the next section explains why.

## Failure modes and idempotency

Because delivery is At-Least-Once, the fire is a fresh side effect that sits outside the Job's Effect-Once fence. The same transition can reach your callback more than once, and making that safe is your job, not BackWave's. The natural dedup key is `(JobId, Attempt)`, or `(JobId, State, Attempt)` if you subscribe to several states. Both are stable per transition, so write your dedup check against them before doing anything irreversible.

A few more behaviors to keep in mind.

- **A callback is bounded by `DeliveryTimeout`.** Run past it and the delivery is recorded failed, retried on the `DeliveryRetryPolicy` backoff, and eventually Dead-Lettered if it keeps failing. Keep the callback fast and non-blocking; do heavy work elsewhere.
- **A throw, a timeout, or a hang never fail-stops anything.** It is contained and turned into a failed delivery. The Workers that run Jobs are untouched, and the pump itself survives. A callback that ignores its `cancellationToken` and runs past the deadline is left to finish in the background, its eventual exception is observed so it cannot crash the process, and the leak is logged loudly as your bug.
- **A poison delivery dead-letters after the ceiling.** A delivery that exhausts `DeliveryRetryPolicy.MaxAttempts` is Dead-Lettered and the cursor advances past it, so one bad row never blocks later notifications forever. The worst-case latency a single fully-hung Observer can add to its own stream is bounded by its batch size times its delivery timeout, after which it self-heals. Other Observers are unaffected.
- **Multi-node is automatic.** Run as many processes as you like. A per-Observer durable cursor serializes claims so each transition is delivered once on the happy path. More processes mean faster delivery, never duplicates by design.

The through-line is the same one Job Handlers live under: write the callback so that running it twice produces the same end state as running it once, and honor the cancellation token so a slow reaction stays contained. [Execution Guarantee](/docs/core-concepts/execution-guarantee) is the canonical statement of that contract.

## Where to go next

- [Observers API](/docs/reference/observers-api): the type-by-type reference for everything in this guide.
- [Job States](/docs/reference/job-states): the full set of states, terminal and not, you can subscribe to.
- [Configure Retries & Error Handling](/docs/guides/configure-retries-and-error-handling): the `RetryPolicy` shape that `DeliveryRetryPolicy` reuses.
- [Execution Guarantee](/docs/core-concepts/execution-guarantee): why delivery is At-Least-Once and why your callback must be idempotent.
- [Retention & Purge](/docs/dashboard-operations/retention-and-purge): why a payload read can come back unavailable.
- [Read Another Job's Output](/docs/guides/read-another-jobs-output): the success-side data channel, the twin of Failure Detail.