# Telemetry Reference

BackWave emits all three OpenTelemetry pillars, traces, metrics, and logs, on the OpenTelemetry
semantic conventions. It builds on the base class library only, so spans, measurements, and log
calls cost nothing until you subscribe a tracer, meter, or logging provider to the source. The
job lifecycle follows the OpenTelemetry **messaging** conventions and each storage adapter's
store round-trips follow the **db** conventions. This page lists every identifier BackWave emits,
byte-for-byte, so you can wire it into your collector by name. For the wiring itself and the
semantics behind these tables, see [Wire Up OpenTelemetry](/docs/guides/wire-up-opentelemetry).

## The source names

Job-lifecycle traces and metrics publish under a single name, `"BackWave"`. Each storage adapter
publishes its store spans and store-fault meter under its own name, and only when you opt that
adapter in.

| Source name | Emits | Constant |
|---|---|---|
| `BackWave` | Job-lifecycle spans, the Core metric instruments, the log catalog | `BackWaveDiagnostics.SourceName` |
| `BackWave.Postgres` | Postgres store spans and store-fault meter | - |
| `BackWave.SqlServer` | SQL Server store spans and store-fault meter | - |
| `BackWave.Sqlite` | SQLite store spans and store-fault meter | - |

The `BackWave` value is exactly that, with a capital B and a capital W, and is exposed as the
public constant `BackWaveDiagnostics.SourceName` so you can subscribe by constant rather than by
literal. The scope version stamped on the source is the package's informational version. The
`BackWave.OpenTelemetry` package wraps each name in a named subscription method; the by-name path
(`AddSource`/`AddMeter` with these strings) is equivalent.

## Metrics

BackWave publishes fifteen Core instruments on the meter named `"BackWave"`, plus one store-fault
counter per adapter on that adapter's own meter. Units are curly-brace annotation units and appear
verbatim on the instrument. The two throughput counters and the process-duration histogram carry
messaging-convention names; the rest are `backwave.*`-owned.

### Core instruments

| Instrument | Type | Unit | Meaning |
|---|---|---|---|
| `messaging.client.sent.messages` | `Counter<long>` | `{message}` | Jobs accepted by enqueue. |
| `messaging.client.consumed.messages` | `Counter<long>` | `{message}` | Executions that succeeded. |
| `backwave.jobs.failed` | `Counter<long>` | `{job}` | Failed Attempts, retried and Dead-Lettered alike. Carries `error.type`. |
| `backwave.job.attempts` | `Counter<long>` | `{attempt}` | Attempts started, one per claimed job. |
| `backwave.jobs.dead_lettered` | `Counter<long>` | `{job}` | Jobs that exhausted their Attempt ceiling and were Dead-Lettered. |
| `backwave.observer.deliveries.attempted` | `Counter<long>` | `{delivery}` | Observer callback invocations started. |
| `backwave.observer.deliveries.succeeded` | `Counter<long>` | `{delivery}` | Observer callbacks that returned without throwing. |
| `backwave.observer.deliveries.dead_lettered` | `Counter<long>` | `{delivery}` | Observer deliveries that exhausted their ceiling and were dead-lettered. |
| `messaging.process.duration` | `Histogram<double>` | `s` | Handler execution time, on success and failure. |
| `backwave.schedule.delay` | `Histogram<double>` | `s` | How late execution started against the job's Due Time. |
| `backwave.job.queue.wait` | `Histogram<double>` | `s` | Due Time to claim, clamped at zero. |
| `backwave.observer.dispatch.duration` | `Histogram<double>` | `s` | Observer callback dispatch time. |
| `backwave.worker.slots.active` | `UpDownCounter<long>` | `{slot}` | Slots currently occupied by running jobs. |
| `backwave.queue.depth` | `ObservableGauge<long>` | `{job}` | Point-in-time job counts by Queue and state. |
| `backwave.worker.slots.capacity` | `ObservableGauge<long>` | `{slot}` | Configured slot capacity per consumer group. |

The segment is singular in `backwave.job.attempts` and `backwave.job.queue.wait` but plural in
`backwave.jobs.failed` and `backwave.jobs.dead_lettered`; both spellings are stable identifiers.

### Histogram buckets

BackWave advises explicit bucket boundaries on its four histograms. Two schemes are in play, and
they do not share a ceiling.

| Histograms | Bucket boundaries (seconds) | Ceiling |
|---|---|---|
| `messaging.process.duration`, `backwave.observer.dispatch.duration` | `0.001, 0.002, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30` | 30 seconds |
| `backwave.schedule.delay`, `backwave.job.queue.wait` | `0.001, 0.01, 0.1, 0.5, 1, 2.5, 5, 10, 30, 60, 300, 3600` | 1 hour |

The wider scheme reaches an hour because scheduling delay and queue wait routinely run to minutes
or longer, whereas an execution or an observer dispatch measured beyond thirty seconds is already
an outlier. These bucket boundaries take effect on .NET 9 and later only; on .NET 8 all four
histograms fall back to the SDK default buckets.

### Counter semantics

The job counters relate in a way that trips up dashboards if you assume the wrong invariant. The
correct one is `attempts ≥ consumed + failed`, and the gap is in-flight and cancelled work. A
handler-thrown `OperationCanceledException` increments `backwave.jobs.failed`, but a
Shell-requested cancellation, where BackWave asks a running job to stop, increments neither the
consumed nor the failed counter and records no `messaging.process.duration`.

`backwave.jobs.failed` carries `error.type` and lumps together failures that will be retried and
failures that were terminally Dead-Lettered. `backwave.jobs.dead_lettered` counts only the
retry-exhausted terminal subset, so it is a subset of the failure count, not a parallel total. To
read how many jobs are Dead-Lettered *right now*, filter the `backwave.queue.depth` gauge to
`backwave.state = DeadLettered` rather than reaching for a counter. Note that
`backwave.observer.deliveries.dead_lettered` is about [Transition Observer](/docs/guides/react-to-job-outcomes)
deliveries, not jobs; do not conflate the two.

### Queue-depth gauge

`backwave.queue.depth` reports current job counts as a point-in-time gauge, one measurement per
Queue-and-state pair. It keeps the pre-rename tag keys: each measurement carries `backwave.queue`,
the Queue name, and `backwave.state`, the [job state](/docs/reference/job-states) as a string.

| `backwave.state` value | Meaning |
|---|---|
| `Scheduled` | Waiting for its Due Time. |
| `AwaitingParent` | Held until a parent job completes. |
| `Leased` | Claimed by a Worker and executing. |
| `Succeeded` | Completed successfully. |
| `Cancelled` | Cancelled before or during execution. |
| `DeadLettered` | Exhausted its Attempt ceiling after running and failing. |
| `Quarantined` | Could not be routed or decoded. |

Under the hosting package this gauge is fed for you by a background service that snapshots the
store on a cadence, so it updates at roughly that granularity rather than in real time, and a
failed depth query retains the last good snapshot rather than halting anything.

#### Registering a depth source

The gauge is not fed automatically outside the hosting package. `RegisterQueueDepthSource` takes a
callback that returns the current per-Queue, per-state counts and returns an `IDisposable`;
disposing that handle removes the source so it stops contributing to the gauge.

```csharp title="RegisterDepth.cs"
// public static IDisposable RegisterQueueDepthSource(Func<IReadOnlyList<QueueStateCount>> source)
IDisposable registration = BackWaveDiagnostics.RegisterQueueDepthSource(
    () => currentQueueStateCounts);

// Later, stop contributing to backwave.queue.depth.
registration.Dispose();
```

The callback runs each time the gauge is observed and returns an `IReadOnlyList<QueueStateCount>`.
`QueueStateCount` is a record with three positional members.

| Member | Type | Meaning |
|---|---|---|
| `Queue` | `string` | The Queue the count is for. |
| `State` | `JobState` | The state the count is for. |
| `Count` | `int` | Number of jobs in that Queue and state. |

Register one source per store. Registering two Worker Groups that share a store would double-count
that store's depths, and BackWave does not deduplicate for you.

### Saturation gauges

`backwave.worker.slots.active` (an UpDownCounter) and `backwave.worker.slots.capacity` (a gauge)
both carry `messaging.consumer.group.name`, so you join them by consumer group for headroom.
`worker.slots.capacity` accumulates per consumer group across pumps: a group running four pumps of
twenty slots each reports eighty, not twenty. A headroom panel that assumes per-pump capacity will
read saturated at a fraction of true capacity. `worker.slots.active` additionally carries the
destination tags.

### Observer counters

The three observer counters and the dispatch histogram always carry `backwave.observer_id`, the
attribution for which observer a delivery belongs to. The `attempted` and `succeeded` series may
additionally carry `backwave.wire_name` and `backwave.queue` when both are cheaply available at the
delivery edge; the `dead_lettered` series carries `backwave.observer_id` only. Do not write queries
that expect the Wire Name or Queue on the dead-letter series.

### Store-fault meter (per adapter)

This instrument is not a Core instrument. Each storage adapter defines its own copy on its own
meter, and it arrives only when you opt that adapter in with its `AddBackWave{Postgres|SqlServer|Sqlite}Instrumentation()`
method. Until then you get no store-fault measurements at all.

| Instrument | Type | Unit | Tag |
|---|---|---|---|
| `backwave.store.faults` | `Counter<long>` | `{fault}` | `backwave.store.fault_kind` = `transient` or `terminal` |

## Traces

BackWave starts three job-lifecycle activities on the activity source named `"BackWave"`, following
the OpenTelemetry messaging conventions. Each storage adapter starts its own db-convention span, on
its own source, covered under [Store spans](#store-spans) below.

| Operation name | Kind | Display name | When it fires |
|---|---|---|---|
| `send` | `Producer` | `send {queue}` | A job is accepted by enqueue. Its context becomes the job's trace correlation. |
| `send` (workflow root) | `Producer` | `send {workflow-name}` (or `send workflow`) | A Workflow is enqueued. Parents the per-member send spans and closes immediately. |
| `send` (per member) | `Producer` | `send {queue}` | Each workflow member is enqueued. Child of the workflow root. |
| `receive` | `Client` | `receive` | One claim round-trip against the store. Its display name is the literal `receive`, with no interpolation. |
| `process` | `Consumer` | `process {queue}` | One Handler execution. |

### Correlation

When BackWave accepts a job it captures the current W3C trace context and stores it with the job.
At execution the `process` span is started as a **forced root** and correlated back to its
originating `send` context with an `ActivityLink`, not a parent edge. A `process` span can hold
more than one link: a workflow member links to its own send context and to one context per fan-in
ancestor, so a fan-in step links to every upstream step it depends on. A job enqueued with no
ambient trace still starts its `process` span as a clean root and is never parented to whatever
`Activity` happens to be current on the Worker. Trace queries must follow the links rather than walk
a parent chain from enqueue to execution.

The stored context has two parts, and both parts survive the hop. BackWave stores the `traceparent`
and the `tracestate`, and the link on the `process` span restores both. Vendor routing and sampling
state therefore reach the Worker. If a captured `tracestate` is too large for the store column that
holds it, BackWave drops the `tracestate` and keeps the `traceparent`. An enqueue never fails
because of the size of a `tracestate`.

### The enqueue call site

The `send` span records where the enqueue call came from. Three tags carry it:
`code.function.name`, `code.file.path`, and `code.line.number`.

The compiler supplies these values at the call site, so BackWave reads no stack trace. The tags cost
nothing at run time and stay correct under Native AOT. The testing harness passes the call site
through, so a span from a test names the test rather than the harness.

### Span status

The `process` span records the outcome of the handler.

| Handler outcome | Span status | Extras | Counter |
|---|---|---|---|
| Returned normally | `Ok` | - | `messaging.client.consumed.messages` |
| Threw `OperationCanceledException` | `Error` | `error.type`, `exception` event | `backwave.jobs.failed` |
| Threw any other exception | `Error` | `error.type`, `exception` event | `backwave.jobs.failed` |
| Shell-requested cancellation | (span ends without an error status) | - | Neither consumed nor failed |

A handler-thrown `OperationCanceledException` is a failure, exactly like any other thrown
exception. Only a Shell-requested cancellation, where BackWave asks the running job to stop, is
neither consumed nor failed. The `send` and `receive` spans set no explicit status; they simply end.

### Store spans

Each adapter emits one `Client`-kind span per store round-trip, on the adapter's own source, and
only when you opt the adapter in. These spans are not force-rooted: they inherit the ambient Core
span (a `send` or `receive`) when there is one, or root under the background sweep otherwise.

| Adapter | Source name | `db.system` |
|---|---|---|
| Postgres | `BackWave.Postgres` | `postgresql` |
| SQL Server | `BackWave.SqlServer` | `mssql` |
| SQLite | `BackWave.Sqlite` | `sqlite` |

All three share the same shape: `ActivityKind.Client`, a display name of `{operation} {collection}`,
and tags `db.system`, `db.operation.name` (one of `claim`, `enqueue`, `complete`, `fail`, or
`expire_leases`), and `db.collection.name` (the effective jobs table, honoring a custom schema or
table prefix). On a store fault the span records `error.type` and an `Error` status.

## Structured logs

BackWave's lifecycle events are emitted through `ILogger` with stable event ids, which the
OpenTelemetry SDK auto-bridges into OTel Logs. Each event is emitted inside a log **scope** that
stamps exactly four keys: `job_id`, `wire_name`, `attempt`, and `queue`. Turn on `IncludeScopes` in
the logging pipeline to carry them through to your backend.

The event ids group by concern: `10xx` covers enqueue and claim, `11xx` execution, `12xx`
settlement, `13xx` store and schema, and `14xx` observers, all in the Core catalog. The Hosting
package adds an operational catalog at `20xx` and `21xx`, and BackWave Pro adds a workflow catalog
at `30xx`.

### Core catalog

| Event id | Event | Level | Meaning |
|---|---|---|---|
| 1001 | `JobEnqueued` | Debug | A job was accepted by enqueue. |
| 1101 | `LeaseAcquired` | Trace | A Worker claimed a lease on a job. |
| 1102 | `ExecutionStarted` | Debug | Handler execution began. |
| 1103 | `ExecutionCompleted` | Debug | Handler execution completed. |
| 1201 | `RetryScheduled` | Information | A failed Attempt was scheduled for retry. |
| 1202 | `LeaseLost` | Warning | A lease was lost mid-execution. |
| 1203 | `DeadLettered` | Error | A job exhausted its Attempt ceiling and was Dead-Lettered. |
| 1204 | `LeasesReclaimed` | Information | Expired leases were reclaimed for re-execution. |
| 1301 | `StoreFaultTransientRetry` | Warning | A transient store fault was retried. |
| 1302 | `MigrationApplied` | Information | A schema migration was applied. Opt-in; off by default. |
| 1401 | `ObserverDeliveryDeadLettered` | Warning | An observer delivery exhausted its ceiling. |

### Hosting operational catalog

| Event id | Event | Level | Meaning |
|---|---|---|---|
| 2001 | `WorkerGroupFailStopped` | Critical | A Worker Group fail-stopped. |
| 2101 | `ObserverPumpFaulted` | Error | The observer pump faulted. |
| 2102 | `ObserverClaimFaulted` | Warning | An observer claim faulted. |
| 2103 | `ObserverReportFaulted` | Warning | An observer report faulted. |
| 2104 | `ObserverCallbackFaulted` | Warning | An observer callback faulted. |
| 2105 | `ObserverCallbackTimedOut` | Error | An observer callback timed out. |
| 2106 | `ObserverLeakedCallbackFaulted` | Error | A leaked observer callback faulted after its delivery returned. |

### Pro workflow catalog

| Event id | Event | Level | Meaning |
|---|---|---|---|
| 3001 | `GateDecided` | Information | A workflow gate reached a decision. |

The `MigrationApplied` event (1302) is the one exception to "it just works": it emits only when the
store's options carry a `LoggerFactory` *and* `AutoMigrate` is on. `LoggerFactory` defaults to null,
so a default host never emits it. See [Wire Up OpenTelemetry](/docs/guides/wire-up-opentelemetry#the-migration-log-is-off-by-default)
for the wiring.

## Attribute keys

The job-lifecycle spans and metrics use the messaging-convention keys; the observer instruments,
the queue-depth gauge, and the store spans keep or add their own `backwave.*` and `db.*` keys. The
lifecycle **metrics** carry `messaging.system`, `messaging.destination.name`, and
`messaging.destination.template` (not `messaging.message.id`, to hold cardinality down), with
`error.type` added on the failure paths.

| Key | Value | Appears on |
|---|---|---|
| `messaging.system` | `backwave` | Job-lifecycle spans and metrics. |
| `messaging.destination.name` | The job's Queue | Job-lifecycle spans and metrics. |
| `messaging.destination.template` | The job's Wire Name | Job-lifecycle spans and metrics. |
| `messaging.message.id` | The job's id | `send` and `process` spans (not the metrics). |
| `messaging.operation.name` | The operation string | Job-lifecycle spans and metrics. |
| `messaging.operation.type` | The operation string (currently the same value as `.name`) | Job-lifecycle spans and metrics. |
| `messaging.consumer.group.name` | The Worker Group | `process` span; the slot gauges. |
| `code.function.name` | The method that called enqueue | `send` span. |
| `code.file.path` | The source file of that call | `send` span. |
| `code.line.number` | The line of that call | `send` span. |
| `backwave.worker_id` | The Worker id | `receive` span. |
| `backwave.claimed_count` | Number of jobs claimed | `receive` span, on completion. |
| `backwave.attempt` | The job's Attempt number | `process` span. |
| `backwave.workflow.name` | The Workflow name | Workflow-root `send` span. |
| `backwave.workflow.member_count` | Member count | Workflow-root `send` span. |
| `backwave.workflow.append` | Whether the enqueue appends to an existing Workflow | Workflow-root `send` span. |
| `backwave.workflow.after` | The member's upstream dependencies | Member `process` span. |
| `error.type` | The exception type | Failed counter, failed process-duration, `process` span, store-fault span. |
| `backwave.queue` | The Queue name | Queue-depth gauge; observer counters when available. |
| `backwave.state` | The job state name | Queue-depth gauge. |
| `backwave.wire_name` | The Wire Name | Observer counters when available. |
| `backwave.observer_id` | The observer id | Observer counters and dispatch histogram, always. |
| `backwave.store.fault_kind` | `transient` or `terminal` | `backwave.store.faults` (per adapter). |
| `db.system` | `postgresql` / `mssql` / `sqlite` | Store spans. |
| `db.operation.name` | `claim` / `enqueue` / `complete` / `fail` / `expire_leases` | Store spans. |
| `db.collection.name` | The effective jobs table | Store spans. |

The `backwave.wire_name` and `backwave.queue` keys survive only on the observer instruments and the
queue-depth gauge. On the job-lifecycle spans and metrics they were renamed to
`messaging.destination.template` and `messaging.destination.name`, so do not treat either old key as
present everywhere.

## Convention stability

The job-lifecycle spans and metrics borrow the OpenTelemetry **messaging** conventions, which are
still at *Development* stability upstream, so a future semantic-conventions release may rename some
`messaging.*` attributes. Only the `backwave.*`-owned names carry BackWave's stability promise; treat
the `messaging.*` keys as tracking an upstream spec rather than as forever-stable. The three `code.*`
keys on the `send` span come from the OpenTelemetry code conventions and carry the same caveat.

## Where to go next

- [Wire Up OpenTelemetry](/docs/guides/wire-up-opentelemetry): the wiring, the exemplars tip, and the Aspire path.
- [Observers API](/docs/reference/observers-api): Transition Observers, the source of the observer delivery counters.
- [Job States & Transitions](/docs/reference/job-states): the states that appear as `backwave.state` on the queue-depth gauge.
- [Execution Model](/docs/core-concepts/execution-model): claiming, leasing, and Attempts, which back the `receive` span and the Attempt counter.
- [Queues](/docs/core-concepts/queues): the Queue every span and metric is tagged by.