# Wire Up OpenTelemetry

Subscribe BackWave to your OpenTelemetry pipeline to light up all three pillars: job traces, metrics, and structured logs.

BackWave instruments itself with the standard .NET diagnostics primitives, an `ActivitySource` for spans, a `Meter` for metrics, and `ILogger` for structured logs. It ships no exporter, no collector, and no configuration of its own. It only emits under known source names, and you point whatever OpenTelemetry pipeline you already run at those names. This guide covers the wiring, the send, receive, and process spans you light up, the metric instruments, the structured-log catalog, and the handful of semantics worth knowing before you build dashboards and alerts on top of them.

One thing is worth knowing before the wiring: the instrumentation costs nothing until you subscribe. It is built on the base class library primitives, so with no listener attached a span is a null no-op, a measurement is never taken, and a log call short-circuits. Ship it in production and you pay for it only once you actually export. Store spans and the store-fault meter go a step further and are opt-in per adapter, so the chattier database signals never crowd the job-lifecycle view unless you ask for them.

## The registration package

BackWave publishes its job-lifecycle traces and metrics under a single source name, `"BackWave"`. Each storage adapter publishes its own store signals under its own name: `"BackWave.Postgres"`, `"BackWave.SqlServer"`, or `"BackWave.Sqlite"`. You could subscribe those names by hand, but the `BackWave.OpenTelemetry` package gives you a named method for each so your wiring survives any future rename.

`AddBackWaveInstrumentation()` subscribes the Core job-lifecycle source. It is defined on both the tracer-provider builder and the meter-provider builder, so the same call name works on each. The per-adapter methods, `AddBackWavePostgresInstrumentation()`, `AddBackWaveSqlServerInstrumentation()`, and `AddBackWaveSqliteInstrumentation()`, opt you in to that adapter's store spans and store-fault meter, and are likewise defined on both builders.

Subscription is by source *name*, so `BackWave.OpenTelemetry` references only the OpenTelemetry API and does not drag any BackWave assembly into your build. Opting into an adapter does not pull that adapter's assembly in either. If you prefer to subscribe by hand, `AddSource("BackWave")` / `AddMeter("BackWave")` (and the `"BackWave.Postgres"` and sibling names) are equivalent and fully supported; the public constant `BackWaveDiagnostics.SourceName` holds the `"BackWave"` string.

## Wire it into OpenTelemetry

Add the registration package plus the OpenTelemetry SDK and whichever exporters you use, then wire tracing, metrics, and logging. `AddBackWaveInstrumentation()` subscribes the Core signals with one call on each builder; add the matching per-adapter method for the store you run to also collect its store spans and store-fault meter.

```csharp title="Program.cs" {10,15,24}
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("my-app"))
    .WithTracing(tracing => tracing
        .AddBackWaveInstrumentation()          // Core job spans: send, receive, process
        .AddBackWavePostgresInstrumentation()  // opt in to the Postgres store spans (or SqlServer / Sqlite)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter())                     // reads OTEL_EXPORTER_OTLP_ENDPOINT by default
    .WithMetrics(metrics => metrics
        .AddBackWaveInstrumentation()          // Core metrics incl. the messaging.process.duration histogram
        .AddBackWavePostgresInstrumentation()  // opt in to the Postgres store-fault meter
        .SetExemplarFilter(ExemplarFilterType.TraceBased)  // trace-id exemplars on the histograms (see below)
        .AddOtlpExporter())
    .WithLogging(
        logging => logging
            .AddOtlpExporter(),
        options =>
        {
            options.IncludeScopes = true;           // carry the job_id / wire_name / attempt / queue scopes
            options.IncludeFormattedMessage = true;
        });
```

A few things about this snippet. Tracing, metrics, and logging are independent subscriptions, so wiring one and forgetting another is the most common mistake; you get spans but no metrics, or the reverse. Add `AddAspNetCoreInstrumentation` (or whatever produces an ambient span where you enqueue) so that the trace context BackWave captures at enqueue is a real, sampled parent, which is what makes the cross-time correlation in the spans section visible end to end. The exporter is your choice: OTLP, Console, Prometheus, anything. BackWave is exporter-agnostic and ships none. Order does not matter; the `AddBackWave...` calls can come before or after other instrumentation and the exporters.

There is nothing to configure on the BackWave side. BackWave has no enable flag, sampler, tag toggles, or telemetry-options record to set; sampling and filtering happen entirely in the OpenTelemetry pipeline, through `SetSampler`, views, and the like.

### Verify it works

Add `.AddConsoleExporter()` to all three pipelines, enqueue one Job, and watch the console. You should see a `send` span paired with a `process` span, the `messaging.client.sent.messages` and `messaging.client.consumed.messages` counters tick, and the lifecycle log events print. Once you see them, point the exporters back at your real backend.

### Exemplars

`SetExemplarFilter(ExemplarFilterType.TraceBased)` attaches a **trace-id exemplar** to each histogram data point that was recorded inside a sampled trace. A spike in a slow bucket of `messaging.process.duration`, `backwave.schedule.delay`, or `backwave.job.queue.wait` then links straight through to the trace that produced it, so a worse p99 is one click away from the exact slow job. Trace and span ids ride only as exemplars, never as metric attributes: putting a trace id on a metric dimension would give every measurement a unique attribute set and explode the metric's cardinality. `TraceBased` records an exemplar only when the measurement happened inside a sampled trace; `AlwaysOn` records one for every measurement.

### The .NET Aspire dashboard

The [.NET Aspire dashboard](https://learn.microsoft.com/dotnet/aspire/fundamentals/dashboard/overview) is a standalone OTLP receiver with a traces, metrics, and logs UI. Because BackWave exports plain OTLP, you get a full local observability UI with no AppHost project and no code change: run the dashboard image, point `OTEL_EXPORTER_OTLP_ENDPOINT` at it, drive a job, and BackWave's spans render as a trace (`send`, `receive`, `process`, with the fan-in links), the metrics appear under Metrics, and the catalogued log events show under Structured logs with their scopes. It is all standard OTLP on standard conventions, so no bespoke config is involved.

## The spans

BackWave emits three job-lifecycle spans on the `BackWave` `ActivitySource`, and they map onto the [Job Lifecycle](/docs/core-concepts/job-lifecycle) you already know: enqueue, claim, execute.

| Operation name | Kind | Display name | When it fires | Key tags |
| --- | --- | --- | --- | --- |
| `send` | Producer | `send {queue}` | A Job is accepted by an enqueue call. | `messaging.destination.name`, `messaging.destination.template`, `messaging.message.id`, `code.function.name`, `code.file.path`, `code.line.number` |
| `receive` | Client | `receive` | One claim round-trip against the store. | `backwave.worker_id`, `backwave.claimed_count` |
| `process` | Consumer | `process {queue}` | One Handler execution. | `messaging.destination.name`, `messaging.destination.template`, `backwave.attempt`, `messaging.consumer.group.name` |

The spans follow the OpenTelemetry **messaging** conventions. `messaging.destination.name` is the [Queue](/docs/core-concepts/queues) name (the default Queue reads `default`), `messaging.destination.template` is the serialization-stable [Wire Name](/docs/core-concepts/jobs-and-handlers) of the payload, `messaging.message.id` is the job id, and `messaging.consumer.group.name` is the [Worker Group](/docs/core-concepts/worker-groups). Every job-lifecycle span also carries `messaging.system=backwave`. The `receive` span's display name is the literal `receive` with no interpolation, unlike `send` and `process`, which interpolate the Queue name. When you enqueue a [Workflow](/docs/core-concepts/workflows), the workflow root is a `send` span named `send {workflow-name}` (or `send workflow` when the workflow is unnamed) that closes immediately and parents one `send` span per member.

The full byte-for-byte span, tag, and metric contract lives in the [Telemetry reference](/docs/reference/telemetry). What follows is the semantics worth internalizing before you build on it.

### A process span is a root linked to its send, not a child

This is the behavior most worth understanding, and it changed in this release. When BackWave accepts a Job, it captures the current W3C trace context and stores it with the Job. Later, possibly hours later and on a different machine, the Worker that runs the Job starts the `process` span as a **forced root** and correlates it back to the originating enqueue with an `ActivityLink`, not a parent edge. So a Handler does not appear nested under the HTTP request that enqueued it; instead its span carries a link you follow to reach that request.

The link model is deliberate. A single `process` span can fan in across many upstream steps: a workflow member links to *every* ancestor step it depends on, and a batch links to each of its enqueues. A hard parent edge can express only one ancestor; a set of `ActivityLink`s expresses all of them. Trace-waterfall tools that assume strict parent/child will not draw enqueue-to-execute as nesting, so build your queries to follow links rather than walk the parent chain.

The correlation works as long as some `Activity` is current at enqueue time, which is exactly why `AddAspNetCoreInstrumentation` (or your framework's equivalent) matters. It also degrades safely: a Job enqueued with no ambient trace still starts its `process` span as a clean root and is never accidentally glued onto whatever unrelated `Activity` happens to be live in the Worker loop.

Both parts of the W3C context survive the hop. BackWave stores the `traceparent` and the `tracestate`, and the link restores both. Your vendor routing and sampling state therefore reach the Worker. If a `tracestate` is too large for the store, BackWave drops the `tracestate` and keeps the `traceparent`. An enqueue never fails because of the size of a `tracestate`.

### The send span points back at your code

The `send` span carries the enqueue call site as three tags: `code.function.name`, `code.file.path`, and `code.line.number`. A job in your trace backend therefore points back at the line that enqueued it.

The compiler fills these values in at the call site. There is no stack walk, so the tags cost nothing at run time and stay correct under a Native AOT publish. The [testing harness](/docs/testing/your-first-test) forwards the call site as well. A span from a test therefore names your test, not the harness.

### Span status

The `process` span carries status. A Handler that returns normally ends `Ok`. A Handler that throws ends `Error`, with `error.type` set and an `exception` event attached. The `send` and `receive` spans set no explicit status; they simply end.

One caution if you alert on the `process` span. A handler-thrown `OperationCanceledException` is classified as a **failure**: it ends the span `Error`, records `error.type`, and increments `backwave.jobs.failed`, exactly as any other thrown exception does. A Handler that surfaces a library timeout as an `OperationCanceledException` (an `HttpClient` deadline, for example) therefore reads as a failure, which is usually what you want, since the Worker retries it as one. Only a Shell-requested cancellation, where BackWave itself asks a running Job to stop, is neither consumed nor failed: it records no `process.duration` and increments neither the processed nor the failed counter. See [Retries & Error Handling](/docs/guides/configure-retries-and-error-handling) for how the Worker classifies that case.

## The metrics

BackWave emits fifteen Core instruments on the `BackWave` `Meter`, plus a per-adapter store-fault counter on each adapter's own meter. The throughput counters and the process-duration histogram follow the OpenTelemetry messaging conventions; the rest are `backwave.*`-owned instruments for the signals the conventions do not cover. The [Telemetry reference](/docs/reference/telemetry) lists every instrument, unit, and bucket set; the semantics below are the ones most likely to surprise you.

- **Throughput is renamed.** Jobs accepted at enqueue count on `messaging.client.sent.messages`; successful executions count on `messaging.client.consumed.messages`. Any dashboard keyed on the old `backwave.jobs.enqueued` / `backwave.jobs.processed` names is broken until you repoint it.
- **Latency is a histogram now, in seconds.** `messaging.process.duration` records execution time in seconds on success and failure, but not on a Shell-requested cancellation. Two more histograms, `backwave.schedule.delay` (how late a Job started against its Due Time) and `backwave.job.queue.wait` (Due Time to claim), round out the timing picture, also in seconds.
- **`backwave.jobs.failed` carries `error.type`.** Alert on failure rate *by exception type*. `backwave.jobs.dead_lettered` counts only the retry-exhausted terminal subset, so it is a subset of failures, not a parallel total.
- **Saturation is two instruments that join by consumer group.** `backwave.worker.slots.active` (an UpDownCounter) and `backwave.worker.slots.capacity` (a gauge) both carry `messaging.consumer.group.name`; join them for headroom. Capacity accumulates per group across pumps, so a group running four pumps of twenty slots reports eighty, not twenty.
- **Queue depth is a store-polled gauge.** `backwave.queue.depth` reports point-in-time Job counts, tagged `backwave.queue` and `backwave.state`. It is the operational primitive for "how much work is backed up, and in what state," and it is where you read Dead-Lettered and Quarantined depth, since there is no per-Job dead-letter counter.

> **Histogram buckets are .NET 9+ only.** The custom bucket boundaries BackWave advises on `messaging.process.duration`, `backwave.schedule.delay`, `backwave.job.queue.wait`, and `backwave.observer.dispatch.duration` take effect only on .NET 9 and later. On .NET 8 those histograms fall back to the SDK default buckets, so a dashboard binned on the documented boundaries will look different there.

### Feed queue depth without the hosting package

`backwave.queue.depth` is the one instrument that needs a feeder. Under the hosting package a background service registers that feeder for you, exactly one per host, and nothing more is needed. If you run BackWave without the hosting package's hosted services, nothing feeds the gauge and it stays empty; that is not an error, just an absence of measurements. To light it up yourself, call `RegisterQueueDepthSource` with a callback that returns the current per-Queue, per-state counts, and dispose the returned handle on shutdown.

```csharp title="Program.cs"
IDisposable depthHandle = BackWaveDiagnostics.RegisterQueueDepthSource(
    () => store.CountJobs());

// on shutdown
depthHandle.Dispose();
```

The callback is synchronous and runs each time the meter provider observes the gauge, so return the counts you have on hand as an `IReadOnlyList<QueueStateCount>`, one entry per Queue-and-state pair. Each `QueueStateCount` carries its `Queue` name, `JobState`, and `Count`. Register exactly one source per store: if two Worker Groups share a store and each registers a depth source, the depths double-count, so dispose the extras. Reach for this only when you run your own host loop; under the hosting package it is already handled, one feeder per host.

## Structured logs

Logs are the third pillar. BackWave's lifecycle events are emitted through `ILogger` with stable event ids, and the OpenTelemetry SDK auto-bridges `ILogger` into OTel Logs, so once you wire `WithLogging` there is no bespoke pipe to build. The full event-id catalog, with every id, level, and meaning, is in the [Telemetry reference](/docs/reference/telemetry).

Two settings on the logging pipeline matter. Set `IncludeScopes = true` so the job scope rides along: every lifecycle log is emitted inside a scope that stamps `job_id`, `wire_name`, `attempt`, and `queue`, which is what lets you filter a backend to one job or one queue. Set `IncludeFormattedMessage = true` so the rendered message text reaches your backend rather than only the message template and its arguments.

### The migration log is off by default

Every log event above needs no store-side wiring except the schema-migration event, which is opt-in and off by default. The store writes it only when both of these hold: the store's options carry a `LoggerFactory`, and `AutoMigrate` is on. `LoggerFactory` defaults to null, which disables the event with no allocation and changes nothing else about the store, so a default host, including the wiring shown above, never emits it no matter how the OTel logging pipeline is configured. Set the factory when you build the store to turn the event on.

```csharp title="Program.cs"
builder.Services.AddBackWave(backwave => backwave
    .UseStore(serviceProvider => new PostgresJobStore(new PostgresStoreOptions
    {
        ConnectionString = connectionString,
        AutoMigrate = true,
        LoggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>(),
    })));
```

The `UseStore` factory overload hands you the `IServiceProvider`, so the app's own `ILoggerFactory` is already in reach with no separate registration. The same property exists on the SQL Server and SQLite store options.

## Convention stability

The job-lifecycle spans and metrics borrow the OpenTelemetry **messaging** conventions, which are still at *Development* stability upstream, so a future semantic-conventions release may rename some `messaging.*` attributes. Only the `backwave.*`-owned names carry BackWave's stability promise. Treat the `messaging.*` keys as tracking an upstream spec rather than as forever-stable, and expect that a convention bump could move one.

## Where to go next

- [Telemetry reference](/docs/reference/telemetry): the exhaustive table of every instrument, span, tag key, and log event id, byte-for-byte.
- [Job Lifecycle](/docs/core-concepts/job-lifecycle): the enqueue, claim, and execute states the three spans map onto.
- [React to Job Outcomes](/docs/guides/react-to-job-outcomes): the Transition Observers behind the `backwave.observer.deliveries.*` counters.
- [Configure Retries & Error Handling](/docs/guides/configure-retries-and-error-handling): how the Worker classifies failures and cancellations that show up in your spans.
- [Health, Fail-Stop & Draining](/docs/dashboard-operations/health-and-fail-stop): the health-check surface that sits alongside, but apart from, OpenTelemetry.
- [Monitor API](/docs/dashboard-operations/monitor-api): query Queue depth and Job state programmatically when a gauge scrape is not enough.