Wire Up OpenTelemetry

Subscribe BackWave to your OpenTelemetry pipeline to light up all three pillars: job traces, metrics, and structured logs.

Subscribe BackWave to your OpenTelemetry pipeline to light up all three pillars: job traces, metrics, and structured logs.

BackWave instruments itself with the standard .NET diagnostics primitives, an ActivitySource for spans, a Meter for metrics, and ILogger for structured logs. It ships no exporter, no collector, and no configuration of its own. It only emits under known source names, and you point whatever OpenTelemetry pipeline you already run at those names. This guide covers the wiring, the send, receive, and process spans you light up, the metric instruments, the structured-log catalog, and the handful of semantics worth knowing before you build dashboards and alerts on top of them.

One thing is worth knowing before the wiring: the instrumentation costs nothing until you subscribe. It is built on the base class library primitives, so with no listener attached a span is a null no-op, a measurement is never taken, and a log call short-circuits. Ship it in production and you pay for it only once you actually export. Store spans and the store-fault meter go a step further and are opt-in per adapter, so the chattier database signals never crowd the job-lifecycle view unless you ask for them.

The registration package#

BackWave publishes its job-lifecycle traces and metrics under a single source name, "BackWave". Each storage adapter publishes its own store signals under its own name: "BackWave.Postgres", "BackWave.SqlServer", or "BackWave.Sqlite". You could subscribe those names by hand, but the BackWave.OpenTelemetry package gives you a named method for each so your wiring survives any future rename.

AddBackWaveInstrumentation() subscribes the Core job-lifecycle source. It is defined on both the tracer-provider builder and the meter-provider builder, so the same call name works on each. The per-adapter methods, AddBackWavePostgresInstrumentation(), AddBackWaveSqlServerInstrumentation(), and AddBackWaveSqliteInstrumentation(), opt you in to that adapter's store spans and store-fault meter, and are likewise defined on both builders.

Subscription is by source name, so BackWave.OpenTelemetry references only the OpenTelemetry API and does not drag any BackWave assembly into your build. Opting into an adapter does not pull that adapter's assembly in either. If you prefer to subscribe by hand, AddSource("BackWave") / AddMeter("BackWave") (and the "BackWave.Postgres" and sibling names) are equivalent and fully supported; the public constant BackWaveDiagnostics.SourceName holds the "BackWave" string.

Wire it into OpenTelemetry#

Add the registration package plus the OpenTelemetry SDK and whichever exporters you use, then wire tracing, metrics, and logging. AddBackWaveInstrumentation() subscribes the Core signals with one call on each builder; add the matching per-adapter method for the store you run to also collect its store spans and store-fault meter.

using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
 
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("my-app"))
    .WithTracing(tracing => tracing
        .AddBackWaveInstrumentation()          // Core job spans: send, receive, process
        .AddBackWavePostgresInstrumentation()  // opt in to the Postgres store spans (or SqlServer / Sqlite)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter())                     // reads OTEL_EXPORTER_OTLP_ENDPOINT by default
    .WithMetrics(metrics => metrics
        .AddBackWaveInstrumentation()          // Core metrics incl. the messaging.process.duration histogram
        .AddBackWavePostgresInstrumentation()  // opt in to the Postgres store-fault meter
        .SetExemplarFilter(ExemplarFilterType.TraceBased)  // trace-id exemplars on the histograms (see below)
        .AddOtlpExporter())
    .WithLogging(
        logging => logging
            .AddOtlpExporter(),
        options =>
        {
            options.IncludeScopes = true;           // carry the job_id / wire_name / attempt / queue scopes
            options.IncludeFormattedMessage = true;
        });

A few things about this snippet. Tracing, metrics, and logging are independent subscriptions, so wiring one and forgetting another is the most common mistake; you get spans but no metrics, or the reverse. Add AddAspNetCoreInstrumentation (or whatever produces an ambient span where you enqueue) so that the trace context BackWave captures at enqueue is a real, sampled parent, which is what makes the cross-time correlation in the spans section visible end to end. The exporter is your choice: OTLP, Console, Prometheus, anything. BackWave is exporter-agnostic and ships none. Order does not matter; the AddBackWave... calls can come before or after other instrumentation and the exporters.

There is nothing to configure on the BackWave side. BackWave has no enable flag, sampler, tag toggles, or telemetry-options record to set; sampling and filtering happen entirely in the OpenTelemetry pipeline, through SetSampler, views, and the like.

Verify it works#

Add .AddConsoleExporter() to all three pipelines, enqueue one Job, and watch the console. You should see a send span paired with a process span, the messaging.client.sent.messages and messaging.client.consumed.messages counters tick, and the lifecycle log events print. Once you see them, point the exporters back at your real backend.

Exemplars#

SetExemplarFilter(ExemplarFilterType.TraceBased) attaches a trace-id exemplar to each histogram data point that was recorded inside a sampled trace. A spike in a slow bucket of messaging.process.duration, backwave.schedule.delay, or backwave.job.queue.wait then links straight through to the trace that produced it, so a worse p99 is one click away from the exact slow job. Trace and span ids ride only as exemplars, never as metric attributes: putting a trace id on a metric dimension would give every measurement a unique attribute set and explode the metric's cardinality. TraceBased records an exemplar only when the measurement happened inside a sampled trace; AlwaysOn records one for every measurement.

The .NET Aspire dashboard#

The .NET Aspire dashboard is a standalone OTLP receiver with a traces, metrics, and logs UI. Because BackWave exports plain OTLP, you get a full local observability UI with no AppHost project and no code change: run the dashboard image, point OTEL_EXPORTER_OTLP_ENDPOINT at it, drive a job, and BackWave's spans render as a trace (send, receive, process, with the fan-in links), the metrics appear under Metrics, and the catalogued log events show under Structured logs with their scopes. It is all standard OTLP on standard conventions, so no bespoke config is involved.

The spans#

BackWave emits three job-lifecycle spans on the BackWave ActivitySource, and they map onto the Job Lifecycle you already know: enqueue, claim, execute.

The spans follow the OpenTelemetry messaging conventions. messaging.destination.name is the Queue name (the default Queue reads default), messaging.destination.template is the serialization-stable Wire Name of the payload, messaging.message.id is the job id, and messaging.consumer.group.name is the Worker Group. Every job-lifecycle span also carries messaging.system=backwave. The receive span's display name is the literal receive with no interpolation, unlike send and process, which interpolate the Queue name. When you enqueue a Workflow, the workflow root is a send span named send {workflow-name} (or send workflow when the workflow is unnamed) that closes immediately and parents one send span per member.

The full byte-for-byte span, tag, and metric contract lives in the Telemetry reference. What follows is the semantics worth internalizing before you build on it.

A process span is a root linked to its send, not a child#

This is the behavior most worth understanding, and it changed in this release. When BackWave accepts a Job, it captures the current W3C trace context and stores it with the Job. Later, possibly hours later and on a different machine, the Worker that runs the Job starts the process span as a forced root and correlates it back to the originating enqueue with an ActivityLink, not a parent edge. So a Handler does not appear nested under the HTTP request that enqueued it; instead its span carries a link you follow to reach that request.

The link model is deliberate. A single process span can fan in across many upstream steps: a workflow member links to every ancestor step it depends on, and a batch links to each of its enqueues. A hard parent edge can express only one ancestor; a set of ActivityLinks expresses all of them. Trace-waterfall tools that assume strict parent/child will not draw enqueue-to-execute as nesting, so build your queries to follow links rather than walk the parent chain.

The correlation works as long as some Activity is current at enqueue time, which is exactly why AddAspNetCoreInstrumentation (or your framework's equivalent) matters. It also degrades safely: a Job enqueued with no ambient trace still starts its process span as a clean root and is never accidentally glued onto whatever unrelated Activity happens to be live in the Worker loop.

Both parts of the W3C context survive the hop. BackWave stores the traceparent and the tracestate, and the link restores both. Your vendor routing and sampling state therefore reach the Worker. If a tracestate is too large for the store, BackWave drops the tracestate and keeps the traceparent. An enqueue never fails because of the size of a tracestate.

The send span points back at your code#

The send span carries the enqueue call site as three tags: code.function.name, code.file.path, and code.line.number. A job in your trace backend therefore points back at the line that enqueued it.

The compiler fills these values in at the call site. There is no stack walk, so the tags cost nothing at run time and stay correct under a Native AOT publish. The testing harness forwards the call site as well. A span from a test therefore names your test, not the harness.

Span status#

The process span carries status. A Handler that returns normally ends Ok. A Handler that throws ends Error, with error.type set and an exception event attached. The send and receive spans set no explicit status; they simply end.

One caution if you alert on the process span. A handler-thrown OperationCanceledException is classified as a failure: it ends the span Error, records error.type, and increments backwave.jobs.failed, exactly as any other thrown exception does. A Handler that surfaces a library timeout as an OperationCanceledException (an HttpClient deadline, for example) therefore reads as a failure, which is usually what you want, since the Worker retries it as one. Only a Shell-requested cancellation, where BackWave itself asks a running Job to stop, is neither consumed nor failed: it records no process.duration and increments neither the processed nor the failed counter. See Retries & Error Handling for how the Worker classifies that case.

The metrics#

BackWave emits fifteen Core instruments on the BackWave Meter, plus a per-adapter store-fault counter on each adapter's own meter. The throughput counters and the process-duration histogram follow the OpenTelemetry messaging conventions; the rest are backwave.*-owned instruments for the signals the conventions do not cover. The Telemetry reference lists every instrument, unit, and bucket set; the semantics below are the ones most likely to surprise you.

• Throughput is renamed. Jobs accepted at enqueue count on messaging.client.sent.messages; successful executions count on messaging.client.consumed.messages. Any dashboard keyed on the old backwave.jobs.enqueued / backwave.jobs.processed names is broken until you repoint it.
• Latency is a histogram now, in seconds. messaging.process.duration records execution time in seconds on success and failure, but not on a Shell-requested cancellation. Two more histograms, backwave.schedule.delay (how late a Job started against its Due Time) and backwave.job.queue.wait (Due Time to claim), round out the timing picture, also in seconds.
• backwave.jobs.failed carries error.type. Alert on failure rate by exception type. backwave.jobs.dead_lettered counts only the retry-exhausted terminal subset, so it is a subset of failures, not a parallel total.
• Saturation is two instruments that join by consumer group. backwave.worker.slots.active (an UpDownCounter) and backwave.worker.slots.capacity (a gauge) both carry messaging.consumer.group.name; join them for headroom. Capacity accumulates per group across pumps, so a group running four pumps of twenty slots reports eighty, not twenty.
• Queue depth is a store-polled gauge. backwave.queue.depth reports point-in-time Job counts, tagged backwave.queue and backwave.state. It is the operational primitive for "how much work is backed up, and in what state," and it is where you read Dead-Lettered and Quarantined depth, since there is no per-Job dead-letter counter.

Histogram buckets are .NET 9+ only. The custom bucket boundaries BackWave advises on messaging.process.duration, backwave.schedule.delay, backwave.job.queue.wait, and backwave.observer.dispatch.duration take effect only on .NET 9 and later. On .NET 8 those histograms fall back to the SDK default buckets, so a dashboard binned on the documented boundaries will look different there.

Feed queue depth without the hosting package#

backwave.queue.depth is the one instrument that needs a feeder. Under the hosting package a background service registers that feeder for you, exactly one per host, and nothing more is needed. If you run BackWave without the hosting package's hosted services, nothing feeds the gauge and it stays empty; that is not an error, just an absence of measurements. To light it up yourself, call RegisterQueueDepthSource with a callback that returns the current per-Queue, per-state counts, and dispose the returned handle on shutdown.

IDisposable depthHandle = BackWaveDiagnostics.RegisterQueueDepthSource(
    () => store.CountJobs());
 
// on shutdown
depthHandle.Dispose();

The callback is synchronous and runs each time the meter provider observes the gauge, so return the counts you have on hand as an IReadOnlyList<QueueStateCount>, one entry per Queue-and-state pair. Each QueueStateCount carries its Queue name, JobState, and Count. Register exactly one source per store: if two Worker Groups share a store and each registers a depth source, the depths double-count, so dispose the extras. Reach for this only when you run your own host loop; under the hosting package it is already handled, one feeder per host.

Structured logs#

Logs are the third pillar. BackWave's lifecycle events are emitted through ILogger with stable event ids, and the OpenTelemetry SDK auto-bridges ILogger into OTel Logs, so once you wire WithLogging there is no bespoke pipe to build. The full event-id catalog, with every id, level, and meaning, is in the Telemetry reference.

Two settings on the logging pipeline matter. Set IncludeScopes = true so the job scope rides along: every lifecycle log is emitted inside a scope that stamps job_id, wire_name, attempt, and queue, which is what lets you filter a backend to one job or one queue. Set IncludeFormattedMessage = true so the rendered message text reaches your backend rather than only the message template and its arguments.

The migration log is off by default#

Every log event above needs no store-side wiring except the schema-migration event, which is opt-in and off by default. The store writes it only when both of these hold: the store's options carry a LoggerFactory, and AutoMigrate is on. LoggerFactory defaults to null, which disables the event with no allocation and changes nothing else about the store, so a default host, including the wiring shown above, never emits it no matter how the OTel logging pipeline is configured. Set the factory when you build the store to turn the event on.

builder.Services.AddBackWave(backwave => backwave
    .UseStore(serviceProvider => new PostgresJobStore(new PostgresStoreOptions
    {
        ConnectionString = connectionString,
        AutoMigrate = true,
        LoggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>(),
    })));

The UseStore factory overload hands you the IServiceProvider, so the app's own ILoggerFactory is already in reach with no separate registration. The same property exists on the SQL Server and SQLite store options.

Convention stability#

The job-lifecycle spans and metrics borrow the OpenTelemetry messaging conventions, which are still at Development stability upstream, so a future semantic-conventions release may rename some messaging.* attributes. Only the backwave.*-owned names carry BackWave's stability promise. Treat the messaging.* keys as tracking an upstream spec rather than as forever-stable, and expect that a convention bump could move one.

Where to go next#

• Telemetry reference: the exhaustive table of every instrument, span, tag key, and log event id, byte-for-byte.
• Job Lifecycle: the enqueue, claim, and execute states the three spans map onto.
• React to Job Outcomes: the Transition Observers behind the backwave.observer.deliveries.* counters.
• Configure Retries & Error Handling: how the Worker classifies failures and cancellations that show up in your spans.
• Health, Fail-Stop & Draining: the health-check surface that sits alongside, but apart from, OpenTelemetry.
• Monitor API: query Queue depth and Job state programmatically when a gauge scrape is not enough.

Found a problem on this page? Report an issue