Wire Up OpenTelemetry

Subscribe BackWave’s ActivitySource and Meter to your OpenTelemetry exporter to get handler spans and metrics.

Subscribe BackWave’s ActivitySource and Meter to your OpenTelemetry exporter to get handler spans and metrics.

BackWave instruments itself with the standard .NET diagnostics primitives, an ActivitySource for spans and a Meter for metrics. It ships no exporter, no collector, and no configuration of its own. It only emits under a known name, and you point whatever OpenTelemetry pipeline you already run at that name. The whole task is two builder calls. This guide covers those calls, the spans and metrics you light up, and the handful of semantics worth knowing before you build dashboards on top of them.

Two things are worth knowing before the wiring. BackWave publishes everything under a single name, "BackWave": one ActivitySource and one Meter, both carrying that string, so you subscribe both with the same constant. And the instrumentation costs nothing until you subscribe. It is built on the base class library primitives, so with no listener attached a span is a null no-op and a measurement is never taken. Ship it in production and you pay for it only once you actually export.

The single subscription name#

Everything lives in the BackWave.Diagnostics namespace, on a single static class, BackWaveDiagnostics.

BackWaveDiagnostics.SourceName is the canonical thing to subscribe. It is the literal "BackWave", and both the ActivitySource and the Meter carry it. Prefer the constant over the bare string so your wiring survives any future rename. The ActivitySource and Meter are also exposed for advanced scenarios, but subscription is by name, so most code never touches them directly. RegisterQueueDepthSource is a hook for self-hosted setups, covered at the end.

Wire it into OpenTelemetry#

Subscribing is two calls on your OpenTelemetry builder: AddSource on the tracer provider and AddMeter on the meter provider, each passed the same BackWaveDiagnostics.SourceName.

using BackWave.Diagnostics;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;
 
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("my-app"))
    .WithTracing(tracing => tracing
        .AddSource(BackWaveDiagnostics.SourceName)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddMeter(BackWaveDiagnostics.SourceName)
        .AddOtlpExporter());

A few things about this snippet. The string passed to AddSource and AddMeter is identical: one name feeds both providers. Tracing and metrics are independent subscriptions, so wiring one and forgetting the other is the most common mistake. You get spans but no metrics, or the reverse. Add AddAspNetCoreInstrumentation (or whatever produces an ambient span where you enqueue) so that the trace context BackWave captures at enqueue is a real, sampled parent. That is what makes the cross-time correlation in the next section visible end to end. The exporter is your choice. AddOtlpExporter, AddConsoleExporter, Prometheus, anything. BackWave is exporter-agnostic and ships none. Order does not matter; AddSource and AddMeter can come before or after instrumentation and exporters.

There is nothing to configure on the BackWave side. No enable flag, no sampler, no tag toggles, no telemetry options record. Sampling and filtering are done entirely in the OpenTelemetry pipeline, through SetSampler, views, and the like.

Verify it works#

Swap .AddConsoleExporter() onto both pipelines, enqueue one Job, and watch the console. You should see a backwave.enqueue span paired with a backwave.execute span, plus the backwave.jobs.enqueued and backwave.jobs.processed counters tick. Once you see the pair, point the exporters back at your real backend.

The spans#

BackWave emits three spans on the BackWave ActivitySource, and they map onto the Job Lifecycle you already know: enqueue, claim, execute.

The tag keys are part of the public observable contract, so you can build queries against them with confidence. backwave.wire_name is the serialization-stable Wire Name of the payload, backwave.queue is the Queue name (the default Queue reads default), backwave.attempt is the 1-based Attempt number (a claim increments it, so the first run is attempt 1), and backwave.claimed_count is how many Jobs the claim batch returned.

The execute span is re-parented to enqueue#

This is the behavior most worth understanding. When BackWave accepts a Job, it captures the current W3C trace context and stores it with the Job. Later, possibly hours later and on a different machine, the Worker that runs the Job starts the backwave.execute span as a child of that stored context. So a Handler shows up in your trace nested under the HTTP request, or whatever ambient operation enqueued it, even though they ran at different times on different nodes.

The correlation works as long as some Activity is current at enqueue time, which is exactly why AddAspNetCoreInstrumentation (or your framework's equivalent) matters. It also degrades safely. When no trace context was stored, BackWave deliberately starts the execute span as an explicit root rather than inheriting whatever ambient Activity happens to be live in the Worker loop. An un-correlated Job never gets accidentally glued onto an unrelated span.

Span status#

The execute span carries status. A Handler that returns normally ends Ok. A Handler that throws ends Error, with the description set to the exception message. A cancellation, an OperationCanceledException, ends Error with the description cancelled. The enqueue and claim spans set no explicit status; they simply end.

One caution if you alert on the execute span. BackWave classifies any OperationCanceledException as a cancellation, so a Handler that surfaces a library timeout as an OperationCanceledException (an HttpClient deadline, for example) produces an Error/cancelled span even though the Worker treats that failure as an ordinary retryable failure. If you rely on the span status to distinguish real cancellations from failures, prefer not to surface library timeouts as OperationCanceledException. See Retries & Error Handling for how the Worker classifies that case.

The metrics#

BackWave emits eight instruments on the BackWave Meter. The units are UCUM annotation units, dimensionless counts with a human label, the OpenTelemetry-conventional way to mark "this counts things."

Counter semantics worth knowing#

The four Job counters relate in a way that trips up dashboards if you assume the wrong invariant. The correct one is attempts ≥ processed + failed. The gap is in-flight work and cancelled work.

Cancelled Jobs count nowhere. A cooperative cancellation increments neither processed nor failed, by design. An operator-cancelled Job leaves those totals untouched and shows up only as an Error/cancelled execute span. So if you build a dashboard on attempts == processed + failed, it will be off by the cancelled count. Use the inequality.

jobs.failed lumps together failures that will be retried and failures that were terminally Dead-Lettered. There is no separate per-Job dead-letter counter. If you want to know how many Jobs are Dead-Lettered right now, that is the backwave.queue.depth gauge filtered to backwave.state = DeadLettered, not a counter. The dead-letter counter that does exist, backwave.observer.deliveries.dead_lettered, is about Transition Observer deliveries, not Jobs. Do not conflate them.

The queue-depth gauge#

backwave.queue.depth reports current Job counts as a point-in-time gauge, one measurement per Queue-and-state pair. Its backwave.state tag carries the state name as a string: one of Scheduled, AwaitingParent, Leased, Succeeded, Cancelled, DeadLettered, or Quarantined. This gauge is the operational primitive for "how much work is backed up, and in what state," and it is where you read Dead-Lettered and Quarantined depth.

When you register BackWave through the hosting package, a background service feeds this gauge for you, once per host. It snapshots the store roughly every 30 seconds and exposes the latest snapshot behind the gauge. That has two consequences. The gauge updates at about 30-second granularity, not in real time, so a scrape between refreshes sees the previous snapshot. And if a depth query fails, the last good snapshot is retained and nothing halts. The metric is non-load-bearing on purpose; it never blocks your Workers.

Observer-delivery tags#

The three observer counters always carry backwave.observer_id, the required attribution for which observer a delivery belongs to. The attempted and succeeded series may additionally carry backwave.wire_name and backwave.queue when both are cheaply available at the delivery edge. The dead_lettered series carries backwave.observer_id only. Do not write queries that expect wire name or queue on the dead-letter series.

Feeding queue depth without the hosting package#

If you run BackWave without the hosting package's hosted services, nothing feeds backwave.queue.depth and the gauge stays empty. That is not an error, just an absence of measurements. To light it up yourself, call RegisterQueueDepthSource with a callback that returns the current per-Queue, per-state counts, and dispose the returned handle on shutdown.

IDisposable depthHandle = BackWaveDiagnostics.RegisterQueueDepthSource(
    () => store.CountJobs());
 
// on shutdown
depthHandle.Dispose();

The callback yields QueueStateCount records, one per Queue-and-state pair, each carrying its Queue name, JobState, and Count.

Register exactly one source per store. If two Worker Groups share a store and each registers a depth source, the depths double-count. Dispose extras. Under the hosting package this is handled for you, one feeder per host, so reach for RegisterQueueDepthSource only when you run your own host loop.

Things that are deliberately not here#

A short boundary, so you do not go looking for an API that does not exist.

• There is no BackWave-side telemetry configuration. No enable or disable flag, no sampler, no tag filter. Do all of that in the OpenTelemetry pipeline through samplers and views.
• The ActivitySource and Meter are created without a version string. If you key OpenTelemetry views on meter version, there is none to match on.
• Referencing the ActivitySource or Meter field in your own code collects nothing. Collection happens only when a provider subscribes the name through AddSource or AddMeter. The fields are not a substitute for subscribing.
• The queue-depth feeder runs whether or not anyone exports. Subscribing the meter is what surfaces its measurements; not subscribing simply means nobody reads them.

Health checks are a separate observability surface from this OpenTelemetry wiring. If you want a liveness signal rather than traces and metrics, see Health, Fail-Stop & Draining and the Monitor API. They answer a different question and do not overlap with the spans and counters here.

Where to go next#

• Telemetry reference: the exhaustive table of every instrument, span, and tag key, byte-for-byte.
• Job Lifecycle: the enqueue, claim, and execute states the three spans map onto.
• React to Job Outcomes: the Transition Observers behind the backwave.observer.deliveries.* counters.
• Configure Retries & Error Handling: how the Worker classifies failures and cancellations that show up in your spans.
• Health, Fail-Stop & Draining: the health-check surface that sits alongside, but apart from, OpenTelemetry.
• Monitor API: query Queue depth and Job state programmatically when a gauge scrape is not enough.