Telemetry Reference

The OpenTelemetry identifiers BackWave emits across all three pillars: the source names, every metric instrument, the send/receive/process spans, the log event-id catalog, and every tag key.

BackWave emits all three OpenTelemetry pillars, traces, metrics, and logs, on the OpenTelemetry semantic conventions. It builds on the base class library only, so spans, measurements, and log calls cost nothing until you subscribe a tracer, meter, or logging provider to the source. The job lifecycle follows the OpenTelemetry messaging conventions and each storage adapter's store round-trips follow the db conventions. This page lists every identifier BackWave emits, byte-for-byte, so you can wire it into your collector by name. For the wiring itself and the semantics behind these tables, see Wire Up OpenTelemetry.

The source names#

Job-lifecycle traces and metrics publish under a single name, "BackWave". Each storage adapter publishes its store spans and store-fault meter under its own name, and only when you opt that adapter in.

The BackWave value is exactly that, with a capital B and a capital W, and is exposed as the public constant BackWaveDiagnostics.SourceName so you can subscribe by constant rather than by literal. The scope version stamped on the source is the package's informational version. The BackWave.OpenTelemetry package wraps each name in a named subscription method; the by-name path (AddSource/AddMeter with these strings) is equivalent. For Oracle that method is AddBackWaveOracleInstrumentation(), on both TracerProviderBuilder and MeterProviderBuilder.

Metrics#

BackWave publishes sixteen Core instruments on the meter named "BackWave", plus one store-fault counter per adapter on that adapter's own meter. Units are curly-brace annotation units and appear verbatim on the instrument. The two throughput counters and the process-duration histogram carry messaging-convention names; the rest are backwave.*-owned.

Core instruments#

The segment is singular in backwave.job.attempts and backwave.job.queue.wait but plural in backwave.jobs.failed and backwave.jobs.dead_lettered; both spellings are stable identifiers.

Histogram buckets#

BackWave advises explicit bucket boundaries on its four histograms. Two schemes are in play, and they do not share a ceiling.

The wider scheme reaches an hour because scheduling delay and queue wait routinely run to minutes or longer, whereas an execution or an observer dispatch measured beyond thirty seconds is already an outlier. These bucket boundaries take effect on .NET 9 and later only; on .NET 8 all four histograms fall back to the SDK default buckets.

Counter semantics#

The job counters relate in a way that trips up dashboards if you assume the wrong invariant. The correct one is attempts ≥ consumed + failed, and the gap is in-flight and cancelled work. A handler-thrown OperationCanceledException increments backwave.jobs.failed, but a Shell-requested cancellation, where BackWave asks a running job to stop, increments neither the consumed nor the failed counter and records no messaging.process.duration.

backwave.jobs.failed carries error.type and lumps together failures that will be retried and failures that were terminally Dead-Lettered. backwave.jobs.dead_lettered counts only the retry-exhausted terminal subset, so it is a subset of the failure count, not a parallel total. To read how many jobs are Dead-Lettered right now, filter the backwave.queue.depth gauge to backwave.state = DeadLettered rather than reaching for a counter. Note that backwave.observer.deliveries.dead_lettered is about Transition Observer deliveries, not jobs; do not conflate the two.

Queue-depth gauge#

backwave.queue.depth reports current job counts as a point-in-time gauge, one measurement per Queue-and-state pair. It keeps the pre-rename tag keys: each measurement carries backwave.queue, the Queue name, and backwave.state, the job state as a string.

Under the hosting package this gauge is fed for you by a background service that snapshots the store on a cadence, so it updates at roughly that granularity rather than in real time, and a failed depth query retains the last good snapshot rather than halting anything.

Registering a depth source#

The gauge is not fed automatically outside the hosting package. RegisterQueueDepthSource takes a callback that returns the current per-Queue, per-state counts and returns an IDisposable; disposing that handle removes the source so it stops contributing to the gauge.

// public static IDisposable RegisterQueueDepthSource(Func<IReadOnlyList<QueueStateCount>> source)
IDisposable registration = BackWaveDiagnostics.RegisterQueueDepthSource(
    () => currentQueueStateCounts);
 
// Later, stop contributing to backwave.queue.depth.
registration.Dispose();

The callback runs each time the gauge is observed and returns an IReadOnlyList<QueueStateCount>. QueueStateCount is a record with three positional members.

Register one source per store. Registering two Worker Groups that share a store would double-count that store's depths, and BackWave does not deduplicate for you.

Saturation gauges#

backwave.worker.slots.active (an UpDownCounter) and backwave.worker.slots.capacity (a gauge) both carry messaging.consumer.group.name, so you join them by consumer group for headroom. worker.slots.capacity accumulates per consumer group across pumps: a group running four pumps of twenty slots each reports eighty, not twenty. A headroom panel that assumes per-pump capacity will read saturated at a fraction of true capacity. worker.slots.active additionally carries the destination tags.

Observer counters#

The three observer counters and the dispatch histogram always carry backwave.observer_id, the attribution for which observer a delivery belongs to. The attempted and succeeded series may additionally carry backwave.wire_name and backwave.queue when both are cheaply available at the delivery edge; the dead_lettered series carries backwave.observer_id only. Do not write queries that expect the Wire Name or Queue on the dead-letter series.

Invariant violations#

backwave.invariant.violations increments once per always-on check that fails. It carries backwave.invariant.trigger, the InvariantTrigger member name, and backwave.invariant.action, which is Halt when the check threw and Degrade when it logged and continued. A Halt raised inside a Worker Group pairs with log event 2001. One raised on the enqueue path or in a dependency read propagates to the caller instead. A 2001 whose trigger reads unclassified has no counter measurement. A Degrade measurement pairs with 2003 inside a Worker Group or 1601 elsewhere. The trigger set is listed on Always-On Assertions & Fail-Stop.

Store-fault meter (per adapter)#

This instrument is not a Core instrument. Each storage adapter defines its own copy on its own meter, and it arrives only when you opt that adapter in with its AddBackWave{Postgres|SqlServer|Sqlite|Oracle}Instrumentation() method. Until then you get no store-fault measurements at all.

Traces#

BackWave starts three job-lifecycle activities on the activity source named "BackWave", following the OpenTelemetry messaging conventions. Each storage adapter starts its own db-convention span, on its own source, covered under Store spans below.

Correlation#

When BackWave accepts a job it captures the current W3C trace context and stores it with the job. At execution the process span is started as a forced root and correlated back to its originating send context with an ActivityLink, not a parent edge. A process span can hold more than one link: a workflow member links to its own send context and to one context per fan-in ancestor, so a fan-in step links to every upstream step it depends on. A job enqueued with no ambient trace still starts its process span as a clean root and is never parented to whatever Activity happens to be current on the Worker. Trace queries must follow the links rather than walk a parent chain from enqueue to execution.

The stored context has two parts, and both parts survive the hop. BackWave stores the traceparent and the tracestate, and the link on the process span restores both. Vendor routing and sampling state therefore reach the Worker. If a captured tracestate is too large for the store column that holds it, BackWave drops the tracestate and keeps the traceparent. An enqueue never fails because of the size of a tracestate.

The enqueue call site#

The send span records where the enqueue call came from. Three tags carry it: code.function.name, code.file.path, and code.line.number.

The compiler supplies these values at the call site, so BackWave reads no stack trace. The tags cost nothing at run time and stay correct under Native AOT. The testing harness passes the call site through, so a span from a test names the test rather than the harness.

Span status#

The process span records the outcome of the handler.

A handler-thrown OperationCanceledException is a failure, exactly like any other thrown exception. Only a Shell-requested cancellation, where BackWave asks the running job to stop, is neither consumed nor failed. The send and receive spans set no explicit status; they simply end.

A process span settles when the batched outcome write that carries its row returns, on the row as the Worker Group applied it. It settles even when that write faulted. An outcome the Worker Group rewrote as a dead-letter (an over-cap output or an over-long tag) settles its span on that rewritten row.

Store spans#

Each adapter emits one Client-kind span per store round-trip, on the adapter's own source, and only when you opt the adapter in. These spans are not force-rooted: they inherit the ambient Core span (a send or receive) when there is one, or root under the background sweep otherwise.

All four share the same shape: ActivityKind.Client, a display name of {operation} {collection}, and tags db.system, db.operation.name (one of claim, enqueue, report_outcomes, expire_leases, or relinquish_leases), and db.collection.name (the effective jobs table, honoring a custom schema or table prefix). On a store fault the span records error.type and an Error status.

Structured logs#

BackWave's lifecycle events are emitted through ILogger with stable event ids, which the OpenTelemetry SDK auto-bridges into OTel Logs. Each event is emitted inside a log scope that stamps exactly four keys: job_id, wire_name, attempt, and queue. Turn on IncludeScopes in the logging pipeline to carry them through to your backend.

The event ids group by concern: 10xx covers enqueue and claim, 11xx execution, 12xx settlement, 13xx store and schema, 14xx observers, 15xx Wake-Up Hints, and 16xx invariants, all in the Core catalog. The Hosting package adds an operational catalog at 20xx and 21xx, and BackWave Pro adds a workflow catalog at 30xx.

Core catalog#

Hosting operational catalog#

Pro workflow catalog#

The MigrationApplied event (1302) is the one exception to "it just works": it emits only when the store's options carry a LoggerFactory and AutoMigrate is on. LoggerFactory defaults to null, so a default host never emits it. See Wire Up OpenTelemetry for the wiring.

Attribute keys#

The job-lifecycle spans and metrics use the messaging-convention keys; the observer instruments, the queue-depth gauge, and the store spans keep or add their own backwave.* and db.* keys. The lifecycle metrics carry messaging.system, messaging.destination.name, and messaging.destination.template (not messaging.message.id, to hold cardinality down), with error.type added on the failure paths.

The backwave.wire_name and backwave.queue keys survive only on the observer instruments and the queue-depth gauge. On the job-lifecycle spans and metrics they were renamed to messaging.destination.template and messaging.destination.name, so do not treat either old key as present everywhere.

Convention stability#

The job-lifecycle spans and metrics borrow the OpenTelemetry messaging conventions, which are still at Development stability upstream, so a future semantic-conventions release may rename some messaging.* attributes. Only the backwave.*-owned names carry BackWave's stability promise; treat the messaging.* keys as tracking an upstream spec rather than as forever-stable. The three code.* keys on the send span come from the OpenTelemetry code conventions and carry the same caveat.

Where to go next#

• Wire Up OpenTelemetry: the wiring, the exemplars tip, and the Aspire path.
• Observers API: Transition Observers, the source of the observer delivery counters.
• Job States & Transitions: the states that appear as backwave.state on the queue-depth gauge.
• Execution Model: claiming, leasing, and Attempts, which back the receive span and the Attempt counter.
• Queues: the Queue every span and metric is tagged by.

Found a problem on this page? Report an issue