Performance & Benchmarks

BackWave's benchmarking methodology, the tuning knobs that move throughput, and what is and isn't published.

BackWave measures throughput with a dedicated Benchmark Harness that drives the real pump against a real database under wall-clock time, deliberately outside the deterministic test world so a noisy run is never mistaken for a bug. Its credibility rests on a published methodology rather than a single hero number. A noop workload is the clearly-labeled overhead ceiling, every competitor is tuned to its own best honest config, and every run self-labels its environment so a laptop figure can never be quoted as official. This page covers the method: what the harness measures, how fairness is enforced, and which dials a consumer actually turns. As of this writing, no official numbers have been published yet.

What the harness measures, and what it deliberately does not#

The Benchmark Harness drives the real BackWave pump against a real Storage Adapter under wall-clock time, with no simulation and no in-memory shortcut, and produces throughput in jobs per second, latency percentiles, and resource numbers. It measures performance and never correctness. A noisy result is expected, and a flaky measurement is never treated as a defect, because correctness lives in a separate, deterministic world; see The Determinism Boundary and How BackWave Is Tested. The harness is the one body of testing that sits outside that boundary: it runs on a real clock with real I/O and real threads, non-deterministic by design.

The systems in scope are BackWave and Hangfire, each on PostgreSQL and SQL Server, the two networked Storage Adapters under test. Two things are excluded on purpose. The In-Memory Store runs on Virtual Time and so has no wall-clock throughput to measure, and SQLite is an embedded single-writer deployment class that must never share a chart with networked databases. The harness is a manual tool kept out of the automated test command; absolute numbers on shared CI runners are too noisy to gate a change, and official numbers need a pinned machine anyway.

The headline is a ceiling, clearly labeled#

The primary figure comes from a noop workload: an immediate handler, a small fixed payload in the ~100-200 byte band, a single queue, and a 100% happy path. That configuration isolates the only thing that differentiates one scheduler from another — the framework and storage cost of claiming, dispatching, writing the outcome, and Lease churn — which a real handler otherwise dominates and washes out. It is always framed as "system overhead supports N jobs/sec; real throughput is min(N, what your handler allows)" and never as a literal production number.

A single realistic anchor point ships alongside the ceiling: the same workload with a roughly 10 ms handler delay, which shows framework overhead becoming negligible against real per-job work. Handler delay of zero is the overhead ceiling; a small positive delay is the anchor.

Two throughput shapes, and metrics that never travel naked#

Throughput is measured in two shapes. A drain run preloads N jobs and then times the cluster emptying them to zero, the clean and reproducible headline. A sustained run runs producers and consumers together at a target arrival rate, exposing the enqueue-versus-claim table contention that production actually hits. Sustained mode fans the producer across several concurrent tasks so the arrival side outpaces the cluster's drain capacity; a single producer would cap sustained throughput at its own enqueue rate and starve the consumers.

No throughput number is published naked. Every cell travels with its latency percentiles and a distribution, so a single lucky run is never the figure quoted.

Latency is always a p50/p99 pair, never an average. Warmup runs, which absorb JIT, connection-pool fill, and database cache warm-up, are discarded before the distribution is computed. Allocation and GC numbers are kept as an internal-only BackWave metric and are never charted across systems. Process-wide GC stats set against a reflection-JSON competitor would be apples-to-oranges, so the harness does not even capture them for the competitor.

Fairness: tuned-to-best on both sides#

The success criterion is deliberately hard on BackWave: on equivalent storage, with both systems tuned to their best honest config, BackWave matches or beats the competitor, and separately demonstrates a high-throughput ceiling and near-linear scale-out. Tuning the competitor to its strongest is self-defense. A win over an under-tuned Hangfire reverses the moment a maintainer re-runs it correctly, and that would discredit everything else published.

The rule is to neutralize configuration differences and surface architectural ones. Worker and pool count, poll interval, connection-pool size, retry policy, and database hardware are matched so they cannot explain the result. The architectural differences are the product, won fairly and never hidden: batch-claim, source-generated serialization, and the row-locking claim (FOR UPDATE SKIP LOCKED on PostgreSQL, UPDLOCK, READPAST, ROWLOCK on SQL Server). Worker count is the single most load-bearing neutralized dial, so the harness derives the competitor's worker count from BackWave's pump pool size and the two are provably matched. Every dial is stamped into each result so a third party can reproduce and challenge any number.

SQL Server is the flagship comparison because Hangfire is first-party and battle-tested there, so a win is the most credible obtainable. PostgreSQL is published too, but footnoted, because its Hangfire adapter is community-maintained.

Note that the harness's aggressive tuning is not the shipped library default. The harness runs a max(8, cores × 4) pool at a 25 ms poll to measure the engine; a consumer out of the box gets a PoolSize of 20 and a one-second PollInterval. Do not read the harness config as a recommended production setting.

The scale-out curve#

Single-node, meaning one process per system, is the apples-to-apples comparison baseline. Separately, BackWave publishes a scale-out curve with no competitor in it, charting aggregate throughput as independent Node processes on one host rise 1 → 2 → 4 → 8 against a shared database. Each swept count resets and preloads the shared backlog, spawns that many Node subprocesses (each a Pump coordinating only through the shared adapter), waits at a readiness barrier until every pump is up and claiming, then times the interval between readiness and the backlog draining empty. Timing only the post-readiness drain keeps per-Node startup out of the window, so fixed startup cannot masquerade as an early saturation knee.

The DB-saturation knee is defined as the first Node count beyond which adding Nodes lifts aggregate throughput by less than 10%; the result records whether saturation was reached within the swept range or throughput was still climbing. True multi-host scale-out is acknowledged future work, not a promise.

Publishable vs indicative: why official numbers are gated#

Every run self-labels its mode and carries a full environment manifest (OS, process architecture, .NET version, CPU count, database engine and version) from which a publishable flag is derived. A number is publishable only when it is run in official mode on a native x86-64 host that is not Rosetta-emulated. SQL Server under Rosetta on Apple Silicon measures the emulator, not the engine, so official mode refuses up front on any ineligible host with a clean error and a non-zero exit rather than running to a silently unpublishable number.

As of this writing the published result matrix is entirely pending an official run. No official throughput numbers exist in the repository yet. Local-mode figures exist only to refine the harness and eyeball changes; they are never transcribed as results. Official runs are invoked manually per release.

How to run the benchmarks#

The single-run harness is a console app invoked with dotnet run -c Release. It takes a --target, a --mode, workload flags, and --out to write a self-labelled JSON result.

# Local mode only produces indicative numbers; never publish these.
dotnet run -c Release --project benchmarks/BackWave.Benchmarks -- \
  --target postgres --mode local \
  --arrival drain --delay-ms 0 --jobs 100000 \
  --warmup 1 --runs 5 \
  --out benchmarks/results/postgres-noop-drain.json

The full published matrix (BackWave versus Hangfire on both databases across noop-drain, noop-sustained, and the 10 ms anchor, plus the BackWave-only scale-out curve) is scripted as one battery that writes a JSON per cell. Connection strings default to the local Docker databases and are overridden by environment variables for the pinned instance.

# Native x86-64 Linux, database co-located on loopback.
export BACKWAVE_POSTGRES_DSN=...           BACKWAVE_SQLSERVER_DSN=...
export BACKWAVE_HANGFIRE_POSTGRES_DSN=...  BACKWAVE_HANGFIRE_SQLSERVER_DSN=...
 
benchmarks/run-official.sh   # one JSON per cell, plus the scale-out curve
# Transcribe only cells whose manifest reads "Publishable": true.

Official numbers require the app and database co-located on loopback, because network round-trip is additive-and-equal for both systems and only adds variance to the engine delta. On Apple Silicon or under Rosetta the harness produces indicative numbers only and official mode is refused.

What the tuning knobs mean for throughput#

A single Pump's store I/O is serial, with one round-trip in flight at a time, so a pump is bounded by store round-trip latency rather than by worker slots or CPU. Job execution itself already runs off the loop, so it is not the single-pump bottleneck; the serial store round-trip is. The dials below live on WorkerGroupOptions; for the mechanism behind them see Throughput & Pump Fan-Out, and for the symptom-to-dial view see Scaling & Throughput Tuning.

Pumps is the lever that raises the single-node ceiling: each pump claims under a distinct identity, so the database hands each a disjoint set of jobs with no extra coordination. Unlike idle pool slots, a pump draws connections whether busy or not, so budget roughly 2-3 connections per pump (a Pumps = 4 group draws on the order of 8-12) and leave it at 1 unless a node is throughput-bound with connection headroom.

builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("default"),
        // Single-node throughput lever. Default is 1.
        // Each Pump draws ~2-3 DB connections whether busy or idle,
        // so Pumps = 4 budgets ~8-12 connections for this group.
        Pumps = 4,
        PoolSize = 20,
    }));

The shape to expect#

Because the repository holds no official numbers, treat every observation as a shape rather than a guarantee. A single pump on a realistic handler sits near-idle on CPU at its ceiling, with the loop parked on database round-trips and large connection headroom. The ceiling is serial store I/O, not the box. In-process pump fan-out scales close to linearly and beats separate node processes at every count above one, because the pumps share a runtime and one connection pool and avoid cross-pool self-contention on the row-locking claim. Spending a single node's spare CPU and connection budget via Pumps is what lets one BackWave node reach and pass a competitor while still under its resource budget. Those are ratios and directions, not published throughput; the honest number is the method above.

Where to go next#

• Throughput & Pump Fan-Out: the serial store-I/O ceiling and how pumps multiply it.
• The Determinism Boundary: why the harness is the one test body outside the deterministic world.
• Scaling & Throughput Tuning: the symptom-to-dial tuning guide.
• Virtual-Time Testing: the in-memory, wall-clock-free world the harness deliberately excludes.