# Performance & Benchmarks

BackWave measures throughput with a dedicated **Benchmark Harness** that drives the real pump against a real database under wall-clock time, deliberately outside the deterministic test world so a noisy run is never mistaken for a bug. Its credibility rests on a published methodology rather than a single hero number. A noop workload is the clearly-labeled overhead ceiling, every competitor is tuned to its own best honest config, and every run self-labels its environment so a laptop figure can never be quoted as official. This page covers the method: what the harness measures, how fairness is enforced, and which dials a consumer actually turns. As of this writing, no official numbers have been published yet. The harness itself is not private: it lives under `benchmarks/` in the [public repository](https://github.com/Back-Wave/BackWave/tree/main/benchmarks), so every command on this page is one you can run against your own database and read the source of.

## What the harness measures, and what it deliberately does not

The **Benchmark Harness** drives the real BackWave pump against a real **Storage Adapter** under wall-clock time, with no simulation and no in-memory shortcut, and produces throughput in jobs per second, latency percentiles, and resource numbers. It measures performance and never correctness. A noisy result is expected, and a flaky measurement is never treated as a defect, because correctness lives in a separate, deterministic world; see [The Determinism Boundary](/docs/advanced/determinism-boundary) and [How BackWave Is Tested](/docs/testing/how-backwave-is-tested). The harness is the one body of testing that sits *outside* that boundary: it runs on a real clock with real I/O and real threads, non-deterministic by design.

The systems in scope are BackWave and Hangfire, each on PostgreSQL and SQL Server, the two networked **Storage Adapters** under test. Two things are excluded on purpose. The **In-Memory Store** runs on [Virtual Time](/docs/testing/behavior-over-time) and so has no wall-clock throughput to measure, and SQLite is an embedded single-writer deployment class that must never share a chart with networked databases. The harness is a manual tool kept out of the automated test command; absolute numbers on shared CI runners are too noisy to gate a change, and official numbers need a pinned machine anyway.

## The headline is a ceiling, clearly labeled

The primary figure comes from a noop workload: an immediate handler, a small fixed payload in the ~100-200 byte band, a single queue, and a 100% happy path. That configuration isolates the only thing that differentiates one scheduler from another — the framework and storage cost of claiming, dispatching, writing the outcome, and **Lease** churn — which a real handler otherwise dominates and washes out. It is always framed as *"system overhead supports N jobs/sec; real throughput is min(N, what your handler allows)"* and never as a literal production number.

A single realistic anchor point ships alongside the ceiling: the same workload with a roughly 10 ms handler delay, which shows framework overhead becoming negligible against real per-job work. Handler delay of zero is the overhead ceiling; a small positive delay is the anchor.

## Two throughput shapes, and metrics that never travel naked

Throughput is measured in two shapes. A **drain** run preloads N jobs and then times the cluster emptying them to zero, the clean and reproducible headline. A **sustained** run runs producers and consumers together at a target arrival rate, exposing the enqueue-versus-claim table contention that production actually hits. Sustained mode fans the producer across several concurrent tasks so the arrival side outpaces the cluster's drain capacity; a single producer would cap sustained throughput at its own enqueue rate and starve the consumers.

No throughput number is published naked. Every cell travels with its latency percentiles and a distribution, so a single lucky run is never the figure quoted.

| Metric | What it captures | How it is reported |
|---|---|---|
| Throughput | Jobs completed over the measured window | min / median / max across repeated runs |
| End-to-end latency | Enqueue → terminal state | p50 and p99, never a bare average |
| Enqueue latency | Enqueue call → committed | p50 and p99 |
| Peak DB connections | Max concurrent connections held under load | Sampled by a background task |
| CPU | Process CPU time ÷ (wall window × core count) | Computed identically for every system |

Latency is always a p50/p99 pair, never an average. Warmup runs, which absorb JIT, connection-pool fill, and database cache warm-up, are discarded before the distribution is computed. Allocation and GC numbers are kept as an internal-only BackWave metric and are never charted across systems. Process-wide GC stats set against a reflection-JSON competitor would be apples-to-oranges, so the harness does not even capture them for the competitor.

## Fairness: tuned-to-best on both sides

The success criterion is deliberately hard on BackWave: on equivalent storage, with both systems tuned to their best honest config, BackWave matches or beats the competitor, and separately demonstrates a high-throughput ceiling and near-linear scale-out. Tuning the competitor to its strongest is self-defense. A win over an under-tuned Hangfire reverses the moment a maintainer re-runs it correctly, and that would discredit everything else published.

The rule is to neutralize configuration differences and surface architectural ones. Worker and pool count, poll interval, connection-pool size, retry policy, and database hardware are matched so they cannot explain the result. The architectural differences are the product, won fairly and never hidden: batch-claim, source-generated serialization, and the row-locking claim (`FOR UPDATE SKIP LOCKED` on PostgreSQL, `UPDLOCK, READPAST, ROWLOCK` on SQL Server). Worker count is the single most load-bearing neutralized dial, so the harness derives the competitor's worker count from BackWave's pump pool size and the two are provably matched. Every dial is stamped into each result so a third party can reproduce and challenge any number.

| Dial | BackWave (harness) | Hangfire (harness) | Kind |
|---|---|---|---|
| Worker / pool size | `max(8, cores × 4)` | Derived from BackWave, matched | Neutralized |
| Poll interval | 25 ms | SQL Server 0 ms; PostgreSQL 200 ms | Neutralized (tuned-equivalent) |
| Connection-pool size | 100 (ADO.NET default) | 100 (ADO.NET default) | Neutralized |
| Retry policy | None exercised (happy path) | Automatic retry disabled | Neutralized |
| Claim strategy | Batch-claim ≤128/poll, row-locking skip | Single-row fetch per worker | Architectural |
| Serialization | Source-generated, no reflection | Reflection + Newtonsoft.Json | Architectural |

SQL Server is the flagship comparison because Hangfire is first-party and battle-tested there, so a win is the most credible obtainable. PostgreSQL is published too, but footnoted, because its Hangfire adapter is community-maintained.

Note that the harness's aggressive tuning is not the shipped library default. The harness runs a `max(8, cores × 4)` pool at a 25 ms poll to measure the *engine*; a consumer out of the box gets a `PoolSize` of 20 and a one-second `PollInterval`. Do not read the harness config as a recommended production setting.

## The scale-out curve

Single-node, meaning one process per system, is the apples-to-apples comparison baseline. Separately, BackWave publishes a scale-out curve with no competitor in it, charting aggregate throughput as independent **Node** processes on one host rise 1 → 2 → 4 → 8 against a shared database. Each swept count resets and preloads the shared backlog, spawns that many Node subprocesses (each a **Pump** coordinating only through the shared adapter), waits at a readiness barrier until every pump is up and claiming, then times the interval between readiness and the backlog draining empty. Timing only the post-readiness drain keeps per-Node startup out of the window, so fixed startup cannot masquerade as an early saturation knee.

The DB-saturation knee is defined as the first Node count beyond which adding Nodes lifts aggregate throughput by less than 10%; the result records whether saturation was reached within the swept range or throughput was still climbing. True multi-host scale-out is acknowledged future work, not a promise.

## Publishable vs indicative: why official numbers are gated

Every run self-labels its mode and carries a full environment manifest (OS, process architecture, .NET version, CPU count, database engine and version) from which a publishable flag is derived. A number is publishable only when it is run in official mode on a native x86-64 host that is not Rosetta-emulated. SQL Server under Rosetta on Apple Silicon measures the emulator, not the engine, so official mode refuses up front on any ineligible host with a clean error and a non-zero exit rather than running to a silently unpublishable number.

| Run mode | Host architecture | Publishable? |
|---|---|---|
| Official | Native x86-64 (not Rosetta) | Yes, the only publishable source |
| Official | Rosetta-emulated x64 on Apple Silicon | No, refused up front |
| Official | Arm64 / any non-x64 | No, refused up front |
| Local | Any | No, indicative only |

As of this writing the published result matrix is entirely pending an official run. No official throughput numbers exist in the repository yet. Local-mode figures exist only to refine the harness and eyeball changes; they are never transcribed as results. Official runs are invoked manually per release.

## How to run the benchmarks

The single-run harness is a console app invoked with `dotnet run -c Release`. It takes a `--target`, a `--mode`, workload flags, and `--out` to write a self-labelled JSON result.

| Flag | Meaning | Default |
|---|---|---|
| `--target` | `postgres` \| `sqlserver` \| `hangfire-postgres` \| `hangfire-sqlserver` | `postgres` |
| `--mode` | `local` (indicative) \| `official` (publishable, native x86-64 only) | `local` |
| `--jobs` | Number of jobs in the stream | `10000` |
| `--arrival` | `drain` \| `sustained` | `drain` |
| `--delay-ms` | Per-job handler delay (0 = noop ceiling, ~10 = realistic anchor) | `0` |
| `--rate` | Sustained enqueue rate, jobs/sec (0 = unpaced) | `0` |
| `--producers` | Concurrent producer tasks | `1` |
| `--warmup` | Warmup runs discarded | `1` |
| `--runs` | Measured runs reported as min/median/max | `3` |
| `--out` | Write result JSON to a file | stdout only |

```bash title="Run a single indicative cell (local)"
# Local mode only produces indicative numbers; never publish these.
dotnet run -c Release --project benchmarks/BackWave.Benchmarks -- \
  --target postgres --mode local \
  --arrival drain --delay-ms 0 --jobs 100000 \
  --warmup 1 --runs 5 \
  --out benchmarks/results/postgres-noop-drain.json
```

The full published matrix (BackWave versus Hangfire on both databases across noop-drain, noop-sustained, and the 10 ms anchor, plus the BackWave-only scale-out curve) is scripted as one battery that writes a JSON per cell. Connection strings default to the local Docker databases and are overridden by environment variables for the pinned instance.

```bash title="Run the official battery on a pinned host"
# Native x86-64 Linux, database co-located on loopback.
export BACKWAVE_POSTGRES_DSN=...           BACKWAVE_SQLSERVER_DSN=...
export BACKWAVE_HANGFIRE_POSTGRES_DSN=...  BACKWAVE_HANGFIRE_SQLSERVER_DSN=...

benchmarks/run-official.sh   # one JSON per cell, plus the scale-out curve
# Transcribe only cells whose manifest reads "Publishable": true.
```

Official numbers require the app and database co-located on loopback, because network round-trip is additive-and-equal for both systems and only adds variance to the engine delta. On Apple Silicon or under Rosetta the harness produces indicative numbers only and official mode is refused.

## What the tuning knobs mean for throughput

A single **Pump**'s store I/O is serial, with one round-trip in flight at a time, so a pump is bounded by store round-trip latency rather than by worker slots or CPU. Job execution itself already runs off the loop, so it is not the single-pump bottleneck; the serial store round-trip is. The dials below live on `WorkerGroupOptions`; for the mechanism behind them see [Throughput & Pump Fan-Out](/docs/advanced/throughput-model), and for the symptom-to-dial view see [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning).

| Property | Default | Effect on throughput | Cost |
|---|---|---|---|
| `Pumps` | `1` | The single-node throughput lever: N independent claim→execute→report loops, each claiming a disjoint job set | ~2-3 DB connections per Pump, busy or idle |
| `PoolSize` | `20` | Max concurrent jobs per node; polling pauses while full | Idle slots cost nothing |
| `MaxClaimBatch` | `32` | Jobs claimed per poll (effectively capped at `PoolSize`) | Larger batches amortize round-trips |
| `MaxOutcomeBatch` | `null` → `MaxClaimBatch` | Buffers completed outcomes into one batched write, keeping the writer single-threaded | Raises throughput without opening more connections |
| `PollInterval` | `1 second` | Shorter interval lowers pickup latency | More frequent store queries |

`Pumps` is the lever that raises the single-node ceiling: each pump claims under a distinct identity, so the database hands each a disjoint set of jobs with no extra coordination. Unlike idle pool slots, a pump draws connections whether busy or not, so budget roughly 2-3 connections per pump (a `Pumps = 4` group draws on the order of 8-12) and leave it at 1 unless a node is throughput-bound with connection headroom.

```csharp title="Program.cs" {8-12}
builder.Services.AddBackWave(bw => bw
    .UseStore(new PostgresJobStore(connectionString))
    .UseJobs(BackWaveJobs.Module)
    .AddWorkerGroup(new WorkerGroupOptions
    {
        Name = "default",
        Policy = new DispatchPolicy.Strict("default"),
        // Single-node throughput lever. Default is 1.
        // Each Pump draws ~2-3 DB connections whether busy or idle,
        // so Pumps = 4 budgets ~8-12 connections for this group.
        Pumps = 4,
        PoolSize = 20,
    }));
```

## The shape to expect

Because no official numbers are published yet, treat every observation below as a shape rather than a guarantee, and run the battery yourself if you need a figure you can stand behind. A single pump on a realistic handler sits near-idle on CPU at its ceiling, with the loop parked on database round-trips and large connection headroom. The ceiling is serial store I/O, not the box. In-process pump fan-out scales close to linearly and beats separate node processes at every count above one, because the pumps share a runtime and one connection pool and avoid cross-pool self-contention on the row-locking claim. Spending a single node's spare CPU and connection budget via `Pumps` is what lets one BackWave node reach and pass a competitor while still under its resource budget. Those are ratios and directions, not published throughput; the honest number is the method above.

## Where to go next

- [Throughput & Pump Fan-Out](/docs/advanced/throughput-model): the serial store-I/O ceiling and how pumps multiply it.
- [The Determinism Boundary](/docs/advanced/determinism-boundary): why the harness is the one test body outside the deterministic world.
- [Scaling & Throughput Tuning](/docs/dashboard-operations/scaling-and-throughput-tuning): the symptom-to-dial tuning guide.
- [Virtual-Time Testing](/docs/testing/behavior-over-time): the in-memory, wall-clock-free world the harness deliberately excludes.