How BackWave Is Tested

The layered strategy behind BackWave's correctness: a deterministic core, virtual-time tests you write, seeded-fault simulation inspired by TigerBeetle's VOPR, an executable storage contract, concurrency torture on real databases, and continuous soak and benchmarks.

BackWave runs your production queue, so correctness is the whole point of the library. That confidence comes from layers, each built to catch what the others structurally cannot, and it starts with a single design choice that makes the strongest layers possible: a deterministic core. You never write any of these tests yourself; they are how the library is built and how it stays correct as it grows. This page is the tour.

It starts with a deterministic core#

The scheduling logic (when a job is due, when to retry and how long to back off, when a lease has expired, which state transition is legal) does no I/O and never reads the wall clock. It is a pure function of state, event, and time: feed it the same events in the same order and you get the same decisions, every single run.

That determinism is not an academic nicety. It is the property that makes every layer below it possible. A test can hand the core events on a clock it fully controls and get identical behavior each time, with no real timers, no sleeping, and no flakiness. The seam that keeps the core pure and pushes all the messy I/O to the edges is described in The Determinism Boundary.

The layer you use: virtual-time tests#

The same determinism is handed to you directly. The BackWave.Testing package gives you a single-node, in-memory harness that pairs a real, deterministic implementation of the storage contract with a clock that only moves when you advance it. A simulated year of retries, recurring schedules, and lease expiries runs in milliseconds, and you assert through the same Monitor API production observability uses.

The point worth internalizing: your handler tests run against the same core that ships. There is no mock of the scheduler, no approximation of retry timing. What you assert in a test is what happens in production, which is why these tests are both fast and trustworthy. Start with Write Your First Test and Test Behavior Over Time.

Deterministic simulation, inspired by TigerBeetle's VOPR#

This is where the deterministic core pays off hardest. Internally, BackWave drives a whole cluster of virtual nodes against an in-memory store through compressed virtual time, and the entire run (every node, every message, every tick of the clock) is determined by a single 64-bit seed.

On top of that it injects faults, the failures a distributed background-job system must survive: nodes crashing and restarting mid-job, clocks skewed so machines disagree about "now", lost heartbeats that let a lease silently lapse, transient store faults, a node cut off from the database while it keeps working, lost acknowledgements, and operator actions (cancel, requeue, pause) racing everything else. After every step, a battery of invariant checks runs and asserts the things that must always be true: no job is ever executed twice, no stale write from a lapsed node ever applies, a queue's concurrency limit is never exceeded, and every job eventually reaches a terminal state. The moment any invariant is violated, the run halts, and because one seed fully determined it, the entire failure replays exactly, fault for fault and interleaving for interleaving.

The approach is borrowed from TigerBeetle, the financial database, and its VOPR, the deterministic simulator they use to test their system. The method travels even though the two systems have nothing else in common. You turn a hard-to-test distributed system into a pure function of a seed, then let a machine search seeds for the interleaving that breaks an invariant. Because the very same core code runs in the simulator and in production, a bug the simulator finds is a bug that would otherwise have shipped.

Two things make this more than a big test suite:

• It is a forever-running search, not a fixed list of scenarios. A pool of workers draws fresh seeds without end, so the space of crash, skew, and loss interleavings is explored continuously rather than sampled by a handful of hand-written cases. When a seed trips an invariant, it is automatically shrunk to the smallest still-failing case and checked in as a permanent regression fixture, so a bug found once can never quietly return.
• The checks themselves are tested. A check that has silently gone dead catches nothing, so each invariant has a deliberately broken twin that must trip it. A run that sabotages a property and still comes back green is treated as its own bug. That is what lets the suite claim its guarantees are alive rather than merely present.

One contract, every database#

Determinism proves the core, but it cannot prove a real SQL query. A wrong Postgres claim statement or an off-by-one in a migration lives below the boundary the simulation can reach. That surface is covered by an executable storage contract: a single specification, with each clause named after the behavior it pins down, run against every store BackWave ships (in-memory, PostgreSQL, SQL Server, and SQLite), each against a real instance of its database.

The in-memory store is the reference implementation and passes the whole contract; any divergence on a real adapter is therefore a bug in exactly one of the two, and the suite tells you which. This is the mechanism behind the claim that every adapter behaves identically: it is verified, not asserted. The details of the contract and how adapters are held to it are in the Storage overview.

Concurrency torture on real databases#

The contract suite proves each clause in isolation; torture testing proves those clauses hold under genuine contention. Swarms of concurrent clients hammer a live database at once (including real multi-process SQLite pounding a single file on disk) to surface the races that no single-threaded or deterministic test can reach: two workers reaching for the same job, a lease expiring in the middle of a claim, a failover landing mid-transaction. These are the bugs that only appear when real processes and a real database scheduler fight over the same rows.

Soaked and benchmarked#

Two final layers run on a real clock. Long-running soak flows keep the system under load for extended periods to catch slow leaks and drift that a fast test never runs long enough to see. Throughput benchmarks track jobs per second and latency percentiles so a performance regression is caught before it reaches a release.

Benchmarks measure performance and never correctness, and that separation is deliberate: correctness already lives in the deterministic world above, so a noisy or flaky measurement is never mistaken for a defect. The methodology and what is measured are covered in Performance & Benchmarks.

What this means for you#

You inherit all of it for free. The core your handlers run against in a virtual-time test is the exact core the simulation tortures with crashes and skew, and the exact core that ships in your app. So the tests you write are fast and non-flaky for the same reason the simulation is even possible, and the guarantees the library makes to you are the ones under continuous attack. You get to write a small number of ordinary tests and trust that the hard, distributed correctness underneath them has already been earned.

Where to go next#

• Write Your First Test: build the harness and run the define → advance → assert loop.
• Test Behavior Over Time: drive delays, retries, and schedules on a clock you control.
• Integration Testing: the handful of real-database tests that cover what the harness cannot.
• The Determinism Boundary: the architectural seam that makes the deterministic layers possible.