How BackWave Is Tested

The layered strategy behind BackWave's correctness: a deterministic core, virtual-time tests you write, seeded-fault simulation inspired by TigerBeetle's VOPR, property-based tests over the scheduling kernels, an executable storage contract, concurrency torture on real databases, tested in-place upgrades, continuous soak and benchmarks, and mutation testing that proves the tests have teeth.

BackWave runs your production queue, so correctness is the whole point of the library. That confidence comes from layers, each built to catch what the others structurally cannot, and it starts with a single design choice that makes the strongest layers possible: a deterministic core. You never write any of these tests yourself; they are how the library is built and how it stays correct as it grows. This page is the tour.

It starts with a deterministic core#

The scheduling logic (when a job is due, when to retry and how long to back off, when a lease has expired, which state transition is legal) does no I/O and never reads the wall clock. It is a pure function of state, event, and time: feed it the same events in the same order and you get the same decisions, every single run.

That determinism is not an academic nicety. It is the property that makes every layer below it possible. A test can hand the core events on a clock it fully controls and get identical behavior each time, with no real timers, no sleeping, and no flakiness. The seam that keeps the core pure and pushes all the messy I/O to the edges is described in The Determinism Boundary.

The layer you use: virtual-time tests#

The same determinism is handed to you directly. The BackWave.Testing package gives you a single-node, in-memory harness that pairs a real, deterministic implementation of the storage contract with a clock that only moves when you advance it. A simulated year of retries, recurring schedules, and lease expiries runs in milliseconds, and you assert through the same Monitor API production observability uses.

The point worth internalizing: your handler tests run against the same core that ships. There is no mock of the scheduler, no approximation of retry timing. What you assert in a test is what happens in production, which is why these tests are both fast and trustworthy. Start with Write Your First Test and Test Behavior Over Time.

Deterministic simulation, inspired by TigerBeetle's VOPR#

This is where the deterministic core pays off hardest. Internally, BackWave drives a whole cluster of virtual nodes against an in-memory store through compressed virtual time, and the entire run (every node, every message, every tick of the clock) is determined by a single 64-bit seed.

On top of that it injects faults, the failures a distributed background-job system must survive: nodes crashing and restarting mid-job, clocks skewed so machines disagree about "now", lost heartbeats that let a lease silently lapse, transient store faults, a node cut off from the database while it keeps working, lost acknowledgements, and operator actions (cancel, requeue, pause) racing everything else. After every step, a battery of invariant checks runs and asserts the things that must always be true: no job is ever executed twice, no stale write from a lapsed node ever applies, a queue's concurrency limit is never exceeded, and every job eventually reaches a terminal state. The moment any invariant is violated, the run halts, and because one seed fully determined it, the entire failure replays exactly, fault for fault and interleaving for interleaving.

The approach is borrowed from TigerBeetle, the financial database, and its VOPR, the deterministic simulator they use to test their system. The method travels even though the two systems have nothing else in common. You turn a hard-to-test distributed system into a pure function of a seed, then let a machine search seeds for the interleaving that breaks an invariant. Because the very same core code runs in the simulator and in production, a bug the simulator finds is a bug that would otherwise have shipped.

Two things make this more than a big test suite:

• It is a forever-running search, not a fixed list of scenarios. A pool of workers draws fresh seeds without end, so the space of crash, skew, and loss interleavings is explored continuously rather than sampled by a handful of hand-written cases. When a seed trips an invariant, it is automatically shrunk to the smallest still-failing case and checked in as a permanent regression fixture, so a bug found once can never quietly return.
• The checks themselves are tested. A check that has silently gone dead catches nothing, so each invariant has a deliberately broken twin that must trip it. A run that sabotages a property and still comes back green is treated as its own bug. That is what lets the suite claim its guarantees are alive rather than merely present.

The time math, tested against every input#

The simulation searches interleavings; a second search goes straight at the pure scheduling kernels. Recurring-schedule math is exactly where hand-written examples miss the case no one thought of: cron parsing, computing the next occurrence, catching up after downtime, and the perennial minefield of time zones and daylight-saving transitions. So generators manufacture thousands of schedules, zones, and DST boundaries and assert the properties that must hold for all of them: every occurrence a schedule produces matches its expression, none is skipped or fired twice across a gap, and a spring-forward or fall-back never drops or duplicates a run. When a property fails, the offending input is shrunk automatically to the smallest schedule that still breaks it, so what lands in front of you is a one-line counterexample instead of a haystack.

One contract, every database#

Determinism proves the core, but it cannot prove a real SQL query. A wrong Postgres claim statement or an off-by-one in a migration lives below the boundary the simulation can reach. That surface is covered by an executable storage contract: a single specification, with each clause named after the behavior it pins down, run against every store BackWave ships (in-memory, PostgreSQL, SQL Server, and SQLite), each against a real instance of its database.

The in-memory store is the reference implementation and passes the whole contract; any divergence on a real adapter is therefore a bug in exactly one of the two, and the suite tells you which. This is the mechanism behind the claim that every adapter behaves identically: it is verified, not asserted. The details of the contract and how adapters are held to it are in the Storage overview.

Concurrency torture on real databases#

The contract suite proves each clause in isolation; torture testing proves those clauses hold under genuine contention. Swarms of concurrent clients hammer a live database at once (including real multi-process SQLite pounding a single file on disk) to surface the races that no single-threaded or deterministic test can reach: two workers reaching for the same job, a lease expiring in the middle of a claim, a failover landing mid-transaction. These are the bugs that only appear when real processes and a real database scheduler fight over the same rows.

Upgrades that don't eat your queue#

A background-job library outlives any single deploy, so its schema changes while your database is full of live jobs. BackWave treats upgrading a populated database with work in flight as a first-class operation, not something left to chance, and two mechanisms hold it to that. Every schema change is additive-first: new columns and tables land in this release, destructive changes wait for a later one. A gate in the pull-request battery then blocks any migration that drops, renames, or tightens something an existing node still reads, which is what lets nodes one release behind keep working through a rolling deploy instead of falling over the moment the schema moves. Separately, an upgrade harness restores a populated database captured from a prior release, runs the migration while a live workload enqueues, claims, and reports against it, then audits that every pre-existing job is still accounted for: none lost, none silently moved to the wrong state by the migration itself. Break a migration on purpose and the harness goes red, which is how it stays honest.

Soaked and benchmarked#

Two final layers run on a real clock. Long-running soak flows keep the system under load for extended periods to catch slow leaks and drift that a fast test never runs long enough to see. Throughput benchmarks track jobs per second and latency percentiles so a performance regression is caught before it reaches a release.

Benchmarks measure performance and never correctness, and that separation is deliberate: correctness already lives in the deterministic world above, so a noisy or flaky measurement is never mistaken for a defect. The methodology and what is measured are covered in Performance & Benchmarks.

Do the tests have teeth?#

Every layer above asserts that the code is correct. One more layer asks the harder question: when the code is wrong, does a test actually fail? A suite can have thousands of green tests and still be asleep, green because nothing checks the part that matters. Mutation testing measures exactly that. A tool called Stryker.NET makes thousands of small, deliberate breakages of the shipping code (flip a comparison, drop a statement, swap an && for an ||) and reruns the covering tests against each one. A breakage that makes a test fail is killed; one that leaves every test green is a survivor, a spot where the assertions have a hole.

Pointed at the storage adapters, the executable contract kills essentially every mutation it is handed (for the SQLite adapter, every one of 1,357), because a black-box round trip through enqueue, claim, heartbeat, report, and read exercises nearly every statement and branch. Pointed at the core state machine, the deterministic unit suite kills roughly three-quarters, and each survivor is triaged by hand: equivalent mutations that change nothing observable are documented and accepted, and the genuine gaps become the next round's targeted tests. Unlike the layers above, this one is a periodic audit rather than a build gate: the tool is too slow, and too intolerant of the randomized simulation suite, to run on every pull request. But each pass is a checkup on whether the tests that guard everything else can still bite.

What this means for you#

You inherit all of it for free. The core your handlers run against in a virtual-time test is the exact core the simulation tortures with crashes and skew, and the exact core that ships in your app. So the tests you write are fast and non-flaky for the same reason the simulation is even possible, and the guarantees the library makes to you are the ones under continuous attack. You get to write a small number of ordinary tests and trust that the hard, distributed correctness underneath them has already been earned.

Where to go next#

• Write Your First Test: build the harness and run the define → advance → assert loop.
• Test Behavior Over Time: drive delays, retries, and schedules on a clock you control.
• Integration Testing: the handful of real-database tests that cover what the harness cannot.
• The Determinism Boundary: the architectural seam that makes the deterministic layers possible.

Found a problem on this page? Report an issue