The Conformance Suite

The published certification package for the Storage Contract: what it proves, how you subclass it to certify an out-of-tree adapter, and how its capability-aware tests skip the guarantees your store doesn't claim rather than fail them.

BackWave makes one strong claim about storage: every adapter, the embedded one included, honors the exact same semantics, so nothing about your choice of database changes what a job does. That claim has to be backed by something other than trust, and the Conformance Suite is what backs it. It is the executable form of the Storage Contract — the dialect-free specification every store must satisfy — with every MUST in the contract mapped to at least one runnable test. This page is about the suite as a thing you can use: it ships as a package you subclass to certify an adapter BackWave doesn't provide, and it is the tool that turns "my store honors the contract" from an assertion into a result you can watch go green. If you are writing an adapter, read Authoring a Storage Adapter first for what the contract demands; this page is how you prove you met it.

Why simulation can't do this job#

BackWave's headline correctness instrument is the Simulator, which drives a virtual cluster through compressed virtual time under a single seed and audits invariants after every step. It is powerful, but it stops at the Storage Contract boundary: it runs the In-Memory Store only, so it proves the Core and how the Core handles storage faults, and it is structurally blind to a real database. A wrong SELECT ... FOR UPDATE SKIP LOCKED, an off-by-one in a migration, a claim that double-hands a row under genuine contention — none of that is reachable by a deterministic replay of the In-Memory Store. Proving it is a different kind of work, done against a real instance of a real database, and that work is the Conformance Suite's. The two instruments partition the space cleanly: the Simulator owns everything above the seam, the Conformance Suite owns everything below it. The Determinism Boundary explains why the line sits exactly where it does.

The In-Memory Store is the reference#

The suite is anchored by a reference implementation. The In-Memory Store — a first-class, publicly shipped implementation of the contract, not a mock — passes the Conformance Suite completely, and it passes it first, before any production adapter is measured. That is what makes the suite a usable oracle rather than a pile of assertions: because the reference passes every test, any test a real adapter fails is a genuine divergence between that adapter and the contract, and the failing test names the clause that diverged. You are never left wondering whether the test or the store is wrong; the reference already settled that.

Every store BackWave ships — In-Memory, PostgreSQL, SQL Server, and SQLite — runs this same suite. The relational adapters run it against a live instance of their database, including real multi-process SQLite pounding a single file. This is the concrete mechanism behind "the embedded adapter honors all the same clauses": it is verified, not claimed.

Certifying an adapter you wrote#

The suite ships as the BackWave.Conformance package, and its shape is deliberately blunt: it is an abstract base class, ConformanceSuite, carrying north of a hundred executable tests, one required override, and a set of optional ones. You certify a store by subclassing it in an xunit test project and teaching it how to build your store:

public sealed class MyStoreConformanceTests : ConformanceSuite
{
    protected override async ValueTask<IJobStore> CreateStoreAsync(JobHistoryPolicy historyPolicy)
    {
        // Return a fresh, empty store per call — e.g. over a new temp database —
        // honoring the requested history policy.
        var store = new MyJobStore(CreateEmptyDatabase(), historyPolicy);
        await store.MigrateAsync();
        return store;
    }
}

Because every test in the base class is an ordinary xunit fact, your test runner discovers and runs the whole suite against your store the moment you compile, with nothing to register or wire into a manifest. The one contract you owe the factory is a genuinely clean slate on every call: a fresh, empty database, schema, or file, honoring the history policy handed in. Every test assumes it starts from nothing, so a factory that leaks state between calls produces failures that have nothing to do with your store's correctness.

That single override is enough to run the suite. Everything below is about the tests that ask for more.

What the suite skips, and why#

The Storage Contract has a mandatory spine and a few honestly-declared options, and the suite mirrors that exactly. A test that exercises an optional behavior your store does not claim does not fail — it returns early and is a no-op. This is the design that lets one suite grade a minimal store and a fully-featured one without punishing the minimal store for guarantees it never advertised. The mechanism is uniform: an optional test asks your subclass for the hook it needs, and if you haven't provided one, the test skips itself as "not applicable to this store."

What you opt into, and what each opt-in unlocks:

You provideYou unlock
Nothing beyond the store factoryThe full mandatory spine — enqueue, claim, outcome reporting, heartbeats, lease expiry, cancel, schedules and minting, monitor reads, retention purge, the transition log, and history-policy behavior
A caller-owned transaction to enlist inThe Transactional Enqueue tests — proving a job commits or rolls back atomically with the caller's own database work (only relevant if your store reports the capability)
A "batch outcomes are atomic" declarationThe whole-batch atomicity test — proving an over-limit row aborts the entire outcome batch rather than applying the rows before it
A fault-armed second store over the same databaseThe crash-mid-write tests — proving each multi-effect operation lands all-or-nothing when interrupted before commit
An instrumented store that can park an operation mid-flightThe forced-interleaving tests — pinning the exact concurrent orderings (a claim racing a queue's first config write, a duplicate tag or workflow-edge insert) that random parallelism almost never hits

The reason the advanced hooks are opt-in rather than mandatory is that some of them are not simulable on some stores, and pretending otherwise would be dishonest. An in-memory store has no transaction to abort, so crash-mid-write interruption cannot be reproduced on it. A store that serializes every writer on one database-wide lock can never interleave two operations, so the forced-interleaving tests have nothing to force. In both cases the honest result is a skip: a green tick would prove nothing, and a red one would punish a store for lacking a race it structurally cannot have. The shipped adapters differ here in exactly the way you'd expect: the networked adapters wire up the fault and interleaving hooks and run those tests; the embedded adapter, serializing writers whole, legitimately skips the ones its design makes impossible.

What the mandatory spine covers#

Even with nothing but the store factory, the suite exercises every operation in the contract and the invariants that make them trustworthy under concurrency. Enqueue creates exactly one visible job, rejects duplicate ids without touching the original, and enforces every named bound by rejection rather than truncation. Claim leases only due jobs in the candidate queues, in the right order, and — proven with concurrent claimers — never hands the same job to two workers. Outcome reporting is fenced by the worker-and-attempt pair, so a write from a lapsed lease changes nothing and reports itself stale rather than clobbering the newer attempt. Lease expiry, cancel, schedule minting (including the guard against double-minting on a raced cursor), monitor reads, and the retention sweep each get the same treatment. The transition log is checked for appending on every state change with the correct resulting state and attempt, staying bounded per job, and being deleted with the job. Observer delivery, where the store advertises it, is certified for at-most-once cursor advance.

The through-line is that the suite spends most of its weight on the adversarial cases — the concurrent, the interrupted, the raced — because those are exactly the failures a single-threaded read of your SQL will never surface and a deterministic simulation cannot reach below the seam. Passing the happy path is table stakes; the suite's value is everything after it.

What passing does and doesn't mean#

A store that passes the Conformance Suite has demonstrated, against a real instance of its database, that it honors the clauses it claims — the whole mandatory spine plus whichever options it opted into. That is a strong, specific statement, and it is the same bar the shipped adapters clear. It is not a substitute for the two things that live on either side of it: the Simulator still owns Core correctness above the seam, and your own integration tests still own the wiring between your application and the store. Think of conformance as the middle layer that proves the store itself, cleanly separated from the layers that prove the engine and prove your usage. Integration Testing covers the consumer-side tests that complement it, and How BackWave Is Tested places all three layers in one picture.

Where to go next#

Previous← Authoring a Storage AdapterNextOverview →