Authoring a Storage Adapter

What it takes to put BackWave on a database it doesn't ship an adapter for: the Storage Contract as the specification, capability flags, fault classification, wake-up hints, and schema versioning duties.

Putting BackWave on a database it doesn't already ship for is not a fork or a plugin hack — it is an implementation project against a written specification. That specification is the Storage Contract, the dialect-free set of operations and invariants that every shipped Storage Adapter honors, expressed in the source as the IJobStore seam. A new adapter earns BackWave's correctness claims exactly to the degree that it satisfies the contract's invariants: nothing about choosing an unusual database weakens them, and nothing about choosing a familiar one grants them for free. This page is the narrative companion to that contract. It owns the judgment call — what the seam actually demands, which capabilities are optional and how to declare them honestly, the duties that separate a demo adapter from a production one, and how far you can verify your own adapter today — rather than the per-operation lookup.

The contract is the product's spine#

IJobStore is the single seam between BackWave's engine and a backing store, whether that store is a database, an embedded file, or an in-memory reference. There are four public implementations of that one specification: the In-Memory Store, which ships in the core assembly, and the PostgreSQL, SQL Server, and SQLite adapters. The In-Memory Store is a first-class, publicly shipped implementation used for tests and local development, never for production — not a mock and not a fake, but the reference implementation the others are measured against.

What makes this seam load-bearing is where determinism stops. The Simulator that BackWave uses to shake out concurrency bugs drives the In-Memory Store only, and it stops at the Storage Contract. A wrong query against a real database is not something a deterministic simulation can catch — that is the Conformance Suite's job. The contract is the exact line the deterministic core is drawn along, which is why an adapter is verified against the contract rather than simulated. The Determinism Boundary explains why that line sits where it does.

The consequence for an author is blunt: your adapter inherits BackWave's correctness claims only insofar as it honors the invariants below. There is no partial credit and no "good enough for our workload" — the engine assumes every clause holds, all the time.

The non-negotiables#

The contract's full operation list is a lookup you read elsewhere. What matters when you sit down to implement is the handful of invariant families that carry the correctness story. Get these right and the rest is mechanical CRUD; get one wrong and the failures are the subtle, only-under-load kind.

Atomic claim with mutual exclusion across nodes. When several workers poll at once, each ready job must be handed to at most one of them — no double-claim under concurrency, ever. The contract mandates the semantics, not the machinery, and the shipped adapters prove it by disagreeing on mechanism. The two networked adapters claim with row-level skip-locked reads, so concurrent pollers step over one another's rows. The embedded SQLite adapter has no skip-locked primitive and instead serializes writers whole, so exactly one node holds the writer at a time. Same guarantee, different machinery. If your target database offers a skip-locked idiom you will likely rhyme with the networked adapters; if it doesn't, whole-writer serialization is a proven fallback.

The worker-and-attempt fence on outcome commits. An adapter must record a job's outcome only when the caller still holds the live Lease for exactly that Attempt. If the lease has already lapsed and the work been re-claimed elsewhere, the write must change nothing and report itself as stale rather than overwrite the newer attempt's result. That worker-and-attempt pair is the fence, and it is what makes Effect-Once hold under At-Least-Once Execution: a worker isolated past its lease expiry and then recovered can try to report a result the system has already moved past, and the fence rejects it harmlessly. The batched outcome path fences every row in the batch independently, so one stale row never poisons the rest.

All-or-nothing multi-row operations. Enqueue must reject duplicate keys and bound violations outright and never leave a half-written job behind. Enqueuing a whole workflow is all-or-nothing, with no orphan members and no dangling workflow record if any part fails. Minting due scheduled work skips an entire decision when it races a stale cursor, so a tick can never double-mint. Each of these is a transaction boundary the adapter owns — a partial write is not a degraded mode, it is a contract violation.

Monotonic schema versioning with a fail-stop on mismatch. The adapter's schema carries a version, and BackWave checks it at startup and refuses to run the affected Worker Group on a mismatch. Version skew is thus caught before it can corrupt job state rather than after. The shipped shape of this is covered under the duties below.

Bounds are enforced by rejection, not truncation. Every adapter enforces the same documented store bounds, exposed through the public StoreBounds type. Their defaults:

The behaviors fall into a few kinds, and the split is the through-line to carry into your own adapter. Request-shaping caps — claim batch, monitor page size, purge batch — clamp down to the maximum, so asking for more just gives you the cap. Rolling-history caps — recorded skipped ticks and per-job transitions — age out their oldest entries, since they are bounded windows by design. Data caps reject: an oversized payload, an over-long wire name, or too many parents fails the operation rather than storing something malformed. Exactly one data cap truncates — failure detail, which is write-only diagnostic text and still readable when clipped. And exactly one that might look truncatable instead rejects: Job Output, whose serialized result blob would be undeserializable if clipped mid-encoding, so an over-limit output is rejected with JobOutputTooLargeException rather than silently corrupted. The rule: truncate only where the data is human-read diagnostics; everywhere a value will be read back by the machine, reject.

The capability flags#

Almost everything in the Storage Contract is mandatory. The one thing an adapter may legitimately decline to do is enlist a job's enqueue in the caller's own database transaction, and that single choice is surfaced through the contract's one capability flag in v1: SupportsTransactionalEnqueue, a boolean the adapter answers honestly. There are no other capability flags — everything else the contract describes, every adapter must implement.

The pattern this flag teaches is worth internalizing, because it is how BackWave treats every optional behavior: a capability flag is an honest declaration, not a feature toggle. An adapter that returns false for transactional enqueue must reject any transaction handed to those methods loudly — never quietly ignore it and run the enqueue outside the caller's transaction, which would look like it worked while silently breaking atomicity. The enqueue client, seeing the flag is false, surfaces a clear not-supported error at the call site rather than degrading. The store never pretends.

The shipped sample makes the shape concrete. Its transactional endpoint checks SupportsTransactionalEnqueue before doing anything, returns a 409 with guidance when the configured store can't honor a transaction, and otherwise rides the new job on the caller's Entity Framework transaction so the job and the row it depends on commit together or not at all. The capability follows the deployment: the In-Memory Store reports false, the relational adapters report true. The client asks the store every time, rather than assuming.

One clarification, because it is easy to miscount: an adapter also self-reports its history policy, but that is a description of how it prunes completed jobs, not a capability the client negotiates. Transactional enqueue is the only capability flag.

The duties beyond CRUD#

Fault classification, and failing closed. Beyond storing and claiming, an adapter may teach BackWave how to read its faults, through the IStoreFaultClassifier capability — detected at runtime by whether the store implements it, with no compile-time dependency either way. The rule it feeds is fail-closed: only a fault the adapter can prove is transient, such as a connection reset, a failover blip, or a deadlock victim, is allowed to degrade-and-retry; everything else, including invariant violations and anything unrecognized, fail-stops the Worker Group rather than retrying against a store that may be in an undefined state. A false positive is the dangerous direction, because it turns a fatal error into an endless retry. Most networked databases already raise their transient conditions through the standard DbException.IsTransient flag, so those adapters need not implement the interface at all — it exists to close provider-specific gaps, and among the shipped adapters only SQLite uses it, to mark its residual busy-and-locked contention as transient. Always-On Assertions & Fail-Stop covers the halt mechanism this feeds.

Wake-Up Hints — latency, never correctness. An adapter may also offer a Wake-Up Hint channel through the IWakeUpHintSource capability, again runtime-detected. A hint only ever makes a worker poll sooner. It carries no delivery guarantee — hints may be dropped, duplicated, delayed, or reordered — and no correctness decision anywhere may depend on one; the system must reach the identical final state with every hint discarded. The shipped example is PostgreSQL's LISTEN/NOTIFY, wired so the notification is transactional: it fires when the enqueuing transaction commits, or never. SQL Server ships polling-only, with no hint channel at all, and is no less correct for it — polling is the guarantee everywhere, and a hint is only a way to shave latency off it. Wake-Up Hints & Latency covers the channel in depth.

Schema and migrations. Each shipped adapter follows the same three-part shape, and yours should too: a set of versioned, idempotent migration scripts that are safe to re-run as no-ops; a single migrator entry point that applies them; and a startup version check the store runs on first use and that refuses to proceed on a mismatch, so version skew fail-stops rather than corrupting job state. Auto-migration is opt-in sugar over exactly those scripts — the sample turns it on for its relational stores — not a separate code path. Objects live under a backwave schema or table-prefix by default, configurable per store so BackWave can share a database without colliding. Schema & Migrations walks the shipped shape.

How the shipped adapters divide the design space#

Three shipped adapters, but really two shapes — and an author's target almost always rhymes with one of them.

The Networked Adapters, PostgreSQL and SQL Server, talk to a database server reachable across hosts, so a BackWave cluster can span machines that all claim from the same store. This is the shape you match when your target is a client-server database.

The Embedded Adapter, SQLite, runs the database engine in-process on a single host. It is durable and held to the very same contract as the networked adapters, but it is bounded to one host, in either a co-resident deployment alongside your app or a dedicated one. It is a supported production target, not a development convenience — worth saying plainly, because a single-file engine invites the opposite assumption.

The mechanism split from the invariants above maps straight onto this divide: skip-locked claiming is a networked-adapter move, whole-writer serialization is the embedded one. But the divide is only about which deployment shapes an adapter supports, never which contract clauses it honors — every adapter honors all of them. See PostgreSQL, SQL Server, and SQLite for how each shipped adapter fills in its half of the space.

Verifying your adapter#

The shipped adapters are held to the contract by a Conformance Suite — the executable form of the specification, where every MUST in the contract maps to at least one test, and which the In-Memory Store passes first as the reference every other adapter is measured against. That suite is how BackWave knows its PostgreSQL, SQL Server, and SQLite adapters actually honor what this page describes.

You certify your own adapter with the same suite. It ships as the BackWave.Conformance package: subclass its abstract base in an xunit test project, override one factory that hands back a fresh, empty instance of your store, and your test runner runs the whole suite against it. The tests that depend on an optional behavior your store doesn't claim — transactional enqueue, batch atomicity, crash-mid-write interruption — skip rather than fail, so a minimal store isn't punished for guarantees it never advertised. This is the same tool the shipped adapters clear, giving you the line-by-line MUST coverage that integration tests alone cannot. The Conformance Suite walks the package, the override points, and what each opt-in unlocks.

Conformance proves the store itself; it does not prove the wiring between your application and the store. For that, integration-test your adapter through BackWave's public surface — stand your store up behind the client and exercise real enqueue, claim, outcome, workflow, and recovery paths against it, asserting on observable behavior. The two are complements: conformance certifies the contract, integration tests certify your usage. Integration Testing covers how to structure them and what to assert.

Where to go next#

• The Conformance Suite: the published package you subclass to certify your adapter against the contract.
• The Determinism Boundary: why the contract is the line the deterministic core is drawn along, and what that means for verification.
• Extension Seams: where IStoreFaultClassifier, IWakeUpHintSource, and the other public extension points sit.
• Schema & Migrations and Wake-Up Hints & Latency: the two optional duties in depth.
• Integration Testing: how to exercise your adapter through the public surface today.
• Always-On Assertions & Fail-Stop: the fail-closed halt that fault classification feeds.