Deploy with Zero Downtime
Roll out a new version with no maintenance window: migrate the schema first, roll nodes under N-1 version skew, evolve payloads additively, and shed the old nodes without losing in-flight work.
Roll out a new version with no maintenance window: migrate the schema first, roll nodes under N-1 version skew, evolve payloads additively, and shed the old nodes without losing in-flight work.
BackWave is built to be upgraded in place. A Mixed-Version Fleet — old nodes and new nodes claiming from the same store — is a supported state, not an accident to race through. This guide is the deploy-day playbook: the order to do things in, what is guaranteed to keep working while versions are skewed, and the three compatibility rules — schema, payload, wire name — that make the rolling window safe. It ties together three pages that each hold one piece, so the value here is the sequencing, not new facts.
If you have not read Execution Guarantee yet, skim it first. It defines the guarantees this playbook leans on: the Lease, At-Least-Once Execution, and what happens to a Job when the node running it disappears mid-Attempt. Those are the same guarantees that make shedding an old node during a roll safe.
Why in-place upgrade is a supported state#
Every node in a BackWave cluster is a stateless peer. There is no leader, no primary, and no coordination protocol between nodes: they are database-authoritative and coordinate only through the shared store. That means adding a new-version node and retiring an old-version one is a membership change, not a failover — there is nothing to promote and no split-brain to avoid.
An In-Place Upgrade is the supported upgrade path for every Storage Adapter: you upgrade the schema on a live database while jobs remain in flight, with no drain and no maintenance window. The transient state this produces is the Mixed-Version Fleet — a cluster running two adjacent BackWave versions at once, which is the normal condition during a rolling deploy.
The skew guarantee is precise. A Mixed-Version Fleet is supported at N-1 skew only: nodes one release behind must keep operating correctly against the upgraded schema. Skew wider than one release is out of contract — do not run nodes two or more releases behind against a new schema and expect it to hold. N-1 is what a rolling deploy actually needs, and holding the compatibility surface to exactly one release is what keeps it testable.
What makes N-1 coexistence safe is a schema discipline: released schema changes are additive. New columns and tables land in release N; destructive changes — drops, renames, and constraint tightening that would break an N-1 reader or writer — are deferred to release N+1 at the earliest. This rule is verified in CI before a release ships, so an old node and a new schema can share a database without the old node hitting a column that vanished under it.
The backstop covers the case where the order goes wrong anyway. On a schema-version mismatch the worker fail-stops before touching data: it refuses to start rather than run against a schema it does not understand. The failure mode of a mis-ordered deploy is therefore a stopped Worker Group, never corrupted job state — a loud, recoverable condition instead of a silent one. See Health & Fail-Stop for how that surfaces operationally and Fail-Stop for the model behind it.
Step 1 — migrate the schema first#
Migrate before you roll any node. Because schema changes are additive, the old binary keeps working against the new schema, so applying the migration first is always safe; rolling a new node against an un-migrated schema is not. Run the migrator out-of-band from your deploy pipeline — as its own step, before the first new node comes up.
Each adapter ships a public migrator whose MigrateAsync applies every schema script in version order. It is idempotent: safe to run on every deploy and harmless against an already-current database, so you do not need to detect whether a migration is pending.
| Adapter | Migrator | MigrateAsync takes |
|---|---|---|
| Postgres | PostgresMigrator | an NpgsqlDataSource (a schema-name overload is available) |
| SQL Server | SqlServerMigrator | a connection string |
| SQLite | SqliteMigrator | a connection string (a table-prefix overload is available) |
Pair the migrate step with a pipeline gate. VerifySchemaVersionAsync reads the deployed schema version and throws if the schema is missing or its version differs from the one this adapter build requires. Call it right after migrating to fail the deploy fast — before any node starts claiming work — if the migration did not take.
Leave AutoMigrate off in production. It defaults to off, and it exists for development loops where letting the store self-apply its scripts on startup is a convenience. Production pipelines apply the same scripts themselves as a deliberate, observable step. Either way, the store verifies the schema version on startup and refuses to run on a mismatch, so the fail-stop backstop from the previous section holds regardless of the flag.
The mechanics of the scripts, the version table, and the per-adapter options live in Schema & Migrations. This step only fixes the ordering: migrate and verify out-of-band, then roll.
Step 2 — make the code change wire-safe before it ships#
Two compatibility rules govern the code change itself, and both are cheapest to enforce in the same CI pipeline that builds the release.
The first rule is additive payloads. Change a payload only by adding fields that have sensible defaults. The payload serializer tolerates both unknown and missing members, which is exactly what makes both directions of skew safe: a field a new node adds is ignored when an old node deserializes the payload, and a field an old payload lacks deserializes to its type default when a new node reads it. Both directions succeed as long as the change stays additive. The failure this rule forecloses is the non-additive change — a payload that no longer decodes lands in Quarantined, a loud terminal state, rather than retrying silently. Evolve Job Payloads covers the additive rules in full.
The second rule is Wire Name stability. A Wire Name is a job type's declared identity in storage, stable across refactors by construction — renaming the class, moving the file, or changing the namespace never changes it. Removing or renaming a Wire Name is different: it strands any in-flight job recorded under the old name, because the Pump that claims it can no longer find a registered handler and Quarantines it. The Job Manifest turns that mistake into a build failure instead of a production incident. It is a committed snapshot of every registered Wire Name, verified by the shipped JobManifest.Verify test helper: the check rewrites additively so a genuinely new Wire Name shows up in the PR diff, and it throws — a red build — when a recorded Wire Name is removed, renamed, or has its payload type changed. The remedy it prescribes is the one to follow: a breaking payload change gets a new Wire Name, and you keep the old handler registered until the jobs under the old name have drained. Guard Wire Compatibility covers wiring the check into your suite.
Step 3 — roll the nodes#
With the schema migrated and the code proven wire-safe, the roll itself is an ordinary rolling replace: bring up new-version nodes and retire old-version ones, batch by batch, checking health between batches (Step 4).
During the window both versions claim from the same Queues. That is the Mixed-Version Fleet doing its job, and it is the normal, supported state — not something to rush. In the dashboard's executing-now view you will see Attempts running under both versions at once; mixed workers there are expected and are not a symptom of anything wrong.
The one thing worth understanding precisely is what happens to an old node's in-flight work when you retire it. On a graceful shutdown the node stops taking new claims and signals its in-flight Attempts to cancel cooperatively: each Attempt's CancellationToken fires, and a well-behaved handler observes it and stops promptly. Cancellation is cooperative — threads are never killed — so the node does not wait for those Attempts to run to completion. Their Leases then lapse, and peer nodes reclaim the work. A hard kill skips the cooperative signal but lands in the same place: the Lease lapses and a peer inherits the Attempt. Either way, At-Least-Once Execution absorbs the interruption. The roll is safe because a shed node's work is reclaimed by its peers, not because the node drains to completion — which is why idempotent handlers matter here as everywhere: a reclaimed Attempt runs again. Health & Fail-Stop covers the shutdown path, and Leases & Crash Recovery covers the reclaim timeline.
Step 4 — verify and roll back#
Gate each batch on a real signal rather than a stopwatch. Register BackWaveHealthCheck in the standard ASP.NET Core health-check pipeline — it is backed by BackWaveHealth, which exposes IsHealthy — and hold the roll if it goes unhealthy. The check reads unhealthy once a Worker Group has fail-stopped, which is exactly the condition a mis-ordered deploy produces. Alongside it, watch that Queue depths are draining and that the count of Quarantined jobs is flat.
That last signal is the one to learn. A spike in Quarantined jobs immediately after a deploy is the signature of a wire-name break: either a Wire Name with no registered handler (an in-flight job whose handler was removed or renamed) or a payload that no longer decodes. Read it as a wire-format regression and roll back, not as a transient the system will retry away. Keep it distinct from Dead-Lettered, which is attempt-exhaustion — a job that failed its way through its Attempt ceiling.
Rollback within the N-1 window is just another roll, with one asymmetry to keep straight: the schema does not roll back. Schema migrations are forward-only and additive, so "rolling back" the release means running the old binary against the new schema. That is precisely the N-1 case the additive discipline already makes safe — the old readers and writers were required to work against the new schema in the first place. The schema staying put is what makes the rollback safe, not a hole in it. Do not try to reverse a migration to recover from a bad binary.
The playbook at a glance#
| Step | Action | Guards against |
|---|---|---|
| Migrate first | Run MigrateAsync out-of-band (idempotent), gate on VerifySchemaVersionAsync, keep AutoMigrate off | An old or new binary hitting a schema it cannot read |
| Prove it wire-safe | JobManifest.Verify in CI; additive payload fields with defaults only | A rename or removal stranding in-flight jobs in Quarantine |
| Roll the nodes | Rolling replace, batch by batch, BackWaveHealthCheck green between batches | An unnoticed fail-stopped Worker Group |
| Shed old nodes | Graceful stop: no new claims, in-flight Attempts cancelled cooperatively, peers reclaim via lease expiry | Lost work — At-Least-Once reclaims it |
| Verify and roll back | Health green, queues draining, no Quarantine growth; rollback is another roll, not a schema reversal | A silent wire-name break, or the schema being blamed for a bad binary |
Where to go next#
- Schema & Migrations: the migrator, the version table, and the per-adapter options behind Step 1.
- Evolve Job Payloads: the additive rules that keep payload reads safe across a skewed fleet.
- Guard Wire Compatibility: wiring the Job Manifest check into your test suite.
- Health & Fail-Stop: the health surface you gate the roll on, and the shutdown path.
- Leases & Crash Recovery: the timeline by which a shed node's work is reclaimed.
- Fail-Stop: why a schema-version mismatch stops the worker before it touches data.