Evolve Job Payloads

Version a job’s payload safely with backward-compatible serialization, and how that differs from renaming its Wire Name.

Version a job’s payload safely with backward-compatible serialization, and how that differs from renaming its Wire Name.

A Job that is enqueued today might not run until tomorrow, or next week, or after the retry that fires three deploys from now. By the time a Worker claims it, your code has moved on. The payload shape you serialized is not necessarily the shape your handler now expects. This guide is about keeping those two in sync without losing in-flight work.

Two things change independently over a Job's life, and confusing them is the source of most payload pain. One is the Wire Name, the explicit string that identifies the Job in storage. The other is the payload shape, the bytes that get serialized when you enqueue and deserialized when a Worker picks the Job up. They evolve under different rules, and they fail in different ways.

How a Job is routed#

When a Worker claims a stored Job, it hands the row to the Job Registry. The row holds two things that matter here: the Wire Name and the payload bytes. The registry does two steps with them.

First it looks up a registration by Wire Name. If nothing is registered for that name, the Job is Quarantined. Second it runs the registration's deserialize delegate over the payload bytes. If decoding throws, the registry catches it at the decode boundary and Quarantines the Job too.

That gives you the taxonomy for the rest of this page. A missing handler and undecodable bytes both land in the same terminal state, Quarantined, and they get there before your handler runs. This is different from Dead-Lettered, which is where a Job goes after it ran and exhausted its attempts. Quarantine is a routing or decoding failure. Dead-letter is an execution failure. A Quarantined Job is loud and visible, not a silent retry loop.

Renaming the type is free; renaming the Wire Name is not#

The Wire Name is declared explicitly on the [Job] attribute. It is never derived from the CLR type name or the method name.

Jobs/OrderCharged.cs
[Job("order-charged")]
public sealed record OrderCharged(Guid OrderId);
 
public sealed class OrderChargedHandler : IJobHandler<OrderCharged>
{
    public Task HandleAsync(OrderCharged job, JobContext context, CancellationToken cancellationToken)
    {
        // ...
        return Task.CompletedTask;
    }
}

Because the identity lives in the string, you can rename OrderCharged to OrderChargedEvent, move it to another namespace, or restructure the class, and storage never notices. Old rows still carry order-charged, the registry still finds the handler, and the Job runs. Refactor the C# freely.

Changing the string is the opposite. The moment you rewrite [Job("order-charged")] to [Job("order-settled")], every in-flight row that still says order-charged has no handler and will Quarantine. The Wire Name is the contract with storage, and storage is full of Jobs you enqueued before you changed your mind.

Wire Name comparisons are ordinal and case-sensitive. order-charged and Order-Charged are two different identities, not one with a typo.

Tolerant decode: what is safe to change#

Under a fixed Wire Name, the generated serializer follows one contract: unknown JSON properties are skipped, and missing ones are defaulted. That single rule covers the additive changes you make most often.

Adding an optional field is safe. Old bytes lack the new property, so the reader leaves it at its missing value.

Jobs/SendEmail.cs
// Before
public sealed record SendEmail(string To);
 
// After: Subject is new and optional
public sealed record SendEmail(string To, string? Subject = null);

When a Worker decodes an old payload that only has To, Subject arrives as null and the Job runs. "Optional" here is not an attribute. It means a nullable type, or a constructor parameter with an explicit default. A nullable reference, a Nullable<T>, or a parameter declared with = something all decode cleanly when the bytes are missing that property. A non-nullable value type with no default falls back to default for its type.

Removing a field is also safe. Old bytes carry a property the new type no longer reads, and the reader skips it. The data in that property is gone, but the Job decodes and runs.

These changes do not require a new Wire Name. Keep the same string, ship the new shape, and old in-flight Jobs continue to route.

What breaks decode and Quarantines the Job#

The decode boundary swallows every exception into a Quarantine. So anything that makes the reader throw turns old in-flight Jobs into terminal, Quarantined rows. A few changes do exactly that, and they are easy to make by accident.

Changing a property's type. If a field was a string and becomes an int, old bytes holding "abc" make the reader throw when it tries to read a number. The Job Quarantines. The generated reader is chosen by the declared CLR type, so any change the reader cannot coerce is a break.

Renaming or removing an enum member. Enums decode strictly. Unlike other kinds, an unrecognized enum token throws instead of defaulting. Old bytes that carry the old member string fail to parse and Quarantine. Adding a new enum member is safe. Renaming or removing an existing one is a breaking payload change.

Removing the handler. If you delete a [Job] declaration while rows for its Wire Name are still in storage, those rows have no handler and Quarantine on the next claim.

A Quarantined Job is terminal. It does not retry itself or recover on the next deploy. Recovering it is an operator action: an operator requeues it once a handler is back. Treat Quarantine as a signal you shipped a breaking change without a plan, not as a state the system clears on its own.

The quiet one: renaming a property#

There is a change that neither decodes cleanly nor Quarantines, which makes it the most dangerous of the set. The JSON keys are the CLR property names, verbatim. The [Job] generator does not read [JsonPropertyName] or any other naming attribute, so there is no way to rename a property while preserving its wire key.

Rename To to Recipient and old bytes still carry "To". The new reader treats To as an unknown property and skips it, then leaves Recipient at its missing value. No exception, no Quarantine. The Job runs with an empty recipient. That is silent data loss. If you must rename a property, treat it like a type change: it is a breaking payload change, not a refactor.

Breaking changes: introduce a new Wire Name#

When a change cannot be made tolerantly, version by Wire Name. You cannot register two payload shapes under one Wire Name. Duplicate Wire Names are a compile error and a runtime registry exception, so there is no in-place upgrade. There is also no payload version field, no schema number, and no migration or upcasting hook. Versioning is a new Wire Name, and nothing else.

The pattern is to run both shapes side by side until the old ones drain.

  1. Leave the old [Job] declaration and its handler registered.
  2. Add a new [Job] with a new Wire Name and the new payload shape, with its own handler.
  3. Switch your enqueue sites to the new type.
  4. Both Wire Names route in parallel. Old in-flight Jobs drain through the old handler.
  5. Once the old queue is empty, delete the old declaration and handler.
Jobs/SendEmail.cs
// Old shape and Wire Name stay registered until in-flight jobs drain.
[Job("send-email")]
public sealed record SendEmail(string To);
 
// New Wire Name carries the breaking shape; new enqueues use this.
// Here the field was renamed from To to Recipient, which old bytes cannot survive.
[Job("send-email-v2")]
public sealed record SendEmailV2(string Recipient, string Subject);

This is the only safe way to make a change that old bytes cannot survive. The cost is carrying two handlers for a while, which is cheap next to Quarantining a backlog.

Guard the wire format with the Job Manifest#

Tolerant decode handles the field-level changes, but it does not stop you from making a breaking one. That is what the Job Manifest is for. It is a committed snapshot, one line per registration in WireName => PayloadType.FullName form, sorted by Wire Name.

You verify it from a test.

JobManifestTests.cs
[Fact]
public void Wire_format_is_unchanged()
{
    var registry = BackWaveJobs.Module.CreateRegistry();
    JobManifest.Verify(registry, "jobs.manifest");
}

On the first run the file does not exist, so Verify writes it and passes, seeding the snapshot. On later runs it compares. If a recorded line is no longer registered, because a Wire Name was removed, renamed, or its payload type changed, Verify throws and names the broken lines. If the change is purely additive, it rewrites the file so the new entry shows up in your diff and passes. The Manifest is a test helper. It does not run at startup, and BackWave ships no runtime gate around it.

The thrown message is the procedure restated: breaking payload changes get a new Wire Name, and you keep the old handler until the queue drains. A wire-format break stops being a production surprise and becomes a red line in a pull request.

A sharp edge on the Manifest#

The Manifest keys on JobType.FullName, which includes the namespace. Renaming the payload type or moving it to another namespace is wire-safe at runtime, because storage only holds the Wire Name and the bytes. The Manifest test, however, will still flag it, since the recorded line no longer matches the new FullName, and the failure message lumps "payload type changed" together with removed and renamed Wire Names.

This is expected. The runtime is fine; the snapshot is stale. Review the diff, confirm it is a pure CLR rename and not a Wire Name or shape change, and accept the rewritten line. The Manifest is conservative on purpose: it would rather make you look at a harmless rename than miss a harmful one.

Note also what the Manifest does not catch. It records the Wire Name and the payload type's full name, not the fields on that type. Adding or removing a field on the same type with the same name does not fail the Manifest test. Field-level safety rests entirely on tolerant decode, not on the snapshot.

A checklist#

  • Renamed or moved the CLR type, same [Job] string: safe at runtime, accept the Manifest rewrite.
  • Added a nullable field or a parameter with a default: safe, same Wire Name.
  • Removed a field: safe, same Wire Name, but the data in it is gone.
  • Changed a field's type, renamed a property, renamed or removed an enum member: breaking, use a new Wire Name.
  • Changed the [Job] string itself: breaking, old rows Quarantine, run both names until drained.

Where to go next#

Previous← Transactional Enqueue with EF CoreNextGuard Wire Compatibility →