Cancel a Job

Cancel pending and running jobs, and write handlers that honor the cooperative cancellation token delivered through the heartbeat.

Cancel pending and running jobs, and write handlers that honor the cooperative cancellation token delivered through the heartbeat.

Cancellation in BackWave has two halves. One is an operator action you call, CancelJobAsync, which asks the system to stop a Job. The other is the Handler contract that makes a running Job actually stop, because BackWave never kills threads. This guide covers both, the difference between cancelling a Job that has not started and one that is mid-flight, and the single gotcha that quietly turns a cancel into a success if you get it wrong.

A cancel is not a failure. A cancelled Job goes terminal as Cancelled, so it does not consume an Attempt and is never rescheduled. That is what separates this from the retry path, where a failing Job keeps coming back until its budget runs out.

Two kinds of cancel#

Whether a cancel happens instantly or has to be negotiated depends entirely on whether the Job has started running.

A Job that has not started yet, one that is Scheduled and waiting for its due time, or one that is held back as AwaitingParent on a dependency, is cancelled immediately and synchronously. Nothing is running, so there is nothing to stop. It transitions straight to terminal Cancelled in the same call.

A Job that is already running has been claimed by a Worker under a Lease, and BackWave cannot force-kill it. Threads are never aborted. Instead the cancel request sets a flag, and the Worker that owns the Job learns about that flag on its next heartbeat, the same periodic signal it uses to keep the Lease alive. At that point the Worker trips the CancellationToken it handed to your Handler. Your Handler is expected to notice and stop. So for a running Job, the call is a request, fulfilled cooperatively a moment later, not an instant kill.

There is no separate out-of-band kill channel. The cancel signal rides the heartbeat into the running Job, which is why a well-behaved Handler that honors its token is the other half of the contract.

Issue the cancel#

Cancellation is an operator action, so it lives on BackWaveOperator, not on the BackWaveClient you enqueue with. The client is enqueue-only. The split is deliberate: the client enqueues, the operator performs administrative actions, and the monitor reads. AddBackWave registers BackWaveOperator as a singleton, so you inject it directly.

public class JobsController(BackWaveOperator backwave) : ControllerBase
{
    [HttpPost("jobs/{jobId:guid}/cancel")]
    public async Task<IActionResult> Cancel(Guid jobId)
    {
        var result = await backwave.CancelJobAsync(jobId, actor: User.Identity?.Name ?? "system");
 
        return result switch
        {
            CancelResult.CancelledImmediately  => Ok("Job cancelled."),
            CancelResult.CancellationRequested => Accepted(value: "Cancellation requested; the job will stop shortly."),
            CancelResult.NotCancellable        => NotFound("Job is unknown or already finished."),
            _ => StatusCode(500),
        };
    }
}

CancelJobAsync takes four arguments:

• jobId is the id you got back from EnqueueAsync.
• actor is the acting operator's identity. It is recorded in the append-only audit log and, for a successful cancel, becomes the Job's terminal cause. Pass something meaningful, a user name or a system identifier, not a placeholder.
• now defaults to the operator's clock. Leave it unless you are stamping the action with a specific instant.
• cancellationToken cancels the request itself, the I/O of issuing the cancel, not the Job you are trying to cancel. This is easy to misread, so do not pass the Job's own token here expecting it to do something.

The call never throws for a missing or already-finished Job. It tells you what happened through its return value instead.

Read the result#

CancelJobAsync returns a CancelResult that tells you exactly which of the two paths you hit, or that nothing happened.

NotCancellable deliberately covers both "no such job" and "already finished" without distinguishing them, so do not try to tell those two cases apart from the result alone. If you need to know which, query the Job first.

Treat CancellationRequested as accepted-but-pending in any UI you build. The Job is still running at that instant. Confirming it actually stopped is a separate read, covered below.

Honor the token in your Handler#

This is the half of the contract your code owns. A running Job stops only as fast as its Handler checks the cancellation token. BackWave delivers the signal, but the Handler has to act on it.

The token is the third parameter of HandleAsync.

That token fires when the Attempt is being torn down, whether from an operator cancel, a lost Lease, or host shutdown. A Handler that honors it looks like this: flow the token into every async call, and check it between units of work.

public class GenerateReportHandler(IReportStore store) : IJobHandler<GenerateReport>
{
    public async Task HandleAsync(GenerateReport job, JobContext context, CancellationToken cancellationToken)
    {
        foreach (var section in job.Sections)
        {
            cancellationToken.ThrowIfCancellationRequested();           // check between units of work
            var rendered = await RenderAsync(section, cancellationToken); // flow it into every async call
            await store.SaveAsync(rendered, cancellationToken);
        }
    }
}

What well-behaved means, concretely:

• Flow the token into every async and IO call. A blocking wait that received the token unblocks promptly when it fires; one that did not will run to its own completion.
• Check ThrowIfCancellationRequested() between units of work. A tight CPU loop the token cannot interrupt on its own needs an explicit check, otherwise it runs to the end no matter what the operator did.
• Let the OperationCanceledException propagate. Do not catch and swallow it. This is the gotcha called out below.
• Stay idempotent. Under at-least-once execution a Handler may run more than once anyway, and a cancelled, partially-completed Attempt is just one more case the work has to tolerate. Guard irreversible steps the same way you would for a retry.

The gotcha: swallowing the cancellation#

If your Handler catches the OperationCanceledException and returns normally, BackWave sees a Handler that completed without throwing, which means success. The Job goes to Succeeded, not Cancelled, and the cancel is silently lost. Nothing logs an error, because from the Worker's point of view nothing went wrong.

The fix is to let the exception propagate. ThrowIfCancellationRequested() and a token flowed into your async calls do this for free. If you have a broad catch around your work, make sure it re-throws OperationCanceledException rather than absorbing it.

The mirror image is a Handler that ignores the token entirely. A cancel never takes effect until the Handler next checks, so a long loop that never looks at the token simply runs to completion and the Job succeeds. Cancellation is cooperative, full stop.

Confirm it stopped#

Because cancelling a running Job is asynchronous, you may want to observe the outcome rather than assume it. Read it through BackWaveMonitor.GetJobAsync, which returns a JobSnapshot.

var snap = await monitor.GetJobAsync(jobId);
 
// running but flagged:  snap.State == JobState.Leased   && snap.CancelRequested == true
// fully cancelled:      snap.State == JobState.Cancelled && snap.TerminalCause == "<actor>"

Between the cancel request and the Handler actually yielding, the Job reads as still running with the cancellation flagged: State is Leased and CancelRequested is true. Once the Handler honors the token and the Attempt ends, the Job reads as terminal Cancelled, and TerminalCause holds the actor string you passed. TerminalAt records when. For the full timeline of transitions, GetJobHistoryAsync returns the ordered list.

If you want to check whether a Job is even cancellable before calling, JobStates.IsTerminal returns true for the four terminal states (Succeeded, Cancelled, DeadLettered, Quarantined). A Job in any of those returns NotCancellable.

How fast does a running cancel land#

A running cancel is not instantaneous. Its latency has two parts.

The first is the heartbeat interval. The cancel flag is only observed by the Worker on its next heartbeat, so the worst-case wait for the signal to reach the running Job is one heartbeat cycle. That cadence is governed by HeartbeatInterval on the Worker Group, which defaults to a third of the Lease duration. The second part is how often your Handler checks its token. Even after the token fires, the Handler keeps running until its next ThrowIfCancellationRequested() or its next token-aware async call returns.

Add those together and you have the practical bound. There is no numeric limit on cancellation itself, only this timing. If you need running Jobs to stop quickly, check the token at a finer grain rather than expecting the platform to interrupt mid-statement.

Cancel is not failure, shutdown, or self-cancellation#

Three different events can trip a Handler's token, and only one of them produces terminal Cancelled. Telling them apart is what lets you predict the outcome.

Host shutdown is not a cancel. When the process stops, the Job's token fires so the Handler can wind down, but BackWave does not mark the Job cancelled. The Lease simply lapses and another node inherits the work, which counts as an Attempt exactly like a lapsed Lease does. This is why swallowing the token is wrong even on shutdown: you want the Handler to stop, not to falsely report success.

A cancellation your own code raised, an HttpClient timeout or an internal deadline that nobody requested, is treated as a plain failure. It retries on the normal backoff and never becomes Cancelled. Only an operator cancel produces the terminal state.

Cancel races with completion#

If a Job finishes on its own at almost the same instant you cancel it, there is no double transition and no corruption. BackWave records the outcome of an Attempt exactly once. If the Handler completes successfully before the cancel flag is observed, the success wins and the cancel becomes a no-op. If the flag is observed first, the token trips and the Job ends Cancelled. One outcome is recorded, never both, even when a cancel and a completion land in the same window.

The same fencing means a stale or lost node cannot apply an outcome to a Job it no longer owns. You do not have to guard against the race yourself; calling cancel during a completion is safe and simply resolves to whichever happened first.

Cancel and dependencies#

An operator cancel is one of two ways a Job becomes Cancelled. The other is a dependency cascade: when an on-success parent fails, its children are cancelled rather than run, because the precondition they were waiting on can never be met. Both causes land in the same Cancelled state, distinguished by the recorded terminal cause.

A Job that has already been cancelled stays cancelled. A parent succeeding later never un-cancels a child, and no automatic path revives a cancelled Job. Reanimating one is an explicit, Pro-only operation covered in the workflow guide.

Audit trail#

Every cancel writes one audit record atomically with the effect, so the action and its result can never disagree. Read the history for a Job through ListAuditRecordsAsync, passing the Job id as the target.

var records = await backwave.ListAuditRecordsAsync(target: jobId.ToString());
// each OperatorAuditRecord carries Actor, Action, Target, and RecordedAt

A cancel shows up with an Action of OperatorAction.Cancel and the Actor you supplied. This is the same audit surface shared by the other operator actions, requeue, pause, resume, and trigger, so the who-did-what story is uniform across all of them. See operator actions for the full set, and note the dashboard exposes a built-in Cancel button gated behind the Cancel permission.

Cancelling a whole workflow#

When you are running workflows with the Pro package, you can cancel every member of a workflow at once instead of cancelling Jobs one by one. The CancelWorkflowAsync extension on BackWaveOperator cancels each member that is still running at the moment of the call. It is a one-time snapshot, not a standing rule, so a member that starts after the call returns is unaffected. Because an operator cancel produces no failed members, the workflow's derived status reads Cancelled rather than Failed. The build-a-workflow guide covers the fan-out semantics and how to reanimate a cancelled workflow.

Where to go next#

• Job lifecycle: every state a Job moves through, with Cancelled as one of the four terminal ones.
• Retries & error handling: why a cancel costs no Attempt, and how that contrasts with dead-lettering.
• Execution guarantee: the at-least-once model behind why your Handler must stay idempotent.
• Dependencies: the cascade that cancels children when an on-success parent fails.
• Operator actions: cancel's sibling actions and the shared audit log.
• Build a workflow: cancelling and reanimating an entire workflow with the Pro package.