Build a Workflow

Author a typed multi-job DAG with the fluent builder: linear chains, parallel fan-out/fan-in, typed output, conditional gates, saga compensation, atomic enqueue, append, and cancel.

Author a typed multi-job DAG, enqueue the whole graph atomically, append to a running workflow, and cancel it as a unit.

A Workflow is the grouping and identity layer over a set of Jobs wired together by Dependencies. It gives a loose pile of dependent jobs a strongly-typed builder, a stable id, an optional name, a graph you can read back, and lifecycle operations over the whole group. This guide walks the full authoring path: building the DAG with the fluent builder, fanning out and back in, reading an upstream step's typed output, guarding a step with a saga compensation, enqueuing in one atomic write, appending live work, cancelling the group, and recovering by re-running. Everything here lives in the BackWave Pro package and hangs off the same BackWaveClient, BackWaveOperator, and BackWaveMonitor you already use for ordinary jobs.

If you have not read Workflows yet, skim it first. It is the conceptual twin of this how-to and defines the vocabulary leaned on here: Workflow, Dependency, derived status, the composition shapes, and the line between a static DAG and durable execution.

What a Workflow is, and is not#

The model here is deliberately narrow.

A Workflow is a static DAG. Every edge is fixed at each member's enqueue time, and the graph never rewrites an existing job's dependencies. It grows in exactly one way, by appending new jobs, and it shrinks never. A dependent may read an ancestor's typed output, and a conditional gate may use that output to decide which pre-declared arm runs, but output never adds, skips, or reorders nodes.

A Workflow is not durable execution. There are no durable waits, signals, or timers, and no replay of steps inside a job. Conditional branching exists as a build-time gate step, not as a durable pause. If you have used Temporal, unlearn that mental model here. The right comparison is the dependency model of River or Hangfire with a typed builder, first-class saga compensation, and conditional gates bolted on top.

Two rules hold for every member and the store enforces both at enqueue:

• A job belongs to at most one Workflow.
• Every gating parent of a member must be a member of the same Workflow. A member cannot depend on a job in another workflow or on a free-standing job.

Status is derived, never stored#

A Workflow's status is always a projection of its member-job states, computed on read. The stored workflow row holds only identity and configuration. Nothing writes an authoritative "the workflow is Running" flag, which is why appending live work to a drained workflow legitimately reopens it.

The status resolves by first-match-wins precedence over the members:

Read that order top to bottom. A workflow with one Dead-Lettered member and ten succeeded members reads Failed, not partially succeeded. Never cache the status as authoritative.

One consequence surprises everyone the first time. If your graph contains a conditional (.If), the gate cancels the arm it did not take, so a member ends Cancelled on every healthy run. That makes the whole-workflow rollup read Cancelled even when every step that ran Succeeded. This is expected and healthy for a conditional workflow. For those graphs, read per-step state from the member list, not the rollup, to tell a healthy conditional run from a genuinely aborted one.

Enable Pro#

Workflows light up by referencing the Pro package and calling AddBackWavePro once at startup, after AddBackWave.

builder.Services
    .AddBackWave(bw => bw
        .UseStore(new PostgresJobStore(connectionString))
        .UseJobs(BackWaveJobs.Module)
        .AddWorkerGroup(new WorkerGroupOptions { Name = "default" }))
    .AddBackWavePro(builder.Configuration["BackWave:ProLicense"]);

The license argument is optional and defaults to null, which is correct for organizations under the revenue threshold on the honor system. A missing, malformed, or out-of-term license soft-fails: you get one startup log warning and a banner on the dashboard, and every feature behaves identically. Referencing the package is the entire boundary. The license never gates whether workflows run.

To get the graph UI, add the optional dashboard package on top.

builder.Services.AddBackWaveProDashboard();

That lights up the Workflows surface on the dashboard: the workflow list, the graph view, and the cancel-workflow action. It requires AddBackWavePro to be wired first.

Mark your jobs as workflow steps#

A workflow step is an ordinary [Job] payload record that additionally wears the IWorkflowStep marker. The marker is the compile-time gate: only a marked type can be added to a workflow, so a mistyped or renamed step is a build error rather than a runtime one. A step that produces a value for downstream steps implements IWorkflowStep<TOut> instead and declares its output type once.

[Job("validate-order")]
public sealed record ValidateOrder(string OrderId) : IWorkflowStep;
 
[Job("price-order")]
public sealed record PriceOrder(string OrderId) : IWorkflowStep<OrderPrice>;   // produces an OrderPrice
 
[Job("reserve-inventory")]
public sealed record ReserveInventory(string OrderId) : IWorkflowStep;

List any output DTO (here OrderPrice) in a JsonSerializerContext; BackWave auto-wires its codec, and a type it cannot find is a BW0007 build error with a one-click fix. You never hand-pass a JsonTypeInfo at a workflow call site.

Author the DAG#

You build a graph through the fluent TypedWorkflowBuilder, started from the client. client.Workflow(name) opens a fresh workflow; the optional name is a human-readable label that also shows up in the graph view. You chain steps by their .NET type: a linear .Then(step) depends on the current frontier and becomes the new frontier, and a step that fans in over several upstreams names them by type with after: [typeof(StepA), typeof(StepB)].

// inject BackWaveClient client
var workflowId = await client.Workflow("nightly-etl")
    .Then(new Extract(date))
    .Then(new TransformA(), after: [typeof(Extract)])   // fan-out from Extract
    .Then(new TransformB(), after: [typeof(Extract)])   // also from Extract
    .Then(new Load(), after: [typeof(TransformA), typeof(TransformB)]) // fan-in
    .Then(new Notify(), mode: DependencyMode.OnAnyTerminal)            // runs regardless
    .EnqueueAsync();

There is no separate build-then-enqueue handshake: .EnqueueAsync() validates the graph and writes it, returning the workflow's Guid. Root steps become due immediately; downstream steps land awaiting their parents and release as those parents reach terminal states, according to each edge's mode.

Each .Then also accepts queue and tags, which behave exactly as their single-job enqueue counterparts: queue overrides the Queue the step type registered with, and tags are unioned on top of the type's default Tags, additive only.

When an edge releases its child#

The mode argument decides what counts as "the parent finished". It is a DependencyMode with two values.

• OnSuccess is the default. The child releases only if every parent Succeeded. Any other terminal outcome on a parent (Dead-Lettered, Quarantined, Cancelled) cancels the child instead of running it.
• OnAnyTerminal releases the child once every parent is terminal, whatever the outcome.

Use OnSuccess for a normal pipeline step that needs its inputs to have worked. Use OnAnyTerminal for cleanup or notification that should run regardless, as the Notify step above does, and to converge past a conditional gate. This is the same on-success versus on-any-terminal choice documented for the base Dependency primitive; a Workflow is the grouping layer over that primitive.

Fan out and fan back in with Parallel#

The after: fan-in above is explicit about its upstreams. When you want to fan out a set of branches from one point and rejoin them, Parallel(...) is the ergonomic form. It fans out one branch per argument from the current frontier and makes the set of all branch tips the new frontier, so the next plain .Then fans in over every tip with no synthetic join node. A branch is either a single step (WorkflowBranch.Step) or a chained sub-builder (WorkflowBranch.Do).

await client.Workflow("checkout")
    .Then(new PriceOrder(orderId))
    .Parallel(
        WorkflowBranch.Step(new ReserveStock(orderId)),                      // single-step branch
        WorkflowBranch.Do(b => b.Then(new NotifyWarehouse(orderId))          // multi-step branch
                                .Then(new ConfirmPick(orderId))))
    .Then(new AuthorizeCharge(orderId))   // fans in over BOTH branch tips (reserve-stock + confirm-pick)
    .EnqueueAsync();

At least one branch is required and every branch must add at least one step, or Build() throws InvalidWorkflowException.

Read an upstream step's typed output#

A step that implements IWorkflowStep<TOut> emits its value with ctx.SetOutput<TStep, TOut>(value), and a downstream handler pulls it with ctx.Output<TStep, TOut>(), type-checked against the producer's declaration. Data flows down the graph by pull, never push.

public Task HandleAsync(PriceOrder job, JobContext context, CancellationToken cancellationToken)
{
    var price = /* compute */;
    context.SetOutput<PriceOrder, OrderPrice>(new OrderPrice(price));
    return Task.CompletedTask;
}

public async Task HandleAsync(AuthorizeCharge job, JobContext context, CancellationToken cancellationToken)
{
    var priced = await context.Output<PriceOrder, OrderPrice>(cancellationToken);
    if (priced.HasOutput)
    {
        var amount = priced.Output!.Cents;
        // charge that amount
    }
}

Absence is normal, never a throw: an ancestor that failed, was cancelled, emitted nothing, or a non-ancestor sibling on a parallel branch all resolve to a clean HasOutput == false. Always branch on HasOutput before touching Output. A read is ambiguous if the same step type appears more than once among the reader's ancestors, so structure the graph so at most one ancestor of a reader is that type.

Guard a step with saga compensation#

.WithCompensation(...) attaches a first-class undo guarding the current frontier. The compensation runs once the protected step is terminal, whatever the outcome, and its handler reads the protected step's decided state to choose whether to act. So the compensation always runs but usually no-ops.

.Then(new AuthorizeCharge(orderId))
.WithCompensation(new RefundCharge(orderId))   // no-ops on success, refunds on failure

public async Task HandleAsync(RefundCharge job, JobContext context, CancellationToken cancellationToken)
{
    var charge = await context.Output<AuthorizeCharge, ChargeResult>(cancellationToken);
    if (charge.AncestorState == JobState.Succeeded) return;   // settled, nothing to undo
    // else reverse charge.Output?.ChargeId
}

The frontier stays on the protected work, so whatever you chain after WithCompensation still depends on AuthorizeCharge, not on the refund. Calling WithCompensation again for earlier steps builds a reverse-order saga: later-protected work undoes first.

Branch at run time with a conditional gate#

.If<TGate, TStep, TOut>(then:, otherwise:) enqueues both arms up front and inserts a gate step that, at run time, reads an upstream step's output, evaluates a named IWorkflowGate type, and cancels the arm it does not take. Register each gate once with AddWorkflowGate, and converge past it with mode: DependencyMode.OnAnyTerminal, because the not-taken arm ends cancelled.

.If<LargeOrderGate, PriceOrder, OrderPrice>(
    then: b => b.Then(new ExpressShip(orderId)),
    otherwise: b => b.Then(new StandardShip(orderId)))
.Then(new PrepareHandoff(orderId), mode: DependencyMode.OnAnyTerminal)   // join past the gate

Remember the status footgun: a healthy run of this workflow derives Cancelled, because one of the two shipping arms is always cancelled. The full conditional surface, including the seed-aware gate variant, is in the Workflows API reference.

Validation happens at enqueue#

The builder validates the graph before anything touches the store, and any failure throws InvalidWorkflowException:

• The graph is non-empty and acyclic.
• Every after: type resolves to exactly one already-added step. A repeated type is ambiguous; disambiguate it with a name: and a WorkflowStepRef.
• No two steps share an identity (type plus optional name).
• Conditional guardrails hold: an OnSuccess join may not span both arms of an .If, and an arm step may depend only on the gate or another step in the same arm.

These are programming errors, caught at build time on your own machine, never in production. Build() returns the prepared WorkflowDefinition if you want it without enqueuing; EnqueueAsync() builds and writes in one call.

Enqueue the whole graph atomically#

.EnqueueAsync() writes every member job and the workflow record in a single all-or-nothing transaction. Either the entire graph lands or nothing does, and it returns the workflow's Guid.

A failed enqueue surfaces as an exception rather than a silent partial write. The store can reject a graph for a duplicate workflow id, a member id that collides, an append whose target does not exist, a containment violation (a member gating on a job outside its workflow), an empty graph, an oversized payload, an over-length wire name, or a member with too many parents. The reason name is interpolated into an InvalidOperationException message.

Limits to design around#

A few numeric bounds shape how wide and how heavy a graph can be. These are the store defaults; an adapter may raise them.

A fan-in node that needs more than 16 inputs is usually a sign to introduce an intermediate aggregation step rather than widening the node. For payload size, the same advice as ordinary jobs applies: carry a reference (an id or a blob key) and have the handler fetch the bytes.

Capture the workflow id#

The builder assigns the workflow's Guid up front, before anything is enqueued, and exposes it as WorkflowId. Capture it when you need to record a correlation between your own domain row and the workflow before it even commits.

var builder = client.Workflow("checkout").Then(new ChargeCard(order.Id));
var workflowId = builder.WorkflowId;   // known before enqueue
 
order.WorkflowId = workflowId;
await builder.EnqueueAsync();

Individual member ids are assigned automatically and are not addressed by name from the builder. When you need a specific member's id later, for example to point an append at it, read it back from the monitor by its Wire Name (see append below).

Enqueue atomically with your own writes#

The workflow path supports Transactional Enqueue: commit the entire graph in the same database transaction as your own data, so there is no outbox and no window where your row exists but the jobs do not. Extract the DbTransaction and pass it to EnqueueAsync.

await using var tx = await context.Database.BeginTransactionAsync();
context.Orders.Add(order);
await context.SaveChangesAsync();
 
await client.Workflow("checkout")
    .Then(new ChargeCard(order.Id))
    .Then(new SendReceipt(order.Id))
    .EnqueueAsync(transaction: context.Database.CurrentTransaction!.GetDbTransaction());
 
await tx.CommitAsync(); // the order and the whole workflow commit together

GetDbTransaction() comes from Microsoft.EntityFrameworkCore.Storage. This requires a store that supports transactional enqueue, which covers networked Postgres and SQL Server, and co-resident SQLite. Passing a transaction to an adapter that cannot do it throws NotSupportedException. The transactional enqueue guide covers the broader pattern and the raw DbTransaction form.

Append live work to a running workflow#

A Workflow grows by appending. client.WorkflowAppend(workflowId) returns a builder whose added steps become new members of an existing workflow. The existing members and their edges are never touched.

New steps express their dependencies two ways, and the distinction matters:

• after: [typeof(NewStep)] points at other new steps in this same append batch, by type.
• afterExisting: [guid] points at existing members enqueued in a prior batch, by their Guid. This is the only way to point at work already in the store.

Resolve an existing member's id from the monitor by its Wire Name, then feed it to afterExisting.

var view = await monitor.GetWorkflowAsync(workflowId);
var loadId = view!.Members.Single(m => m.WireName == "load").JobId;
 
await client.WorkflowAppend(workflowId)
    .Then(new Reconcile(), afterExisting: [loadId])
    .EnqueueAsync();

The builder validates after: types against the current batch, but it does not validate afterExisting ids against the live store. An id that is not actually a member of the target workflow passes Build() cleanly and surfaces only at enqueue time as a containment-violation InvalidOperationException. Double-check the ids you feed into afterExisting.

Appending live work to a workflow that had already drained reopens its derived status from Succeeded back to Running, which is exactly the behavior the projection model is built to allow. One caveat: appended members carry no Workflow Input, so a ctx.Input<T>() in an appended step throws at runtime. Give appended steps their constant data through their own payload.

Cancel the workflow as a unit#

CancelWorkflowAsync cancels every member that is non-terminal at the moment of the call. It is the whole-graph counterpart of single-job cancel and uses the same cooperative-token mechanism underneath.

// inject BackWaveOperator op
var result = await op.CancelWorkflowAsync(workflowId, actor: "ops@acme.com");
// result.Found, result.CancelledImmediately, result.CancellationRequested

The actor is recorded in the audit log against each member cancelled. The result reports how the cancel landed:

• Found is false, via WorkflowCancelResult.NotFound, when the id matches no workflow.
• CancelledImmediately counts members that had not started yet and went straight to terminal Cancelled.
• CancellationRequested counts members that were leased and running and were asked to stop cooperatively. They stop only when their handler next observes its CancellationToken, delivered on the next heartbeat. Threads are never killed.

Two properties of this call are easy to misread. First, it is a one-time snapshot, not a standing rule. It acts on what is non-terminal at read time, so for a graph still spawning members a single cancel may leave later-starting members running. Second, because an operator cancel produces no failed members, the workflow's derived status reads Cancelled, not Failed. A member that happens to terminate on its own between the read and its per-member cancel is simply counted in neither bucket.

Inspect status and graph#

Read the workflow back through BackWaveMonitor. List them oldest-first, or pull one full view.

// inject BackWaveMonitor monitor
var all  = await monitor.ListWorkflowsAsync();          // oldest first
var view = await monitor.GetWorkflowAsync(workflowId);  // null if absent

ListWorkflowsAsync returns a lightweight snapshot per workflow: the id, optional name, creation time, derived WorkflowStatus, a member count, and the RestartedFrom lineage pointer if the workflow was restarted. GetWorkflowAsync returns the full WorkflowView for one graph, or null if the id is unknown. The view carries:

• Status, the derived WorkflowStatus (Running, Failed, Cancelled, or Succeeded), recomputed on every read. For a conditional workflow, read the member states below rather than this rollup alone.
• Members, a per-member snapshot list. Each entry exposes the member's job id, Wire Name, Queue, current State, Attempt, due time, lease details, cancel-requested flag, terminal time and cause, and Tags. Snapshots never carry payload bytes.
• Edges, the structural parent-to-child edges. Unlike the live gating edges that resolve away as parents go terminal, these stay complete for the whole life of the workflow, so the graph view always renders the full shape.
• RestartedFrom, set when this workflow was produced by restarting an earlier one.

The Pro dashboard surface renders all of this visually, so reach for the monitor API when you need the data in code and the dashboard when a human is looking.

Recover by re-running#

Recovery is a full redo, not a resume-from-failure. BackWave records nothing inside a handler, so it cannot resume a graph part-way. To re-run a failed workflow, build the same graph again with the typed builder and EnqueueAsync it. That re-runs every step from the start, including the ones that already Succeeded, so non-idempotent steps double-fire and the same idempotency discipline that protects your handlers under at-least-once execution protects them here.

For a shape-preserving redo that carries lineage, Build() returns a WorkflowDefinition, and WorkflowDefinition.RestartAsNew() mints a fresh definition with a new workflow id, fresh member ids, the identical shape, and RestartedFrom pointing at the original. It only creates new jobs; it never reanimates a terminal one. The new workflow's RestartedFrom lets you trace the lineage in the monitor views.

There is no in-place Workflow Retry that reanimates the failed members under their original ids. That capability sits below the determinism boundary and is not part of the shipped public surface. A full redo with new identities is the supported recovery operation today.

Where to go next#

• Workflows: the conceptual model behind derived status, the composition shapes, and the boundary against durable execution.
• Workflows API reference: the exhaustive signatures for every method and type touched here.
• Chain Jobs with Dependencies: the base Dependency primitive a Workflow groups over, for single-edge cases that need no workflow.
• Read Another Job's Output: how a dependent member consumes an ancestor's Job Output in an ETL-style graph.
• Testing Workflows: driving workflow authoring through the harness and asserting derived status and the member graph.
• Cancel a Running Job: the single-job cancel whose cooperative-token mechanism the whole-graph cancel shares.

Found a problem on this page? Report an issue