Build a Workflow

Author a multi-job DAG, enqueue the whole graph atomically, append to a running workflow, and cancel it as a unit.

Author a multi-job DAG, enqueue the whole graph atomically, append to a running workflow, and cancel it as a unit.

A Workflow is the grouping and identity layer over a set of Jobs wired together by Dependencies. It gives a loose pile of dependent jobs four things they would otherwise lack: a stable id, an optional name, a graph you can read back, and lifecycle operations over the whole group. This guide walks the full authoring path: building the DAG, enqueuing it in one atomic write, correlating member ids, appending live work, cancelling the group, and recovering by restart. Everything here lives in the BackWave Pro package and hangs off the same BackWaveClient, BackWaveOperator, and BackWaveMonitor you already use for ordinary jobs.

If you have not read Workflows yet, skim it first. It is the conceptual twin of this how-to and defines the vocabulary leaned on here: Workflow, Dependency, derived status, and the line between a static DAG and durable execution.

What a Workflow is, and is not#

The model here is deliberately narrow.

A Workflow is a static DAG. Every edge is fixed at each member's enqueue time. The graph never rewrites an existing job's dependencies. It grows in exactly one way, by appending new jobs, and it shrinks never. A dependent may read an ancestor's Job Output, but output never reshapes the graph.

A Workflow is not durable execution. There are no signals, waits, conditionals, step results that branch the graph, or replay. BackWave never records or replays steps inside a job. If you have used Temporal, unlearn that mental model here. The right comparison is the dependency model of River or Hangfire with a stable group identity bolted on top.

Two rules hold for every member and the store enforces both at enqueue:

• A job belongs to at most one Workflow.
• Every gating parent of a member must be a member of the same Workflow. A member cannot depend on a job in another workflow or on a free-standing job.

Status is derived, never stored#

A Workflow's status is always a projection of its member-job states, computed on read. The stored workflow row holds only identity and configuration. Nothing writes an authoritative "the workflow is Running" flag, which is why appending live work to a drained workflow legitimately reopens it.

The status resolves by first-match-wins precedence over the members:

Read that order top to bottom. A workflow with one Dead-Lettered member and ten succeeded members reads Failed, not partially succeeded. Never cache the status as authoritative.

Enable Pro#

Workflows light up by referencing the Pro package and calling AddBackWavePro once at startup, after AddBackWave.

builder.Services
    .AddBackWave(bw => bw
        .UseStore(new PostgresJobStore(connectionString))
        .UseJobs(BackWaveJobs.Module)
        .AddWorkerGroup(new WorkerGroupOptions { Name = "default" }))
    .AddBackWavePro(builder.Configuration["BackWave:ProLicense"]);

The license argument is optional and defaults to null, which is correct for organizations under the revenue threshold on the honor system. A missing, malformed, or out-of-term license soft-fails: you get one startup log warning and a banner on the dashboard, and every feature behaves identically. Referencing the package is the entire boundary. The license never gates whether workflows run.

To get the graph UI, add the optional dashboard package on top.

builder.Services.AddBackWaveProDashboard();

That lights up the Workflows surface on the dashboard: the workflow list, the graph view, and the cancel-workflow action. It requires AddBackWavePro to be wired first.

Author the DAG#

You build a graph through a WorkflowBuilder, started from the client. BuildWorkflow opens a fresh workflow; the optional name is a human-readable label that also shows up in the graph view.

// inject BackWaveClient client
var wf = client.BuildWorkflow("nightly-etl");
 
wf.AddJob("extract", new Extract(date));
wf.AddJob("transform-a", new Transform("a"), dependsOn: ["extract"]);
wf.AddJob("transform-b", new Transform("b"), dependsOn: ["extract"]);     // fan-out from "extract"
wf.AddJob("load", new Load(), dependsOn: ["transform-a", "transform-b"]); // fan-in
wf.AddJob("notify", new Notify(), dependsOn: ["load"], mode: DependencyMode.OnAnyTerminal);
 
var definition = wf.Build();

Each AddJob takes a name, the job payload, and a handful of optional arguments. The name is a handle that is unique within the workflow, compared ordinally. It doubles as the dependency reference key and the label in the graph view, so pick something readable. The payload must be a type registered with BackWave the same way any other job is.

dependsOn lists the names of same-workflow jobs that must finish before this one is eligible. Omit it for a root job. There is no special fan-out or fan-in construct: fan-out is one parent named in several children's dependsOn, and fan-in is one child listing several parents. Both are just edges. In the example, extract fans out to two transforms, and load fans those back in.

AddJob returns the same builder, so you can chain calls fluently if you prefer that to the statement-per-job style above.

When an edge releases its child#

The mode argument decides what counts as "the parent finished". It is a DependencyMode with two values.

• OnSuccess is the default. The child releases only if every parent Succeeded. Any other terminal outcome on a parent (Dead-Lettered, Quarantined, Cancelled) cancels the child instead of running it.
• OnAnyTerminal releases the child once every parent is terminal, whatever the outcome.

Use OnSuccess for a normal pipeline step that needs its inputs to have worked. Use OnAnyTerminal for cleanup or notification that should run regardless, as the notify job above does. This is the same on-success versus on-any-terminal choice documented for the base Dependency primitive; a Workflow is the grouping layer over that primitive.

Per-member Queue and Tags#

AddJob also accepts queue and tags. The queue argument overrides the Queue the job type registered with, for this member only. The tags argument is unioned on top of the job type's default Tags; it is additive only and never removes a default. Both behave exactly as their single-job enqueue counterparts.

Validation happens at Build()#

Build() validates the graph and emits a WorkflowDefinition. Treat that definition as an opaque token: you produce it here and hand it to the enqueue call, nothing more. The validation runs three checks, and any failure throws InvalidWorkflowException:

• The graph is non-empty.
• Every dependsOn name resolves to a job in this workflow. A name that points at nothing produces a message like Job 'x' depends on 'y', which is not a job in this Workflow.
• The graph is acyclic. A cycle, including a self-edge, is rejected with a message naming the offending nodes.

These are programming errors, caught at build time on your own machine, never in production. A duplicate name is caught even earlier, at the AddJob call that introduces the collision, and a null or empty name throws ArgumentException there too.

Enqueue the whole graph atomically#

One call writes every member job and the workflow record in a single all-or-nothing transaction. Either the entire graph lands or nothing does.

var workflowId = await client.EnqueueWorkflowAsync(definition);

It returns the workflow's Guid. Root members (those with no dependsOn) become due immediately and start running as Workers pick them up. Downstream members land awaiting their parents and release as those parents reach terminal states, according to each edge's mode.

A failed enqueue surfaces as an exception rather than a silent partial write. The store can reject a graph for several reasons, and the reason name is interpolated into an InvalidOperationException message:

• A duplicate workflow id, or a member id that collides in the store or appears twice in the batch.
• An append whose target workflow does not exist.
• A containment violation, where a member gates on a job that is not a member of the same workflow.
• An empty graph, a payload over the size limit, a wire name over the length limit, or a member declaring too many parents.

Limits to design around#

A few numeric bounds shape how wide and how heavy a graph can be. These are the store defaults; an adapter may raise them.

A fan-in node that needs more than 16 inputs is usually a sign to introduce an intermediate aggregation step rather than widening the node. For payload size, the same advice as ordinary jobs applies: carry a reference (an id or a blob key) and have the handler fetch the bytes.

Correlate a member id#

Each member's id is assigned the moment you call AddJob, before anything is enqueued, and it stays stable for the life of the builder. Read it back by name with GetJobId.

var chargeJobId = wf.GetJobId("charge"); // the Guid assigned to that member at AddJob time

This is how you record a correlation between your own domain row and a specific job before the workflow even commits. Calling GetJobId with a name you never added throws InvalidWorkflowException.

Enqueue atomically with your own writes#

The workflow path supports Transactional Enqueue: commit the entire graph in the same database transaction as your own data, so there is no outbox and no window where your row exists but the jobs do not. Unlike the single-job EF Core path, there is no EF-specific workflow overload, so you extract the DbTransaction yourself and pass it in.

await using var tx = await context.Database.BeginTransactionAsync();
context.Orders.Add(order);
await context.SaveChangesAsync();
 
var definition = client
    .BuildWorkflow("checkout")
    .AddJob("charge", new ChargeCard(order.Id))
    .AddJob("receipt", new SendReceipt(order.Id), dependsOn: ["charge"])
    .Build();
 
await client.EnqueueWorkflowAsync(
    definition,
    transaction: context.Database.CurrentTransaction!.GetDbTransaction());
 
await tx.CommitAsync(); // the order and the whole workflow commit together

GetDbTransaction() comes from Microsoft.EntityFrameworkCore.Storage. This requires a store that supports transactional enqueue, which covers networked Postgres and SQL Server, and co-resident SQLite. Passing a transaction to an adapter that cannot do it throws NotSupportedException. The transactional enqueue guide covers the broader pattern and the raw DbTransaction form.

Append live work to a running workflow#

A Workflow grows by appending. Start an append builder against an existing workflow id, add new members, and enqueue. The existing members and their edges are never touched.

var append = client.BuildWorkflowAppend(workflowId);
append.AddJob("reconcile", new Reconcile(), after: [wf.GetJobId("load")]);
await client.EnqueueWorkflowAsync(append.Build());

Appends use two different ways to express edges, and the distinction matters:

• after takes the ids of existing members the new job should depend on. This is the only way to point at work that was enqueued in a prior batch.
• dependsOn takes the names of other new jobs in this same append batch.

after is exclusively for appends. Passing it on a fresh BuildWorkflow builder is silently ignored, so within a fresh graph always wire edges with dependsOn. There is one sharp edge here: the builder validates dependsOn names against the current batch, but it does not validate after ids against the live store. An after id that is not actually a member of the target workflow passes Build() cleanly and surfaces only at enqueue time as a containment-violation InvalidOperationException. Double-check the ids you feed into after.

Appending live work to a workflow that had already drained reopens its derived status from Succeeded back to Running, which is exactly the behavior the projection model is built to allow.

Cancel the workflow as a unit#

CancelWorkflowAsync cancels every member that is non-terminal at the moment of the call. It is the whole-graph counterpart of single-job cancel and uses the same cooperative-token mechanism underneath.

// inject BackWaveOperator op
var result = await op.CancelWorkflowAsync(workflowId, actor: "ops@acme.com");
// result.Found, result.CancelledImmediately, result.CancellationRequested

The actor is recorded in the audit log against each member cancelled. The result reports how the cancel landed:

• Found is false, via WorkflowCancelResult.NotFound, when the id matches no workflow.
• CancelledImmediately counts members that had not started yet and went straight to terminal Cancelled.
• CancellationRequested counts members that were leased and running and were asked to stop cooperatively. They stop only when their handler next observes its CancellationToken, delivered on the next heartbeat. Threads are never killed.

Two properties of this call are easy to misread. First, it is a one-time snapshot, not a standing rule. It acts on what is non-terminal at read time. A member that becomes eligible and starts running after the call returns is unaffected, so for a graph still spawning members a single cancel may leave later-starting members running. Second, because an operator cancel produces no failed members, the workflow's derived status reads Cancelled, not Failed. A member that happens to terminate on its own between the read and its per-member cancel is simply counted in neither bucket.

Inspect status and graph#

Read the workflow back through BackWaveMonitor. List them oldest-first, or pull one full view.

// inject BackWaveMonitor monitor
var all  = await monitor.ListWorkflowsAsync();          // oldest first
var view = await monitor.GetWorkflowAsync(workflowId);  // null if absent

ListWorkflowsAsync returns a lightweight snapshot per workflow: the id, optional name, creation time, derived WorkflowStatus, a member count, and the RestartedFrom lineage pointer if the workflow was restarted. GetWorkflowAsync returns the full WorkflowView for one graph, or null if the id is unknown. The view carries:

• Status, the derived WorkflowStatus (Running, Failed, Cancelled, or Succeeded), recomputed on every read.
• Members, a per-member snapshot list. Each entry exposes the member's job id, Wire Name, Queue, current State, Attempt, due time, lease details, cancel-requested flag, terminal time and cause, and Tags. Snapshots never carry payload bytes.
• Edges, the structural parent-to-child edges. Unlike the live gating edges that resolve away as parents go terminal, these stay complete for the whole life of the workflow, so the graph view always renders the full shape.
• RestartedFrom, set when this workflow was produced by restarting an earlier one.

The Pro dashboard surface renders all of this visually, so reach for the monitor API when you need the data in code and the dashboard when a human is looking.

Recover by Restart#

When a workflow fails, the available recovery path is Restart. RestartAsNew re-instantiates the whole graph as a brand-new workflow: a fresh workflow id, fresh member job ids with all the internal parent references remapped, the identical shape, and RestartedFrom pointing at the original. It only creates new jobs; it never reanimates a terminal one. To run it, take the original definition, call RestartAsNew, and enqueue the result.

var restartDefinition = definition.RestartAsNew(); // fresh ids, RestartedFrom = original
var newWorkflowId = await client.EnqueueWorkflowAsync(restartDefinition);

Be clear about what Restart does: it re-runs every step from the start, including the ones that already Succeeded. It is a full redo, not a resume-from-failure. Non-idempotent steps will double-fire, so the same idempotency discipline that protects your handlers under at-least-once execution protects them here. The new workflow's RestartedFrom lets you trace the lineage in the monitor views.

There is no in-place Workflow Retry that reanimates the failed members under their original ids. That capability sits below the determinism boundary and is not part of the shipped public surface. Restart, with new identities and a full redo, is the supported recovery operation today.

Where to go next#

• Workflows: the conceptual model behind derived status, the static DAG, and the boundary against durable execution.
• Workflows API reference: the exhaustive signatures for every method and type touched here.
• Chain Jobs with Dependencies: the base Dependency primitive a Workflow groups over, for single-edge cases that need no workflow.
• Read Another Job's Output: how a dependent member consumes an ancestor's Job Output in an ETL-style graph.
• Testing Workflows: driving workflow authoring through the harness and asserting derived status and the member graph.
• Cancel a Running Job: the single-job cancel whose cooperative-token mechanism the whole-graph cancel shares.