Handle Sensitive Data

Keep PII out of the job store: payload minimization, tuning the Job History Policy and failure-detail capture, the no-PII-in-tags rule, the dashboard's sensitive-data gate, and retention as a data-lifecycle control.

Keep PII out of the job store: payload minimization, tuning the Job History Policy and failure-detail capture, the no-PII-in-tags rule, the dashboard's sensitive-data gate, and retention as a data-lifecycle control.

A background-job system quietly becomes a second database: payloads, failure messages, and tags all persist in your store and surface in a dashboard. This guide collects BackWave's data-protection levers in one place — the page a security reviewer reads. Each lever exists and is documented individually elsewhere; here they are walked as one ordered checklist, following the data as it flows: what you put in a payload, what gets recorded alongside it, who can see it, and how long it survives.

If you have not read Job Lifecycle yet, skim it first. It defines the vocabulary this guide leans on: terminal states, the Transition Log, Failure Detail, and the Transition Observer.

Minimize the payload#

The strongest control is the one that stores nothing. BackWave treats a payload as opaque bytes: it never inspects the contents, but it persists whatever you serialize, and it holds those bytes for the life of the job row. Anything you put in a payload is data at rest in your store, so the first question for every field is whether it needs to be there at all.

The pattern that keeps PII out is to enqueue identifiers rather than data. Put an order id or a customer id on the payload and have the Handler fetch the current record at run time. The privacy win is that the id is meaningless on its own, but there are two bonuses beyond privacy: the Handler always sees fresh data rather than a stale snapshot captured at enqueue, and the payload shape stays small and stable, which makes it far easier to evolve later.

Payloads are also bounded. BackWave caps a serialized payload at 64 KB, and an over-limit payload is rejected at enqueue, not silently truncated — the enqueue call throws and tells you to store a reference to the data instead of the data itself. The cap is a backstop, not a target: if a payload is approaching it, that is usually a sign real data is riding along that should be fetched by id instead.

Failure detail is the leak you forget#

Payloads are the data you chose to store. Failure Detail is the data that stores itself. When an Attempt throws, BackWave captures the exception at the edge — its type, message, and stack trace — and records that text against the job. Exception messages love to interpolate user data: an argument value, an email address, the row that failed validation. That text lands in your store just like a payload, but you never explicitly wrote it there, which is why it is easy to overlook in a review.

How much gets recorded is governed by the Job History Policy, which has three levels that form a ladder — each rung adds to the one below it:

The default is TransitionsAndFailureDetail, the top rung — capture is on unless you turn it down. You configure the policy on your store, so it is a storage-level decision rather than something set per job.

There are two mitigations, in order of preference. The first is to write Handlers whose exceptions do not embed user data in the first place — throw on a stable, non-interpolated message and keep the offending values out of the exception. That is the durable fix, because it holds no matter how history is configured. The second is to turn capture down. Dropping the policy from TransitionsAndFailureDetail to Transitions keeps the state history but stops exception text from ever being recorded.

For an ops-level control that needs no code change, set the environment variable BACKWAVE_DISABLE_FAILURE_DETAIL to a truthy value (1, true, yes, or on). When it is set, BackWave automatically downgrades an effective TransitionsAndFailureDetail to Transitions: transition rows still record, but failure-detail capture is suppressed. Two properties are worth stating precisely. First, it gates recording, not viewing — the text never lands in the database at all, which is a stronger guarantee than merely hiding it in the dashboard. Second, it only ever turns capture down; it has no effect when the configured policy is already Transitions or Off, so it can never raise a level.

One constraint ties this section to your outcome-reaction wiring. Transition Observers require a history policy of at least Transitions, because an observer fires off a recorded transition. Registering an observer while the policy is Off throws at startup — during container composition, not on the first tick — with a message that names the Job History Policy, so the failure is loud and early rather than a silent no-op in production. This is why the two mitigations above stay distinct: the BACKWAVE_DISABLE_FAILURE_DETAIL kill-switch downgrades to Transitions, which is still at least Transitions, so observers keep working; only choosing Off — dropping everything — breaks them.

Nothing regulated in a tag#

A Tag is an observational annotation — a Label (a bare string) or a Keyed Tag (a key paired with a string value) — that exists so you can search, filter, and group jobs in the Monitor and Dashboard. Tags are purely descriptive: the Core never reads them to make decisions.

That descriptive role has a hard consequence for sensitive data. Tags are stored in plaintext and must stay queryable to do their job, so they are deliberately left outside the sensitive-data gate that covers raw content in the next section. A tag value shows up in dashboard lists, in filter queries, and in facet counts, in the clear, to anyone who can view jobs. The rule is therefore absolute: never put regulated data in a Tag. No names, no emails, no account numbers.

Correlate by opaque id instead. A Keyed tenant tag carrying a tenant id lets you filter and group every job for that tenant without exposing who the tenant is. The id is a handle; the identity behind it lives in your own database, behind your own access controls. See Tags for the Label-versus-Keyed distinction and Tag & Find Jobs for the filter and facet queries.

Gate what the dashboard shows#

Payloads, Failure Detail, and Job Output are the three channels that can carry secrets or PII, and the dashboard puts a single gate over all three. The gate has two layers, and both must agree before any of that content is read for display.

The first layer is a host-wide master switch, ExposeSensitiveData, which defaults to true. Set it to false and sensitive content never leaves storage for the dashboard, regardless of who is asking. The same switch has an environment override, BACKWAVE_DASHBOARD_DISABLE_SENSITIVE_DATA, that forces exposure off when set to a truthy value (1, true, yes, or on) — an ops-level kill-switch that needs no redeploy. Effective exposure is the option and the absence of that environment kill-switch: either one turning off is enough to close the gate.

The second layer decides who may see the content while exposure is on. AuthorizeViewSensitiveData is a per-request delegate that defaults to deny — even with ExposeSensitiveData left at its default, no one sees raw content until you supply a delegate that says yes for a given user. So exposure decides whether sensitive content can leave storage at all; the delegate decides who sees it when it can.

This sits inside a broader dashboard posture worth stating in one line: viewing jobs is default-allow, while every state-changing action — requeue, cancel, pausing a queue, triggering a schedule — and viewing sensitive content are default-deny, and a control renders only when its permission passes. Do not re-derive that model here; the Dashboard's permission model documents it in full.

Retention is a privacy control#

The final lever is time. Sensitive data you did store does not have to live forever, and bounding its lifetime is a genuine data-lifecycle control rather than mere disk hygiene — when a job row is purged, the payload and the whole Transition Log go with it. For regulated data, "we delete it after N days" is often the compliance story, and this is the knob that enforces it.

Retention is governed by two spans on the RetentionPolicy:

The retention clock starts at the instant a job reached its terminal state, not when it was enqueued, so a long-running job is measured from when it finished. Two operational facts belong in any review. Retention is configured per Worker Group, so different groups can hold data for different windows, and the default policy applies unless you set your own. Setting a group's retention to null disables sweeping entirely for that group — rows, and any PII in them, then persist indefinitely. If your compliance story depends on purging, confirm no group has retention switched off. The Retention & History Purge page covers the sweep and manual purge in full.

The reviewer's checklist#

These are the levers a deployment review should walk, end to end — from what enters a payload to how long a terminal job's row survives.

Where to go next#

• The Dashboard: the full default-allow reads, default-deny writes model behind the dashboard gate.
• Retention & History Purge: how the retention sweep runs and how to purge history on demand.
• React to Job Outcomes: Transition Observers and the history-policy floor they require.
• Tags: the Label-versus-Keyed distinction behind the no-PII-in-tags rule.
• Evolve Job Payloads: keep payloads small and by-reference without breaking in-flight jobs.