Every automation needs an exit route.

An automated workflow is usually specified as a sequence: receive an invoice, extract the fields, match it to a purchase order, post it to the ledger, notify the approver. That sequence is the part everyone agrees on, and it is rarely the part that causes trouble. The trouble lives in the questions the sequence does not answer.

• What happens when the ledger API times out after it has already posted the entry?
• What happens when the same invoice arrives twice, once by email and once through the supplier portal?
• What happens when the purchase order number is missing, or matches two orders?
• Who finds out when a step has been failing quietly since Tuesday?

A workflow is only as dependable as its behavior in those cases. Each one needs a deliberate outcome, designed up front: retry, skip, escalate, or stop. If the design does not choose, production will, usually by retrying forever or dropping the work silently.

Failures come in three kinds, and they need different treatment. Confusing them is behind most unreliable automation.

Retrying a business exception wastes capacity and buries the real issue. Escalating a transient timeout to a person trains people to ignore the queue. The classifier deserves code review as careful as the happy path's.

type Outcome =
  | { kind: "retry"; delayMs: number }
  | { kind: "needs_review"; reason: string }
  | { kind: "halt"; reason: string };

const TRANSIENT_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);
const MAX_ATTEMPTS = 6;

export function classify(error: StepError, attempt: number): Outcome {
  if (error.kind === "business_rule") return { kind: "needs_review", reason: error.message };
  const transient = error.kind === "network" || TRANSIENT_STATUS.has(error.status ?? 0);
  if (!transient) return { kind: "halt", reason: `Permanent failure: ${error.message}` };
  if (attempt >= MAX_ATTEMPTS) return { kind: "needs_review", reason: `Gave up after ${attempt} attempts` };
  return { kind: "retry", delayMs: error.retryAfterMs ?? fullJitter(attempt) };
}

// "Full jitter": a random delay between zero and an exponentially growing cap.
function fullJitter(attempt: number, baseMs = 500, capMs = 60_000) {
  return Math.random() * Math.min(capMs, baseMs * 2 ** attempt);
}

Two details in that function matter. First, it honors a Retry-After value when the server sends one; the server knows its own recovery time better than any backoff curve. Second, the delay is randomized. Without jitter, every client that failed at the same moment retries at the same moment, and the recovering service is hit by a synchronized wave. The AWS Architecture Blog's analysis of exponential backoff and jitter shows how much contention full jitter removes.

Retries create a second problem. If the ledger timed out after it posted the entry, a retry posts it again. Across a network you cannot always tell "the request failed" from "the response was lost", so every step with a side effect must be safe to repeat. That property is idempotency, and it has to be designed in.

The standard mechanism is an idempotency key: a value derived from the work itself, sent with the request, and recorded by the receiver so that a repeat returns the original result instead of acting again. Stripe's API popularized the pattern, and an IETF draft standardizes the Idempotency-Key header. When you own the receiving side, a unique constraint does the same job:

CREATE TABLE ledger_postings (
  idempotency_key text        PRIMARY KEY,   -- e.g. 'invoice:INV-20931:post'
  invoice_id      text        NOT NULL,
  amount_cents    bigint      NOT NULL,
  posted_at       timestamptz NOT NULL DEFAULT now()
);

-- Safe to run any number of times: a repeat inserts nothing.
INSERT INTO ledger_postings (idempotency_key, invoice_id, amount_cents)
VALUES ($1, $2, $3)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING posted_at;

If RETURNING produces no row, the posting already exists and the step can report success without acting again. Derive the key from the business identity of the work (this invoice, this step), never from a value generated per attempt, or every retry will look like new work.

The same reasoning applies on the trigger side. Most queues and webhook senders deliver at least once, so consumers must tolerate duplicates. And when a step must both update your database and publish an event, write the event to an outbox table in the same transaction and publish it from there. The transactional outbox pattern removes the case where one succeeds and the other does not.

Most workflow engines can retry. Fewer make failure visible. The difference is whether failure is a first-class state that the system stores, counts, and shows to someone, or a line in a log nobody reads.

Store each item's workflow state and history durably, so that a crash in the middle resumes from the last completed step instead of starting over. Record which step it is in, how many attempts it has made, and the last error. That record is what lets an operator answer "where is invoice INV-20931?" in seconds. Messaging systems call the terminal version of this a dead letter channel. The name matters less than the fact that someone owns what lands there.

Automation removes repetitive work. It should not remove control. The people who run the process need a small set of operations they can use without an engineer:

• Pause a workflow or a single integration, so that a known outage at a supplier does not flood the review queue.
• Inspect an item: its current state, history, inputs, and the exact error.
• Retry or replay an item, or a batch, once the cause is fixed. Idempotency is what makes this button safe to press.
• Resolve manually, recording who did what and why, so that the audit trail survives the exception.

Then measure the exits. How many items reach "needs review" each day, how long they wait, and the most common reasons are the most honest metrics an automation has. A rising count is either a process change nobody mentioned or a rule worth automating next. Either way the workflow is telling you something, and the design should make sure someone is listening. This exception-first approach is how we scope AI automation and data integration work.