design·lab

Distributed systems

Workflow Engine

A long-running, multi-step process must survive crashes — a workflow engine persists progress after every step so it can resume exactly where it left off.

✗ Problem

Cron + status flags is brittle

Order fulfillment is a long-running process: pack → ship → notify, with retries, timeouts, waits, even a human approval step. Hand-roll it with a cron job polling a status column and you're one crash away from a lost order.

function pollOrders() {
  const order = db.next('status=packing');
  pack(order);              // crash HERE →
  order.status = 'packed';  // ← never runs, state lost
  db.save(order);
  // retries? timeouts? waits? all hand-rolled, ad-hoc
}
A crash mid-way loses progress — did it pack twice? Ship never? Retries and timeouts are bolted on by hand, and there's no clean way to pause for a human approval.
✓ How it works

The engine persists every step's result

A workflow engine (Temporal, AWS Step Functions, Airflow, Camunda) runs a workflow defined as code. After each step completes, its result is written to a durable state store. On crash or restart, the engine replays that history and resumes at the next step — not from scratch.

async function orderWorkflow(order) {
  await step('pack',   () => pack(order));
  await step('ship',   () => ship(order),
    { retries: 3, timeout: '30s' });
  await step('notify', () => notify(order));
  // engine persists after EVERY await
}
⚙️ Workflow Engine
drives the steps
📦 pack → 🚚 ship → ✉️ notify
retries / timeouts / waits
💾 Durable state store
persists after each step

You write the workflow logic; the engine provides durability, retries, timeouts, waits, and human-approval tasks. It's the orchestrator that can run a saga.

✓ See it live

Watch a step fail, retry, and still finish

The engine runs each step and persists the result to the state store. Notice ship times out once — the engine retries automatically instead of losing the whole order.

Each "persist" write is the crash-resumable checkpoint — if the process died right after "persist: packed", a restart would resume at ship, not pack.

✓ Takeaway

Durable, resumable orchestration

  • Durable & resumable: every step's result is persisted; a crash resumes exactly where it left off, not from step one.
  • Built-in primitives: retries, timeouts, waits, and human-approval tasks — no more hand-rolled cron + status flags.
  • Workflow-as-code: you write plain control flow; the engine handles the durability underneath.
  • It's the orchestrator that runs sagas — coordinating compensating actions across services.
  • Related: steps often hand off via queues, and the state store is itself a form of event sourcing.