Distributed systems
Workflow Engine
A long-running, multi-step process must survive crashes — a workflow engine persists progress after every step so it can resume exactly where it left off.
Cron + status flags is brittle
Order fulfillment is a long-running process: pack → ship → notify, with retries, timeouts, waits, even a human approval step. Hand-roll it with a cron job polling a status column and you're one crash away from a lost order.
function pollOrders() {
const order = db.next('status=packing');
pack(order); // crash HERE →
order.status = 'packed'; // ← never runs, state lost
db.save(order);
// retries? timeouts? waits? all hand-rolled, ad-hoc
}
The engine persists every step's result
A workflow engine (Temporal, AWS Step Functions, Airflow, Camunda) runs a workflow defined as code. After each step completes, its result is written to a durable state store. On crash or restart, the engine replays that history and resumes at the next step — not from scratch.
async function orderWorkflow(order) {
await step('pack', () => pack(order));
await step('ship', () => ship(order),
{ retries: 3, timeout: '30s' });
await step('notify', () => notify(order));
// engine persists after EVERY await
}
You write the workflow logic; the engine provides durability, retries, timeouts, waits, and human-approval tasks. It's the orchestrator that can run a saga.
Watch a step fail, retry, and still finish
The engine runs each step and persists the result to the state store. Notice ship times out once — the engine retries automatically instead of losing the whole order.
Each "persist" write is the crash-resumable checkpoint — if the process died right after "persist: packed", a restart would resume at ship, not pack.
Durable, resumable orchestration
- Durable & resumable: every step's result is persisted; a crash resumes exactly where it left off, not from step one.
- Built-in primitives: retries, timeouts, waits, and human-approval tasks — no more hand-rolled cron + status flags.
- Workflow-as-code: you write plain control flow; the engine handles the durability underneath.
- It's the orchestrator that runs sagas — coordinating compensating actions across services.
- Related: steps often hand off via queues, and the state store is itself a form of event sourcing.