design·lab

Distributed systems

Circuit Breaker

When a downstream dependency is failing, a circuit breaker stops the bleeding — it fails fast instead of piling retries onto a service that's already down.

✗ The problem

Retrying a sick service makes it sicker

A downstream call starts failing. The caller keeps retrying — every retry blocks a thread waiting on a service that won't answer.

while (true) {
  try {
    return callDownstream();
  } catch (e) {
    // retry immediately —
    // piles more load on a
    // service that's already down
  }
}
Caller
retrying…
→→→
Service
down, overloaded
Threads pile up waiting on a dead dependency. The caller's own thread pool exhausts, and the failure cascades upstream to every service that depends on it.
✓ How it works

Three states, one wrapper around the call

The breaker sits in front of the call and tracks its own health, independent of the caller.

class CircuitBreaker {
  call(fn) {
    if (this.state === 'OPEN')
      throw new Error('fail fast');
    try {
      const r = fn();
      this.onSuccess();
      return r;
    } catch (e) {
      this.onFailure();  // may trip OPEN
      throw e;
    }
  }
}
CLOSED
calls flow, count failures
↓ failure threshold reached
OPEN
fail fast, no call made
↓ cooldown elapsed
HALF-OPEN
allow one trial call
↑ success → CLOSED · failure → OPEN
✓ See it live

Trip it, then watch it recover

Fail 3 calls in a row to trip the breaker OPEN — further calls fail fast with no call made. Then cooldown to HALF-OPEN and send one trial call.

CLOSED
calls flow
OPEN
fail fast
HALF-OPEN
trial call

State: CLOSED  •  Failures: 0 / 3

✓ Takeaway

Fail fast, recover gracefully

  • Fail fast: once tripped, callers get an immediate error instead of waiting on a timeout.
  • Give the dependency time to recover: no load reaches it while OPEN.
  • Prevent cascades: the caller's own threads and resources stay free for other work.
  • Pair it up: combine with retries, timeouts, and fallbacks for full resilience.
  • You already use it: Netflix Hystrix, Resilience4j, Polly.
🎯 Combines / relates: it's the State pattern applied at the system boundary (a Proxy around the call), and teams with rate limiting for resilience.