Circuit Breakers, Bulkheading, and Why Retrying POST Is a Junior Mistake

Distributed systems fail in predictable ways. The question isn't whether a backend service will fail — it's whether your gateway handles the failure gracefully or cascades it into a full outage. throttleGate implements three resilience patterns that work together: circuit breakers, bulkheading, and smart retries.

Circuit Breaker State Machine

The circuit breaker is a state machine with three states: closed, open, and half-open. In the closed state, requests flow normally while the breaker tracks error rates over a rolling window. When the error rate exceeds a configurable threshold, the breaker trips to open — blocking all requests instantly and returning 503.

CLOSED ──(errors > threshold)──▶ OPEN
OPEN   ──(recovery timeout)────▶ HALF-OPEN
HALF-OPEN ──(all succeed)──────▶ CLOSED
HALF-OPEN ──(any failure)──────▶ OPEN

After a recovery timeout, the breaker transitions to half-open and allows a limited number of probe requests. If all probes succeed, the breaker resets to closed. If any probe fails, it snaps back to open and the recovery timeout restarts.

The key implementation detail: circuit breaker state is local per gateway instance. This is intentional. Using Redis for CB state would add latency and complexity with no benefit. Each instance independently tracks its experience with the backend, which is correct behavior — one instance may be seeing errors while another is fine.

Bulkheading — Don't Let One Service Take Down the Rest

Bulkheading means isolating resources so a failure in one component doesn't exhaust shared resources. In throttleGate, each upstream target gets its own connection pool via a dedicated http.Agent with keep-alive enabled.

const agent = new http.Agent({
  keepAlive: true,
  maxSockets: 100,
  maxFreeSockets: 20,
});

When the billing service slows down, it can only exhaust its own 100-socket pool. The user service, with its own independent pool, continues serving normally. Without bulkheading, a single slow backend can consume all gateway connections and take down routing to every service.

Retry With Exponential Backoff and Jitter — But NEVER POST

Retries sound like an easy win: request failed, try again. But blind retries on non-idempotent requests are one of the most common junior-level mistakes in distributed systems.

Why not retry POST? A POST request that creates a resource might succeed on the server but the response times out before reaching the client. Retrying the POST creates a duplicate resource. Orders get charged twice, tickets get created in duplicate, databases get corrupted. The HTTP spec is clear: POST is not idempotent unless the endpoint explicitly guarantees it.

In throttleGate, retries only trigger for idempotent methods: GET, PUT, DELETE, HEAD, and OPTIONS. The retry uses exponential backoff with jitter to prevent thundering herd problems:

function backoff(attempt, baseMs, maxMs) {
  const delay = Math.min(baseMs * 2^attempt, maxMs);
  const jitter = Math.random() * delay * 0.5;
  return Math.floor(delay + jitter);
}

The jitter prevents the retry stampede that happens when N clients all retry at exactly the same interval. Adding 0-50% random jitter spreads retries across the window, giving the backend time to recover.

Testing Resilience

The chaos tests kill a backend mid-request and verify the circuit breaker opens correctly. They simulate Redis going down and verify fail-open behavior kicks in. These aren't unit tests that pass with mocks — they spin up real Redis instances and mock backend services to verify the system behaves correctly under real failure conditions.