Failure handling: retries, backoff, circuit breakers, idempotency

How it works

1. Timeouts on every call

Always set a timeout, based on the dependency's normal latency (for example a bit above its p99). For a chain of calls, pass a deadline down so inner calls know how much time is left and don't keep working on a request the user has already given up on.

2. Retries, only for the right failures

Retry errors that are likely temporary: timeouts, connection resets, 503 Service Unavailable, 429 Too Many Requests (after the Retry-After delay). Don't retry errors that will fail again: 400 Bad Request, 401, 404, validation errors.

Limit attempts (2–3 is typical) and retry at one layer only. If every layer of a five-service chain retries three times, one failure becomes 3⁵ = 243 calls.

3. Exponential backoff with jitter

Wait longer between each attempt: 100 ms, 200 ms, 400 ms, 800 ms… That's exponential backoff. Add randomness, jitter, so clients don't move in lockstep. A common form is “full jitter”: wait a random time between 0 and the current backoff cap.

async function callWithRetry<T>(fn: () => Promise<T>, attempts = 3, baseMs = 100, capMs = 2000) {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i + 1 >= attempts || !isRetryable(err)) throw err;
      const backoff = Math.min(capMs, baseMs * 2 ** i);
      await sleep(Math.random() * backoff); // full jitter
    }
  }
}

4. Circuit breakers

A circuit breaker wraps calls to a dependency and tracks failures.

While open, calls fail instantly, often with a fallback (cached data, a default, “try again later”). That keeps your threads free and gives the dependency time to recover.

5. Idempotency: make retries safe

An operation is idempotent if doing it twice has the same effect as once. GET, PUT and DELETE are naturally idempotent; POST usually isn't. Make it so with an idempotency key:

1. The client generates a unique key per logical operation and sends it with the request (and every retry).
2. The server stores the key with the result when it first processes the request.
3. A repeat with the same key returns the stored result instead of doing the work again.

Payment APIs work this way. The same idea makes queue consumers safe (lesson 17).

6. Other tools

• Bulkheads: separate thread or connection pools per dependency, so one slow dependency can't use up everything.
• Load shedding: under overload, reject low-priority work early (with 503) instead of slowing everything down.
• Fallbacks and graceful degradation: show cached or partial results.
• Dead-letter queues for async work that keeps failing.
• Chaos testing: inject failures deliberately to check that all of the above actually works.