Retry

Retry is the first resilience pattern every engineer writes, usually in about four lines, usually inside a for loop, and usually wrong. The idea is disarmingly simple: the network is unreliable, some failures are momentary, so when a call fails you try it again. That instinct is correct — a large majority of remote-call failures in a healthy distributed system are transient, and the single most effective response to a transient failure really is to make the call a second time.

What makes retry subtle is that the same mechanism which rescues you from a dropped packet will, in a different failure mode, be the thing that finishes off a struggling dependency. A retry adds load to a system that just told you it could not cope. Getting retry right is entirely about knowing when that is a reasonable thing to do, and building in the brakes that stop it when it is not.

The problem

Start with the failure retry is designed for. A service calls a downstream API over TCP. The connection is reset mid-flight — a load balancer recycled a node, a pod was rescheduled, a leader election moved a partition, a switch dropped a frame. Nothing is broken; a specific attempt was unlucky. Returning an error to the user here is a waste: the very next attempt would almost certainly have succeeded. This is the transient failure, and retry is its correct treatment.

Now consider the failure mode that punishes naive retry. The downstream service is not unlucky, it is overloaded — its queues are full, its latency is climbing, and it starts shedding requests with 503s. Your caller sees failures and dutifully retries each one three times. So does every other caller. The dependency's offered load instantly triples at precisely the moment it is least able to absorb it. Requests that would have completed now time out; those timeouts trigger more retries. The system converges on a state where nearly all traffic is retries of work that will never succeed. This is the retry storm, and it is one of the most reliable ways to convert a partial degradation into a total outage.

There is a second, quieter amplifier: synchronization. Suppose a dependency blips for two seconds and a thousand callers all fail at once. If every caller waits exactly one second and retries, all thousand retries arrive in the same millisecond. Fixed-delay backoff does not spread load — it aligns it, creating a thundering herd that arrives in periodic waves. Each wave can knock the recovering dependency back down, producing an oscillation that outlives the original fault by minutes.

The third problem is not about load at all. It is correctness. Your call to POST /payments times out. Did the payment fail, or did it succeed and the response get lost? You cannot tell from a timeout — that is precisely what a timeout means. Retry it and you may charge the customer twice. Retry is safe only for operations where doing it twice is indistinguishable from doing it once, and most interesting operations are not naturally like that.

Finally, retries interact badly with latency budgets. A user-facing request with a 3-second budget calls a service with a 1-second timeout and 3 retries. In the worst case that single dependency consumes 4 seconds on its own, blowing the budget while the user stares at a spinner. Retries buried deep in a call stack multiply: three layers each retrying three times is up to 27 attempts for one user action.

When to use it

Retry is the right tool when all of the following hold:

• The failure is plausibly transient. Connection resets, timeouts, 502/503/504, throttling responses, brief DNS failures, leader elections. These are failures of an attempt, not of the request. If the same call would fail identically forever, retrying only wastes time.
• The operation is idempotent, or you can make it so. Reads are naturally safe. Writes need an idempotency key, a conditional update, or a natural primary key that makes a duplicate a no-op. "Make it so" is usually achievable and always worth the effort.
• The dependency is not failing from overload — or your policy can tell the difference. A 429 or 503 with Retry-After is the dependency explicitly telling you when to come back; honor it. A retry policy that ignores backpressure signals is a load generator.
• You have latency budget to spend. Background jobs and async consumers have enormous budgets and should retry generously. A synchronous request already 2.5 seconds into a 3-second budget has no room for another attempt, and attempting one only guarantees a slower failure.

The strongest home for retry is asynchronous work: message consumers, batch pipelines, webhooks, and background jobs, where nobody is waiting, the budget is minutes rather than milliseconds, and a queue provides natural backpressure. In synchronous request paths, retry should be conservative — one or two attempts, tight timeouts, always bounded by the caller's deadline.

When NOT to use it

• The error is deterministic. A 400, 401, 403, 404, or 422 will fail identically on every attempt. Retrying a validation error is pure waste that converts a fast, clear failure into a slow, confusing one. Classify errors and retry only the retryable class.
• The operation is not idempotent and cannot be made idempotent. Sending an email, charging a card, or appending a non-deduplicated ledger entry — without an idempotency key, a retry after a timeout risks a duplicate side effect. Correctness beats availability here; fail and surface it.
• The dependency is in sustained failure. Retry assumes the next attempt might work. When a dependency has been down for minutes, every retry is a wasted round trip and added load. This is exactly the boundary where a circuit breaker takes over — see Related patterns below.
• The caller's deadline has effectively expired. If the remaining budget is smaller than the expected duration of one more attempt, do not start it.
• You are retrying on top of a layer that already retries. Nested retry policies multiply. Decide, deliberately, on exactly one layer that owns retry for a given call and make the others fail through.

Architecture

A correct retry policy is four decisions composed: classify the error, check the budget and deadline, wait for a jittered backoff interval, then re-attempt — with a hard cap on attempts.

flowchart TD
    A[Call attempt] --> B{Success?}
    B -->|Yes| C[Return result]
    B -->|No| D{Retryable error class?}
    D -->|No — 4xx, validation| E[Fail immediately]
    D -->|Yes| F{Attempts left AND deadline remains AND budget allows?}
    F -->|No| G[Fail: exhausted]
    F -->|Yes| H[Sleep: jittered exponential backoff]
    H --> A

The backoff schedule is where most of the engineering lives. Exponential backoff — doubling the delay after each failure — gives a struggling dependency geometrically more room to recover. But exponential backoff alone still synchronizes callers, because they all double from the same base at the same moment. Jitter is the fix: randomize each delay so a herd of callers spreads across the window instead of arriving together.

sequenceDiagram
    participant C as Caller
    participant D as Dependency
    C->>D: attempt 1
    D--xC: 503 (overloaded)
    Note over C: sleep random(0, 200ms)
    C->>D: attempt 2
    D--xC: 503
    Note over C: sleep random(0, 400ms)
    C->>D: attempt 3
    D-->>C: 200 OK

The variant shown — sleeping a uniformly random duration between zero and the exponential ceiling — is full jitter, and it is the one to reach for by default. It minimizes the peak concurrent load on the recovering dependency, at the cost of occasionally retrying sooner than a pure exponential schedule would.

One structural point that matters more than the schedule: retry belongs inside the timeout for a single attempt and outside nothing else. Each attempt gets its own timeout; the whole policy is bounded by the caller's deadline. A retry loop without a per-attempt timeout can hang forever on attempt one and never reach its own backoff logic.

Code walkthrough

Here is a production-shaped retry in TypeScript with no dependencies. Start with the two things every correct policy needs and most naive ones lack: an explicit notion of which errors are retryable, and a deadline.

interface RetryPolicy {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  isRetryable: (error: unknown) => boolean;
}

class RetryExhaustedError extends Error {
  constructor(readonly attempts: number, readonly lastError: unknown) {
    super(`Retry exhausted after ${attempts} attempts`);
    this.name = "RetryExhaustedError";
  }
}

isRetryable is deliberately injected rather than hardcoded. Retryability is a property of your dependency's error vocabulary — an HTTP client, a database driver, and a message broker each signal transience differently, and a policy that guesses will retry validation errors and give up on throttling.

Next, the backoff calculation. This is full jitter: an exponential ceiling that grows with the attempt number, capped, with the actual delay drawn uniformly from zero to that ceiling.

function backoffDelayMs(attempt: number, policy: RetryPolicy): number {
  const exponentialCeiling = Math.min(
    policy.maxDelayMs,
    policy.baseDelayMs * 2 ** (attempt - 1),
  );
  return Math.random() * exponentialCeiling;
}

The Math.random() factor is the entire anti-herd mechanism. Remove it and a thousand callers that failed together will retry together, forever, in lockstep waves.

Now the loop itself. Note what it checks before sleeping — a policy that sleeps first and checks the deadline afterwards wastes the budget it was supposed to protect.

async function withRetry<T>(
  operation: (signal: AbortSignal) => Promise<T>,
  policy: RetryPolicy,
  deadline: AbortSignal,
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 1; attempt <= policy.maxAttempts; attempt++) {
    try {
      return await operation(deadline);
    } catch (error) {
      lastError = error;

      const isLastAttempt = attempt === policy.maxAttempts;
      if (isLastAttempt || !policy.isRetryable(error) || deadline.aborted) {
        throw error;
      }

      await sleep(backoffDelayMs(attempt, policy), deadline);
    }
  }

  throw new RetryExhaustedError(policy.maxAttempts, lastError);
}

Three guards share one if, and each closes a distinct failure mode. isLastAttempt enforces the hard cap. !policy.isRetryable(error) fails fast on deterministic errors instead of burning the budget. deadline.aborted stops the moment the caller no longer cares — without it, a retry loop happily continues working on a request whose user gave up two seconds ago.

The sleep must be interruptible, or the deadline check above is decorative:

function sleep(ms: number, signal: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(signal.reason);
    }, { once: true });
  });
}

Classification for an HTTP dependency, showing the distinction that matters:

const httpRetryPolicy: RetryPolicy = {
  maxAttempts: 3,
  baseDelayMs: 100,
  maxDelayMs: 2_000,
  isRetryable: (error) => {
    if (error instanceof TypeError) return true; // fetch network failure
    if (!(error instanceof HttpError)) return false;
    return error.status === 429 || error.status === 503 || error.status >= 500;
  },
};

A 400 is not retried; a 503 is. That single function is the difference between a policy that recovers from blips and one that hammers a server about a malformed request.

Finally, idempotency — the correctness half of the pattern, and the half that gets skipped. The key is generated once, outside the retry loop, so that all attempts of the same logical operation carry the same key and the server can collapse duplicates:

async function chargeCustomer(amount: Money, deadline: AbortSignal) {
  const idempotencyKey = crypto.randomUUID(); // once per logical charge

  return withRetry(
    (signal) => paymentsClient.charge(amount, { idempotencyKey, signal }),
    httpRetryPolicy,
    deadline,
  );
}

Move that randomUUID() inside the loop and you have rebuilt the double-charge bug with extra steps.

Two refinements belong in a mature system. A retry budget — a token bucket, shared across the process, that caps retries at some fraction (commonly 10%) of base request volume — converts a retry storm into graceful shedding: when everything is failing, most calls stop retrying automatically. And Retry-After awareness: when a dependency tells you when to return, that instruction should override your computed backoff, because the dependency knows more about its own recovery than your exponent does.

Performance characteristics

The figures above are indicative, meant to calibrate intuition rather than describe a specific benchmark.

Recovery rate. For genuinely transient faults, the first retry does nearly all the work. A dropped connection or a rescheduled pod resolves on the next attempt the overwhelming majority of the time, which is why the marginal value of attempts falls off a cliff: attempt two is enormously valuable, attempt three modestly so, attempt six almost never. Policies with maxAttempts: 10 are usually a sign that retry is being used to paper over a dependency that is actually broken.

Amplification. This is retry's characteristic danger and the reason budgets exist. A 3-attempt policy has a worst-case amplification factor of 3× — and that worst case arrives precisely during an incident, when every call is failing. A retry budget breaks the feedback loop by making the retry rate a function of success volume rather than failure volume: when the success rate collapses, so does the permitted retry rate.

Jitter and peak load. Without jitter, N callers failing simultaneously produce a retry spike of N concurrent requests at each backoff boundary. With full jitter over a window W, those N retries spread across W, cutting instantaneous peak load by roughly the ratio of the window to the arrival width. This is the cheapest reliability improvement in the entire pattern — one multiplication by a random number — and the most commonly omitted.

Latency cost. Retry trades latency for success rate, and the trade is only sometimes good. A successful retry converts a fast failure into a slower success, which users generally prefer. An unsuccessful retry sequence converts a fast failure into a slow failure, which is strictly worse than failing immediately. This asymmetry is why deadline propagation matters: the goal is never to spend budget on attempts that cannot land in time.

Live demo

An interactive retry sandbox is coming soon. You will be able to set a dependency's failure rate and recovery profile, then watch fixed-delay, exponential, and full-jitter policies side by side — observing the herd waves that fixed delay produces, the amplification factor climbing during an induced overload, and a retry budget cutting in to shed retries and let the dependency recover. Check back shortly.

Related patterns

Circuit Breaker is retry's necessary counterpart, and the pair is frequently misunderstood as alternatives. They answer different questions on different timescales. Retry asks "was this attempt unlucky?" and responds by adding load. A breaker asks "is this dependency broken?" and responds by removing load. A system with retry but no breaker keeps paying for attempts long after the answer is clearly no. A system with a breaker but no retry fails on blips a single re-attempt would have absorbed. Layer them with the breaker on the outside: the breaker sees the aggregate health signal and, when open, prevents the retry loop from running at all. Invert that order and the retry loop runs to exhaustion inside a breaker that never gets clean signal, which is the configuration that produces retry storms in production.

Message Queue makes retry dramatically safer by changing where it happens. A broker with redelivery and a dead-letter queue gives you durable, out-of-band retry with an enormous budget and natural backpressure — nobody is holding a connection open, and poisoned messages land in the DLQ instead of cycling forever. When an operation can be made asynchronous, moving retry from the request path to a consumer is usually a better answer than tuning the synchronous policy.

Saga is what you reach for when retry cannot succeed. Retry handles a step that failed transiently; a saga handles a step that failed permanently by compensating the steps already committed. In practice a saga step retries first and compensates only once retry is exhausted — the two nest naturally, with retry as the inner loop.

API Gateway and service meshes are where retry policy increasingly lives, and for good reason: centralizing it makes budgets global rather than per-caller, which is the only place a budget can genuinely prevent a storm. The caveat is that a mesh retrying beneath an application that also retries reintroduces multiplication — if the platform owns retry, application code must stop.