Rate Limiting

Every rate limiter is a policy about fairness wearing an algorithm's clothing. The code is small — a counter and a clock — and it is genuinely not the hard part. The hard part is that a limiter's entire job is to refuse someone, and every design decision in it is really a decision about who.

Get that framing wrong and you build something that measures traffic accurately and protects nothing: it throttles the customer whose integration is working correctly while the runaway script that caused the incident sails through on a different key.

The problem

A service has finite capacity. Demand does not respect it.

The demand that exceeds it is rarely malicious. In order of how often they actually cause incidents: a customer's nightly batch job that used to run at 10 requests per second and now runs at 3,000 after their own scaling change; a retry storm from a downstream service that has no retry budget; a crawler; a mobile client with a bug that polls in a tight loop after a failed response; and only then, occasionally, someone hostile.

Without a limit, the outcomes are all bad and all indiscriminate:

• Everyone degrades together. The service does not slow down for the noisy caller specifically; it slows down for everybody, so one caller's bug becomes every caller's outage.
• Costs follow traffic. Serverless billing, per-request database costs, and third-party API quotas all scale with the flood. Auto-scaling will faithfully spend real money serving a client's infinite loop.
• Queues grow instead of failing. Unbounded queuing converts an overload into latency and then into memory exhaustion — a slower, more expensive way to fail than refusing early.

So you add a limit. And immediately the real question arrives, because a limit is worthless without a key: rate-limited per what? Per IP address punishes an entire office behind one NAT and does nothing about a distributed source. Per API key is usually right and is exactly what a legitimate high-volume customer trips first. Per user is right for interactive traffic and unavailable before authentication — which is precisely where you most want a limit, since login and password-reset endpoints are the abuse targets.

The key is the policy. The algorithm is an implementation detail of it, and choosing the algorithm before choosing the key is the most common way to build a limiter that measures the wrong thing very precisely.

When to use it

• A shared resource serves callers who can each affect the others. Any multi-tenant API. This is the base case, and the limit is what makes tenancy safe.
• Some traffic is worth refusing. Free-tier consumption of an expensive endpoint, crawlers, batch jobs during peak hours. If all traffic is equally valuable, a limiter can only choose which valuable traffic to lose, and provisioning is the better answer.
• Downstream costs are real and per-request. Third-party API quotas, per-invocation billing, a database that gets expensive under concurrency. Here the limiter is a budget control as much as a resilience control.
• Abuse-sensitive endpoints exist. Login, password reset, signup, anything that sends an email or an SMS. These want tight per-identity and per-IP limits, and this is one of the few cases where a limiter is genuinely a security control.
• You need a fairness guarantee you can state. "No tenant may consume more than 10% of capacity" is a property a limiter can enforce and nothing else can.

When NOT to use it

• Capacity is the problem and the traffic is legitimate. A limiter protecting an under-provisioned service converts an infrastructure shortfall into a customer-visible refusal, and it does so silently. Scale first; limit to protect a correctly sized system from abnormal demand.
• The caller cannot handle a refusal. A 429 is only useful if someone acts on it. A client that treats 429 as a generic error and retries immediately makes the overload worse — the limiter now costs a rejection per attempt and the attempt rate has gone up. If you cannot ship Retry-After handling into the clients, a queue with backpressure may serve you better.
• A concurrency limit is what you actually need. These are different questions and they are constantly conflated. Rate limiting bounds requests per unit time; a bulkhead bounds requests in flight. If the resource being exhausted is threads or connections, ten slow requests can exhaust it while sitting far under any per-second rate. Limit the rate to control cost and fairness; limit the concurrency to control resource exhaustion.
• The limit would be per-node in a fleet you cannot coordinate. N nodes each enforcing the limit locally admit up to N times it. Either accept that arithmetic explicitly and set the per-node limit to limit/N, or share state — but do not ship a "1,000 requests per second" limit across twenty nodes and believe it means 1,000.

Architecture

Four algorithms are in common use, and they differ in exactly one respect that matters: what they do at a boundary.

flowchart LR
    A[Request] --> B{Key: tenant, user, IP}
    B --> C[Token bucket]
    C -->|token available| D[Serve, decrement]
    C -->|empty| E[429 + Retry-After]
    F[Refill: rate/sec] -.->|up to capacity| C

Fixed window — count requests per calendar minute, reset at the boundary. Trivially simple, and it admits twice the limit across a boundary: 100 requests at 10:00:59 and 100 more at 10:01:00 is 200 in one second, entirely within policy. For a limiter whose job is protecting capacity, that is a hole in the middle of the thing.

Sliding window log — store a timestamp per request, count those inside the trailing window. Exactly correct, and it stores one entry per request per key, which is unbounded memory proportional to your traffic. Correct and unaffordable at any scale where it matters.

Sliding window counter — keep the current and previous fixed windows and interpolate between them by how far into the current window you are. Two counters per key, no boundary doubling, small and bounded error. This is the pragmatic default for most APIs.

Token bucket — a bucket of capacity C refilling at R tokens per second; each request takes one. This is the one to reach for when traffic is bursty, because capacity and rate are separate knobs: R is the sustained rate you will allow, C is how large a burst you will absorb. A client averaging well under the limit but arriving in clumps — which describes almost every real client — is served rather than refused.

sequenceDiagram
    participant C as Client
    participant L as Limiter
    participant S as Service
    C->>L: 50 requests (burst)
    L->>S: 50 served (bucket had 50)
    C->>L: request 51
    L--xC: 429, Retry-After: 1
    Note over L: refills at 20/s
    C->>L: retry after 1s
    L->>S: served

The distinction between the last two is the practical one: a sliding window says "no more than N per interval, evenly"; a token bucket says "N per interval on average, and I will forgive a burst of C". Real clients burst. A limiter that refuses bursts it had the capacity to serve is a limiter that generates support tickets.

Code walkthrough

A token bucket does not need a background timer, and implementations that use one are doing extra work to be less accurate. Refill lazily from the clock:

export interface BucketState {
  readonly tokens: number;
  readonly lastRefillMs: number;
}

export interface BucketPolicy {
  readonly capacity: number;
  readonly refillPerSecond: number;
}

export function takeToken(
  state: BucketState,
  policy: BucketPolicy,
  nowMs: number,
): { allowed: boolean; state: BucketState; retryAfterMs: number } {
  const elapsedMs = Math.max(0, nowMs - state.lastRefillMs);
  const refilled = Math.min(
    policy.capacity,
    state.tokens + (elapsedMs / 1000) * policy.refillPerSecond,
  );

  if (refilled < 1) {
    const deficitMs = ((1 - refilled) / policy.refillPerSecond) * 1000;
    return {
      allowed: false,
      state: { tokens: refilled, lastRefillMs: nowMs },
      retryAfterMs: Math.ceil(deficitMs),
    };
  }

  return {
    allowed: true,
    state: { tokens: refilled - 1, lastRefillMs: nowMs },
    retryAfterMs: 0,
  };
}

This is a pure function of state, policy and time — no clock, no storage, no I/O — which is what makes the boundary behaviour testable by passing timestamps rather than sleeping. Four details earn their place:

Math.max(0, ...) on elapsed time. A clock that steps backwards (NTP correction, a container's monotonic clock reset) would otherwise produce a negative refill and silently remove tokens from every bucket at once.

Clamping to capacity. Without it, an idle key accumulates tokens forever and returns after a quiet hour able to fire an unlimited burst. The clamp is what makes C mean "maximum burst".

Fractional tokens are kept. Rounding down on each call loses a fraction every time and drifts the effective rate below the configured one — a limiter that is 8% stricter than advertised, for no stated reason.

retryAfterMs is computed, not guessed. It is the exact time until one token exists. Returning it is what makes the limit cooperative instead of adversarial: a client that respects it returns precisely when it can be served, rather than hammering and being refused repeatedly.

The HTTP boundary is where a limiter becomes usable or useless:

export function rateLimit(deps: LimiterDeps) {
  return async (c: Context, next: Next) => {
    const key = deps.keyOf(c);
    const policy = deps.policyFor(key);
    const verdict = await deps.store.take(key, policy, deps.now());

    c.header('RateLimit-Limit', String(policy.capacity));
    c.header('RateLimit-Remaining', String(Math.floor(verdict.tokens)));
    c.header('RateLimit-Reset', String(Math.ceil(verdict.resetMs / 1000)));

    if (!verdict.allowed) {
      c.header('Retry-After', String(Math.ceil(verdict.retryAfterMs / 1000)));
      return c.json(
        { ok: false, error: 'rate_limited', retryAfterSeconds: Math.ceil(verdict.retryAfterMs / 1000) },
        429,
      );
    }

    await next();
  };
}

The headers go out on every response, not just the refused ones. A client can only pace itself if it can see its remaining budget before it hits zero; telling it only at the moment of refusal guarantees the refusal.

policyFor(key) rather than a constant is the other thing worth building in from the start. Limits are per-tier in every system that survives contact with customers, and retrofitting per-key policy into a limiter with a hardcoded constant means touching every call site.

Distributed state is where correctness is usually lost. Per-node in-memory buckets across twenty nodes enforce twenty times the limit. The standard fix is a shared store with an atomic read-modify-write — in Redis, a small Lua script, because GET then SET from concurrent nodes loses updates and admits more than the limit under exactly the load that made you want a limiter. The cost is a network round trip on every request, which is why high-volume systems often accept per-node limits of limit/N and live with the imprecision when nodes are unevenly loaded.

Performance characteristics

A local token bucket is two arithmetic operations and a comparison — sub-100 µs, invisible next to any real request. A Redis-backed limiter adds a round trip: 0.5-2 ms in the same availability zone, and it puts Redis on the critical path of every request, which makes the limiter's availability the service's availability. Fail open, not closed, unless the endpoint is abuse-sensitive: a limiter that refuses everything when its store is unreachable has converted a Redis blip into a full outage, which is a strictly worse failure than briefly serving unlimited traffic.

State is 16 bytes per key for a token bucket versus one timestamp per request for a sliding window log — the difference between a million tracked keys costing megabytes and costing gigabytes.

Set limits from measurement, not intuition. Instrument first, look at the p99 of per-key request rates over a week, and set the limit above the legitimate peak with headroom — then watch what would have been refused before enforcing anything. Ship every limiter in observe-only mode first. Nearly every rate limiter that has caused an incident was correct arithmetic applied to a limit nobody had checked against real traffic, and the observe-only period is where you discover that your largest customer's normal Monday looks exactly like abuse.

Then alert on refusals by key. A single key at its limit is the system working. A sudden broadening in the number of distinct keys being refused is the signal that the limit is wrong, and it is invisible in an aggregate refusal count.

Live demo

A live token bucket — burst absorption, refill, Retry-After honoured by a well-behaved client and ignored by a badly-behaved one — is on the roadmap here. The pure function above is the load-bearing part, and its testability is the point: boundary behaviour is verified by passing timestamps, never by sleeping.

Related patterns

API Gateway is where rate limiting usually belongs, and for a structural reason rather than convenience: a limit is only a fairness guarantee if it is enforced at a point that sees all of a caller's traffic. Enforced per-service, a caller fanning out across five services gets five times the budget, and no service can tell. The gateway is also the only place that can refuse before the request costs anything downstream.

Bulkhead answers the neighbouring question and is not a substitute. Rate limiting bounds requests per unit time; a bulkhead bounds requests in flight. Ten simultaneous eight-second requests exhaust a connection pool while sitting far below any per-second rate limit, and a thousand fast requests per second blow a rate limit without ever exhausting a pool. Systems that need protection usually need both, on different axes.

Auto-scaling is the alternative to reach for when the traffic is legitimate. Scaling says "meet the demand"; limiting says "refuse it". The choice is economic and it is worth making explicitly: scale for demand you want to serve and can profitably serve, limit demand that is abusive, unprofitable, or growing faster than your ability to provision. The failure mode of choosing wrong is expensive in one direction and customer-visible in the other, and a limiter placed in front of an auto-scaler is what stops a client's infinite loop from becoming an unbounded bill.

Retry is the pattern most likely to defeat a limiter. A client retrying immediately on 429 converts one refusal into many and multiplies the load the limiter exists to shed. Retry-After is the contract between the two, and it only works if both sides honour it — which is why a limiter's headers are part of its design and not decoration.