Load Balancing

Load balancing is the practice of spreading incoming work across several backends so that no single one becomes the bottleneck and no single failure becomes an outage. It is the most universally deployed pattern in this catalog and the one most often treated as a solved checkbox — you put a load balancer in front of your service, choose round robin because it is the default, and move on.

That works until it does not, and the failures are instructive. A backend that is up but slow will happily accept its full share of round-robin traffic and turn your p99 into a cliff. A health check that only tests TCP connectivity will keep routing to a process that has deadlocked its worker pool. A "smarter" algorithm can behave worse than a dumb one under partial failure, because the fastest backend is often the one that is fast because it is failing immediately. Load balancing is where distribution theory meets operational reality, and the reality has sharp edges.

The problem

One instance of a service can handle a bounded amount of work and can fail. Both facts push toward running several, which immediately raises the question this pattern answers: which backend should this request go to?

The naive answer — pick them in turn — is a decent default and quietly assumes a great deal. Round robin is optimal when every request costs the same and every backend is equally capable. Real systems violate both.

Requests are not uniform. In a typical API, most requests are cheap and a few are enormously expensive. Round robin distributes request counts evenly, not work. Give one backend three consecutive heavy queries by chance and it is saturated while its peers idle, having received exactly its fair share of requests.

Backends are not identical. Even in a homogeneous fleet, instances differ: one sits on a noisy neighbor, one has cold caches after a restart, one is mid-garbage-collection, one runs on a slightly older instance type after an autoscaling event. Treating them as interchangeable sends work to whichever is currently least able to do it.

Failure is rarely binary. The failure mode that hurts most is not a backend that is down — that one is easy, you stop routing to it. It is the backend that is sick: responding, passing health checks, and taking ten times as long. Every request routed there consumes a caller's connection and blows the latency budget. Worse, a backend failing fast — returning 500s in a millisecond — looks like the least-loaded, fastest-responding member of the fleet to any naive load-aware algorithm, so the smarter the balancer, the more traffic it funnels into the black hole. This is the black hole or death star failure, and it is the canonical argument for treating load metrics with suspicion.

Distributed balancers make inconsistent decisions. Several load balancers, or many clients doing client-side balancing, each hold a partial view. All of them independently concluding that backend 7 is least loaded will collectively overwhelm backend 7 — the herd behavior that load-aware algorithms are supposed to prevent, reintroduced by the fact that each decider acts on stale information.

When to use it

Load balancing is effectively mandatory whenever you run more than one instance of anything, but the deliberate choices worth making arise when:

• You need horizontal scale. More capacity means more replicas, and replicas need traffic distribution. This is the base case and needs little justification.
• You need availability across instance failure. A balancer with health checking turns an instance failure into a brief, partial degradation instead of an outage. For many systems this matters more than the scaling.
• Backends are heterogeneous or request costs vary widely. This is where algorithm choice stops being cosmetic. Least-connections or a latency-aware policy meaningfully outperforms round robin when work is unevenly sized.
• You are terminating TLS or routing on content. Path-based routing, header-based routing, canary weighting, and per-route timeouts all require an L7 balancer that understands HTTP.
• You need traffic shaping for releases. Weighted routing is the mechanism underneath canary releases and blue-green cutovers — the balancer is where a release strategy is actually implemented.

When NOT to use it

The pattern itself is not usually optional, but specific choices within it are frequently wrong:

• Do not add an L7 balancer where L4 suffices. Terminating and parsing HTTP costs latency and CPU. For raw TCP services, databases, or internal traffic that needs no content-based routing, an L4 balancer is substantially cheaper and simpler.
• Do not use sticky sessions to avoid fixing state. Session affinity is occasionally necessary but is fundamentally a scaling constraint: it defeats even distribution, breaks rebalancing on scale-out, and turns an instance failure into lost sessions for the users pinned to it. Externalizing session state is nearly always the better fix.
• Do not use a load-aware algorithm without failure-aware signals. As above — least-connections or least-latency, applied naively, actively concentrates traffic on failing backends. If you cannot distinguish "fast because healthy" from "fast because erroring," round robin is safer.
• Do not load balance across regions for latency without understanding data gravity. Sending a request to a distant region because it is less loaded is a loss if it then makes five cross-region database calls.
• Do not treat a load balancer as capacity management. It distributes load; it does not create capacity. A perfectly balanced overloaded fleet is still overloaded — that is auto-scaling's job, and conflating the two leads to teams tuning algorithms when they need more instances.

Architecture

Two structural choices dominate: where the balancing decision is made, and what information it uses.

flowchart TD
    subgraph Server-side
        C1[Clients] --> LB[Load balancer]
        LB --> B1[Backend 1]
        LB --> B2[Backend 2]
        LB --> B3[Backend 3]
    end
    subgraph Client-side
        C2[Client + embedded balancer] --> R[(Service registry)]
        C2 --> D1[Backend 1]
        C2 --> D2[Backend 2]
        C2 --> D3[Backend 3]
    end

Server-side balancing puts a dedicated hop in the path — an ELB, an nginx tier, a hardware appliance. It centralizes policy and is invisible to clients, at the cost of an extra network hop and a component that must itself be made highly available.

Client-side balancing pushes the decision into the caller, which resolves backends from a service registry and picks one directly. It removes the hop and the central bottleneck, which is why service meshes adopt it via sidecars — but each client now holds an independent, partial view of fleet health, which is precisely the condition that produces herd behavior.

The algorithm layer is where the real trade-offs live:

flowchart LR
    A[Request] --> B{Algorithm}
    B --> C[Round robin<br/>no state, ignores cost]
    B --> D[Least connections<br/>tracks in-flight, cost-aware]
    B --> E[Two random choices<br/>near-optimal, herd-resistant]
    B --> F[Consistent hashing<br/>cache affinity, key-stable]
    B --> G[Weighted<br/>heterogeneous capacity, canary]

Round robin is stateless and fair in request count. It is the right default and a poor fit for uneven request costs.

Least connections routes to the backend with fewest in-flight requests, which is a good proxy for actual load because expensive requests stay in flight longer. It is the best simple upgrade over round robin — and it is the algorithm most vulnerable to the black-hole failure, since a backend erroring instantly always has the fewest connections.

Power of two random choices is the quietly brilliant one: pick two backends at random, send the request to the less loaded of the two. It captures almost all the benefit of full load awareness while being robust to stale information, because no decider ever concludes "backend 7 is globally best" — randomness prevents synchronized herds by construction. This is the default in several modern meshes for good reason.

Consistent hashing routes by a key so the same key lands on the same backend, which is what makes local caches effective. Its virtue is that adding or removing a backend remaps only a small fraction of keys rather than reshuffling everything.

Weighted variants layer on top of any of the above and are how canary and blue-green traffic splitting is expressed.

Underneath every algorithm sits the mechanism that matters more than all of them: health checking.

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Suspect: check fails
    Suspect --> Unhealthy: N consecutive failures
    Suspect --> Healthy: check passes
    Unhealthy --> Probing: after eject interval
    Probing --> Healthy: M consecutive passes
    Probing --> Unhealthy: check fails

Two kinds run together in mature setups. Active health checks probe a dedicated endpoint on a schedule — simple, but they only test what the endpoint tests, and a shallow check that returns 200 from a process whose thread pool is exhausted is worse than useless. Passive checks (outlier detection) observe real traffic and eject backends that accumulate errors or latency, catching precisely the sick-but-answering case that active checks miss. The hysteresis in the diagram — N failures to eject, M successes to restore — exists to stop a backend flapping in and out of rotation on transient noise.

Code walkthrough

Balancing usually lives in infrastructure you configure rather than write, but the algorithms are short enough to make the trade-offs concrete. Start with the shared shape.

interface Backend {
  id: string;
  address: string;
  inFlight: number;
  healthy: boolean;
  weight: number;
}

interface LoadBalancingStrategy {
  pick(backends: Backend[]): Backend | undefined;
}

Every strategy filters to healthy backends first, and returning undefined when none are healthy is deliberate — failing fast when the whole pool is down is better than routing into a fleet you know is broken.

Round robin, with the detail people get wrong:

class RoundRobin implements LoadBalancingStrategy {
  private cursor = 0;

  pick(backends: Backend[]): Backend | undefined {
    const healthy = backends.filter((b) => b.healthy);
    if (healthy.length === 0) return undefined;
    return healthy[this.cursor++ % healthy.length];
  }
}

The cursor advances over the filtered list. Indexing into the full list and skipping unhealthy entries silently biases traffic toward whichever backends follow the unhealthy ones in the array — a real bug that shows up as inexplicable uneven distribution during partial outages.

Power of two random choices, which is barely longer and much better behaved:

class TwoRandomChoices implements LoadBalancingStrategy {
  pick(backends: Backend[]): Backend | undefined {
    const healthy = backends.filter((b) => b.healthy);
    if (healthy.length === 0) return undefined;
    if (healthy.length === 1) return healthy[0];

    const first = healthy[Math.floor(Math.random() * healthy.length)];
    const second = healthy[Math.floor(Math.random() * healthy.length)];
    return first.inFlight <= second.inFlight ? first : second;
  }
}

The randomness is load-bearing. A globally-informed "always pick the minimum" strategy makes every balancer in the fleet choose the same backend simultaneously; sampling two at random means the chance of many deciders converging on one backend is negligible, while still steering work away from busy nodes.

Now the guard that separates a toy from something you would run. Raw inFlight cannot distinguish a healthy idle backend from one returning instant errors, so the load signal must account for outcomes:

interface BackendHealth {
  recentErrorRate: number;
  recentLatencyMs: number;
}

const ERROR_PENALTY_FACTOR = 10;

function effectiveLoad(backend: Backend, health: BackendHealth): number {
  const errorPenalty = 1 + health.recentErrorRate * ERROR_PENALTY_FACTOR;
  return backend.inFlight * errorPenalty;
}

Multiplying by an error penalty makes a backend that fails 50% of requests look heavily loaded rather than attractively idle. This one adjustment is what stops least-connections and least-latency policies from funnelling traffic into a black hole, and its absence is a recurring cause of outages that look, from the graphs, like a load balancer deliberately targeting the broken instance — because it was.

Outlier detection closes the loop by ejecting sustained offenders from rotation entirely:

class OutlierDetector {
  private consecutiveFailures = new Map<string, number>();

  record(backendId: string, succeeded: boolean, pool: Map<string, Backend>): void {
    if (succeeded) {
      this.consecutiveFailures.set(backendId, 0);
      return;
    }

    const failures = (this.consecutiveFailures.get(backendId) ?? 0) + 1;
    this.consecutiveFailures.set(backendId, failures);

    if (failures >= EJECT_THRESHOLD) {
      const backend = pool.get(backendId);
      if (backend) backend.healthy = false;
      this.scheduleProbe(backendId, pool);
    }
  }
}

This is passive health checking: the verdict comes from real traffic rather than a synthetic probe, so it catches the backend whose /health endpoint returns 200 while its actual work path is broken. scheduleProbe must return the backend gradually — restoring it to full traffic immediately after an ejection window sends a burst at an instance that has just recovered and has cold caches, which frequently re-breaks it.

Performance characteristics

The figures above are indicative and describe shape rather than a specific benchmark.

L4 versus L7 is a real cost. An L4 balancer forwards packets after a connection-level decision; an L7 balancer terminates TLS, parses HTTP, and can route on path, header, or cookie. The per-request CPU difference is roughly an order of magnitude, and the added latency is small in absolute terms but not free. The rule: pay for L7 when you use L7 features, and use L4 for everything else.

Algorithm choice matters most under skew. With uniform request costs and identical backends, round robin is essentially optimal and nothing else can beat it meaningfully. As variance in request cost rises, the gap widens sharply — least-connections and two-random-choices track actual load while round robin tracks only counts. Two random choices is the notable result: it achieves maximum load close to the theoretical optimum with no shared state and no herd risk, which is why it is a better default than "least connections across a globally shared view."

Detection time is the availability bottleneck, not distribution. When a backend dies, every request routed to it during the detection window fails. That window is the health-check interval multiplied by the unhealthy threshold, and it dominates the availability calculation — a 10-second interval with three required failures means up to 30 seconds of routing into a dead instance. Tightening it costs probe traffic and raises the risk of ejecting healthy backends on transient blips, which is why passive detection on real traffic is such a good complement: it reacts within a few real requests instead of waiting for the next scheduled probe.

Session affinity costs distribution quality. Pinning users to backends means load follows user activity rather than balancer policy. A backend holding several heavy users stays hot while others idle, and rebalancing after a scale-out event cannot move existing sessions. The cost is real and grows with session lifetime.

Balancing does not create capacity. Perfect distribution across an undersized fleet yields uniformly overloaded backends. Distribution and capacity are orthogonal problems, and improving the algorithm when the real issue is insufficient instances produces graphs that look better while users see no change.

Live demo

An interactive balancer sandbox is coming soon. You will be able to run round robin, least connections, and two-random-choices against a fleet with configurable request-cost variance and watch the maximum-load gap open up — then inject the interesting failure: a backend that returns errors in one millisecond. The demo shows least-connections steering more traffic into it as it fails, and the error-penalty adjustment correcting that in real time. Check back shortly.

Related patterns

Auto-scaling is the complementary half and the contrast worth being precise about. Load balancing decides where a request goes among the capacity you have; auto-scaling decides how much capacity there is. Neither substitutes for the other, and the failure of conflating them runs both ways: teams tune balancing algorithms when the fleet is simply too small, and teams scale out to paper over a balancer that is concentrating traffic on a subset of instances. They also interact operationally — new instances from a scale-out event arrive cold, and a balancer that immediately gives them full share can push them over before their caches warm, which is why slow-start ramping exists.

Service Discovery supplies the input every balancer needs: the current set of backend addresses. In a static deployment this is a config file; in a dynamic one it is a registry that changes continuously as instances start, stop, and fail health checks. Client-side load balancing is essentially discovery plus an algorithm embedded in the caller, which is exactly the shape a service mesh sidecar implements.

API Gateway is frequently the same physical component as the edge load balancer, but the responsibilities are distinct and worth separating mentally: the gateway owns authentication, rate limiting, request transformation, and routing between services; the balancer owns distribution across instances of one service. Conflating them produces gateways with too many jobs and no clear ownership boundary.

Canary Release is implemented on top of weighted balancing. The canary controller is doing nothing more exotic than adjusting backend weights based on telemetry — which makes weighted routing the load balancer feature that most directly enables progressive delivery.

Circuit Breaker operates on the same signals from the other side of the call. Outlier detection in a balancer is, in effect, a circuit breaker per backend, run centrally rather than per caller. Where a caller-side breaker protects one client from one sick dependency, balancer-side ejection protects all clients at once — which is why pushing this concern into shared infrastructure tends to win over per-application breakers in mature deployments.