Message Queue

Some work does not need to happen now — it needs to happen reliably, eventually, and without making the caller wait for it. A user uploads a video that needs transcoding; a checkout completes and an invoice PDF must be rendered and emailed; a batch import drops ten thousand records that each need enrichment. If you do that work inline, on the request that triggered it, you have chained the caller's latency to the slowest thing you do and coupled their success to a downstream that may be slow, overloaded, or briefly down. The upload times out because the transcoder is busy. Checkout fails because the mail server hiccuped. The import blocks the browser for a minute.

A message queue puts a durable buffer between the work and the workers. The producer writes a task to the queue and immediately moves on — its job is done the moment the task is safely stored. On the other side, a pool of workers pulls tasks off the queue, each task going to exactly one worker, and processes them at whatever pace the workers can sustain. The queue holds the backlog when producers outrun consumers, redelivers a task if the worker handling it dies, and lets you drain a spike faster simply by adding more workers. The producer is decoupled from the consumer in time (it does not wait), in space (it does not know which worker runs the task), and in rate (a burst of enqueues does not have to be a burst of processing).

The problem

Consider any workload where the thing that triggers work and the thing that does work run at different speeds. A request comes in fast; the work it implies is slow. Traffic is bursty; processing capacity is fixed. The obvious implementation — do the work inline, synchronously, on the triggering request — couples three things that want to be independent, and each coupling is a distinct failure mode:

• Latency coupling. The caller waits for the slowest step. A checkout that must render a PDF and call a mail server before returning is as slow as the PDF renderer and the mail server combined, on their worst day. Work that the user does not need to see happen is nonetheless charged to the user's wait.
• Availability coupling. The caller's success now depends on the downstream's health. If the transcoder is down, the upload fails — even though the upload itself succeeded and the transcoding could perfectly well have waited five minutes. A momentary outage in a non-critical dependency becomes a user-facing error in a critical path.
• Rate coupling. A spike of triggers becomes a spike of work with no shock absorber. Ten thousand imports arriving at once means ten thousand simultaneous enrichment calls hammering a database that can handle a few hundred. The load has nowhere to pool, so it lands all at once on whatever is least able to absorb it, and the system browns out precisely when it is busiest.

Underneath all three is a single mismatch: the producer's timeline is not the consumer's timeline, but synchronous processing forces them to be. What you want is for the trigger to record "this work needs doing" cheaply and durably, and for the doing of it to proceed independently — buffered when there is too much, parallelized when there is a backlog, retried when it fails, and never lost when a worker crashes mid-task. That buffer, with those semantics, is a message queue.

When to use it

A message queue fits when work is asynchronous, point-to-point, and worth doing reliably even if not immediately:

• The trigger does not need the result. The producer's obligation ends once the task is durably accepted. Whether the transcode finishes in two seconds or two minutes, and on which worker, is not the uploader's concern. Fire-and-forget with a durability guarantee is exactly the queue's shape.
• Producer and consumer run at different, uncorrelated rates. Bursty ingestion, steady processing — or the reverse. The queue is a buffer that lets each side run at its own natural pace, holding the backlog when they diverge and draining it when they reconverge. This is load leveling: converting a spike in arrival into a smooth, sustainable rate of work.
• The work parallelizes across interchangeable workers. Each task is independent and can be handled by any worker in the pool. Because a queue delivers each message to exactly one consumer, adding workers adds throughput with no coordination code — the competing-consumers model gives you horizontal scale for free.
• Failures should be retried, not dropped. A durable queue redelivers a task until it is acknowledged. A worker can crash halfway through and the task survives, to be picked up by another worker. Work that must not be silently lost — payments, provisioning, anything with a side effect that matters — wants this at-least-once floor.
• You need to protect a fragile or rate-limited downstream. The queue is a natural throttle. However fast tasks arrive, the workers pull them at a rate the downstream can survive; a third-party API with a strict quota, or a database with a modest write ceiling, sits safely behind a queue that meters the pressure.

When NOT to use it

The queue buys asynchrony and durability, and both cost something. Avoid it when:

• The caller needs the result to proceed. A price quote, an authorization decision, a validated write the user is waiting on — that is request/response, and a queue is the wrong tool. Bolting a "reply queue" and a correlation id onto a broker to simulate a synchronous call reinvents RPC with more moving parts, worse latency, and worse debuggability. Make the call.
• One event has many interested consumers. A queue delivers each message to exactly one worker — that is its defining property and the whole point of competing consumers. If a single business fact needs to fan out to several independent subsystems, you want publish/subscribe, where every subscriber gets its own copy. Forcing fan-out through a work queue means either one consumer that does everything or a fragile web of queues you feed by hand.
• The work is trivial and reliably fast. If a step takes a millisecond and essentially never fails, routing it through a broker adds a durable write, a network hop, a worker fleet, and a new operational surface to buy back nothing. Asynchrony is worth its complexity only when the work is slow, bursty, or failure-prone enough to justify decoupling. Introduce the queue when inline processing actually hurts, not speculatively.
• You require strict end-to-end ordering at high throughput. Standard queues optimize for throughput and make no ordering promise; strict-FIFO queues preserve order but only within an ordering key and at a fraction of the throughput, because ordering and parallelism are in direct tension. If global order is a correctness requirement, the competing-consumers model — many workers pulling concurrently — is fighting you, and you must give up either the ordering or the parallelism.
• Latency end-to-end must be tight and predictable. A queue trades immediacy for resilience. A task may sit in the buffer behind a backlog, wait out a visibility timeout after a failed attempt, and only then complete. For a path with a hard, low latency budget, that variable tail is a defect, not a feature.

Architecture

A producer enqueues a task and returns. The broker stores it durably. Workers in a pool each pull tasks, and the broker hands any given task to exactly one of them, making it invisible to the others while that worker holds it. The worker processes the task and acknowledges it, at which point the broker deletes it. If the worker fails to acknowledge within a visibility timeout — because it crashed, hung, or errored — the broker makes the task visible again and another worker picks it up. A task that fails repeatedly is diverted to a dead-letter queue so one poison message cannot block the stream forever.

flowchart LR
  P[Producer<br/>enqueue + return] --> Q[(Queue<br/>durable buffer)]
  Q -->|exactly one| W1[Worker 1]
  Q -->|exactly one| W2[Worker 2]
  Q -->|exactly one| W3[Worker 3]
  W1 -->|ack| Q
  W2 -->|ack| Q
  W3 -.repeated failure.-> DLQ[(Dead-letter<br/>queue)]

The lifecycle of a single message is where the delivery guarantee actually lives. The message is not deleted when it is delivered — it is hidden, then deleted only when it is acknowledged. That gap between delivery and acknowledgement is what makes the queue survive a crashed worker, and it is also why duplicates are possible.

sequenceDiagram
  participant P as Producer
  participant Q as Queue
  participant W as Worker
  P->>Q: enqueue(task)
  Note over Q: stored durably
  W->>Q: pull
  Q-->>W: task (now invisible for T)
  alt worker acknowledges in time
    W->>Q: ack
    Note over Q: task deleted
  else worker crashes / times out
    Note over Q: visibility timeout expires
    Q-->>W: task redelivered to another worker
  end

The structural decision that matters most is the same one the hexagon teaches everywhere else: the worker is a thin adapter, and the work is a pure function it calls. The broker's message shape and the ack protocol are just a boundary. Parse the message into a domain command, run the domain logic, and let the adapter translate the Result back into an ack or a retry. The business logic never learns it was triggered by a queue rather than an HTTP request or a test.

Code walkthrough

Start with the work as a pure domain function — it has no idea a queue exists. It takes a command, returns a Result, and could run inline, in a test, or behind any transport unchanged:

interface TranscodeJob {
  readonly uploadKey: string;
  readonly profile: 'sd' | 'hd';
}

type TranscodeResult =
  | { ok: true; renditionKey: string }
  | { ok: false; reason: 'source-missing' | 'unsupported-codec' };

async function transcode(job: TranscodeJob, media: MediaStore): Promise<TranscodeResult> {
  const source = await media.get(job.uploadKey);
  if (!source) return { ok: false, reason: 'source-missing' };
  if (!source.isTranscodable()) return { ok: false, reason: 'unsupported-codec' };

  const rendition = source.encodedAs(job.profile);
  const renditionKey = derivedKey(job.uploadKey, job.profile);
  await media.put(renditionKey, rendition);
  return { ok: true, renditionKey };
}

The producer side is almost nothing: build the task, enqueue it, return. It does not wait for the transcode, and it does not know which worker will run it. The endpoint stays fast and stays up even when the transcoder pool is saturated — the backlog simply grows and drains later:

export async function requestTranscode(uploadKey: string, queue: TaskQueue): Promise<void> {
  await queue.enqueue({ uploadKey, profile: 'hd' });
}

The worker is the adapter. It pulls a message, parses it into a domain command, runs the pure function, and — critically — translates the outcome into the broker's protocol: acknowledge on success so the message is deleted, and on a transient failure let the message become visible again for retry rather than acknowledging a task that did not actually complete:

async function handleMessage(message: QueueMessage, media: MediaStore): Promise<Ack> {
  if (isDuplicate(message.id)) return 'ack';

  const result = await transcode(toJob(message.body), media);
  switch (result.ok) {
    case true:
      recordProcessed(message.id);
      return 'ack';
    case false:
      return result.reason === 'source-missing' ? 'ack' : 'retry';
  }
}

Note the two different failure dispositions, because they are the crux of using a queue correctly. A source-missing task is permanently unprocessable — retrying it forever accomplishes nothing, so acknowledge it (and, in a real system, route it to a dead-letter queue for inspection). A transient fault — the media store briefly unavailable — should not be acknowledged, so the visibility timeout returns the task for another attempt. Acknowledging on transient failure silently drops work; retrying on permanent failure wedges the queue behind a poison message. The disposition is a domain judgement the adapter makes from the Result, and getting it wrong is how queues lose data or grind to a halt.

The duplicate check on the first line is not optional bookkeeping — it is the price of at-least-once delivery, which the next section is about.

Delivery semantics and idempotency

A durable queue's default guarantee is at-least-once: every task is delivered until it is acknowledged, so nothing is lost when a worker dies mid-task — but a task can be delivered more than once. The exact scenario is mundane and unavoidable: a worker pulls a task, processes it fully, and crashes in the instant before its acknowledgement lands. The broker never heard the ack, the visibility timeout expires, and the task is redelivered to another worker. The work runs twice.

This is not a bug you can configure away; it is a consequence of the same mechanism that makes the queue reliable. The only robust response is to make processing idempotent — safe to run more than once with the same effect as running it once. Key the write by an idempotency token, use a conditional put, dedupe on the message id (the isDuplicate / recordProcessed pair above). If handling the same task twice double-charges a card, sends two emails, or writes a duplicate row, you have a latent data-integrity bug that surfaces exactly when the system is already under stress and redelivering. Idempotency is not a nicety on a message queue; it is a correctness requirement the delivery guarantee imposes on you.

The stronger-sounding alternative, exactly-once, is mostly a marketing word for at-least-once delivery plus consumer-side deduplication — the broker still delivers more than once; the effect is made to happen once by the idempotency you wrote. Some platforms provide dedup windows or transactional acks that narrow the gap, but the durable, portable assumption is at-least-once, and the durable, portable defense is an idempotent handler.

Backpressure and load leveling

The queue's depth — how many tasks are waiting — is the pattern's most useful and most underrated signal. It is producers outrunning consumers, made visible as a single number, and it does two jobs at once.

First, the depth is a shock absorber. A spike in arrivals raises the depth instead of raising the load on the workers. The workers keep pulling at their sustainable rate; the excess pools in the queue and drains once the spike passes. This is load leveling: the workers never see the spike, only a slightly longer backlog. A downstream that would have toppled under a synchronous burst instead sees a smooth, bounded stream — the queue converts a peak into a plateau.

Second, the depth is a control signal. A queue that is growing means the workers cannot keep up; a queue that is empty means they are over-provisioned. This is precisely the input an autoscaler wants — scale the worker pool on queue depth (or on the age of the oldest message) and the system self-tunes to its backlog, adding competing consumers when work piles up and shedding them when it drains. It is a cleaner scaling signal than CPU: queue depth measures unfinished work directly, whereas CPU is a lagging proxy for it. This is the seam where the message queue and auto-scaling compose into a self-regulating pipeline.

There is a discipline the buffer demands in return: an unbounded queue in front of a permanently-too-slow consumer does not fix the mismatch, it hides it — the backlog grows without limit, latency climbs toward infinity, and the "buffer" becomes a slowly-filling reservoir of stale work. A queue absorbs transient imbalance. A sustained one is a capacity problem the queue can only defer, and the depth metric is exactly what tells you which you have.

Performance characteristics

The metrics above are indicative — orders of magnitude to build intuition, not measurements from a specific broker.

Enqueue latency is the producer's whole cost, and it is small. Publishing a task is one durable write; the producer returns in single-digit milliseconds regardless of how long the work itself takes. This is the number that justifies the pattern for latency-sensitive callers: the expensive work is moved off the critical path, and the caller pays only for the act of recording that it needs doing.

Throughput scales with the number of competing consumers. Because each message goes to exactly one worker, a backlog drains faster by adding workers — up to the throughput ceiling of the broker itself and of whatever shared downstream the workers all hit. This is horizontal scaling with no coordination code: the queue is the work distributor, and the workers need not know about each other.

The delivery guarantee is at-least-once, which trades duplicates for durability. You will not lose a task to a crashed worker; you may process a task twice. The cost of the guarantee is the idempotency you must build into the consumer — real engineering effort, but the honest price of never silently dropping work that mattered.

Buffer depth is the tuning surface. It is the shock absorber that keeps a spike off the workers and the signal an autoscaler steers on — but it is bounded by what you provision and by your tolerance for staleness. Depth and end-to-end latency move together: a deep queue means tasks wait, and past some depth a "buffered" system is really just a slow one. Watching the depth, and its rate of change, is watching the health of the whole pipeline.

Live demo

An interactive Message Queue playground is coming soon. You will be able to drive a stream of tasks into a queue and watch the buffer breathe: producers enqueuing faster than a fixed worker pool can drain, the depth climbing as the backlog pools, and the workers pulling steadily underneath — load leveling made visible. A slider will let you add and remove competing consumers and see the backlog drain in proportion; a "worker crash" button will kill a worker mid-task and show the message reappear and complete elsewhere, at-least-once delivery in action; and an autoscaling toggle will bind the worker count to the queue depth so you can watch the pipeline self-regulate against a spike. Check back shortly.

Related patterns

Pub/Sub is the pattern a message queue is most often confused with and most sharply contrasted against — they are the two halves of asynchronous messaging. A queue is point-to-point: each message is consumed by exactly one worker, and the model is work distribution — spread a backlog of tasks across a pool of interchangeable consumers. Pub/sub is fan-out: each message is delivered to every subscriber, and the model is event notification — tell many independent subsystems that something happened. Reach for a queue when you have work that needs doing once, by someone; reach for pub/sub when you have a fact that many parties need to hear. The machinery underneath rhymes — a durable broker, at-least-once delivery, dead-letter queues, idempotent consumers — but the delivery cardinality is the whole difference, and choosing the wrong one means either duplicated work or missed notifications.

Auto-scaling and the message queue compose into a self-regulating pipeline, because queue depth is the cleanest scaling signal there is. Depth measures unfinished work directly, where CPU utilization is only a lagging proxy for it; scale the worker pool on the backlog (or on the age of the oldest message) and the system adds competing consumers when work piles up and sheds them when it drains. The queue absorbs the spike while the autoscaler grows the capacity to drain it — the buffer buys the time that the scaler needs to react.

Serverless turns a message queue into a trigger. A function subscribed to a queue is invoked once per message, scales its concurrency with the queue depth, and idles to zero when the queue is empty — the competing-consumers model with the worker fleet handed entirely to the provider. It also inherits the queue's contract: at-least-once delivery makes handler idempotency mandatory, and the queue decouples a burst of enqueues from the rate at which functions drain it. Queue-plus-function is the canonical shape of event-driven async work with no fleet to run.

Saga rides on message queues to make a multi-step distributed workflow reliable. Each step of a saga — and each compensating action when a step fails — is a durable message a worker processes and acknowledges, so a crash mid-workflow resumes rather than corrupts. The queue supplies the at-least-once delivery and the retry semantics; the saga supplies the orchestration, state, and compensation that raw message-passing lacks. The pattern is the workflow logic; the queue is the reliable transport it runs on.