Message queues and pub/sub

Think of it like this

Analogy

A busy restaurant doesn't make the waiter stand in the kitchen until each dish is cooked. The waiter clips the order ticket to the rail and goes back to the tables. Cooks take tickets in order when they're free. If a rush comes in, tickets pile up on the rail rather than customers being turned away, and the manager can call in another cook.

That rail is a message queue. A pub/sub topic is more like the announcement “table 12 has arrived!”: the host, the waiter and the bartender all hear it and each does their own job.

The problem it solves

When one service calls another directly and waits, the two are tightly linked:

If the slow part takes seconds (sending email, resizing images, charging a card), the user waits.
If the downstream service is down or slow, the caller fails too.
A traffic spike hits every service in the chain at the same time.
Adding one more thing that should happen on an event means changing the caller.

A real example

Example

What happens after you click “Place order”

Checkout saves the order and puts an OrderPlaced message on a queue, then immediately shows you the confirmation page. It doesn't wait for anything else.

Three workers read from the queue. One sends the confirmation email, one reserves stock in the warehouse, and one builds the invoice PDF. At 14:00 the email provider goes down for 10 minutes. Emails just wait in the queue and go out when it recovers; checkout never slows down.

Because a message can occasionally arrive twice, the email worker records the order IDs it has already handled and skips repeats. Nobody gets two confirmation emails.

How it works

A message queue sits between a producer, which sends messages, and a consumer, which processes them. The producer returns as soon as the message is safely stored. Consumers pull messages when they have capacity.

Point-to-point queue vs pub/sub

Architecture

Top: a work queue, where each job goes to one worker. Bottom: pub/sub, where each subscriber gets its own copy of every event.

Work queue: many workers share one queue; each message is processed by one worker. Used for background jobs.
Pub/sub: producers publish to a topic; each subscriber gets its own copy. Used to announce events. A common pattern combines them: a topic fans out to one queue per subscribing service.

Delivery guarantees

A consumer takes a message, processes it, then acknowledges it. Until it's acknowledged, the message is hidden from other consumers for a visibility timeout. If the consumer crashes, the message reappears and is retried.

At-most-once: acknowledge before processing. Fast, but a crash loses the message.
At-least-once: acknowledge after processing. Nothing is lost, but duplicates happen. The usual default.
Exactly-once: truly exactly-once is very hard across systems. In practice you get it with at-least-once delivery plus idempotent consumers, for example recording processed message IDs or using upserts.

Ordering

Standard queues may deliver out of order. If order matters (all events for one account), use FIFO queues or ordering keys, which keep order per key while still processing different keys in parallel.

When things go wrong

Retries with backoff: a failing message is retried a few times, with growing delays.
Dead-letter queue: after N failures, the message moves aside for inspection, so one bad message doesn't block or loop forever.
Backpressure: watch queue depth and the age of the oldest message. Rising backlog means consumers can't keep up: scale them (lesson 8) or slow producers.

Trade-offs

What you gain

Fast responses: slow work happens in the background.
Absorbs spikes: the queue buffers bursts while workers process at a steady pace.
Failure isolation: if consumers are down, messages wait instead of being lost.
Easy to add new consumers of an event without changing the producer.

What it costs you

Results are eventually done, not immediately; users may need status updates.
Duplicates, retries and ordering need careful handling.
Harder to debug: a request's work is spread over time and services.
The queue is one more critical piece of infrastructure.

When not to use it

The caller genuinely needs the answer right now to continue, like checking a password. Use a direct call.
The work is quick and failure is cheap; a queue just adds latency and parts.

In the cloud

Message queues and pub/subEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS3

Amazon SQS (opens vendor docs)
Fully managed message queue; standard (at-least-once) or FIFO (ordered, exactly-once processing).
Amazon SNS (opens vendor docs)
Pub/sub topics that fan a message out to queues, functions, HTTP endpoints, email or SMS.
Amazon MQ (opens vendor docs)
Managed ActiveMQ and RabbitMQ brokers for apps that already use them.

Azure2

Azure Service Bus (opens vendor docs)
Enterprise message broker with queues, topics, sessions and dead-lettering.
Azure Queue Storage (opens vendor docs)
Simple, cheap queues inside a storage account.

Google Cloud2

Pub/Sub (opens vendor docs)
Global messaging with topics and push or pull subscriptions.
Cloud Tasks (opens vendor docs)
Queues of individual tasks dispatched to HTTP handlers, with rate limits and retries.

How you’d set it up

The web tier drops a message and returns at once; workers pull from the queue at their own pace.

API
SQS queue
Worker group
Dead-letter queue

Interview questions

Try answering out loud first, then open the model answer.

How would you design sending a welcome email when a user signs up?

Model answer

The signup request writes the user to the database and publishes a “user-signed-up” event, ideally using the outbox pattern: write the event to an outbox table in the same transaction, and have a relay publish it, so you never create a user without the event or vice versa. An email service subscribes through its own queue, renders the template and calls the email provider. It's idempotent, keyed by user ID and email type, so a retry doesn't send twice. Failures retry with backoff and land in a dead-letter queue after a few attempts, with an alert.

What's the difference between a message queue and pub/sub?

Model answer

In a queue, each message is consumed by exactly one of the competing consumers. It distributes work. In pub/sub, each message is delivered to every subscriber. It broadcasts events. Many systems combine them: a topic fans out to one queue per subscribing service, and within each service, workers compete on that queue.

How do you make message processing idempotent?

Model answer

Give each message a unique ID (or derive one from the business key, like order ID plus action), and record processed IDs in the same transaction as the side effect, skipping any ID already seen. Or design the effect itself to be idempotent: “set status to shipped” instead of “increment shipped count”, upserts instead of inserts, and idempotency keys when calling external APIs like payment providers.

Quiz

Pick an answer to see whether it’s right and why.

0 of 7 answered

Key terms

Message queue: A buffer where producers leave messages and consumers pick them up later, so neither waits for the other.
Producer: The part of a system that sends messages or events.
Consumer: The part of a system that receives and processes messages or events.
Pub/sub: Publish/subscribe. A publisher sends a message to a topic and every subscriber gets its own copy.
Dead-letter queue: A side queue for messages that keep failing, so they can be inspected instead of retried forever.
At-least-once delivery: Every message is delivered, but sometimes more than once, so consumers must handle duplicates.
Idempotency: Doing the same operation twice has the same effect as doing it once, which makes retries safe.