Design a chat app

Think of it like this

Analogy

A switchboard operator at an old telephone exchange. Everyone who's “online” has a line plugged into the board. When Alice wants to talk to Bob, the operator finds Bob's line and connects them. If Bob's line isn't plugged in, the operator writes a note and leaves it in his mailbox, and rings his house to say there's mail. Every conversation is also written into a logbook so it can be read later.

The problem it solves

Chat looks simple, but at scale it combines several hard problems: hundreds of millions of connections that stay open for hours, messages that must arrive in order and exactly once from the user's point of view, users with several devices, people going offline in the middle of a conversation, and group chats where one message goes to hundreds of people.

A real example

Example

A family group chat

Eight people share a family group. You send “Dinner at 7?”. The chat server saves the message with a new ID for that conversation first, and only then delivers it.

Five relatives are online, each with an open WebSocket to some chat server. Their servers get the message through a pub/sub channel for the group and push it down instantly. The other three are offline, so they get a push notification instead.

When your cousin's phone reconnects an hour later, it asks for messages in this group after the last ID it has, and catches up in one request. As each phone shows the message, it reports back, and you see the little “read by 6” update.

How it works

1. Requirements

Functional: one-to-one and group chats (up to, say, 500 members); real-time delivery; message history across devices; online status (presence) and “typing…”; delivered and read receipts; push notifications for offline users; media attachments.

Non-functional: low latency (under a few hundred ms end to end); no lost messages; order preserved within a conversation; highly available.

2. Estimates

100 M daily users × 40 messages ≈ 4 billion messages a day ≈ 46,000 per second on average, perhaps 150,000 at peak. At 200 bytes each that's about 0.8 TB of new text per day, before replicas. Up to tens of millions of concurrent connections.

3. Connections

Clients keep a WebSocket open to a chat server (lesson 4). Each server can hold tens to hundreds of thousands of connections. A load balancer that supports long-lived connections spreads users across servers, and a session registry (a fast key-value store) records which server each user's devices are on.

4. Sending a message

1. The client sends the message with a client-generated ID (for idempotent retries).
2. The chat server appends it to a partitioned event stream keyed by conversation ID. That gives a per-conversation order and a sequence number.
3. The server acknowledges to the sender as soon as the message is durable. That's the single tick.
4. Consumers persist the message and route it to each recipient's chat server(s), which push it down the open connections.
5. If a recipient has no connection, a notification is sent through the platform push service instead.

Delivery is at-least-once; clients drop duplicates by message ID and fill gaps by asking for “everything after sequence N” when they reconnect.

5. Storage

Messages are written constantly and read as “latest page of this conversation”. A wide-column store fits: partition key conversation_id, sort key sequence. Very old messages can move to cheaper storage. Media goes to object storage, and messages hold a link.

6. Group chats

For small groups, fan out on write: route the message to every member's server. For very large groups or channels, store once and have members pull, or fan out through a dedicated pub/sub channel per group.

7. Presence

Clients send a heartbeat every so often; the server records “last seen” in a fast cache with a short TTL. Online status changes are published only to people who have that contact open, not to all contacts, which would be enormous fan-out.

8. The whole picture

Trade-offs

What you gain

• Persistent connections give instant delivery without polling.
• A per-conversation log gives ordering, durability and replay for reconnecting devices.
• Wide-column storage scales writes horizontally.

What it costs you

• Long-lived connections make deploys and scaling harder (connections must drain and reconnect).
• At-least-once delivery requires de-duplication everywhere.
• Presence and typing indicators generate a lot of small traffic.
• End-to-end encryption, if required, limits server-side features like search.

When not to use it

• For low-volume comment threads or support tickets, simple polling or Server-Sent Events plus a relational database is far simpler.
• Don't broadcast presence to every contact; it doesn't scale.

In the cloud

Message queues and pub/subEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS3

• Amazon SQS (opens vendor docs)

  Fully managed message queue; standard (at-least-once) or FIFO (ordered, exactly-once processing).

• Amazon SNS (opens vendor docs)

  Pub/sub topics that fan a message out to queues, functions, HTTP endpoints, email or SMS.

• Amazon MQ (opens vendor docs)

  Managed ActiveMQ and RabbitMQ brokers for apps that already use them.

Azure2

• Azure Service Bus (opens vendor docs)

  Enterprise message broker with queues, topics, sessions and dead-lettering.

• Azure Queue Storage (opens vendor docs)

  Simple, cheap queues inside a storage account.

Google Cloud2

• Pub/Sub (opens vendor docs)

  Global messaging with topics and push or pull subscriptions.

• Cloud Tasks (opens vendor docs)

  Queues of individual tasks dispatched to HTTP handlers, with rate limits and retries.

The web tier drops a message and returns at once; workers pull from the queue at their own pace.

AWSAzureGoogle Cloud

1. API
2. SQS queue
3. Worker group
4. Dead-letter queue

Interview questions

Try answering out loud first, then open the model answer.

How do you deliver a message to a user who has three devices, two online and one offline?+

Model answer

Each device has its own connection and its own read position (last delivered sequence per conversation). When a message arrives, look up all of the user's active connections in the session registry and push to each. For the offline device, send a push notification; when it reconnects, it asks for all messages after its last sequence and the server pages them from the message store. Read receipts are synced across devices as events.

How do you guarantee messages aren't lost?+

Model answer

The server only acknowledges the sender after the message is durably appended to a replicated log. Clients keep unacknowledged messages and resend with the same client message ID, so duplicates are dropped. Delivery to recipients is at-least-once with acknowledgements; anything not acknowledged is re-sent, and reconnecting clients sync from their last sequence number. Gaps in sequence numbers tell a client it's missing something.

How would you scale to 50 million concurrent connections?+

Model answer

Spread connections across many stateless-ish chat servers, each holding up to a few hundred thousand connections (memory per connection matters, so use efficient event-driven servers). Use a load balancer that supports long-lived connections, a sharded session registry mapping users to servers, and a partitioned message log so delivery scales horizontally. Plan for reconnect storms: when a server restarts, its clients reconnect with backoff and jitter. Deploy gradually so connections drain instead of dropping all at once.

Quiz

Pick an answer to see whether it’s right and why.

1. 01Alice and Bob are connected to different chat servers. How does Alice's message reach Bob's connection?

   Alice's server opens a connection to Bob's phoneThrough a pub/sub or routing layer that knows which server holds Bob's connectionThe database pushes it to Bob
2. 02How do you keep messages in order within a conversation?

   Use the sender's phone clockAssign each message a sequence number per conversation on the serverSort alphabetically
3. 03A phone loses signal just as it sends a message, then resends it. How do you avoid a duplicate?

   Ignore all resendsThe client attaches a unique message ID; the server drops any message ID it has already storedAsk the user
4. 04A phone was offline for an hour. How does it catch up when it reconnects?

   The server resends every message ever sentIt asks for messages after the last message ID it hasIt only waits for new messages
5. 05Why save a message before delivering it?

   So it's never lost if delivery fails or the recipient is offlineTo make delivery fasterBecause WebSockets require it
6. 06Why do chat apps use WebSockets instead of polling every second?

   Polling is more secureWebSockets don't use TCPThe server can push messages instantly over one open connection, without constant empty requests

0 of 6 answered

Key terms

WebSocket

A long-lived, two-way connection so the server can push messages to the client at any time.

Presence

Knowing whether a user is online right now.

Event stream (log)

An append-only, ordered record of events that many consumers can read and replay, like Kafka.

Wide-column store

A database that groups data by a partition key and sorts rows inside each partition; built for huge write volumes.

At-least-once delivery

Every message is delivered, but sometimes more than once, so consumers must handle duplicates.

Idempotency

Doing the same operation twice has the same effect as doing it once, which makes retries safe.