Walkthrough: Chat System

"Design WhatsApp/Messenger" is the canonical real-time question. The heart of it: user A is connected to server 1, user B to server 2 — how does a message cross? Everything else hangs off that.

Step 1: Requirements

Functional:

1. Send and receive 1:1 messages in real time.
2. Messages are delivered even if the recipient is offline (delivered on reconnect).
3. Group chats up to 100 members.

Non-functional:

• Low delivery latency (feels instant — sub-second).
• No lost messages — delivery is at-least-once, and we'll handle duplicates.
• Scale: 10M concurrent connections.

Step 2: Estimation where it decides something

A modern app server holds 100k+ concurrent connections, so 10M concurrent users ≈ ~100 chat servers — a modest fleet, not thousands. This number matters because it tells you the connection-routing problem is real but tractable.

Step 3: Core entities

User, Chat (1:1 or group, with members), Message (message_id, chat_id, sender_id, content, sequence, created_at), Inbox (per-recipient queue of undelivered message IDs).

Step 4: Interface

Chat is truly bidirectional — the server must push messages down without being asked — so this is the textbook case for WebSockets (a persistent two-way connection that stays open after one HTTP handshake). Over the socket:

client -> server: send_message { chat_id, client_msg_id, content }
server -> client: new_message  { message_id, chat_id, sender, content, seq }
server -> client: ack          { client_msg_id, message_id }

Plain REST still handles login and fetching chat history. One infra note that earns points: WebSockets are long-lived TCP connections, so balance them with a Layer 4 load balancer (routes on TCP, doesn't inspect HTTP).

Step 5: High-level design

Each chat server keeps an in-memory connection map: user_id → socket handle, for the users connected to it. The cross-server hop uses Redis pub/sub (a broadcast channel: publish once, every subscriber hears it):

1. A's message arrives at server 1.
2. Server 1 persists it (details next), then publishes to channel user:B.
3. Server 2, which subscribed to user:B when B connected, receives it and pushes it down B's socket.

The alternative — consistent-hashing users to specific servers so you always know where B lives — avoids the pub/sub hop but makes reconnects and scaling messier. Name both; pub/sub is the cleaner mid-level pick.

Deep dive 1: Durability — persist, then publish

Here is the nuance that separates candidates: pub/sub is fire-and-forget. If B's server misses the publish (B offline, server restarting), the message is simply gone — pub/sub alone is at-most-once. So durability must come from the database, not the channel:

1. Write the message to the Messages table and an entry to B's Inbox table (DynamoDB works well — simple key-value access by recipient).
2. Then publish for real-time push.
3. When B's client receives it, it sends an ack, which deletes the inbox entry.
4. When B reconnects after being offline, the client drains its inbox — everything undelivered is still there.

An inbox per client (not per user) supports multi-device. Purge delivered/expired entries with a TTL (~30 days).

Deep dive 2: Ordering and duplicates

Timestamps from phones can't be trusted (clocks skew), so order by server receive time, and give each message a per-chat sequence number. Clients use sequence numbers for gap detection: if you have seq 41 and 43, request 42.

Delivery is at-least-once (persist + push + inbox drain can overlap), so clients dedupe by message_id — receiving the same ID twice displays once. At-least-once plus idempotent handling is the standard guarantee; say the phrase.

Deep dive 3: Presence and dead connections

TCP won't tell you promptly that a phone dropped off — timeouts take minutes. Use application-level heartbeats: the client pings every few seconds; miss ~3 and the server marks the user offline, updates last-seen, and cleans the connection map.

Deep dive 4: Groups and media

Group message (≤100 members): persist the message once, then fan out one inbox entry per member and publish to each — at most 100 cheap writes, which is exactly why capping group size in requirements was smart. Media (photos/videos) never flows through chat servers: the client gets a presigned URL (a temporary direct-upload link to blob storage) and sends only the resulting URL as the message body.

Wrap-up

Recap: real-time 1:1 via WebSockets + pub/sub routing; no lost messages via persist-then-publish and inbox drain on reconnect; groups fan out to ≤100 inboxes. We chose at-least-once delivery with client dedupe, and L4 balancing for the sockets. With more time: end-to-end encryption and typing indicators.