vault

WhatsApp Messenger System Design

29 min read

WhatsApp Messenger System Design

Design an end-to-end encrypted (E2EE) instant-messaging service at the scale of WhatsApp (~2 billion monthly active users, ~100 billion messages per day per Meta’s 2020 disclosures) or Signal (the gold-standard E2EE reference implementation). Functional surface: one-to-one chat, group chat (up to 1024 members on WhatsApp as of 2023), delivery and read receipts, presence (online / last-seen), media transfer (photo, video, voice note), end-to-end encryption such that the operator cannot read message plaintext, multi-device sync (companion devices linked to one phone identity). Non-functional: sub-second message delivery for online recipients, durable offline queueing for recipients who are offline, bandwidth-efficient on cellular networks, robust to NAT and intermittent connectivity. The interview-relevant gravity is: how do you reconcile mass scale with E2EE — the operator can route, queue, and fanout but cannot inspect — and the answer is the Signal Protocol (X3DH key agreement plus the Double Ratchet) layered on top of an Erlang/XMPP-style persistent-connection backend (Rick Reed, Erlang Factory 2014, summarized by High Scalability).

1. Requirements

1.1 Functional

  • One-to-one chat. Send and receive text and structured messages between two registered users identified by phone numbers (E.164 format).
  • Group chat. Multi-party conversations of N members (typical N ≤ 256, large groups up to 1024 in WhatsApp, communities up to ~5000 with sub-groups).
  • Delivery semantics surface. A message progresses through visible states: sent (server received from sender), delivered (server pushed to recipient device and got an ack), read (recipient opened the chat — the famous blue double-checkmark on WhatsApp).
  • Presence. Online / typing / last-seen-at indicators, with privacy controls (a user can hide last-seen).
  • Media. Image, video, voice note, document, location pin. Media itself goes through an object store; the message carries a reference + decryption key.
  • End-to-end encryption. Server stores only ciphertext; only the participating clients can decrypt. Compromise of the operator should not yield message plaintext.
  • Multi-device. A phone (the “primary”) plus up to four “companion” devices (Web, Desktop, tablet) that can each send/receive without the primary being online.
  • Group administration. Add / remove members, change group metadata, leave group.
  • Push notifications. When the app is backgrounded, deliver a notification through the platform push channel (APNS = Apple Push Notification service for iPhones; FCM = Firebase Cloud Messaging for Android).

1.2 Non-functional

  • Scale. ~2 × 10⁹ Monthly Active Users (MAU); WhatsApp’s 2020 disclosure cited ~100 × 10⁹ messages/day.
  • Latency. End-to-end delivery between two online clients in < 1 s P95 across continents; < 200 ms intra-region.
  • Availability. Message send must not block on global consensus — async durability is acceptable so long as the sender sees an ack and the message eventually delivers. Target ≥ 99.99 % availability for the send path.
  • Durability. A message accepted by the server must be delivered or eventually acknowledged delivered, even across recipient device restarts and SIM swaps.
  • Bandwidth. Cellular-data efficient; binary frames not JSON over wire (WhatsApp uses a custom binary protocol on top of WebSockets / older XMPP-derived stack).
  • Battery. A single persistent connection per device; minimize wakeups.
  • Privacy. No server-side plaintext access; metadata minimization (Signal’s Sealed Sender hides “from” from the server).

2. Capacity Estimation

Assume the following inputs (round numbers chosen to make arithmetic legible; production engineers would refine with actual product telemetry).

  • Daily Active Users (DAU): D = 1.5 × 10⁹ (75 % of MAU).
  • Messages per DAU per day: m = 60 (a mix of one-to-one and group sends; lower than WhatsApp’s reported figures but conservative).
  • Message size on the wire (ciphertext + header): s = 200 B average (text), with a long tail for media payload references.
  • Media share: 10 % of messages are media; each media object averages M = 500 KB (heavily skewed; voice notes ~50 KB, videos ~5 MB).

2.1 Message-rate arithmetic

Total messages/day = D × m = 1.5 × 10⁹ × 60 = 9 × 10¹⁰ messages/day (~10¹¹/day, comparable to WhatsApp’s public 100 B/day figure).

Average messages per second: 9 × 10¹⁰ / 86 400 s ≈ 1.04 × 10⁶ msgs/s sustained.

Peak-to-average ratio for diurnal traffic ≈ 3×, so peak ≈ 3 × 10⁶ msgs/s during evening hours in major regions.

2.2 Storage arithmetic for the message queue

Server-side, only undelivered messages need durable storage (delivered messages are evicted — operator does not retain plaintext history; clients keep their own copy). Assume the average undelivered backlog per recipient is N_q ≈ 5 messages waiting at the inbox.

  • Per recipient: 5 × 200 B = 1 KB.
  • Total: 1.5 × 10⁹ × 1 KB ≈ 1.5 TB of in-flight queue storage. This is small enough to fit in RAM across a few hundred queue nodes, consistent with WhatsApp’s Erlang-based message store, which kept offline messages in memory with on-disk fallback (Reed 2014, via High Scalability).

2.3 Bandwidth arithmetic

Text-message ingress: 1.04 × 10⁶ msgs/s × 200 B = 208 MB/s ≈ 1.66 Gbps average; 5 Gbps peak. Trivially handled by a few hundred edge gateways.

Media ingress (separate object-store path, not the chat backbone): 0.1 × 1.04 × 10⁶ msgs/s × 500 KB = 52 GB/s ≈ 416 Gbps average; 1.2 Tbps peak. This is why media is offloaded to a dedicated CDN-fronted object store rather than transiting the chat servers.

2.4 Connection arithmetic

Each online user holds one persistent TCP/TLS connection (WebSocket frame layer). Concurrent connections at peak ≈ 0.4 × DAU ≈ 6 × 10⁸ live sockets. WhatsApp’s per-server connection density is the classic benchmark here, but the often-quoted “2 million per server” needs care: in a 2012 Erlang Factory talk the WhatsApp team demonstrated ~2–2.8 million connections on a single tuned FreeBSD host as a stress-test record, whereas by the 2014 talk the production fleet ran ~150 chat servers each holding on the order of ~1 million phones, alongside ~250 multimedia servers, for ~465 million monthly users across >11,000 cores (Reed 2014, via High Scalability). Taking the conservative production figure of ~1 × 10⁶ sockets/server, 6 × 10⁸ / 1 × 10⁶ = ~600 connection servers minimum, with a comfortable provisioning factor giving ~1000+ across many data centers.

3. API Design

The wire protocol is binary; what follows is a logical surface a client SDK exposes. WhatsApp historically used a customized XMPP (Extensible Messaging and Presence Protocol; RFC 6120) profile; Signal uses a custom protobuf-over-WebSocket frame.

3.1 Connection establishment

POST /v1/auth/register        { phone_number, sms_code }       → { user_id, auth_token }
WSS  /v1/connect              Header: Bearer <auth_token>      → bidirectional binary frames

Once connected, the client runs an authentication handshake (the auth token plus a device-specific signed prekey bundle from the Signal Protocol layer; see §8.2).

3.2 Sending a message (frame, conceptual)

message OutboundMessage {
  string  conversation_id = 1;     // 1:1 thread id or group id
  string  client_msg_id   = 2;     // UUID; idempotency key
  bytes   ciphertext      = 3;     // X3DH/Double-Ratchet output
  uint64  client_ts_ms    = 4;
  enum    Type { TEXT=0; MEDIA_REF=1; SYSTEM=2; } type = 5;
  // For multi-device: a separate per-recipient-device ciphertext is bundled
  repeated DeviceCiphertext per_device = 6;
}
 
message DeviceCiphertext {
  string device_id = 1;
  bytes  ciphertext = 2;            // encrypted under the per-device session
}

Server reply:

message Ack {
  string  client_msg_id = 1;
  string  server_msg_id = 2;        // monotonic per conversation; used for ordering
  uint64  server_ts_ms  = 3;
  enum Status { ACCEPTED=0; DUPLICATE=1; REJECTED=2; } status = 4;
}

The client_msg_id is the idempotency key: the server stores a short-window dedup map so that retries from the client (after a flaky network) don’t double-deliver. This makes the channel at-least-once with idempotent client retry, which the receiving side then resolves to exactly-once at user perception by deduping on server_msg_id.

3.3 Receiving (server-pushed)

The server pushes inbound frames over the same socket:

{ server_msg_id, conversation_id, sender_id, ciphertext, server_ts_ms }

The client acks with {server_msg_id, status: DELIVERED}; later, on user opening the chat, the client emits {server_msg_id, status: READ}. The server forwards both back to the original sender.

3.4 Group operations

POST /v1/group                { name, member_user_ids }                          → { group_id }
POST /v1/group/{id}/members   { add: [...], remove: [...] }                      → { ok, version }
POST /v1/group/{id}/leave                                                         → { ok }

Group operations are themselves messages (control messages) — the same delivery channel propagates the membership change so every member’s local view converges.

3.5 Media upload

POST /v1/media/upload         multipart, encrypted blob        → { media_id, cdn_url }

The client encrypts the media with a fresh symmetric key K_media before upload (server stores ciphertext blob); the chat message then carries a reference {media_id, cdn_url, K_media} encrypted under the Double Ratchet so only the recipient can decrypt. (WhatsApp Encryption Whitepaper, §“Transmitting Media and Other Attachments”.)

4. Data Model

The key observation: the operator should hold the minimum metadata required to route messages, and never plaintext message bodies.

4.1 User and device tables

CREATE TABLE users (
  user_id        UUID PRIMARY KEY,
  phone_number   TEXT UNIQUE,           -- E.164, hashed in some deployments
  registration_id INTEGER,              -- Signal Protocol per-install id
  created_at     TIMESTAMP
);
 
CREATE TABLE devices (
  device_id      UUID PRIMARY KEY,
  user_id        UUID REFERENCES users,
  identity_pub   BYTEA,                 -- long-term Curve25519 public key
  signed_prekey  BYTEA,                 -- rotated weekly
  signed_prekey_sig BYTEA,
  one_time_prekeys BYTEA[],             -- a queue, consumed on session-init
  last_seen_at   TIMESTAMP,
  push_token     TEXT
);
CREATE INDEX devices_user_idx ON devices(user_id);

The one_time_prekeys queue is critical: each prekey is consumed exactly once during the first handshake with a new contact (X3DH), giving forward secrecy even if the long-term identity key is later compromised.

4.2 The inbox queue

CREATE TABLE inbox (
  recipient_device_id UUID,
  server_msg_id       BIGINT,           -- monotonic per recipient-device
  sender_user_id      UUID,
  conversation_id     UUID,
  ciphertext          BYTEA,            -- opaque to server
  enqueued_at         TIMESTAMP,
  PRIMARY KEY (recipient_device_id, server_msg_id)
);

Sharded by recipient_device_id via Consistent Hashing — each device’s inbox lives entirely on one shard, so all the recipient’s pending messages arrive in server_msg_id order from one source. After a device has acked DELIVERED the row is deleted; the server retains nothing beyond a small audit ledger.

4.3 Conversation membership (groups)

CREATE TABLE groups (
  group_id    UUID PRIMARY KEY,
  name        TEXT,
  created_by  UUID,
  version     BIGINT
);
 
CREATE TABLE group_members (
  group_id    UUID,
  user_id     UUID,
  role        TEXT,                     -- 'admin' | 'member'
  joined_at   TIMESTAMP,
  PRIMARY KEY (group_id, user_id)
);

Groups are “pairwise sender-keys” on the client side (§8.3) — the server’s view is just the membership roster used for fanout addressing.

4.4 Conversation IDs and message ordering

Per-conversation message ordering uses monotonic server_msg_id assigned by the inbox shard owner. Because each recipient’s inbox is owned by one shard, a single writer assigns IDs and there is no distributed-consensus problem for ordering; that’s the reason the architecture pins per-recipient routing rather than per-conversation routing. (For groups, each member receives an independent stream from the fanout, and order is reconciled client-side using sender-key chain counters; see §8.3.)

5. High-Level Architecture

flowchart LR
    subgraph "Sender Phone"
        A1[Client App<br/>Signal Protocol]
    end
    subgraph "Edge"
        EG[Connection Gateway<br/>WebSocket / Erlang]
    end
    subgraph "Core"
        AUTH[Auth + Identity<br/>Service]
        ROUTE[Routing / Fanout<br/>Service]
        INBOX[(Inbox Queue<br/>Sharded by recipient)]
        GMEMB[(Group Membership<br/>DB)]
        PREKEY[Prekey Bundle<br/>Server]
    end
    subgraph "Recipient Phone"
        B1[Client App<br/>Signal Protocol]
    end
    subgraph "External"
        PUSH[APNS / FCM]
        OBJ[Encrypted Media<br/>Object Store + CDN]
    end
    A1 -- "send (ciphertext)" --> EG
    EG --> ROUTE
    ROUTE --> GMEMB
    ROUTE --> INBOX
    INBOX --> EG
    EG -- "deliver" --> B1
    EG --> PUSH
    A1 -. "fetch prekey<br/>before first chat" .-> PREKEY
    A1 -. "upload encrypted media" .-> OBJ
    B1 -. "fetch encrypted media" .-> OBJ

What this diagram shows. Two clients exchange ciphertext through an edge gateway tier (the persistent-connection layer), a routing tier that consults group membership and writes to per-recipient inbox queues, and supporting services for authentication, prekey distribution (necessary for the X3DH handshake before two users have ever spoken), and media upload to a separate CDN. Note that the chat-message cryptographic boundary is at the client app: every message the server sees is opaque ciphertext. The server’s only roles are routing, queueing for offline recipients, and signaling pushes to the platform push providers. The dotted lines indicate occasional, not per-message, interactions: prekey bundles are fetched once per new contact pair, and media flows out-of-band to a CDN to keep the chat backbone bandwidth-light.

6. Request Flow

6.1 One-to-one message, both online

sequenceDiagram
    participant A as Alice (sender)
    participant GA as Gateway-A
    participant R as Routing
    participant Q as Inbox(Bob)
    participant GB as Gateway-B
    participant B as Bob (recipient)

    A->>GA: SEND ciphertext, client_msg_id
    GA->>R: route(to=bob_device_id, cipher)
    R->>Q: enqueue, assign server_msg_id
    Q-->>R: ack(server_msg_id)
    R-->>GA: ack
    GA-->>A: ACK(server_msg_id)
    Q->>GB: push(server_msg_id, cipher)
    GB->>B: deliver(cipher)
    B-->>GB: DELIVERED
    GB->>R: relay status
    R->>GA: status
    GA->>A: DELIVERED check-mark
    Note over B: User opens chat
    B->>GB: READ(server_msg_id)
    GB->>R: relay
    R->>GA: status
    GA->>A: READ blue check-mark

Walk-through. Alice’s send is first persisted into Bob’s inbox shard, which assigns a monotonic server_msg_id. Only after that durable persistence does Alice receive an ACK — this is the system’s commit point. Bob’s gateway, holding Bob’s persistent socket, gets a notification from the inbox shard (via an internal pubsub topic keyed by bob_device_id) and pushes the frame down. Bob’s client cryptographically verifies and decrypts the frame using its current Double-Ratchet receiving chain and emits DELIVERED; later, on UI open, READ. Both statuses flow back to Alice as separate small frames. The two-stage check-mark UI is just the surfacing of these two acknowledgments.

6.2 Sender online, recipient offline

If Bob’s gateway has no live socket for bob_device_id, the inbox shard queues the message and triggers a side channel:

  1. The routing service notices Bob has no connected gateway.
  2. It calls APNS or FCM with a wakeup payload (no plaintext content; just “you have a message”).
  3. Bob’s OS wakes the WhatsApp app process; the app reconnects via WSS.
  4. On connect, the gateway flushes Bob’s inbox starting from the last acked server_msg_id.

This makes the queue at-least-once: a flaky reconnect may re-deliver an entry that was actually delivered but the ACK was lost, and the client dedups on server_msg_id.

7. Deep Dive

7.1 Persistent connections and Erlang

Every online user holds one TCP/TLS socket. WhatsApp’s choice of Erlang on FreeBSD is the classic case study: Erlang processes are lightweight (a few hundred words of initial heap), scheduled by the BEAM virtual machine, and each socket is modeled as one Erlang process. WhatsApp’s engineers publicly stress-tested a single tuned FreeBSD host to ~2 million (and in one demo ~2.8 million) connections; in steady-state production around 2014 they ran roughly one million phones per chat server across ~150 chat servers (Reed 2014, via High Scalability), with the binding constraint being kernel TCP/socket state and memory bandwidth rather than the Erlang scheduler. The supervision-tree model (a process owns a connection; if it crashes, its supervisor restarts it without affecting siblings) maps almost trivially onto the “many independent users” workload.

A modern alternative is one socket per CPU core in a Go/Rust gateway with epoll/io_uring, sharding sockets by user-id-hash to a core. The architectural shape is the same; Erlang’s specific advantage is the cheap per-process isolation that sidesteps a class of correctness bugs.

7.2 The Signal Protocol — X3DH plus Double Ratchet

The cryptographic core, used by both WhatsApp and Signal, is the Signal Protocol (Marlinspike & Perrin). It has two pieces.

X3DH (Extended Triple Diffie-Hellman) establishes a shared secret between Alice and Bob’s first conversation, even when Bob is offline. Bob has previously uploaded a prekey bundle to the server: identity public key IK_B, signed prekey SPK_B (signed by IK_B), and a stack of one-time prekeys OPK_B,1..n. Alice fetches Bob’s bundle, generates her ephemeral key EK_A, and computes:

DH1 = DH(IK_A, SPK_B)        // Alice identity ↔ Bob signed prekey
DH2 = DH(EK_A, IK_B)         // Alice ephemeral ↔ Bob identity
DH3 = DH(EK_A, SPK_B)        // Alice ephemeral ↔ Bob signed prekey
DH4 = DH(EK_A, OPK_B)        // Alice ephemeral ↔ Bob one-time prekey (if available)
SK  = KDF(DH1 || DH2 || DH3 || DH4)

where:

  • DH(x, y) is X25519 elliptic-curve Diffie-Hellman: given private key x and the public key matching y, produce a shared scalar.
  • || is byte concatenation.
  • KDF is HKDF-SHA256 (Hash-based Key Derivation Function), turning the entropy in the four DH outputs into a uniform 256-bit master secret SK.
  • The four DHs (the “triple” in the name being slightly misleading; the fourth is the one-time prekey component) give cryptographic deniability and forward secrecy: an attacker would have to compromise both identity keys and one-time keys and ephemerals to derive SK.

The OPK is consumed (deleted from the server) the moment it is used. If Alice retries because Bob hadn’t uploaded fresh OPKs, the protocol falls back to using only DH1..DH3, with slightly weaker forward-secrecy guarantees that the formal analysis of Cohn-Gordon et al. (EuroS&P 2017) explicitly enumerates.

Double Ratchet. Once SK is established, the per-message keys evolve via two ratchets. The Diffie-Hellman ratchet runs once per direction-change: when Alice replies, she generates a fresh DH key pair, attaches the public part to her message, and both sides compute a new chain key from DH(new, peer's-current). This gives post-compromise security: even if an attacker steals the current chain state, the next DH ratchet step replaces it. The symmetric ratchet runs once per message within a chain: each message key is derived MK_n = KDF(CK_{n-1}, "msg"), and CK_n = KDF(CK_{n-1}, "chain") is the next chain key. This gives forward secrecy: deleting MK_n after use means past ciphertexts cannot be decrypted even if the current state leaks.

sequenceDiagram
    participant A as Alice
    participant B as Bob
    Note over A,B: Initial handshake (X3DH)<br/>SK established
    A->>B: DH-ratchet pubkey + ciphertext_1
    Note over A,B: Both derive new chain key CK_A
    A->>B: ciphertext_2 (from CK_A's ratchet)
    A->>B: ciphertext_3
    B->>A: DH-ratchet pubkey + ciphertext_1
    Note over A,B: Both derive new chain key CK_B

What this diagram shows. Each arrow is one frame on the wire. Whenever the direction of traffic flips, a fresh DH ratchet step rotates the long-lived state; in between, the symmetric ratchet provides per-message forward secrecy at no additional round trips. The cleverness is that no extra round-trip is required beyond piggybacking the fresh DH public key on the outgoing message.

7.3 Group chat — sender keys vs pairwise

Naïvely, an N-member group chat could be N(N-1) pairwise Double-Ratchet sessions: each sender encrypts the same message N-1 times (once per recipient). This is what WhatsApp originally did. Bandwidth cost is O(N) per message, which becomes painful at N = 256.

The optimization is the sender-key scheme (Signal’s group protocol; WhatsApp adopted a variant). Each sender maintains one sender key for the group: a chain of symmetric keys advanced once per outgoing message. To bootstrap, the sender distributes their current sender key to each member via the pairwise Double-Ratchet (so this is a one-time O(N) cost when the sender first speaks or rejoins). Thereafter, sending one message means encrypting once under the current chain key and broadcasting the same ciphertext to all members; each member advances their copy of the sender’s chain to decrypt. Bandwidth becomes O(1) per message + O(N) per new sender per epoch.

The trade-off is that sender-keys give weaker post-compromise security than pairwise ratchets — a leaked sender key compromises the whole future stream from that sender until a new key is distributed. Modern group-messaging research (e.g., Cohn-Gordon et al. CCS 2018; the IETF MLS protocol) addresses this with tree-based group key agreement. As of mid-2026 neither WhatsApp nor Signal had shipped MLS in production: Wire shipped a full MLS implementation, Google committed MLS for Android Messages, and Meta stated it was “in active discussions with IETF chairs to accelerate MLS adoption” — driven largely by the EU Digital Markets Act interoperability mandate — but had not migrated WhatsApp’s group protocol off sender-keys (TechPolicy.Press 2024).

7.4 Multi-device sync

The challenge: a user has phone + laptop + tablet, all of which should send and receive without the phone being online. The Signal Protocol, in its 2014 form, was strictly per-device-pair. A sender to user Bob has to know all of Bob’s devices and send a separate per-device ciphertext (because each device has its own X3DH-derived session).

WhatsApp’s 2021 multi-device architecture (engineering blog, July 2021) introduced device identity vectors: each device of a user has its own identity-key pair and is independently a participant in the Signal Protocol. A message from Alice’s phone to Bob is encrypted N_Bob times (once per Bob’s device) and N_Alice times (once per Alice’s other devices, so they see the message in their UI). The bundling in §3.2’s per_device field is exactly this. The companion devices are bootstrapped onto the user’s identity by the primary device QR-code-scanning a one-time pairing nonce; after that, the primary signs an attestation that the companion’s identity belongs to this user.

This costs O(devices_in_sender × devices_in_recipient) bandwidth per message; for typical small-device fleets (≤ 5 each) this is negligible compared to the cellular floor.

7.5 Presence

Per-user presence (online / typing) is broadcast to “interested” peers — users who currently have an open chat thread with this user. A naive design publishes “Alice online” to every contact every time Alice’s app comes to foreground, which is O(contacts) per state change.

WhatsApp uses a subscription model: a client subscribes to presence for the conversation it is currently viewing, and the server pushes presence transitions only to active subscribers. “Last seen at” is a single field updated on disconnect, served on demand to subscribed peers, subject to the privacy setting. This is gossip-flavored at the operator level — there is no need for distributed consensus.

For huge groups (community-style 1000+ member rooms), presence is suppressed entirely or shown only as aggregate counts, because the fanout cost is unmanageable.

7.6 Media path

Media doesn’t transit the chat backbone. The flow:

  1. Sender generates a fresh symmetric key K_m and IV; encrypts the bytes locally; uploads the ciphertext to the media object store via a signed URL, getting back media_id, cdn_url, hash.
  2. The chat message body is {type=MEDIA_REF, media_id, cdn_url, K_m, hash}, and that whole structure is then encrypted under the Double Ratchet — so only the recipient learns K_m and can decrypt the blob from the CDN.
  3. Recipient fetches cdn_url, verifies the hash (binds the blob to the message), decrypts with K_m.
  4. The CDN holds ciphertext blobs with a short TTL (typically 14 days for WhatsApp); after expiry the message body still exists in the recipient’s local DB but can’t be re-fetched if the recipient reinstalls.

This gives a clean separation: chat backbone is small, low-latency, persistent-connection-driven; media is an HTTP CDN problem. Cross-link: Content Delivery Network System Design for the CDN side.

8. Scaling Strategy

The system has three distinct scaling problems with different shapes.

8.1 Connection layer scales horizontally and is bound by socket count

Each gateway server holds K connections (K ≈ 2 × 10⁶ on Erlang/FreeBSD, K ≈ 10⁵ on a generic Go/Rust gateway). Total servers ≈ live_users / K. Adding a server is a matter of registering it in the gateway-discovery service (Distributed Lock Service System Design-style) and bringing it up. Users hash to gateways via Consistent Hashing, so a server addition disturbs ≈ users / num_servers connections.

8.2 Inbox-shard layer scales horizontally on user-id

Each shard owns a contiguous arc of recipient_device_id. A hot user (a celebrity in a public broadcast list) is the only worry; mitigation is to split popular accounts into virtual sub-shards or rate-limit broadcast lists explicitly.

8.3 Group fanout scales linearly with member count

A 1000-member group send produces 1000 inbox writes. To keep this cheap:

  • Fanout is parallelized per-shard.
  • The fanout initiator (one routing service node) does not block on each write; it dispatches asynchronously and reconciles late failures with retry.
  • Sender-key encryption (§7.3) is applied client-side, so the server doesn’t pay encryption cost; it is paying only routing cost.

WhatsApp historically capped group size at 256, then 512, then 1024, partly to keep fanout cost bounded and partly to keep cellular bandwidth bounded for senders (each send is N ciphertexts up the wire).

8.4 What breaks first

In capacity testing the typical first breakage is a thundering-herd reconnect after a regional incident: 50 M users all try to reconnect their sockets within seconds, hammering the auth service and the prekey-bundle service. Mitigations include staggered reconnect with jitter, client-side exponential backoff, and a reconnect-flow that skips full re-auth when the prior session token is still valid.

The second breakage is an “everyone sends to one celebrity” pattern (a viral message addressed to a single popular account), which concentrates inbox-shard writes on one shard. Sharded counters and broadcast-list semantics mitigate this.

9. Real-World Example

WhatsApp’s Erlang/FreeBSD architecture (Rick Reed, Erlang Factory SF Bay 2014). Reed — a single named engineer, not the “Reed & Hastings” pairing that an earlier draft of this note erroneously cited — reported figures captured in High Scalability’s write-up of the talk: ~550 servers plus standby gear (roughly 150 chat servers and 250 multimedia servers), >11,000 cores running the Erlang system, ~342K inbound and ~712K outbound messages per second at peak, ~19 billion inbound and ~40 billion outbound messages per day, serving ~465 million monthly users — all managed by on the order of ten engineers (Reed 2014, via High Scalability). Each chat server held on the order of a million phones (the higher “2 million per server” number was a tuned single-host stress-test record, not the production steady state). The entire backend was Erlang on the BEAM virtual machine; the architectural decision that enabled this was the per-process isolation model: a malformed packet from one client crashes one supervised process, not the server.

Signal’s open-source server (whispersystems/Signal-Server on GitHub). Java/Kotlin on AWS, Redis for the inbox queue, AWS S3 for media, with much smaller deployment scale than WhatsApp but the same protocol stack. Sealed Sender (Signal blog, 2018) is a metadata-minimization extension where the sender’s identity is itself encrypted inside the envelope — the server learns “someone sent to Bob” but not who. WhatsApp does not implement Sealed Sender as of 2024.

Telegram’s MTProto. A self-rolled cryptographic protocol that has drawn academic criticism (Albrecht, Mareková, Paterson & Stepanovs, “Four Attacks and a Proof for Telegram,” IEEE S&P 2022 — two attacks on MTProto 2.0, a timing side-channel in three official clients, plus a positive security proof for a slightly modified variant). Default chats on Telegram are not end-to-end encrypted; only “secret chats” are. This is a deliberate product trade-off (server-side message search, multi-device convenience) made differently from WhatsApp/Signal. Worth knowing as a contrast in the interview.

iMessage. End-to-end encrypted with per-device keys via Apple’s IDS (Identity Service); architecturally similar in spirit but specific cryptographic details differ (Apple’s PQ3 upgrade, rolled out in iOS/iPadOS/macOS 17.4/14.4 in February 2024, added Kyber-based post-quantum key establishment alongside the classical elliptic-curve handshake, plus ongoing rekeying).

10. Tradeoffs

ChoiceProConWhen chosen
End-to-end encryptionOperator cannot read; strong privacyNo server-side message search; no cloud-side spam scoringWhatsApp, Signal, iMessage
Sender-key groups vs pairwise ratchetsO(1) bandwidth per send instead of O(N)Weaker post-compromise security per sendAll large-group E2EE messengers
One persistent connectionSub-second push, low batteryConnection setup cost; gateway must be statefulAll real-time messaging
Per-device subkeys (multi-device)Each device independently onlineO(devices²) ciphertext per messageWhatsApp 2021+, Signal Sesame
At-least-once + idempotency keySimple recovery on net flakeClient must dedupStandard for messaging
Erlang process per connection2 M sockets per host, per-conn fault isolationErlang skill rare; harder to integrate non-Erlang stacksWhatsApp’s signature choice
Media via separate CDNChat backbone stays tinyTwo systems to operate; client must orchestrateAll major messengers
No server-side historyCompromise of server yields no plaintextUser loses messages on phone loss without backupWhatsApp pre-2021 (after 2021, encrypted backups)

11. Pitfalls

  1. Treating “delivered” and “read” as system invariants. They are best-effort UI signals. A receipt can be lost or suppressed (recipient privacy setting); building business logic that depends on them is a mistake.
  2. Assuming TCP ordering is enough. TCP guarantees in-order delivery on one connection; messages can still arrive out-of-order because they may transit different gateways across reconnects. The server_msg_id is the only reliable ordering signal.
  3. Reusing the same Diffie-Hellman ephemeral. A bug that lets Alice reuse EK_A across two different X3DH handshakes destroys forward secrecy guarantees. Audit the prekey-rotation code carefully.
  4. Forgetting one-time prekey exhaustion. If Bob’s OPK queue is depleted (he’s been heavily contacted while offline), new contacts fall back to the SPK-only handshake with reduced security. Servers must alert clients to top up the OPK pool eagerly.
  5. Over-eager presence broadcasts. Pushing “online” to every contact on every foreground event is the path to wasted gateway CPU. Subscribe-on-view is the discipline.
  6. Naïve group fanout that sequences writes. A 1000-member group should fanout in parallel; sequencing turns one send into a 1000× latency event. Use a dispatch fanout pattern with bounded concurrency per shard.
  7. Storing message ciphertext indefinitely. The whole point of E2EE is that the operator deletes ciphertext after delivery + a short retention buffer. Indefinite retention turns the operator into a target for legal compulsion.
  8. Multi-device companion key leakage. A laptop with weaker security than the phone can leak per-device keys. The protocol’s per-device sessions limit the blast radius — only that device’s stream is exposed — but client-side OS protections matter.
  9. Push-notification privacy leaks. APNS / FCM payloads are not E2EE; sending message preview text in the push alert leaks plaintext to the platform vendor. WhatsApp/Signal send only “you have a message” in the platform push, with the actual text fetched by the woken-up app over the encrypted socket.
  10. Trusting clock skew for presence “last seen”. Client clocks drift; use server-stamped timestamps for any UI ordering or time-window logic.
  11. Group membership races. “Alice was just removed” + “Alice sent a message at the same time” can produce a message visible to recently-removed members. The version number on the membership update + the requirement that senders attach the group version to each message lets the server reject stale-membership sends.

12. Common Interview Variants and Follow-Ups

  • “How would you support encrypted backups?” WhatsApp’s 2021 encrypted-backup feature (WhatsApp Engineering 2021): the backup is encrypted with a unique, randomly generated key; the user either records that 64-digit key themselves or chooses a password. In the password path, the key is sealed inside a Backup Key Vault built on hardware security modules (HSMs) that WhatsApp operates and distributes geographically across multiple data centers. The vault enforces a limited number of password attempts and renders the key permanently inaccessible after too many failures (mitigating offline brute force against a weak password); WhatsApp learns only that a key exists in the HSM, never the key itself. The backup ciphertext lives on the platform cloud (Google Drive / iCloud), but the wrapping key lives only in the HSM vault. A good interview answer covers the key-escrow tradeoffs and the brute-force-resistance role of the HSM.

    Uncertain no claim of Apple/Google enclave attestation — an earlier draft of this note asserted "verifiable via Apple/Google's auditable enclave attestation," which is unsupported by the primary source and has been removed. To resolve: a WhatsApp/Meta security disclosure or audit report specifying HSM topology and any external verification. #uncertain

    Verify: the precise count of HSMs/data centers behind the Backup Key Vault, and any third-party attestation or transparency-log mechanism. Reason: WhatsApp’s 2021 engineering post states only that the vault is “geographically distributed across multiple data centers” and makes

  • “How do you stop spam without reading messages?” Metadata heuristics: send-rate per number, number-of-recipients per send, freshly-registered numbers, abuse reports from recipients. The operator sees rates, not content, but rates are highly diagnostic. WhatsApp infamously rate-limits forwarded messages to 5 chats at a time after the 2018 misinformation crisis.
  • “How do you implement disappearing messages?” Client-enforced TTL: the message body carries an expiration policy; both clients agree to delete at TTL. The operator can’t enforce it (no plaintext access), so it is a soft guarantee. Forward-secrecy of past message keys means the operator cannot recover deleted messages either.
  • “How do you design the message reactions feature?” A reaction is itself a small encrypted message referencing a parent_server_msg_id; the recipient client merges reactions into the parent message’s UI. Same delivery channel; same encryption.
  • “How would Signal’s MLS migration look?” MLS (Messaging Layer Security; RFC 9420, 2023) replaces sender-keys with a tree-based group key agreement that scales to 50 000-member groups with O(log N) rekey cost. The architectural impact is mostly client-side; the server gains a new “tree state” object that mediates group operations.
  • “What if the database holding inboxes loses a partition?” With at-least-once + client-side dedup, the impact is some duplicate redeliveries (recoverable) plus loss of any messages durably written but not yet replicated (lost forever for that small window). Per-shard cross-region replication mitigates the durability-loss case.
  • “How would you design a federated messenger like Matrix?” Each home server owns its users’ inboxes; cross-server messages flow over an HTTPS-based federation API with eventual consistency on room state via a CRDT-like algorithm (Matrix’s “state resolution v2”). Reasonable contrast with WhatsApp’s centralized model.

13. Open Questions / Uncertain

The deployment-number, MLS-status, and encrypted-backup flags from earlier drafts have been resolved against primary sources and folded into §7.3, §9, and §12 respectively. The one remaining open verification item (HSM topology / attestation for encrypted backups) is flagged inline in §12. Note also that all WhatsApp scale figures (§2, §7.1, §9) carry their as-of context — the production numbers are point-in-time as of Reed’s 2014 talk and the user-count figures are necessarily larger today (WhatsApp passed ~2 billion users by 2020).

The genuinely open design questions:

  • Is sender-keys’ weaker post-compromise security an acceptable trade for low-bandwidth groups, or should MLS be adopted broadly?
  • How much of WhatsApp’s Erlang advantage survives in a world of io_uring + per-CPU-pinned modern Go/Rust servers?
  • For groups in the 5000+ range, is the right architecture sender-keys, MLS, or a fundamentally different protocol (e.g., a federated room-with-server model)?
  • What is the right trade-off between metadata minimization (Sealed Sender) and the need for spam control?

14. See Also

Graph View

Backlinks