Slack Team Messaging System Design

Design a workplace team-messaging service at the scale of Slack (10–20 million daily active users; ~100 000 paid workspaces as of 2023) or Microsoft Teams (≥ 280 million MAU per Microsoft 2022 disclosures). The functional surface differs sharply from a consumer messenger like WhatsApp: workspaces are multi-tenant (every team is its own logical universe — channels, members, files, search index, retention policies are scoped to one workspace), conversations are searchable on the server side (no end-to-end encryption by default; the operator must read message bodies to index them), and integrations are first-class (apps, bots, slash commands, webhooks, OAuth-installed third parties). The interview-relevant gravity is multi-tenant data isolation at scale combined with real-time fanout to many concurrent WebSocket connections per workspace — a problem Slack famously hit a wall on around 2016 and rebuilt with Flannel (an edge cache) and later a cellular architecture (Slack engineering blog, 2024).

1. Requirements

1.1 Functional

• Workspaces. A workspace (“team” in older Slack terminology) is the top-level tenant. A user has separate identities per workspace, separate membership, separate channels.
• Channels. Public (visible to all workspace members), private (invitation-only), and direct messages (1:1) and group DMs (small N). A workspace typically has hundreds to thousands of channels.
• Threads. Replies that branch off a parent message into a side panel; parent stays in channel.
• Messages. Rich text, code blocks, attachments, mentions (@user, @channel), reactions (emoji), edits and deletions, pinned messages, message-bookmark / save-for-later.
• Search. Full-text across all messages a user has access to in the workspace (and sometimes across multiple workspaces if they belong to several). Constrained to the user’s permission set: a user must not see private-channel messages they are not a member of.
• File attachments. Uploads stored in object storage; previewable inline; searchable on filename and OCR/extracted text.
• Notifications. In-app, push (mobile), email digest, mentions vs all-messages settings, do-not-disturb hours, channel-specific overrides.
• Apps and integrations. Bots (operating via the same API as users), slash commands (e.g., /giphy), interactive message components (buttons, dropdowns), incoming webhooks (a third party can post into a channel via a posted URL), outgoing webhooks (Slack POSTs to a third-party URL on events).
• Voice / huddles. Lightweight audio (and optionally video) calls scoped to a channel or DM.
• Enterprise admin. Single sign-on via Security Assertion Markup Language (SAML) or OpenID Connect, audit logs, retention policies, data-loss-prevention hooks, on-prem connector (Enterprise Key Management).
• Shared channels. A channel that lives in two workspaces (Slack Connect, 2019), with membership joining across the boundary.

1.2 Non-functional

• Multi-tenancy. Hard isolation between workspaces. A bug in one workspace’s data path must not leak content to another workspace.
• Latency. Send-to-deliver in < 200 ms within region; UI feedback (“message sent”) < 500 ms P95.
• Search latency. < 1 s for typical queries on workspaces up to 10⁷ messages.
• Availability. ≥ 99.99 % for the messaging path; search can degrade gracefully (return partial results from in-region indexes).
• Compliance. SOC 2, HIPAA-readiness, GDPR. Retention policies must purge data verifiably; audit logs must be append-only and tamper-evident.
• Scale. Per-workspace ranges from a 5-user free team to a 200 000-user enterprise; the architecture must be roughly invariant across that range.

2. Capacity Estimation

The arithmetic differs from a consumer messenger because traffic is concentrated in waking hours and clustered by workspace: a workspace has a bounded user count and writes that fanout only to its own members.

2.1 Workload sizing

• Daily Active Users (DAU): D = 1.5 × 10⁷ (Slack’s 2023 disclosure of ~18 M).
• Workspaces: W = 1 × 10⁵ paid + millions free. Median active workspace ≈ 10 users; large enterprise tail ≈ 10⁵ users.
• Messages per DAU per day: m = 80. Slack’s load is read-heavy: read:write ratio ~50:1 (every message is read by ~K members in the channel, plus mentions, plus indexer).
• Average message size (ciphertext-equivalent + metadata): s = 500 B for text (richer than chat; carries blocks, mentions, reactions).
• File attachments: 5 % of messages, average F = 200 KB.

Total writes/day = D × m = 1.5 × 10⁷ × 80 = 1.2 × 10⁹ messages/day, roughly 14 K/s sustained, 40 K/s peak (3× peak factor). Two orders of magnitude lighter than WhatsApp on the write side.

2.2 Fanout arithmetic

The unit cost of a send is “deliver to all members of the channel who have an active socket and have not muted the channel + index it for search + persist for replay.” If a channel has C subscribed online members, one send produces C real-time pushes plus 1 search-index write plus 1 storage write.

Assume average online subscribers per channel send ≈ 20. Real-time pushes/second ≈ 14 K × 20 = 280 K pushes/s (peak ~1 M/s). This is what saturates the WebSocket fleet.

2.3 Storage

Persistent message storage = 1.2 × 10⁹/day × 500 B = 600 GB/day, ≈ 220 TB/year before compression and indexing. With compression ≈ 100 TB/year. Over five years that’s ≈ 0.5 PB on the message store.

Search index roughly 0.5–1× message bytes (inverted index, plus stored fields, plus compression), so add another ≈ 100 TB/year for the index.

File store grows at 0.05 × 1.2 × 10⁹/day × 200 KB = 12 TB/day; with 5-year retention ≈ 22 PB. This is the largest data tier and lives in object storage with cold-tier moves over time.

2.4 Connection arithmetic

Concurrent WebSockets at peak ≈ 0.4 × DAU = 6 × 10⁶ live sockets. At 50 K sockets per gateway server (Slack’s documented Envoy-based reverse proxy fleet handles more), this is ~120 servers; with regional and overhead factors ≈ 500 gateway nodes globally.

3. API Design

Slack’s public surface is two-layer: a REST/JSON Web API for the bulk of CRUD operations, and a Real-Time Messaging (RTM) WebSocket for push events. Modern Slack uses the Events API (HTTP webhooks to subscriber apps) for many push paths, but the RTM channel (or its successor, the Socket Mode WebSocket) remains for connected user clients.

3.1 Auth and connect

POST /api/auth.test         { token } → { user_id, team_id, ok }
POST /api/rtm.connect                  → { url: "wss://wss-primary.slack.com/?token=...&teams=..." }
WSS  <url>                             → bidirectional event frames

The rtm.connect step is what made Slack’s old architecture famous for “boot time”: before Flannel, the response payload contained the entire workspace’s user roster, channel list, and bots. For a 10 K-member workspace this was megabytes per connect, and during reconnect storms the payload alone was the bottleneck.

3.2 Send a message

POST /api/chat.postMessage
{
  "channel": "C0123",
  "text": "Hello team",
  "blocks": [...],                       // rich-block UI structure
  "thread_ts": "1683551200.000123",      // optional, ties to parent
  "client_msg_id": "uuid-...",           // idempotency key
  "unfurl_links": true
}
→ { ok, channel, ts: "1683551300.000456", message: {...} }

ts (timestamp) is the canonical message id within the channel: Unix-seconds with microsecond suffix. It functions both as a sort key and as a stable handle for edits, reactions, and threads. Two sends in the same microsecond resolve via the suffix counter — the channel’s owner shard atomically allocates ts.

3.3 RTM push event format

{
  "type": "message",
  "channel": "C0123",
  "user": "U0456",
  "text": "Hello team",
  "ts": "1683551300.000456",
  "team": "T0789",
  "client_msg_id": "uuid-...",
  "blocks": [...]
}

Other push event types: reaction_added, presence_change, channel_joined, team_join, pin_added, app_mention. The RTM stream is firehose-shaped — every event the user has access to, in real time, on one socket.

3.4 Search

GET /api/search.messages?query=...&page=...&sort=score|timestamp
→ { ok, messages: { total, paging, matches: [...] } }

The query is a small DSL: from:@alice in:#general before:2024-01-01 has:link "exact phrase". The backend parses this into an Elasticsearch-style query with permission-filtering applied before scoring. Cross-link: Distributed Search System Design.

3.5 Webhooks (incoming and outgoing)

Incoming: a third party gets an opaque URL and POSTs JSON; Slack treats the JSON as a synthetic message in a channel.

Outgoing (Events API): Slack POSTs JSON to a registered URL on events; the receiver must respond 200 within 3 s, otherwise Slack queues a retry with exponential backoff.

4. Data Model

4.1 Tenant-rooted schema

Every row is rooted in a team_id (workspace id). The unwritten rule: no query is permitted that is not filtered by team_id first. This is enforced at the data-access layer, not just at SQL level.

CREATE TABLE messages (
  team_id        BIGINT,
  channel_id     BIGINT,
  ts             NUMERIC(20, 6),     -- e.g. 1683551300.000456
  user_id        BIGINT,
  text           TEXT,
  blocks         JSONB,
  thread_ts      NUMERIC(20, 6),
  client_msg_id  UUID,
  edit_count     INT,
  deleted        BOOLEAN,
  PRIMARY KEY (team_id, channel_id, ts)
);
CREATE INDEX msg_thread_idx ON messages(team_id, channel_id, thread_ts) WHERE thread_ts IS NOT NULL;
CREATE INDEX msg_user_idx   ON messages(team_id, user_id, ts);

The (team_id, channel_id, ts) primary key gives:

• Channel timeline scan: range-scan on (team_id, channel_id, ts) ascending or descending. O(log N + K) for K messages.
• Thread expansion: index lookup on thread_ts.
• “Recent activity by user”: index lookup on user_id.

The schema deliberately denormalizes team_id everywhere to enable per-tenant sharding.

4.2 Channels and membership

CREATE TABLE channels (
  team_id     BIGINT,
  channel_id  BIGINT,
  name        TEXT,
  is_private  BOOLEAN,
  is_shared   BOOLEAN,             -- Slack Connect
  created_at  TIMESTAMP,
  PRIMARY KEY (team_id, channel_id)
);
 
CREATE TABLE channel_members (
  team_id     BIGINT,
  channel_id  BIGINT,
  user_id     BIGINT,
  joined_at   TIMESTAMP,
  PRIMARY KEY (team_id, channel_id, user_id)
);

Channel membership is the permission boundary: when search runs, the query must filter to channels in the user’s membership set, computed at query time or maintained as a denormalized “user → channel set” mapping in a fast cache (Flannel-style, see §7.1).

4.3 Sharding

Slack famously sharded by team_id using Consistent Hashing with virtual nodes; later (2024 cellular blog) they migrated to a cell-based architecture where each cell is a fully independent stack and a workspace is assigned to one cell. The migration is motivated by blast-radius: a database failure in one cell affects only the workspaces assigned to that cell, and a software rollout can canary in one cell.

team_id → cell_id (lookup table)
cell_id → { app servers, db cluster, search index, websocket gateway, storage }

The lookup table is itself replicated globally and cached at the edge so the WebSocket gateway can route a connecting user to their cell on the basis of the user’s team_id.

4.4 Search index

Per-workspace inverted index, sharded into segments. For small workspaces a single segment per workspace is enough; for huge ones the segments shard by (team_id, channel_id, ts_bucket). Slack’s 2017 blog on “Search at Slack” describes a Solr-based, time-bucketed index design where new messages go to a hot shard and older messages migrate to cold shards.

Index segment per (workspace, time-bucket=month)
Inverted entries: term → [doc_id, position, channel_id, ts, ...]
Permission filter applied as a post-filter on the (channel_id IN user's set) condition.

The deliberate trade-off: query time is faster than building permissions into every index entry (that would explode the index for users-in-many-channels), at the cost of needing a fresh permission-set lookup at query time. Flannel caches the permission set for fast retrieval.

5. High-Level Architecture

flowchart TB
    subgraph Edge
        LB[L7 Load Balancer<br/>Envoy]
        WS[WebSocket Gateway<br/>connection state]
        FL[Flannel<br/>workspace cache]
    end
    subgraph "App Tier (per cell)"
        WEBAPI[Web API<br/>Hack/PHP]
        EVENTBUS[Event Bus<br/>Kafka]
        JOBQ[Job Queue<br/>Kafka + worker pool]
        WEBHOOK[Webhook<br/>Dispatcher]
    end
    subgraph "Data Tier (per cell)"
        DB[(MySQL Vitess<br/>messages + meta)]
        SEARCH[(Solr Cluster<br/>per-workspace shards)]
        OBJ[(File Storage<br/>S3)]
        AUDIT[(Audit Log<br/>append-only)]
    end
    subgraph External
        APN[APNS / FCM]
        EMAIL[SES]
        SAML[SAML / OIDC IDP]
    end

    Client --> LB
    LB --> WS
    LB --> WEBAPI
    WS --> FL
    WS --> EVENTBUS
    WEBAPI --> DB
    WEBAPI --> EVENTBUS
    WEBAPI --> SEARCH
    EVENTBUS --> JOBQ
    JOBQ --> SEARCH
    JOBQ --> WEBHOOK
    JOBQ --> APN
    JOBQ --> EMAIL
    DB --> AUDIT
    WEBAPI --> OBJ
    WEBAPI --> SAML

What this diagram shows. The architecture has three tiers per cell. The edge tier (load balancer, WebSocket gateway, Flannel cache) terminates client connections and serves frequent reads from cache. The app tier (Web API, event bus, job queue, webhook dispatcher) handles writes and orchestrates fanout. The data tier (sharded MySQL via Vitess, Solr search, S3 object storage, audit log) holds durable state. External services for push, email, and identity are integrated at the boundaries. The event bus (Kafka) is the backbone for fanout: a write goes to MySQL, then publishes to Kafka; consumers include the WebSocket gateway (for real-time push), the search indexer, the webhook dispatcher, the push-notification service, and the audit log. Decoupling these lets each scale independently and lets the message-write path commit fast.

6. Request Flow

6.1 chat.postMessage

sequenceDiagram
    participant C as Client
    participant LB as Edge LB
    participant API as Web API
    participant DB as Vitess DB
    participant K as Kafka
    participant WS as WebSocket Gateway
    participant IDX as Search Indexer
    participant PUSH as Push Worker

    C->>LB: POST chat.postMessage
    LB->>API: forward
    API->>API: auth, permission check
    API->>DB: INSERT INTO messages
    DB-->>API: ack, ts assigned
    API->>K: publish(message_event, team_id, channel_id, ts)
    API-->>C: 200 { ok, ts }
    K->>WS: subscribe (per gateway)
    WS->>WS: lookup channel members in cache
    WS-->>C2: push event (each online subscriber)
    K->>IDX: subscribe
    IDX->>IDX: index message
    K->>PUSH: subscribe
    PUSH->>PUSH: dispatch APNS/FCM to mobile subscribers

Walk-through. The Web API (running in Hack on the application servers) handles the synchronous portion: authentication of the bearer token, permission check (is the user a member of the channel?), and the durable write to MySQL via Vitess. The ts is allocated by the channel’s primary shard. After the write, the API publishes a single event to Kafka and returns 200 to the caller. Asynchronously and in parallel, the WebSocket gateway pushes to online channel members, the search indexer adds the message, and the push-notification service handles APNS/FCM/email. Each of these is its own consumer group on the Kafka topic, providing at-least-once semantics with consumer-side dedup on (team_id, channel_id, ts).

The split is essential for two reasons. First, latency: the user-perceived ack happens after only the DB write, not after fanout. Second, isolation: a slow push provider does not block the message-send path.

6.2 Search query

sequenceDiagram
    participant C as Client
    participant API as Web API
    participant FL as Flannel
    participant SEARCH as Solr Cluster
    C->>API: GET search.messages?query=...
    API->>FL: get user's channel-membership set
    FL-->>API: { c1, c2, ... } (cached)
    API->>SEARCH: query + channel filter
    SEARCH-->>API: ranked hits
    API-->>C: results

The membership-set lookup is the load-bearing optimization: a user in 200 channels can have their permission filter computed once and cached in Flannel, so search queries don’t re-compute it on every request.

7. Deep Dive

7.1 Flannel — the application-level edge cache

Slack’s biggest 2016 scale crisis was that every WebSocket connection initial-payload contained the entire workspace’s user roster, channel list, and bot list — multi-megabyte for large workspaces — and reconnect storms (a switch flap, or a Slack deploy that bounced sockets) hit the database tier with thousands of these full snapshots simultaneously. The original code was Hack/PHP (Slack’s evolution: PHP → Hack since 2014, see Hack docs), and the database queries serializing the payload pinned the Vitess shards.

Flannel (Slack engineering, 2017) is a smart edge cache: an application-level service deployed to Slack’s edge points-of-presence that keeps a complete in-memory copy of each workspace’s metadata (users, channels, bots). Each Flannel host maintains its model by opening its own WebSocket to Slack’s main AWS region, through which it receives the live event stream; it then serves a slimmed-down startup payload to clients and lazily answers query APIs for the rest, hitting the backend only on a cache miss. It is more than a passive cache: it prefetches — e.g. when a user is mentioned in a channel, Flannel proactively pushes that user’s profile to the client before the client asks. Clients are routed to a Flannel host by Consistent Hashing to preserve per-team cache affinity. At the time of the post Flannel handled roughly 4 million simultaneous connections and ~600K client queries per second, and it cut the startup payload by ~7× for a 1.5K-user team and ~44× for a 32K-user team — the reduction is what made large-team reconnects survivable.

The architectural principle: moving cache from per-row to per-tenant is what unlocks scale for tenant-scoped systems. A row cache (e.g., memcached on (team_id, user_id)) churns; a tenant cache (Flannel) is mostly stable and the deltas are cheap.

7.2 The fanout job queue

Sending a message at scale means dispatching tens to hundreds of pushes for one write. Slack’s first job queue was Redis-based; it failed under load (Lin, Slack engineering blog, 2017, “Rebuilding Slack’s Job Queue”). The replacement is Kafka-backed with stateful workers: each worker holds an in-memory backlog and acks to Kafka as it completes, with bounded prefetch so a slow downstream cannot pile up uncommitted work.

For fanout to subscribers, the WebSocket gateway is itself the consumer: each gateway subscribes to the Kafka topic partitions for the cells it serves and matches incoming events to its local socket map. That makes the fanout O(matching-sockets-on-this-gateway) per event; horizontal scaling of gateways linearly scales the fanout capacity.

For fanout to webhooks, a dedicated webhook dispatcher pool reads from the topic and POSTs to subscriber URLs with retries (exponential backoff with jitter, dead-letter after 24 hours). Cross-link: Webhook Delivery System Design.

7.3 Search at Slack

Slack’s search backend (engineering blog 2017) is Solr-based, with these load-bearing decisions:

• Per-workspace sharding. Each workspace’s index is segregated; queries from one workspace never touch another’s segments.
• Time-bucketed segments. Hot bucket (last week) is replicated for query throughput; cold buckets (older months) live on cheaper hardware with less replication.
• Permission filter as a post-filter. Index entries do not embed channel membership; instead the query rewrites to (...) AND channel_id IN (user's channel set). The set is precomputed in Flannel.
• Boost recent and most-engaged messages. Slack’s relevance score weights recency, mentions of the user, and engagement signals (replies, reactions) — not pure TF-IDF.

The challenge of permission post-filtering is that highly active users with thousands of channel memberships have a long IN list. Solr handles this via a Lucene TermsFilter that compiles to a bit-set; performance scales linearly in the membership-set size, which empirically is bounded enough to be acceptable.

7.4 WebSocket migration to Envoy (2020)

For most of Slack’s history HAProxy was the load balancer for all incoming traffic, including WebSockets (Slack engineering, 2020). The pain point was operational rather than raw throughput: HAProxy required complex tooling to reload its configuration whenever backend endpoints changed, and because reloads could not preserve in-flight connections, old HAProxy processes had to be kept alive for hours to let long-lived WebSocket connections drain gracefully. Slack migrated to Envoy as the ingress termination layer because Envoy supports dynamically configured clusters and endpoints (updated without a restart) and a “hot restart” that preserves existing connections and statistics across binary upgrades — directly attacking the drain problem. (The blog describes dynamic, file-watched configuration rather than naming the xDS gRPC discovery APIs explicitly, and does not publish exact socket counts beyond “millions of simultaneously connected users at peak.”)

Architectural lesson: the load balancer is the load-bearing component for any persistent-connection architecture. Choose one with first-class WebSocket support, dynamic routing, and good observability hooks.

7.5 Cellular architecture (2024)

The 2024 Slack engineering blog describes a migration from a “everything in one giant deployment per region” architecture to a cell-based deployment: each region has many cells, each a complete vertical slice (web tier + DB + search + Redis + storage). Workspaces are pinned to a cell; a cell has a hard capacity ceiling (workspaces × users × messages/s).

Benefits:

• Blast radius. A bad deploy or DB failure affects only workspaces in that cell.
• Canary. A new release rolls out to one cell first; impact bounded.
• Migration rebalancing. Moving a workspace between cells is a defined operation.
• Compliance isolation. A regulated customer can be pinned to a cell with stricter operational controls.

Costs:

• Cross-workspace operations (e.g., Slack Connect shared channels between two workspaces in different cells) require cross-cell calls, which are slower and more failure-prone.
• Operational complexity. More cells, more deploys, more monitoring surface.

Slack’s bet is that the blast-radius and canary benefits outweigh the operational tax. This is a recurring pattern in mature SaaS — Stripe, Shopify, GitHub have similar partitioning schemes.

7.6 Multi-tenant integrity

Multi-tenancy bugs that leak data across workspaces are existentially bad for a B2B SaaS. Mitigations:

• Mandatory team_id filter. Every database query goes through a layer that asserts team_id is in the WHERE clause. Code review tools and lint rules enforce.
• Per-workspace encryption keys (Enterprise Key Management). A workspace’s data is symmetrically encrypted with a key the customer holds; Slack cannot read at rest without the customer’s KMS key. (Useful for compliance, but operationally complex and not enabled by default.)
• Row-level access tokens. Even within a workspace, private channels enforce per-message permission; the API path threads the user’s identity through the permission check.
• Audit logs. Append-only log of administrative actions, available for export.

7.7 Slack Connect (shared channels)

A shared channel lives in two workspaces simultaneously. The implementation: the channel has two team_id values; each workspace stores an independent membership and permissions list; messages are dual-indexed in both workspaces’ search indexes; sharing rules govern what each side sees of the other (e.g., user profile fields). Internally, this is one of the trickier features because almost every “always filter by team_id” assumption needs careful refinement. Slack’s 2018 engineering blog walks through the implementation.

7.8 Voice / huddles

Slack added “huddles” — lightweight always-on audio (and later video) — in 2021. The implementation reuses standard WebRTC plumbing (Selective Forwarding Unit, see Video Conferencing System Design). The architectural decision: huddles are scoped to a channel and a small number of participants, so the SFU doesn’t need to scale to large-meeting territory.

8. Scaling Strategy

8.1 What breaks first

In practice the first scaling pressure hits the WebSocket gateway and its initial-payload database queries during reconnect storms. Slack’s history of incidents reads as a sequence of “the database for the connection-init query got hammered when N% of sockets reconnected at once.” The fix is the Flannel cache (§7.1) plus carefully staggered reconnect backoff in the client SDK.

8.2 Scaling axes

• Cellular partitioning. Add cells horizontally; each cell is a self-contained scaling unit.
• Read replicas for MySQL. For per-channel timeline reads (the heaviest read path), read replicas with eventual consistency are sufficient.
• Sharding the search index per workspace, then per time bucket. Time bucketing concentrates query load on hot recent data; cold data scales out without affecting hot throughput.
• Caching tiers. Flannel for workspace metadata; Redis for per-user state; CDN edge caches for static asset paths and emoji.
• Asynchronous everything. Search indexing, push notifications, webhook delivery, and audit logging are all consumer groups on the Kafka backbone; each scales independently.

8.3 Hot workspaces

A 200 000-user enterprise workspace is one tenant with the load of an entire region’s worth of small teams. Mitigations: place hot workspaces on dedicated cells; scale the cell’s gateway count proportionally; allow per-workspace rate limits on hot operations (mass mentions, large file uploads).

8.4 Hot channels

#general in a 100K user company is a hot channel: a single message produces 100K push fanouts plus 100K WebSocket pushes. Mitigations: rate-limit mass mentions (@channel triggers a confirmation prompt above N members); fanout in batches with bounded concurrency; suppress “presence change” events in very large channels (the noise cost exceeds the value).

9. Real-World Example

Slack’s PHP → Hack evolution. Slack started in 2013 on PHP 5; by 2014 they migrated the language to Hack (Facebook’s gradually-typed PHP dialect; hacklang.org). The driver was incremental static typing: large codebases benefit hugely from compile-time type checks, and Hack let Slack add types file-by-file. Performance also improved (HHVM JIT). The 2020s migration of select subsystems to Go (and some to Rust) addresses CPU-heavy paths the Hack runtime is poor at.

Slack’s job-queue rebuild (Lin, 2017). The initial Redis job queue had no durability story — Redis crashes lost jobs — and lacked backpressure. The replacement uses Kafka as the durable log and a worker pool that consumes from Kafka with explicit commit-on-completion. Worker failures replay from the last committed offset; the system survives Kafka broker restarts; backpressure is automatic (Kafka’s retention is the buffer).

Search at Slack (Johnson, 2017). The blog post details Solr architecture, time-bucketed sharding, the permission post-filter, and operational lessons (how Solr replication interacts with rolling deploys; the cost of aggressive segment merging). It is the canonical Slack search reference.

Microsoft Teams architecture (Microsoft Docs, 2023). Teams is built on the Office 365 plumbing: Azure Active Directory for identity, SharePoint for files, Exchange for calendar, OneNote, etc. The chat backbone is a separate service (the “messaging service”) with WebSockets to clients and a media stack for voice/video. The integration-heavy nature reflects Microsoft’s strategy: Teams as the front door for the existing Microsoft 365 estate.

Discord is interesting as a contrast: similar feature surface (channels, voice, fanout), built in Elixir on Erlang’s BEAM runtime (Discord engineering blog, 2017). Discord’s “distributing requests across servers” problem is solved by Consistent Hashing of guild IDs to backend processes.

10. Tradeoffs

Choice	Pro	Con	When chosen
Operator-readable messages (no E2EE by default)	Server-side search, anti-spam, compliance hooks, integrations	Operator is a target for legal compulsion; E2EE-style privacy unavailable	Slack, Teams (default); not WhatsApp
Cellular architecture	Bounded blast radius, canary deploys, compliance isolation	Cross-cell features more complex (Slack Connect)	Slack 2024+, Stripe, GitHub
Per-workspace search sharding with time buckets	Scales linearly with workspace count; hot/cold tiering possible	Cross-workspace search is expensive	Slack (Solr)
Permission post-filter at search query time	Index small, doesn’t depend on membership	Membership-set must be fetched per query (Flannel cache)	Slack
Kafka as fanout backbone	Durable, multiple consumers, backpressure	Operational complexity (cluster ops)	Slack post-2017
Flannel edge cache	Fast reconnects, DB protection	Stateful; cache-coherence engineering required	Slack
Hack/PHP for app tier	Massive existing codebase reuse, fast iteration	CPU-heavy paths slow	Slack
Webhook integrations (Events API)	Third-party ecosystem	Retry/idempotency complexity, security (signing)	Slack
WebSockets via Envoy	Modern routing, rich observability	New layer to operate; migration effort	Slack 2020+

11. Pitfalls

1. Forgetting the team_id filter. A query that misses it can leak data across tenants. Enforce mandatorily at the data-access layer.
2. Naïve initial-payload on connect. Including the full workspace state on every reconnect inflates database load during storms. Use Flannel-style edge cache + delta updates.
3. Synchronous fanout on send. Doing search indexing + push fanout + audit logging on the request thread couples the user-facing latency to the slowest downstream. Decouple via Kafka.
4. @channel in megachannels without rate-limiting. A single mass mention can saturate the push fleet. Require admin permission or confirmation prompts above N members.
5. Storing reactions as message edits. Edits invalidate all caches and reindex the message. Store reactions as a separate denormalized counter table that aggregates lazily.
6. Treating ts as a wall-clock timestamp. It is — but two messages at the same microsecond resolve via the suffix counter. Code that compares ts as a Number loses precision (microseconds beyond float64 mantissa); compare as string.
7. Search index that doesn’t carry the latest edits. A message that gets edited must reindex; a message that gets deleted must un-index. The indexer must consume all CRUD events, not just creates.
8. Dropped Kafka consumer offsets. A consumer that loses its offset replays a huge backlog and re-fanouts everything. Store offsets transactionally; alert on consumer lag.
9. Webhook retries that hammer recipients. Exponential backoff with jitter; circuit-breaker per endpoint; capped retry duration.
10. Slack Connect cross-cell calls. A shared channel between cells turns every send into a cross-cell call, which is slower and adds a failure dimension. Option: pin shared channels to one cell with a “home” workspace; the other side reads via a thin federation API.
11. Compliance retention purges that don’t actually delete. A “delete in 30 days” policy that leaves data in a search index or cold storage tier violates regulatory promises. The purge must be transactional across all data tiers or use a tombstone-and-reap pattern with audited completion.
12. Mobile push notification leaks message preview. APNS / FCM payloads are not encrypted at the platform level; sending message body in the push alert leaks plaintext. Many enterprise customers require preview-disabled mode (push wakes app; app fetches message from server).

12. Common Interview Variants and Follow-Ups

• “How do you support enterprise customers with strict compliance (HIPAA, FedRAMP)?” Dedicated cells; Enterprise Key Management with customer-held keys; on-prem connector for data residency; expanded audit log with tamper-evident append-only storage; SAML SSO; data loss prevention hooks.
• “How do you prevent a third-party app from leaking data?” OAuth scopes per app per workspace; admin approval flow; rate-limit per app; sandbox slash commands; signed webhooks (the receiver verifies a per-app HMAC); revoke-everywhere on app removal.
• “Design a per-channel encryption mode.” Possible but loses search. Per-channel symmetric key, distributed via the workspace’s KMS. Search becomes client-side (the Mattermost / Wickr approach) or fully unavailable.
• “How would you design @here vs @channel?” @here only notifies currently online users; @channel notifies everyone. Implementation: the push pipeline filters by presence at fanout time. Care needed in megachannels (still rate-limit).
• “How would you build a ‘mark all as read’ button?” Each user has a per-channel last_read_ts. The button sets last_read_ts = now() for every channel. With 1000 channels per user this is 1000 writes; batch them into one transaction. Cross-link Distributed Counter System Design for the unread-count sub-problem.
• “How does Slack handle giant emoji-reaction storms?” A reaction is a counter on a message. Hot messages (a viral one) get reactions from 10K users; the row-update path becomes a contention hot spot. Mitigation: shard the counter across N rows, sum on read. See Distributed Counter System Design.
• “Compare to Discord.” Discord’s traffic is heavier on voice and lower on persistent text history; Discord uses Cassandra for messages (append-only by (channel_id, ts)), Erlang/Elixir for the gateway, Consistent Hashing for guild routing. The architectural shape is similar; the data-store choices reflect different load profiles.
• “Compare to MS Teams.” Teams piggybacks on Office 365 infrastructure (Azure AD identity, SharePoint files, Exchange calendaring). Slack is more vertically integrated; Teams is more horizontally integrated. Trade-off: Teams has stronger enterprise ecosystem; Slack has tighter UX.

13. Open Questions / Uncertain

Uncertain

Verify: Slack’s exact Vitess/MySQL shard topology, cell counts, and per-cell capacity ceilings. Reason: the 2024 cellular-architecture blog describes the framework (cells as self-contained vertical slices, workspace-pinned) but discloses no concrete shard counts or cell sizes — these are genuinely not public. To resolve: a Slack engineering disclosure or conference talk with topology numbers. Secondary corroboration: the cellular model (bounded blast radius, canary-per-cell) is confirmed by the blog; only the numbers are missing. #uncertain

Uncertain

Verify: whether Slack still runs Solr for search, or has migrated to Elasticsearch / OpenSearch / a custom engine since the 2017 “Search at Slack” post. Reason: the 2017 post unambiguously documents a Solr/Lucene backend (Lucene relevance scoring, TermsFilter permission post-filter), and no later Slack engineering post publicly announces a migration — so the historical Solr claim is confirmed but the current engine is unverified. To resolve: a post-2017 Slack search-infrastructure write-up. #uncertain

Uncertain

Verify: Microsoft Teams’ internal chat-backbone architecture. Reason: Microsoft publishes integration-level architecture (Azure AD identity, SharePoint files, Exchange calendaring) in customer-facing docs/posters, but the real-time messaging service internals are not disclosed. The Office 365 integration claims here rest on those Microsoft docs; the messaging-backbone specifics are inferred. To resolve: a Microsoft engineering deep-dive on the Teams messaging service. #uncertain

• Is server-side end-to-end encryption (where the operator holds keys but compartmentalizes them via HSMs) an acceptable middle ground for compliance customers, or do they always demand customer-held keys?
• How does the cellular model evolve when very large enterprise workspaces grow to dominate a single cell? Does Slack split a workspace’s data across cells, breaking the “one workspace one cell” rule?
• What is the right architecture for Slack Connect federation across cells? A pin-to-cell rule? A proper federation protocol?

14. See Also

• Major System Designs MOC
• SWE Interview Preparation MOC
• WhatsApp Messenger System Design — consumer messaging contrast (E2EE, no server-side search)
• Real Time Collaborative Editor System Design — adjacent real-time problem (Google Docs, Figma)
• Video Conferencing System Design — voice / huddles uses similar SFU plumbing
• Notification Service System Design — push / email / in-app pipeline
• Distributed Search System Design — Solr / Elasticsearch architecture details
• Distributed Log System Design — Kafka, the fanout backbone
• Webhook Delivery System Design — third-party event delivery semantics
• API Gateway System Design — Envoy at the edge
• Consistent Hashing — Vitess sharding by team_id
• Token Bucket — rate-limiting integrations and webhooks
• Distributed Counter System Design — reactions, unread counts