Scaling Patterns

Scaling is the practice of making a system handle more — more users, more requests, more data, more failures — without rewriting it. The patterns below are the standard moves every senior engineer should be able to deploy and trade off in System Design Interview Framework Phase 6. They form a progression: vertical → horizontal → caching → CDN → read replicas → sharding → async → microservices. Each step buys you something specific and costs you something specific. The mistake junior candidates make is jumping to “microservices” or “Kafka” as if they’re free; the senior view is that every step has a cost (operational complexity, latency, consistency surrender) and the right answer is the cheapest step that gets you past the current bottleneck. This note is the bottleneck → fix mapping, the CAP theorem / PACELC framing, and the worked decision rules — comprehensive enough to drive Phase 6 of every system-design round.

1. Why This Matters

Scaling is the senior signal in System Design Interview Framework. Junior candidates produce a high-level diagram; senior candidates take that diagram and walk through what breaks at 10× traffic, 100× traffic, 1,000× traffic. The candidate who can fluently say “the first thing that breaks is the write path because we’re at the single-machine ceiling; the fix is sharding by user_id; the cost is cross-shard timeline queries, which we’ll mitigate with denormalization” is demonstrating staff-level reasoning.

In production, scaling decisions dominate a senior engineer’s time. New systems are often greenfield-easy; scaling the system from 10K users to 10M to 100M is the multi-year job. Knowing the patterns lets you (a) predict the next bottleneck, (b) pick the right fix proactively, (c) avoid premature complexity. Patterns are mnemonics for the trade-off space.

The discipline behind the patterns is make the binding constraint explicit, then attack it specifically. Generic “we’ll add more servers” answers signal weakness; “we’ll add read replicas because the binding constraint is read QPS not write throughput, and we accept eventual consistency on the read path because the use case tolerates ~1 second staleness” signals senior-grade thinking.

2. The Scaling Progression

flowchart TD
    Start[Single Machine<br/>Monolith + RDBMS] --> Vert[Vertical Scaling<br/>bigger machine]
    Vert --> Horiz[Horizontal Scaling<br/>+ Load Balancer]
    Horiz --> Cache[Caching Layer<br/>e.g., Redis, Memcached]
    Cache --> CDN[CDN<br/>for static assets]
    CDN --> Replicas[Read Replicas<br/>primary + replicas]
    Replicas --> Shard[Sharding / Partitioning<br/>split data by key]
    Shard --> Async[Async Processing<br/>queues + workers]
    Async --> Micro[Microservices<br/>organizational split]

What this diagram shows. The canonical path that a successful product walks as it grows. Each transition costs operational complexity and often consistency; each buys throughput or latency or resilience. The path is not strictly linear — some systems skip steps (a serverless-first design jumps directly to horizontal); some loop back (microservices that should have stayed monolith). But for interview purposes, this ordering reflects the typical progression and the correct order of mitigations to propose. Skipping ahead — proposing microservices when you haven’t yet added a cache — is the most common anti-pattern.

2.1 Vertical Scaling — Bigger Machine

The simplest scaling: rent a bigger box. CPU 2× → 4× → 32 cores. RAM 8 GB → 64 GB → 1 TB. Disk 100 GB → 10 TB. Modern cloud VMs reach absurd sizes (AWS x1e.32xlarge: 128 vCPUs, 3.9 TB RAM).

What it buys you: zero code changes. Your application remains a monolith on one box; you just give it more resources. Operational simplicity is the top virtue.

What it costs you:

• Hard ceiling. Even the biggest cloud VM has limits. ~10× capacity, then you’re stuck.
• No fault tolerance. If the one machine dies, the system is down.
• Cost. Per-CPU prices skyrocket on the largest VMs; 10× the capacity often costs 30–100× the price.

When to use: before you’ve outgrown a single machine, vertical scaling is almost always the right answer. Don’t introduce horizontal complexity until you must.

Real-world note: Stack Overflow famously runs on a tiny number of large machines (single-digit servers handling all traffic for years). Vertical scaling + careful engineering kept them off Kubernetes-style architectures for over a decade.

2.2 Horizontal Scaling — More Machines

You can no longer fit on one machine, or you need fault tolerance. Add machines and put them behind a Load Balancer that distributes incoming requests across them.

Prerequisite — stateless services. Horizontal scaling works trivially only for stateless services. A stateless service is one where any request can be handled by any instance, with no instance-local state required (sessions, caches, in-memory data). Stateful services require either sticky sessions (route a user to the same instance every time — fragile) or state extraction (pull state out into a backing store — extra latency).

What it buys you:

• Effectively unlimited compute. Add more boxes; cluster scales near-linearly for stateless workloads.
• Fault tolerance. Lose one machine; load balancer routes around it.
• Rolling deploys. Update a few machines at a time without downtime.

What it costs you:

• Load balancer complexity. L4 vs L7; health checks; failure detection.
• Stateful services break. All state must move to a backing store, which becomes the new bottleneck.
• Distributed-systems failure modes. Partial failure, retries, idempotency.

Practical pattern: for an HTTP API, the load balancer (Nginx, HAProxy, ALB, GCLB, or Cloudflare) sits at the edge. Behind it, 3–N stateless application servers. Behind them, a shared database. As traffic grows, you add more app servers; the LB and DB remain.

2.3 Caching Layer

The application servers do work — query DB, transform results — that gets repeated for popular requests. A cache (in-memory KV store like Redis or Memcached, or a process-local cache) saves the result so subsequent requests skip the work.

What it buys you:

• Reduced DB load. A 99% cache hit rate means the DB sees 1% of read traffic. That’s the difference between needing to shard and not.
• Lower latency. RAM-resident cache hits at ~1ms (or ~100 µs for in-process); DB queries at ~10ms+.
• Capacity headroom. Same DB now serves 100× the original QPS.

What it costs you:

• Cache invalidation (one of the two famously hard things in computer science, per Phil Karlton). When the underlying data changes, the cache entry must be updated or evicted.
• Cache stampede. When a popular cache entry expires, thousands of requests simultaneously miss → all hit the DB → DB falls over.
• Consistency. Cache may serve stale data for the TTL window.

Strategies:

Cache eviction policies — see Least Recently Used Cache, Least Frequently Used Cache, Adaptive Replacement Cache. LRU is the default; LFU favors long-tail popularity; ARC adapts.

Cache write strategies — see Write-Through vs Write-Back:

• Write-through: every write goes to cache and DB simultaneously; cache is always fresh; writes are slower.
• Write-back: writes go to cache only; cache flushes to DB asynchronously; writes are fast but data loss on cache failure.
• Write-around: writes bypass cache; cache populated by reads; good for write-heavy workloads where written data is rarely read.
• Cache-aside (lazy loading): application reads from cache; on miss, reads from DB and populates cache; most common pattern.

Cache invalidation: TTL (Time-To-Live) expiry; explicit invalidation on write; pub/sub-based propagation across cache nodes.

Stampede prevention: request coalescing (only one request to DB per missing key; others wait); probabilistic early refresh (recompute before TTL expires for hot keys); negative caching (cache “not found” results too).

2.4 Content Delivery Network (CDN)

Static assets (images, JS, CSS, video) and increasingly dynamic-but-cacheable content (HTML, API responses with TTL) get cached at geographically distributed edge servers. A user in Tokyo gets served from a Tokyo edge; the origin in us-east-1 sees only cache misses.

What it buys you:

• Massive bandwidth offload. YouTube, Netflix, Cloudflare-fronted sites push 95%+ of traffic from edges; the origin handles a tiny fraction.
• Low latency globally. Speed of light is unavoidable for cross-continent requests; edges put content close to the user.
• DDoS absorption. CDNs front-line attacks before they hit your origin.

What it costs you:

• Cost. CDN egress is metered, often expensive at scale (though hyperscalers’ own CDNs subsidize this for their customers).
• Cache invalidation across the edge. When you publish a new version of app.js, every edge needs to refresh — typically via versioned URLs (app.v123.js) and short TTL on the index that references them.
• Dynamic content limits. API responses are harder to cache (per-user, time-sensitive).

Common providers: Akamai (the original, founded by Consistent Hashing paper authors), Cloudflare, AWS CloudFront, Fastly, Google Cloud CDN.

Anti-pattern: putting a CDN in front of a non-cacheable API and being surprised that nothing speeds up. CDN value comes from cache hits.

2.5 Read Replicas — Primary-Replica Replication

For read-heavy workloads, the database becomes the bottleneck even after caching. Replicate the database to multiple read replicas; route read queries to replicas; route writes to a single primary.

flowchart LR
    App[Application] -->|writes| Primary[(Primary DB)]
    App -->|reads| R1[(Replica 1)]
    App -->|reads| R2[(Replica 2)]
    App -->|reads| R3[(Replica 3)]
    Primary -.->|async replication| R1
    Primary -.->|async replication| R2
    Primary -.->|async replication| R3

What it buys you:

• Linear read scaling. Adding replicas adds read capacity.
• Geographic distribution. Replicas in multiple regions reduce read latency for global users.
• Failover capacity. Replicas can be promoted on primary failure.

What it costs you:

• Replication lag. Replicas are seconds-to-minutes behind the primary; reads see stale data.
• Write bottleneck unchanged. Writes still go to one primary; this doesn’t help write QPS.
• Read-after-write inconsistency. A user writes a comment, then immediately re-reads — but the read goes to a replica that hasn’t seen the write yet → “where did my comment go?” Solutions: route the read to the primary for the immediate-after-write window; or use read-your-writes logical-clock-based routing.

Common configurations:

• Primary + N replicas (typical: 1 primary + 2–5 replicas).
• Hot standby (one replica synchronously replicated; ready to take over).
• Multi-master (writes accepted at multiple nodes; resolves conflicts on replication; e.g., Raft-based or CRDT-based — see CRDTs Basics).

Database-specific: MySQL has primary-replica natively; PostgreSQL via streaming replication; MongoDB via replica sets; Cassandra is multi-master by design (every node is a replica).

2.6 Sharding / Partitioning — Splitting Data by Key

When even the primary can’t keep up with writes, or when the data is too big for one machine, partition the data across multiple databases. Each shard owns a subset of the keys; queries are routed to the right shard.

What it buys you:

• Linear scaling for writes. N shards = N× write QPS.
• Linear scaling for storage. N shards = N× capacity.
• Failure isolation. A bad query on one shard doesn’t affect others.

What it costs you:

• Cross-shard queries. “Show me all tweets” now hits every shard (scatter-gather). For aggregation, this is expensive.
• Re-sharding is hard. Adding a shard means moving data; minimizing how much moves is what Consistent Hashing solves.
• Joins become difficult for SQL stores; usually you denormalize or move to a NoSQL design.
• Distributed transactions are needed if a write touches multiple shards (see Two-Phase Commit).

Sharding strategies:

Range-based sharding: keys 0..1M go to shard 1, 1M..2M go to shard 2, etc. Simple but vulnerable to hot-spots if data isn’t uniformly distributed.

Hash-based sharding: shard = hash(key) mod N. Uniform distribution; bad for re-sharding (changing N moves nearly every key).

Consistent Hashing: hash both keys and shards onto a ring; only ~K/N keys move when shards are added/removed. The standard for horizontally-scalable KV stores (DynamoDB, Cassandra, Memcached client libraries).

Directory-based sharding: a lookup service maps key → shard. Flexible but adds a hop.

Choosing a partition key: the most important data-model decision. The partition key determines which queries are cheap (single-shard) and which are expensive (cross-shard). If you partition tweets by tweet_id, then “tweets by user X” requires fan-out across all shards. If you partition by user_id, then tweets by user X are cheap, but global queries are expensive.

2.7 Async Processing — Queues and Event-Driven

Synchronous request paths must complete within the user’s request budget (often < 200 ms). Anything that takes longer — sending an email, transcoding a video, generating a report, indexing a document for search — should be handed off to a queue and processed asynchronously.

flowchart LR
    User -->|HTTP| API[API Server]
    API -->|"enqueue 'send email'"| Q[(Queue<br/>e.g., Kafka, RabbitMQ, SQS)]
    API -->|"return 200"| User
    Q --> W1[Worker 1]
    Q --> W2[Worker 2]
    Q --> W3[Worker 3]
    W1 --> Email[Email Service]

What it buys you:

• Fast user-facing response times. API returns in ms; the slow work happens later.
• Spike absorption. Queue buffers traffic spikes; workers process at sustainable rate.
• Failure isolation. If the email service is down, jobs sit in the queue until it recovers; the API isn’t affected.
• Decoupling. Producers and consumers are independent; they can be scaled independently.

What it costs you:

• Eventual consistency. The user sees the action as “done” but it’s still pending; UI must reflect this.
• Idempotency required. Workers may process the same message multiple times (queue redelivery on failure); operations must be idempotent or use deduplication.
• Operational complexity. A queue is another component to monitor — depth, lag, dead-letter queues.

Common queue technologies: Kafka (high-throughput log), RabbitMQ (rich routing), AWS SQS / SNS (managed), Google Pub/Sub, Redis Streams (lightweight), NATS.

Patterns:

• Work queue: producer puts jobs; competing consumers pick them up; one worker per job (no duplicates).
• Pub/sub (broadcast): producer publishes; multiple consumers each see all messages.
• Event sourcing: application state derived from a log of events; replay reconstructs state.
• CQRS (Command Query Responsibility Segregation): writes go to one model (commands → events); reads come from another (denormalized projections).

2.8 Microservices — Organizational Split

The final stage: split a monolith into many small services, each owning a slice of functionality. This is an organizational pattern, not (primarily) a performance pattern.

What it buys you:

• Independent deployment. Team A pushes service A; team B pushes service B; they don’t block each other.
• Polyglot stacks. Service A in Python; service B in Go; choose tech per service.
• Failure isolation (when done well — see costs).
• Team scaling. Conway’s law in reverse: org structure shapes architecture; microservices let many teams ship in parallel.

What it costs you:

• Distributed-systems complexity everywhere. Network calls instead of in-process function calls. Latency, partial failure, retries.
• Operational overhead. Each service has its own deployment, monitoring, on-call.
• Data partitioning. Each service owns its data; cross-service queries become API calls or event-replay.
• Latency amplification. A single user request that touches 10 services = 10 network hops; tail latencies multiply.

When to use: when team size has outgrown a single codebase. Premature microservices are the canonical anti-pattern of the 2010s — Uber, Netflix, and others have written publicly about scaling back.

Conway’s Law (Melvin Conway, 1968): “Organizations design systems that mirror their communication structures.” Microservices match an org of independent product teams; monoliths match a single coordinated team.

3. The CAP Theorem and PACELC

Eric Brewer (PODC 2000): a distributed system can guarantee at most two of three properties:

• Consistency — every read sees the latest write.
• Availability — every request gets a response (success or failure).
• Partition tolerance — the system continues operating despite network partitions.

Network partitions are not optional in real systems (the network will fail), so P is required; the choice is between C and A.

CP systems (consistency over availability under partition): traditional RDBMS (PostgreSQL, MySQL when configured for sync replication), HBase, MongoDB (with default settings post-3.6), most relational systems, Raft-based systems, Spanner. During partition, some requests fail to preserve consistency.

AP systems (availability over consistency under partition): Cassandra (default), DynamoDB (eventually consistent reads), Riak, CouchDB. During partition, all requests succeed but may return stale data.

The 2002 Gilbert-Lynch paper formally proved Brewer’s conjecture and clarified subtleties: CAP only applies during a partition; when there’s no partition, you can have all three.

3.1 PACELC — The Underdiscussed Refinement

Daniel Abadi (2010, formalized 2012): even when there’s no partition, distributed systems still trade latency for consistency. PACELC: Partition → A/C, Else → L/C.

When there’s a Partition, choose Availability or Consistency. Else, choose Latency or Consistency.

A system that requires every read to see the latest write must coordinate (consensus, quorum reads), which adds latency. A system that tolerates eventual consistency can serve reads from the nearest replica, low latency.

Example classifications:

• Cassandra: PA/EL (during partition: available; else: low latency, eventual consistency).
• PostgreSQL with sync replication: PC/EC (during partition: consistent — fails some writes; else: consistent — slower).
• DynamoDB (default): PA/EL.
• DynamoDB with strongly consistent reads: PC/EC.
• MongoDB (post-3.6 default): PC/EC for primary reads; PC/EL for secondary reads.
• Spanner (Google): PC/EC (uses TrueTime + Paxos for global consistency; pays latency tax).

The PACELC framing is more useful than CAP alone because most of the time, there’s no partition; the latency-vs-consistency trade-off applies all the time. Senior interview answers reference PACELC explicitly: “for the timeline read path, I’m choosing low latency (EL); for the payment write path, I need strong consistency (EC).“

3.2 Tunable Consistency

Many modern systems let you choose consistency per request rather than globally:

• Cassandra / DynamoDB: specify the read/write quorum (ONE, QUORUM, ALL). Quorum-based reads (e.g., R + W > N) guarantee strong consistency; weaker quorums trade for latency.
• MongoDB: read concern (local, majority, linearizable).
• CockroachDB / Spanner: all reads are strongly consistent by default; an opt-in “stale read” allows trading.

Tunable consistency lets you mix CAP positions in a single application — CP for payments, AP for likes. This is increasingly the norm.

4. The Bottleneck → Fix Mapping (The Money Table)

This is the table interview candidates should have memorized. When the interviewer says “what if X breaks at 10× scale?”, the answer should come reflexively.

Bottleneck	First fix	Second fix	Cost
DB read overload	Read replicas	Cache layer (Redis)	Replication lag; cache invalidation
DB write overload	Sharding	Async write path via queue	Cross-shard queries; eventual consistency
Network bandwidth (egress)	CDN for static	Compression; client-caching	CDN cost; cache invalidation
Compute (stateless service)	Horizontal scale + LB	Autoscale	LB complexity
Compute (stateful service)	Extract state to backing store	Shard state	Latency; state-store becomes new bottleneck
Hot key (one row dominant traffic)	Replicate hot key	Key splitting	Application-level complexity
Storage exhaustion	Archive cold data	Partition by date; drop old partitions	Slow queries on archived data
Latency tail (p99 spikes)	Hedged requests	Per-tenant rate limits	More request volume; complexity
Failure of dependency	Circuit breakers	Retries with backoff + jitter	Cascading retries can amplify load
Cross-region latency	Region-local replicas	Geo-routing	Eventual consistency between regions
Single-point-of-failure	Replicate to 3+ nodes	Multi-region failover	Cost; consistency complexity
Cold cache after restart	Pre-warm from neighbor	Persistent cache (e.g., Redis with disk)	Slow startup; staleness
Thundering herd on cache miss	Request coalescing	Probabilistic early refresh	Implementation complexity
Slow leader in consensus	Switch to multi-leader	Quorum-based without leader	Conflict resolution complexity
Schema migration during peak	Online schema change	Dual-write + backfill	Operational risk
Search index ballooning	Tiered storage; archive shards	Aggregate older periods	Slow queries on archive

4.1 Worked Examples From The Table

“DB writes can’t keep up at 50K QPS” → first fix: shard. By what key? Whichever key dominates the write path. For a tweet system: by tweet_id (or by user_id if your queries are mostly per-user). Second fix: introduce a write-buffer queue (Kafka) so writes are async; the DB writes happen at sustainable rate.

“User reports their comment didn’t show up” → cache or replication lag. Diagnose: was the read served from a stale replica? If so, route reads-after-write to the primary for the next few seconds (read-your-writes consistency). Or: was the cache stale? If so, add cache invalidation on write.

“Black Friday traffic 100× normal” → hedged requests + autoscale + CDN absorption + queue buffering for non-essential operations. The point: spikes overwhelm because every component must scale simultaneously; staggered defenses (CDN absorbs first; autoscale catches up; queues buffer the rest) handle them.

“Service A → Service B chain is slow” → check tail latency. Service A’s p99 is the integral of Service B’s p99 plus Service A’s own work. If Service B’s p99 is 100ms and you call it twice, your p99 includes 200ms minimum. Fix: cache, reduce calls, parallelize.

5. The Stateless / Stateful Split

The single most important architectural decision for horizontal scaling.

Stateless service: every request can be handled by any instance. No instance-local memory carries between requests.

Stateful service: instance memory (or local disk) carries state between requests; cannot be freely replicated.

5.1 Why Stateless Matters

For a stateless service, scaling is trivial:

• Spin up more instances behind the LB.
• Lose an instance → its in-flight requests retry on another.
• Rolling deploys are safe.

For a stateful service, every operational concern becomes hard:

• New instance: needs to inherit state from somewhere.
• Failure: state on the failed instance is lost or trapped.
• Rolling deploys: must drain the instance first; or accept downtime.

5.2 The State Extraction Pattern

Take your stateful service and extract the state to a backing store. The service becomes stateless; the backing store is now the stateful piece — but it’s a single, well-understood piece (a database, a cache, a queue) that has its own scaling story.

Example: session state.

• Bad: store sessions in in-memory hashmap on each app server. → can’t horizontally scale; sticky sessions are fragile.
• Good: store sessions in Redis. → app servers are stateless; any instance can handle any session.

Example: local cache.

• Bad: each app instance has its own cache. → cache hit rate drops to 1/N as you scale; thundering-herd on cache restart.
• Good: shared cache (Redis) plus a small per-instance L1. → cache hit rate is global; per-instance L1 absorbs the hottest entries.

5.3 Where State Genuinely Belongs

Some services must be stateful — databases, caches, queues, search indexes. For these, the patterns are:

• Replication (multiple copies for failover).
• Sharding (partition the state across many machines).
• Consensus (so the replicas agree on the order of writes — see Raft, Paxos High-Level).

Stateful services are the hard part of distributed systems. Everything else is easier than they are.

6. Worked Decision Rules

These are the questions to ask yourself when designing or scaling a system.

6.1 Read-Heavy vs Write-Heavy

If reads >> writes (10:1 or more), invest in caching and read replicas first. Sharding can come later. If writes >> reads, invest in LSM Tree-based stores, sharding, and async write paths. If 1:1, design carefully — neither pattern fully solves your problem.

6.2 Strong Consistency or Eventual?

Default to eventual unless explicitly required. Strong consistency required: payments, inventory (“did the last item really sell out?”), user-account creation. Eventual is fine: like counts, view counts, timeline updates, search index refreshes.

6.3 When to Add a Cache

Cache adds complexity (invalidation, stampedes, consistency). Don’t add until DB load proves it’s needed. When DB CPU > 60% sustained, cache is justified. Skewed access (Pareto: 1% of keys = 99% of traffic) makes caches very effective; uniform access makes them ineffective.

6.4 When to Shard

Shard when (a) data exceeds single-machine capacity, or (b) write QPS exceeds single-machine capacity. Don’t shard pre-emptively. Sharding is a one-way door — once you’ve sharded, joining and global queries are expensive forever.

6.5 When to Go Microservices

Go microservices when team size > ~20 engineers in one codebase, not before. Premature microservices is the #1 architectural anti-pattern of the 2010s. Start with a modular monolith; extract services as team boundaries clarify.

6.6 When to Use a Queue

Use a queue when (a) operation is slow (> ~200ms) and not user-blocking, or (b) operation needs decoupling, or (c) operation needs spike absorption. Don’t queue every operation — synchronous is simpler and lower-latency for fast operations.

6.7 Pull or Push?

Pull (consumer-driven): consumer queries a producer; simple; consumer can pace itself. Push (producer-driven): producer notifies consumer; lower latency; harder to manage backpressure. For high-fanout (one event → many consumers): pub/sub (push). For low-fanout: either is fine.

7. Common Failure Modes — The Anti-Patterns

7.1 Premature Microservices

Symptom: 5-engineer startup designs 30-microservice architecture. Why it fails: operational complexity dwarfs the benefit; deployment, debugging, and on-call become nightmares for a team this size.

Fix: start with a modular monolith. Use packages to enforce boundaries within the codebase. Extract services only when an organizational boundary clearly exists.

7.2 Buzzword Architectures

Symptom: “We’ll use Kafka, Spark, Flink, Cassandra, Elasticsearch, Kubernetes.” Why it fails: each component is justified by fashion not requirement; the team doesn’t know how to operate any of them; debugging requires cross-system expertise.

Fix: every component must justify its place by a specific bottleneck it solves. “We use Cassandra because we need write QPS > 100K and tolerate eventual consistency” is engineering. “We use Cassandra” is fashion.

7.3 Skipping the Cache

Symptom: design with sharded DB but no cache. Why it fails: a read-heavy workload doesn’t need sharding yet — cache would have bought 10–100× headroom at 1/10 the operational complexity.

Fix: order matters. Cache before shard.

7.4 Forgetting Consistency at the Cache

Symptom: cache layer added; writes go to DB only; cache is updated lazily on next read. Why it fails: writes are invisible to subsequent reads for the cache TTL window — confusing UX bugs.

Fix: write-through, invalidation on write, or short TTLs aligned with user expectations. Don’t leave consistency unspecified.

7.5 Ignoring Replication Lag

Symptom: write to primary, immediately read from replica, expect to see the write. Why it fails: replication is async; replica is seconds behind. Read returns stale data.

Fix: route reads-after-write to primary; or use causal-consistency mechanisms (e.g., session tokens that pin reads to writes).

7.6 Sharding by the Wrong Key

Symptom: shard by tweet_id, then build a “tweets by user” feature → scatter-gather across all shards on every query. Why it fails: most queries become global, defeating the point of sharding.

Fix: shard by the key that dominates the query pattern. If user-scoped queries dominate, shard by user_id; if tweet-scoped queries dominate, by tweet_id. Pick once; switching is expensive.

7.7 Synchronous What Could Be Async

Symptom: send confirmation email during the user’s request, blocking response by 500ms. Why it fails: user-facing latency suffers; flaky email provider takes down your API.

Fix: queue it. Return success to the user immediately; the email gets sent (eventually) by a worker.

7.8 No Backpressure

Symptom: a service receives more traffic than it can handle, queue grows unbounded, eventually OOM (Out Of Memory) or slow → cascading failure. Why it fails: no mechanism to slow producers down.

Fix: bounded queues, load shedding (drop excess requests with 429 / 503), rate limiting upstream (see Token Bucket, Sliding Window Rate Limiter, Leaky Bucket).

7.9 Retries Without Jitter

Symptom: client retries failed requests on a fixed interval. Why it fails: 1000 clients all retry at exactly the same time → herd → service falls over again → herd again. Failure becomes self-amplifying.

Fix: exponential backoff with jitter. Each retry waits longer; jitter spreads retries randomly across a window. The pattern from AWS (“Exponential Backoff and Jitter,” AWS Architecture Blog) is the canonical reference.

7.10 No Circuit Breaker

Symptom: dependency Service B is down; your Service A keeps trying every request, exhausting threads and dragging itself down. Why it fails: A becomes unhealthy because of B’s failure.

Fix: circuit breaker (per Hystrix / Polly patterns). After N failures, “open” the circuit — fail fast for some time without making the call. Periodically test (half-open) to detect recovery.

8. Sample Phrases / Scripts

Diagnosing the bottleneck:

• “The first thing that breaks at 10× is the database read path. Specifically: …”
• “I’m assuming the binding constraint is read QPS, not write — does that match the workload?”
• “Let me trace a request: it goes A → B → C → D. The slowest hop is C, so C is where I’d start.”

Proposing fixes:

• “Cheapest fix first: cache. If that doesn’t suffice, replicas. Then sharding. Microservices only if team size demands it.”
• “I’d shard by user_id because most queries are user-scoped; cross-user queries are rare.”
• “I’m willing to accept eventual consistency on the read path because the use case tolerates ~1 second of staleness.”

Trade-off articulation:

• “This buys us X but costs us Y; for this product, X matters more.”
• “I’m choosing CP over AP for this subsystem because correctness matters more than uptime here.”
• “Cache hit rate of 99% is a big claim; let me justify it: the access distribution is heavily skewed.”

Recovering from a wrong choice:

• “On reflection, sharding by tweet_id is wrong — let me revise. The hot query is ‘tweets by user X’, so I’ll shard by user_id instead.”
• “Let me back up — I conflated two consistency requirements.”

9. Decision Diagram — Picking the Next Move

flowchart TD
    Start{"What's the<br/>bottleneck?"}
    Start --> R{"Read-heavy?"}
    R -->|"yes"| C{"Cache hit rate<br/>achievable<br/>(skewed access)?"}
    C -->|"yes, 90%+"| Cache["Add cache layer<br/>(Redis/Memcached)"]
    C -->|"no, uniform"| Replicas["Read replicas;<br/>geographic distribution"]
    R -->|"no, write-heavy"| W{"Write QPS<br/>> single-machine?"}
    W -->|"yes"| Shard["Shard by partition key<br/>+ async write path"]
    W -->|"no"| Vert["Vertical scaling first;<br/>profile + optimize"]
    Start --> S{"Storage exhausted?"}
    S -->|"yes"| Tier["Hot/warm/cold tiering;<br/>archive cold partitions"]
    Start --> L{"Latency p99 too high?"}
    L -->|"yes"| Async["Move slow ops to queue;<br/>respond to user fast"]

What this diagram shows. Picking the next scaling move is a function of which axis is the binding constraint. Read-heavy with skewed access → cache wins; read-heavy with uniform access → replicas. Write-heavy past single-machine → sharding. Storage-exhausted → tiering. Latency-bound → async. Different bottlenecks demand different fixes; the diagram is the decision skeleton. The mistake to avoid: applying a fix because it sounds sophisticated, not because it solves the actual bottleneck.

10. Open Questions

• Is multi-master replication ever worth it for non-CRDT use cases? Cassandra and DynamoDB say yes; most relational systems say no. Verify against current literature.
• How do serverless platforms (Lambda, Cloud Run) change the “horizontal scaling” pattern? They abstract it but don’t eliminate the cost — cold starts and per-request overhead are the new bottleneck.
• When does Spanner-style globally-strong-consistency become economically feasible vs. eventual-consistency systems? The TrueTime hardware (atomic clocks + GPS) was once Google-only but is increasingly commoditized.
• Have service meshes (Istio, Linkerd) made microservices’ operational tax materially lower? Anecdotal data is mixed; some teams report yes, others report adding more complexity than they removed.

11. See Also

• System Design Interview Framework — Phase 6 is where these patterns deploy
• Back-of-Envelope Estimation — the numbers that drive the bottleneck identification
• Edge Cases Checklist — also for deep-dives
• Debugging Under Pressure — when the design isn’t working
• STAR Framework — behavioral analog
• CAR Framework — behavioral analog
• Picking Stories — Coverage Matrix — selecting examples
• Problem Decomposition in Interviews — fuzzy-prompt skill
• Talking Through Code — narrating decisions
• Time Management in Interviews — clock discipline
• FAANG Interview Loops — where scaling is graded
• Levels and Calibration — per-level signal weight
• How to Practice — Spaced Repetition for Algorithms — drilling
• Negotiating Offers — what comes after
• Resume for SWE Roles — getting to the loop
• Consistent Hashing — sharding building block
• Least Recently Used Cache — cache eviction primitive
• Least Frequently Used Cache — cache eviction primitive
• Adaptive Replacement Cache — better cache policy
• Bloom Filter — low-storage probabilistic primitive
• LSM Tree — write-optimized storage
• Inverted Index — search primitive
• Token Bucket — rate limiter
• Sliding Window Rate Limiter — rate limiter
• Leaky Bucket — rate limiter
• Fixed Window Rate Limiter — rate limiter
• Raft — consensus primitive
• Paxos High-Level — consensus primitive
• Two-Phase Commit — distributed-transaction primitive
• Write-Through vs Write-Back — caching write strategy
• CRDTs Basics — conflict-free replicated types
• Vector Clocks — causal-consistency primitive
• Gossip Protocol — decentralized state propagation
• CAP theorem — consistency vs availability framing
• B-Tree — read-optimized storage primitive
• B+ Tree — read-optimized storage primitive
• SWE Interview Preparation MOC