Leader-Follower Replication Architecture
Leader-follower replication is the distributed-systems topology in which exactly one node in a replica group accepts writes (the leader — also called the primary or, in deprecated terminology, the master) and all other replicas (followers, replicas, secondaries, or, in deprecated terminology, slaves) passively receive a replication stream of those writes and apply them locally. Reads can be served by the leader, by followers, or by both depending on the chosen consistency mode. This is the dominant replication architecture in relational databases (PostgreSQL, MySQL), in document databases (MongoDB replica sets), in per-partition replication of distributed logs (Apache Kafka), and in caching tiers (Redis with Sentinel). It is conceptually simple — there is one source of truth at any instant, conflict resolution is unnecessary because there is only one writer — but the simplicity carries a hidden cost: failover is a hard distributed-systems problem because demoting a dead leader and promoting a new one without producing a split-brain (two leaders simultaneously) requires an underlying consensus protocol such as Raft or Paxos High-Level, or an external coordinator such as ZooKeeper / etcd. Many of the most-cited production outages of the past fifteen years (the GitHub October 2018 outage, the GitLab January 2017 incident, dozens of MySQL replication breakages) are stories about leader-follower failover going wrong — which is why this architecture, despite being the textbook default, deserves careful study.
1. When to Use, When Not to Use
Use leader-follower replication when:
- Writes have a natural single owner. A relational database with a single ACID-transactional write workload fits this naturally. There is one logical “current state” and one node responsible for evolving it; followers exist to absorb read traffic and to serve as hot standbys.
- You need read scaling but not write scaling. Read-heavy workloads (typical of most operational databases — read-write ratios of 10:1 to 1000:1 are common) can fan reads to followers and let the leader concentrate on writes. This linearly scales read throughput at the cost of replication lag.
- You can tolerate brief unavailability during failover. When the leader fails, the cluster is write-unavailable for the seconds-to-minutes that election + promotion takes. If your business can absorb a 30-second write outage, leader-follower is straightforward. If it cannot, you need Multi-Leader Replication Architecture or Leaderless Replication Architecture.
- You require strong consistency on the write path. Single-leader is the simplest road to linearizable writes — there is one ordering authority, so writes are trivially totally ordered. Reads from the leader inherit that linearizability. (Reads from followers require additional machinery to remain linearizable.)
- The data model has cross-key consistency. Multi-row transactions with foreign keys, joins, and constraints are dramatically simpler when one node is the sole writer. Leaderless systems sacrifice this for availability.
Avoid leader-follower replication when:
- Writes must remain available during partitions. If your network occasionally splits and a write outage is unacceptable (e-commerce shopping carts, IoT telemetry ingestion), the leader’s unavailability under partition is fatal. The Dynamo paper’s central design statement — “an unavailable shopping cart is worse than a stale shopping cart” — is exactly this rejection.
- Writes originate from multiple geographies. Cross-region writes that all must funnel to one leader pay a full Round Trip Time (RTT) penalty per write. A user in Tokyo writing to a leader in Virginia adds ~150 ms per write. Multi-Leader Replication Architecture or sharded multi-leader (CockroachDB, Spanner) addresses this.
- The write workload exceeds a single node’s capacity. A leader is a bottleneck. Once you are CPU-bound or IO-bound on the leader, adding followers does not help — you need Sharded Architecture or a leaderless design that spreads writes.
- The data is naturally partitioned with no cross-partition consistency. If you have many independent users / tenants / shards, running a leader-follower group per shard is correct, but a single global leader is wrong; that is a Sharded Architecture composition issue, not a leader-follower issue per se.
The default heuristic remains: if you have any doubt, start with leader-follower. It is the architecture used by every major operational database, every major message queue (in some form), and every major cache. The alternatives are corrections to specific failure modes, not general improvements.
2. Structure
flowchart TB Client1[Client A] -->|writes| Leader Client2[Client B] -->|writes| Leader Client3[Client C] -->|reads| Leader Client4[Client D] -->|reads<br/>(may be stale)| F1 Client5[Client E] -->|reads| F2 Leader[(Leader<br/>writes + reads)] F1[(Follower 1<br/>read-only)] F2[(Follower 2<br/>read-only)] F3[(Follower 3<br/>read-only)] Leader -->|replication stream<br/>WAL / oplog / binlog| F1 Leader -->|replication stream| F2 Leader -->|replication stream| F3 Coord[Coordinator / Consensus<br/>e.g. Sentinel, etcd,<br/>internal Raft group] -.->|monitors| Leader Coord -.->|monitors| F1 Coord -.->|monitors| F2 Coord -.->|monitors| F3 style Leader fill:#fab,stroke:#933 style F1 fill:#bdf,stroke:#358 style F2 fill:#bdf,stroke:#358 style F3 fill:#bdf,stroke:#358
What this diagram shows. A replica group of four nodes: one leader (highlighted) and three followers. All client writes go to the leader (the red arrows from Client A and Client B). Reads can be served by the leader (Client C, getting fresh data) or by any follower (Clients D and E, possibly seeing slightly stale data due to replication lag). The leader streams a continuous replication log — the Write-Ahead Log (PostgreSQL), the binary log (“binlog”, MySQL), the oplog (MongoDB), or the partition log (Kafka) — to every follower over a long-lived TCP connection. Followers apply the stream in order, mutating their local state. A separate coordinator (sometimes a single coordinator like Redis Sentinel, sometimes a distributed consensus group like an embedded Raft cluster, sometimes an external system like etcd or ZooKeeper) monitors all nodes via heartbeats and decides who is leader. The coordinator is not on the write path during normal operation — it is consulted only at startup, on configuration change, or when a leader appears to have failed. The dotted arrows from the coordinator are out-of-band control plane traffic, not data plane.
The critical design property visible in the diagram: there is exactly one red node. Every other shape is a function of the leader’s stream. This single-source-of-truth property is what makes consistency reasoning tractable and what makes failover difficult.
3. Core Principles
Principle 1: One writer at a time. At any moment, exactly one replica is authoritative for new writes. This is the entire foundation of the architecture. Violations — two replicas accepting writes simultaneously — produce split-brain, where each ends up with a divergent history that cannot be reconciled by simple replay. Splitting brain is not an exotic failure; it is the routine consequence of a leader being declared dead while still alive (a classic outcome of garbage-collection pauses, hung disk IO, or asymmetric network partitions). Every leader-follower system exists in tension with this risk.
Principle 2: Replication is asynchronous by default, synchronous by exception. The replication stream from leader to follower is, in nearly all production systems, asynchronous — the leader commits a write locally, replies to the client, and propagates the change to followers as a background activity. This is fast (one local commit + reply) but leaks: if the leader crashes after replying but before propagating, the write is lost. Synchronous replication waits for at least one follower’s acknowledgement before replying, eliminating the loss window at the cost of doubling write latency and making writes unavailable when no follower can acknowledge. Semi-synchronous (MySQL’s term) compromises: wait for one follower, but fall back to async after a timeout.
Principle 3: The replication stream is a totally ordered log. Every change applied at the leader is captured as an entry in a per-leader sequential log (Log Sequence Number / LSN in PostgreSQL, GTID in MySQL, oplog timestamp in MongoDB, offset in Kafka). Followers apply entries in order; the log is the only source of replicable state. This single ordering is what makes followers eventually converge to the leader’s state — without it, applying writes in different orders would produce different results.
Principle 4: Followers are typically eventually consistent with the leader. Replication takes nonzero time (the replication lag). A follower is, at any instant, somewhere between up to date with the leader and up to N seconds behind, where N depends on workload, network, and the slowest follower step. Reads from followers will eventually see writes confirmed by the leader, but not necessarily immediately. This is the read-after-write consistency problem of §7.4 below.
Principle 5: Failover is the entire game. Normal operation of leader-follower is boring and well-understood. Failover — the moment a leader is lost and a follower must be promoted — is where 95% of production incidents live. Picking the right new leader, fencing off the old leader, redirecting client traffic, and dealing with the leader’s possibly-not-yet-replicated tail of writes are each individually nontrivial.
4. Request Flow
sequenceDiagram actor C as Client participant L as Leader participant F1 as Follower 1 participant F2 as Follower 2 Note over C,F2: Synchronous Write Path (semi-sync, W=1+1) C->>L: write(x = 5) L->>L: append to WAL (durable on local disk) L->>F1: replicate(LSN=42, x=5) L->>F2: replicate(LSN=42, x=5) F1-->>L: ack Note over L: at least one follower acknowledged L-->>C: success F2-->>L: ack (lazy; client already replied to) Note over C,F2: Read from leader (linearizable) C->>L: read(x) L-->>C: x = 5 Note over C,F2: Read from follower (eventually consistent) C->>F2: read(x) F2-->>C: x = 5 (assuming replication caught up) Note over C,F2: If F2 had not caught up, response = previous value
What this diagram shows. A semi-synchronous write: the client sends write(x = 5) to the leader, which (1) writes the change to its Write-Ahead Log (WAL) — a sequential append-only log that is fsynced to disk so the change survives a crash — (2) ships the WAL entry to both followers in parallel, (3) waits for at least one follower (F1, in this example) to acknowledge, then (4) replies to the client. F2’s eventual acknowledgement is lazy — by the time it arrives, the client has already moved on. The same key, read from the leader, gives the just-written value. The same key read from F2 might give the just-written value (if replication has caught up) or the previous value (if it has not). This staleness window is the replication lag, and bounding it is the operational concern.
The pure-async variant of this flow simply omits the wait for F1’s ack — the leader replies to the client immediately after its own local commit, and replication proceeds entirely in the background. The pure-sync variant waits for all followers, not just one, and refuses the write if any cannot ack within a timeout. Each is a point on the durability-vs-latency curve.
5. Variants
The conceptual scheme above admits a sprawl of practical variants distinguished by replication semantics, topology, and failover machinery.
5.1 Synchronous, Asynchronous, and Semi-Synchronous
The choice along this axis is the most consequential single decision in deploying leader-follower:
Pure asynchronous replication. Leader replies as soon as the local WAL is durable; followers receive updates in the background. Pros: lowest write latency (one disk fsync + one network round trip to client); no follower can hold up writes; any number of followers can be added without affecting write performance. Cons: if the leader crashes between local commit and replication, the most-recent writes are lost. The crash window is small in practice — milliseconds — but in high-write-rate systems, “milliseconds of writes” can be hundreds of transactions. Default for MySQL classical replication, PostgreSQL when no standby is configured synchronous.
A useful framing of async-replication’s data-loss window: at a sustained write rate of 10K transactions per second, every millisecond of un-replicated lag holds 10 transactions worth of un-durable data. A 100-millisecond replication delay (typical for cross-rack async) is 1,000 transactions; a 1-second delay (typical when network is degraded) is 10,000 transactions. The absolute magnitude depends on the application’s write rate, but the lesson holds: async’s “small window” can be a meaningful business loss for high-throughput systems. Operators of high-write-rate workloads (advertising auctions, IoT ingestion) typically refuse pure async in favor of semi-sync.
Pure synchronous replication. Leader waits for all followers to acknowledge before replying. Pros: zero data loss if the leader crashes; every confirmed write is on every replica. Cons: one slow or dead follower stalls every write; latency is the slowest follower’s RTT plus disk fsync; write availability requires all followers to be healthy. This is rarely used in pure form because a 5-replica synchronous setup means losing any one replica halts writes.
The pure-sync model’s worst-case latency is determined by the slowest follower in the set. If one follower is on a struggling disk, the whole cluster’s write latency tracks that follower until it recovers or is removed from the configuration. This “tail-latency contagion” is why pure-sync is rarely deployed beyond two replicas: with three or more, the probability that some replica is having a bad moment grows linearly, making the cluster’s tail latency unbearable.
Semi-synchronous replication. Leader waits for at least one (configurable, typically one or quorum) follower to acknowledge before replying, then proceeds. Pros: bounded data-loss window (the un-acked-by-followers pre-commit cache is still in danger, but anything acked by at least one follower survives any single failure); reasonable latency. Cons: still subject to “stall if too few followers respond”; tail latency tracks the fastest of N followers for N=1+ but this can still be jittery. MySQL Group Replication and PostgreSQL with synchronous_commit=on plus synchronous_standby_names implement variants of semi-sync.
The numerical math for semi-sync with one follower: the tail latency is roughly the median of N followers’ RTTs (for the quickly-responding one to win the race), which is much better than pure-sync’s worst-case. Write throughput is bounded by the same calculation. With 3 followers in the same DC, semi-sync-of-one tail-latency is typically within 2× of a single-replica setup; with 1 follower, it equals the single-replica RTT. The “auto-degrade-on-timeout” behavior in MySQL’s implementation is what turns the failure mode into the silent one — the system returns to async on prolonged timeout, hiding the durability degradation from the operator.
Quorum replication (sometimes called “synchronous-quorum”). Leader waits for a majority of followers (e.g., 2 of 4 followers, plus the leader itself, totaling 3 of 5 replicas). This is what Raft does at its core (Ongaro & Ousterhout, USENIX ATC 2014) and what modern systems like CockroachDB, MongoDB (with majority write concern), and Kafka (with acks=all and min.insync.replicas set to majority) use. The cluster tolerates (N-1)/2 follower failures without halting writes. The key insight from Raft’s perspective: a write committed to a majority is preserved across any single failure, because any future leader’s election majority must overlap the write majority by at least one node — ensuring the write is propagated forward.
The relationship between these four points is best understood as a spectrum on a single axis: how many replicas must acknowledge before the leader replies? The answer is 0 (pure async, no acks), 1 (semi-sync), majority (quorum), or all (pure sync). Each step up that ladder buys more durability at the cost of latency and per-acknowledgement availability. Most production deployments default to “leader fsync + 1 follower ack” or “leader fsync + majority follower ack” — the sweet spot where bounded data loss is acceptable and latency is within the application’s budget.
Per-system tradeoff comparison: Postgres streaming vs MySQL semi-sync vs MongoDB write concern. The semantic differences between supposedly-equivalent replication options across major systems are surprisingly large in practice and trip up engineers who assume one mental model suffices. PostgreSQL streaming replication (Postgres docs, Warm Standby) treats synchronous_commit as a per-transaction GUC: setting it to on (the default since 9.1) means the local WAL must be fsynced before commit, but does not by itself wait for any standby; setting synchronous_standby_names to a list (e.g., 'FIRST 1 (standby_a, standby_b)' since 9.6) makes the leader wait for the named standby’s received WAL position. The hierarchy local → remote_write → remote_apply (introduced in 9.6) provides finer control: remote_write waits only for the standby’s OS-level write (not the standby’s fsync), remote_apply waits for actual replay on the standby (so the standby is queryable with the new state). Postgres has no native automatic failover; Patroni, repmgr, or pg_auto_failover provide it via external consensus. MySQL semi-synchronous replication (rpl_semi_sync_master_enabled plus the semisync_master.so plugin since 5.5; loss-less variant since 5.7) waits for one replica’s relay-log write before commit but with a critical caveat — there’s a rpl_semi_sync_master_timeout (default 10 seconds) after which the master automatically falls back to async and emits a warning. This auto-degradation is the source of countless production incidents: operators believe they have semi-sync durability but during a 10+ second network blip, the master silently downgrades and the next crash loses writes. MySQL Group Replication (8.0+) addresses this with Paxos-derived consensus (no async fallback) but at substantially higher operational complexity. MongoDB write concerns (MongoDB docs, Write Concern) operate at per-write-call granularity: {w: 1} (default) acks after primary’s local write, {w: "majority"} acks after a majority of voting members, {w: 5} acks after 5 specific members, {j: true} requires journal fsync (independent of w), {wtimeout: 5000} bounds how long the primary will wait. The crucial subtlety: write concern majority does not by itself guarantee read-after-write consistency from a secondary; that requires combining {w: "majority"} writes with {readConcern: "majority"} reads. Many MongoDB applications use {w: 1} (write to primary’s memory) and discover after their first failover that recently-acked writes were lost.
Worked tradeoff example. Consider an e-commerce checkout writing an order: the application calls db.insert(order). On Postgres with synchronous_commit = off, the call returns in ~1 ms after writing to the OS buffer; a kernel panic before the next checkpoint loses the order. On Postgres with synchronous_commit = on, the call returns in ~3 ms after local fsync; a leader crash before WAL ships to standbys loses the order. On Postgres with synchronous_commit = remote_apply and one standby, the call returns in ~5-8 ms after the standby has replayed the change; failover to that standby is lossless and the new primary is immediately queryable for the order. On MySQL semi-sync with default 10s timeout, the call returns in ~3-5 ms during normal operation but during a 30-second network partition silently degrades to async, returning in ~1 ms — and a leader crash during that window loses the order with no operator notification beyond an error-log warning. On MongoDB with {w: "majority"} on a 3-node replica set, the call returns in ~5-10 ms; this is genuinely lossless across single-node failure. The lesson: identical-sounding options across systems have meaningfully different semantics, and the “auto-degradation under timeout” trap (MySQL semi-sync) is the most common source of latent data-loss incidents.
5.2 Single Global Leader vs Leader-Per-Shard vs Leader-Per-Region
A “leader-follower architecture” can be deployed at different granularities:
Single global leader. One leader for the entire dataset. Simple but doesn’t scale writes beyond the leader’s capacity, and doesn’t help geo-distribution. Suitable for small-to-medium operational databases.
Leader-per-shard. The dataset is partitioned (see Sharded Architecture) and each shard has its own leader-follower group. Different shards can have leaders on different physical nodes, distributing the write load. This is what Kafka does (per-partition leader), what MongoDB does (per-shard primary), what most large MySQL deployments do (sharded leader-follower). Adds the complication that cross-shard transactions require Two-Phase Commit or a higher-level coordinator.
Leader-per-region. A geo-distributed deployment where each region has its own leader and writes are routed to the nearest leader. Cross-region conflicts must be reconciled, which crosses into Multi-Leader Replication Architecture territory. Active-passive regional setups (one region’s leader is “primary” globally; others are warm standbys) are technically still single-leader but need careful failover orchestration during region outages.
5.2.1 Cross-System Generalization: Per-Partition Leadership
The “leader-per-shard” deployment, when generalized, gives the most important modern composition: each unit of work (partition, tablet, range, key-prefix) has its own leader-follower group, and the cluster runs many such groups simultaneously. The choice of “unit” varies — Kafka uses topic partitions, CockroachDB and Spanner use ranges of contiguous keys, MongoDB sharded clusters use shards, Vitess uses tablets — but the structural pattern is identical. A 100-partition Kafka topic on 10 brokers has 100 leader-follower groups distributed roughly 10 per broker; a 100-range CockroachDB cluster has 100 Raft groups distributed across the nodes by leaseholder placement; a 100-shard MongoDB cluster has 100 replica sets each electing their own primary.
The architectural property this pattern unlocks is horizontal scaling without sacrificing leader-follower’s consistency model. Each leader is responsible for a small slice of the data; multiple leaders across the cluster collectively scale write throughput linearly with the partition count; failure of any single leader affects only its slice. The price is that cross-partition operations (queries, transactions) must coordinate multiple leaders — typically via Two-Phase Commit for transactions, or via fan-out for queries (see Sharded Architecture for the partitioning side of this story). The pattern is the default modern leader-follower deployment at scale; pure single-cluster-leader leader-follower is increasingly an artifact of small-scale deployments.
5.3 Statement-Based, Row-Based, and Logical Replication
How is the replication stream encoded?
Statement-based replication. The leader logs the SQL statement; followers re-execute it. Compact, but breaks on non-deterministic statements (NOW(), RAND(), autoincrements with concurrent insertion) — replicas can diverge. Older MySQL default; deprecated in modern setups.
Row-based replication. The leader logs each modified row’s before-image and after-image; followers apply the row changes directly. Bigger log volume, but deterministic. MySQL since 5.6, PostgreSQL via logical replication.
Physical / WAL-based replication. The leader streams its raw on-disk change log; followers replay byte-for-byte. PostgreSQL streaming replication, Oracle redo, SQL Server log shipping. Tightly coupled to the storage engine version (followers must run the same major version), but extremely fast — followers don’t re-do parsing, planning, or constraint checking.
Logical replication. A higher-level event stream describing logical changes (INSERT, UPDATE, DELETE on table X with these column values), independent of storage format. Allows cross-version replication, partial replication, and downstream consumers (e.g., Debezium reading PostgreSQL’s logical decoding stream into Kafka). PostgreSQL since 10, MySQL via row-based binlog. Logical replication is the backbone of the modern Change Data Capture (CDC) ecosystem: by decoupling the replication stream from the storage format, the same stream can feed analytics warehouses, search indexes, cache invalidation, and audit logs — turning the leader-follower replication channel into a general-purpose event source.
The Postgres physical-vs-logical replication debate. Within the PostgreSQL community this is one of the longest-running design tensions, well-summarized in Robert Haas’s PostgreSQL Replication Roadmap-style posts. Physical (streaming) replication, the older built-in, ships raw 8 KB page changes from the WAL: lightning-fast (followers don’t re-parse SQL or re-check constraints, they just byte-replay), absolutely faithful (you cannot have a logical bug if you’re copying the bytes), but rigidly coupled — followers must run the same major Postgres version with the same compile-time settings, the same operating system page size, the same character encoding. You cannot replicate from Postgres 14 to Postgres 16 with physical replication; you cannot replicate a single table; you cannot replicate from Postgres to anything that isn’t Postgres. Logical replication (Postgres 10+, via the pgoutput plugin in core or wal2json and decoderbufs for external consumers) decodes the WAL into logical events — “row inserted into table users with these values” — that are version-agnostic and consumer-agnostic. Logical replication enables: major-version upgrades with zero or near-zero downtime (build a new-version follower via logical replication, promote it), cross-database replication (Postgres → Postgres of different versions, Postgres → Kafka via Debezium, Postgres → analytics warehouse), and selective replication (replicate only specific tables, skip large append-only logs). The cost is operational complexity and some performance overhead for the decoding step. The community’s consensus by ~2023: logical replication is the future for migrations and cross-system integration; physical replication remains the default for high-availability follower replicas because of its faithfulness and lower latency. Stripe, GitHub, and Heroku have all publicly described using logical replication for major-version upgrades on multi-TB Postgres clusters with downtime measured in seconds rather than hours.
Trigger-based replication. Older approach: triggers on the leader’s tables fire on each modification, writing the change to a replication-staging table; a separate process ships the staging-table rows to followers. Slow, fragile, but version-agnostic. Used by older PostgreSQL replication tools (Slony, Bucardo) before logical replication was integrated into the core. Modern systems avoid this in favor of logical replication on the database’s native stream.
5.4 Failover Mechanisms
The component nobody wants to think about until it doesn’t work:
Manual failover. A human operator notices the leader is dead and runs a script to promote a follower. Robust but slow; suitable when the operator’s RTT-to-pager is acceptable as MTTR. Surprisingly common in older deployments.
External coordinator-based failover. A separate system (Redis Sentinel, ZooKeeper, etcd, Consul) monitors the leader’s heartbeat and orchestrates promotion. The coordinator is itself usually a Raft/Paxos High-Level cluster to avoid split-brain in the coordinator itself.
Embedded consensus failover. The replicas themselves run a consensus protocol (typically Raft) and the “leader” of the consensus protocol is the replication leader. MongoDB’s election protocol v1, MySQL Group Replication, etcd, CockroachDB. This is the modern correct default.
STONITH (“Shoot The Other Node In The Head”) / fencing. Before promoting a new leader, force-kill the old leader (often via out-of-band power control) so it cannot be revived as a zombie. Common in HA cluster software like Pacemaker. Brutal but correct. The sober reality of distributed-systems engineering: sometimes the safest way to ensure a node has stopped accepting writes is to physically power it off via remote management interfaces (IPMI, iDRAC). Without fencing, the cluster relies on the deposed leader cooperating (“oh, I see I have a higher-term message; I’ll demote myself”) — a guarantee that fails when the deposed leader is partitioned, paused, or buggy.
Lease-based leadership. Rather than failover via heartbeats, each leader holds a lease of bounded duration (e.g., 10 seconds). The leader continuously renews the lease via consensus; followers will not promote a new leader until the old leader’s lease expires. This eliminates the split-brain window: even if the old leader is alive but unable to renew, it knows to stop accepting writes when its lease expires. Spanner uses lease-based leadership built on top of TrueTime (Corbett et al., OSDI 2012); CockroachDB uses range leases similarly. Lease-based schemes are cleaner than heartbeat-based ones but require synchronized-or-bounded clocks. The Spanner approach: every range has a leaseholder who is also the Paxos leader; the leaseholder must renew its lease before TrueTime’s latest exceeds the lease expiration, otherwise it ceases to act as leader without waiting for any external command. This is the cleanest known fencing mechanism — the leader self-fences via clock — but it is critically dependent on TrueTime giving bounded clock skew. Without TrueTime (which Spanner builds with atomic clocks and GPS receivers in every datacenter), the lease bound must be generous (tens of seconds), bloating the worst-case failover latency.
Pre-vote and PreVote in Raft extensions. A subtle issue with classical Raft: a partitioned follower whose election timer fires will increment its term and start an election. When the partition heals, its higher term causes the legitimate leader to demote itself, triggering an unnecessary re-election. The fix, “PreVote” (described in Ongaro’s PhD thesis, 2014, as a Raft extension), is for the candidate to first poll peers asking “would you vote for me?” without actually incrementing its term. Only if a majority would vote does the candidate increment its term and start a real election. This eliminates the spurious-disruption case. Most production Raft implementations (etcd 3.0+, CockroachDB) include PreVote.
6. Real-World Examples
PostgreSQL streaming replication. The leader (primary) streams its WAL to one or more standbys via a long-lived TCP connection. Standbys can serve read-only queries (hot_standby = on). Synchronous behavior is configured via synchronous_standby_names; default is async. Failover is not automatic in plain Postgres — it requires an external tool such as Patroni (which uses etcd / Consul / ZooKeeper for coordination), repmgr, or pg_auto_failover.
MySQL primary-replica replication. The leader writes to its binlog (binary log of changes); replicas connect via a long-lived IO_THREAD to copy binlog events to a local relay log, and a separate SQL_THREAD applies them. Statement, row, and mixed-format binlogs are supported. Group Replication (8.0+) adds a consensus layer and automatic failover; without it, failover is manual or orchestrated by tools like MHA, Orchestrator, or ProxySQL. The consensus engine inside Group Replication is XCom, which the MySQL manual describes precisely as “a Paxos variant” — by default every member of the group acts as a leader (proposer) for its own slots in the totally-ordered message stream, and from MySQL 8.0.27 a single consensus leader can drive consensus in single-primary mode for better performance (MySQL 8.0 manual, Single Consensus Leader). MySQL’s own engineering write-up is careful about lineage: the multi-proposer design “has some similarities to Mencius, for example uses skip messages, but the overall design is closer to the original Paxos” (MySQL Server Blog, Our homegrown Paxos-based consensus). The accurate one-line summary is therefore “a multi-proposer Paxos variant, Mencius-influenced but not Mencius” — not, as is sometimes loosely claimed, a Mencius implementation.
MongoDB replica sets. A replica set is a leader-follower group of 3-7 nodes. The leader is elected via an Raft-derived protocol (election protocol v1, since MongoDB 3.2). Writes go to the leader by default; reads can be routed to followers via read preferences. Write concerns (w: 1, w: majority, w: 5) control durability; read concerns (local, majority, linearizable) control freshness. The election runs entirely within the replica set — no external coordinator.
Apache Kafka per-partition leader. Each topic partition has one leader broker and a configurable number of follower brokers (the “in-sync replicas” or ISR list). Producers write to the leader; followers fetch from the leader. With acks=all and min.insync.replicas=2 (on a replication factor of 3), the leader requires at least 2 ISR acks before confirming the write. Leader election for a partition was historically done via ZooKeeper; KRaft mode (Kafka 3.3+) replaced ZK with an internal Raft cluster. Kafka’s per-partition leadership generalizes the leader-follower architecture in an important way: rather than one leader per cluster, the cluster has one leader per partition, and the controller (in KRaft, the active controller broker) deliberately spreads leaderships across brokers for load balancing. A 100-partition topic on a 10-broker cluster will have ~10 leaderships per broker; a leader broker’s death triggers re-election for only its 10 leaderships, not the entire cluster’s worth. The Kafka pattern — sharded leader-follower with leader load-balancing — is the same structure used by Vitess (per-keyspace tablets), CockroachDB (per-range Raft groups), and MongoDB sharded clusters (per-shard primaries), and it is the modern default for any system that needs both leader-follower’s consistency model and horizontal scale.
Redis Sentinel. Redis is a single-leader-multi-follower system. Sentinel is a separate fleet of monitor processes (typically 3 or 5) that watch Redis instances, agree among themselves (via a quorum protocol) when the leader has failed, and then promote a follower. Clients are aware of Sentinel and discover the current leader via Sentinel queries. Redis Cluster (the sharded variant) uses a different gossip-based mechanism per shard.
Microsoft SQL Server Always On Availability Groups. Synchronous or asynchronous replication; failover via Windows Server Failover Clustering. Up to 8 readable secondaries. Each secondary can be configured for sync or async replication independently, giving operators a per-replica durability choice. SQL Server’s “automatic seeding” simplifies adding new secondaries by streaming the full database over the existing replication transport, eliminating the historic need for manual backup-and-restore-then-attach.
Oracle Data Guard. Oracle’s primary-standby replication. Maximum Protection (sync to all standbys, halt on any failure), Maximum Availability (sync to ≥1 standby, fall back to async on failure), Maximum Performance (async only) — the trinity that maps cleanly onto the sync/semi-sync/async distinction. Failover (called “switchover” for planned, “failover” for unplanned) is orchestrated by the Data Guard Broker.
Google Spanner internal architecture. Each Spanner tablet is replicated via Paxos with one leader and several followers. The leader-follower architecture is global at the per-tablet level; the global system is sharded leader-follower with strong consistency. Spanner’s TrueTime API gives a globally-bounded clock that allows linearizable reads across tablets without explicit cross-tablet coordination on each read — see Distributed SQL Database System Design.
Amazon RDS / Aurora. AWS-managed leader-follower for major OSS databases. Aurora’s twist: rather than streaming WAL to follower compute nodes, it stores the WAL in a shared distributed log (separation of compute and storage). Followers (“Aurora Replicas”) read pages from the shared storage rather than replaying WAL on their own; this dramatically reduces follower lag because they don’t have to do the redo work themselves. This is a meaningful architectural variation on the classic leader-follower scheme.
Etcd. Although usually presented as a Raft-based KV store, etcd implements a leader-follower architecture under the hood: its Raft layer elects a leader, and that leader is the sole writer for the etcd state machine. Reads can be linearizable (route to leader and verify leadership) or stale (any follower). The etcd codebase is a particularly good study reference for a clean leader-follower implementation with explicit consensus-protected failover.
The pattern is overwhelming: nearly every operational database is a leader-follower system at the per-shard or per-replica-set level. The differences are in the failover machinery and replication semantics. The variations between PostgreSQL streaming replication, MySQL binlog replication, MongoDB oplog replication, Kafka per-partition replication, and Redis Sentinel are mostly differences in encoding (what’s in the replication stream) and failover (how a new leader is chosen) — not differences in the underlying topology, which is universally one writer and many followers.
Worked Example: The GitHub October 21, 2018 Outage
The single most-cited leader-follower failure in the past decade is GitHub’s October 21, 2018 incident, which produced 24 hours and 11 minutes of degraded service and offers a complete cautionary timeline of what happens when leader-follower failover misbehaves on top of MySQL with cross-region replication (GitHub October 2018 post-incident analysis). The setup at the time: GitHub ran MySQL primary-replica clusters with Orchestrator as the failover automation tool, with a US-East-Coast primary, several US-East-Coast and US-West-Coast replicas, and asynchronous replication between regions.
The trigger (22:52 UTC, October 21). A routine maintenance operation introduced a 43-second connectivity disruption between the East-Coast and West-Coast networks. Heartbeats from the East-Coast primary stopped reaching some West-Coast nodes. Orchestrator’s failure detector concluded the East-Coast primary had failed.
The failover decision (~22:53 UTC). Orchestrator promoted a West-Coast replica to primary. From the West-Coast cluster’s perspective, this was a clean failover — they now had a writable primary. From the East-Coast cluster’s perspective, the original East-Coast primary was still running and accepting writes locally, because connectivity between East and West was disrupted but the East-Coast primary itself was healthy and reachable to East-Coast clients.
Split-brain (~22:53–22:54 UTC). For approximately 40 seconds, both the original East-Coast primary and the newly-promoted West-Coast primary accepted writes simultaneously. Each accepted writes to the same logical tables for the same MySQL clusters. Neither was aware of the other’s existence during this window; the partition prevented cross-cluster communication.
Detection and clamp-down (~22:54 UTC). When network connectivity restored, the cluster topology was inconsistent. The promotion algorithm’s safety check noticed that two nodes claimed to be primary; Orchestrator demoted the East-Coast primary to read-only. But by then, both primaries had accumulated divergent writes: 954 binlog events of writes had landed on the East-Coast primary that the West-Coast primary had no record of, and vice versa.
The 24-hour recovery. The system was now in a state where the West-Coast (officially the primary) had writes that East-Coast didn’t have, and East-Coast had writes that West-Coast didn’t have — and these writes touched the same rows in some cases. There was no way to mechanically merge them, because MySQL row-based binlog has no conflict-resolution semantics. The recovery required:
- Stopping all writes globally (full write-availability outage for hours).
- Identifying the divergent transactions on each side via binlog comparison.
- Replaying the East-Coast-only writes to a parallel new primary, then carefully re-applying the West-Coast-only writes, hand-resolving conflicts where they touched the same rows.
- Bringing services back online incrementally as data integrity was verified per-table.
Root cause analysis. Three architectural deficiencies combined to produce the outage: (1) No fencing: the original East-Coast primary was not forcibly demoted before the West-Coast was promoted. Without fencing (STONITH or equivalent), a partitioned old leader continues to accept writes because it has no way to know it’s been deposed. (2) Asynchronous cross-region replication: even in steady state, writes to the East-Coast primary took seconds to reach West-Coast replicas. When the West-Coast replica was promoted, it was not up-to-date with the East-Coast primary’s most recent writes — those writes were only visible on the East-Coast cluster. (3) The failover detector trusted heartbeats over consensus: Orchestrator’s promotion decision was based on local heartbeat timeout, not on a consensus protocol that would have required a quorum to confirm the original primary’s death. Modern systems with embedded Raft (CockroachDB, Spanner, etcd) avoid this exact failure: a Raft leader can only be deposed if a majority of the consensus group agrees, and that majority cannot exist if the original leader is still in contact with a majority of replicas.
The lesson, distilled. Leader-follower’s correctness depends entirely on the failover protocol’s correctness. A 43-second partition is not a rare event — partitions of this duration happen many times per year in any reasonably-sized fleet. The architecture must be hardened against them. Specifically: any failover protocol must (a) require consensus-style quorum agreement (not just heartbeat timeout) before promoting, and (b) fence the old leader (forcibly or by design — fencing tokens, lease expiration, STONITH) so it physically cannot continue to accept writes. Failing either of these conditions, every multi-region leader-follower deployment is one network blip away from the GitHub incident.
MongoDB Write Concern in Detail
MongoDB’s write concern API is the most user-facing of the major systems’ replication-semantics knobs and merits a careful walk-through. A MongoDB write call accepts a writeConcern object with three fields: w (acknowledgement level), j (journal fsync requirement), and wtimeout (maximum wait). The w field can be a number (e.g., w: 3 requires 3 voting members to ack), a string "majority" (the most common; requires a majority of voting members), or a custom write-concern tag (referencing replica-set tags for geo-aware write concerns, e.g., w: { region: 1 } to require at least one replica in the same region).
Crucially, write concern w: 1 (the historical default before MongoDB 5.0) acks after only the primary’s in-memory write — not even the primary’s journal fsync. A primary crash before the next checkpoint loses the write entirely, even though the application received an ack. This default produced enough data-loss incidents that MongoDB 5.0 (released 2021) changed the default to w: "majority" for new deployments — but legacy deployments retain w: 1 and operators must explicitly migrate. Stripe’s 2018 engineering post on MongoDB migrations references this exact issue.
The interaction with readConcern is critical: a write with w: "majority" is durable across single-node failures, but a read with readConcern: "local" (the default) reads from the primary’s in-memory state, which may include unacknowledged writes from earlier transactions that are not yet on majority and could be lost on failover. To get linearizable read-after-write across failure, the application must combine w: "majority" writes with readConcern: "majority" reads — and MongoDB’s readConcern: "majority" reads are slower because they wait for the read to be confirmed against a majority view.
The lesson for application developers: write concern alone is half the picture; read concern completes it. A “I’m using majority writes so I’m safe” assumption is incorrect if reads continue to use the default read concern. The MongoDB docs are explicit on this but operational teams routinely discover the gap during incident review.
Worked Example: Read-After-Write on E-Commerce Checkout
The other common leader-follower production pain point is read-after-write consistency. Consider a typical e-commerce flow: the customer clicks “Place Order”; the application writes the order to the primary; the customer is redirected to an order-status page that displays the new order. With async replication and read-from-followers for scaling, the order-status page query can hit a follower that has not yet seen the order — producing a “no order found” error or a blank page on the customer’s first refresh.
The lag scenario. Primary writes at LSN 8000. Follower A is at LSN 7995 (50 ms behind). Follower B is at LSN 7980 (200 ms behind). Customer’s checkout commits at LSN 8000 against the primary, which acks the write. The customer’s browser is redirected to /orders/abc123. The browser’s request hits the load balancer, which routes the read query to Follower B (300 ms after the write). Follower B is still at LSN 7980 — behind the order’s LSN 8000 — so the query returns “order not found.” The customer panics, refreshes the page, this time the request goes to Follower A (now at LSN 8005), and the order appears.
Mitigation 1: Read your writes via leader for a window. The simplest fix: for a TTL after each user’s write, route their reads to the primary. Implement this as a session attribute “last_write_at = T0”; for now < T0 + 5 seconds, route the user’s reads to the primary. Costs read scaling for a small window per user but eliminates the bug entirely for typical user journeys. The 5-second TTL must exceed the maximum expected replication lag plus a safety margin.
Mitigation 2: LSN-aware read routing. Track the LSN of the user’s last write in their session: session.last_write_lsn = 8000. On every read for that session, query the load balancer’s per-follower lag and pick a follower at LSN ≥ 8000, falling back to the primary if no follower has caught up. This preserves read scaling for caught-up followers but requires the application to plumb LSNs through every write/read path, which is invasive. PostgreSQL’s pg_last_wal_replay_lsn() exposes the per-follower LSN; MySQL’s Read_Master_Log_Pos plus Exec_Master_Log_Pos does the same for binlog positions.
Mitigation 3: Monotonic-read sessions (sticky to a follower). Pin each user to one specific follower for the duration of their session. The follower may be slightly stale, but reads from it are monotonically consistent — the user never sees the data go backward. Doesn’t solve “I just wrote, why don’t I see it?” without combining with mitigation 1, but does solve the secondary “I saw it before, now I don’t see it again” problem caused by load-balancer-driven follower-hopping.
Mitigation 4: Synchronous-replication-on-recent-writes. A pragmatic compromise some teams adopt: for tables where read-after-write matters (orders, posts), write with synchronous_commit = on; for tables where it doesn’t (analytics events, logs), use async. PostgreSQL’s per-transaction GUC makes this easy. Trade off: writes to the “important” tables take slightly longer; reads from any follower are guaranteed consistent.
Mitigation 5: Replication-lag monitoring with alerting. Independently of the application, the operations team monitors pg_stat_replication.replay_lag continuously, alerting on any follower exceeding a threshold (e.g., 1 second). When followers fall behind, they’re temporarily removed from the read pool until they catch up. This doesn’t fix individual user-visible bugs but prevents the bugs from happening systemically when replication degrades.
The pattern across all five mitigations: the architecture itself does not give read-after-write consistency from followers without explicit machinery; the machinery must be built deliberately. The default behavior — async replication plus naive read-balancing — is a bug factory. Most production systems use mitigation 1 (route recent readers to leader) as their primary defense, sometimes augmented with mitigation 5 (lag monitoring).
7. Tradeoffs
Sync replication: durability vs latency. Synchronous replication eliminates the “leader crashes after local commit but before replication” data-loss window — but every write must wait for at least one follower’s ack, doubling write latency from one disk fsync to one fsync plus one network RTT plus the follower’s own fsync. In practice, semi-sync with one follower-ack is the sweet spot for most production workloads.
To make the latency math concrete: a local NVMe fsync is 100-500 microseconds; an intra-datacenter network round trip is 200-500 microseconds; a follower’s own fsync is another 100-500 microseconds. So a sync-replicated write goes from ~300 microseconds (local-only) to ~1-1.5 milliseconds (sync to one follower) — a 3-5× latency multiplier. Cross-datacenter sync adds 5-50 milliseconds of WAN RTT, taking the multiplier to 50-100×. Most production teams refuse cross-datacenter sync for this reason; semi-sync to one local follower is the common compromise.
Sync replication: durability vs availability. A sync follower that goes down stalls writes. With pure synchronous-all-followers replication, losing any single follower freezes the cluster’s write capacity. Quorum replication (the Raft approach) avoids this by waiting for majority rather than all, tolerating up to (N-1)/2 follower failures without write impact.
Read scaling vs read freshness. Followers can serve reads, multiplying the cluster’s read throughput. But followers are by replication-lag amount stale. Reads from followers therefore cannot be assumed to reflect the most recent leader-confirmed write. Applications must either route reads to the leader (no read scaling), accept staleness (eventual consistency), or implement read-after-write affinity (route reads of recently-written keys to the leader for a freshness window).
Failover speed vs split-brain risk. Aggressive failover (short heartbeat timeouts, fast promotion) reduces unavailability but increases the chance of premature failover when the old leader is actually still alive (just paused, partitioned, or slow). Conservative failover reduces split-brain risk but produces longer outages. The classic incident is the GitHub October 2018 outage: a 43-second network partition triggered failover, but the old primary’s writes during the partition were not lost — they were replayed against the new primary during recovery, producing corruption. The lesson: failover protocols must have fencing — once a new leader is promoted, the old leader must be prevented (forcibly if necessary) from accepting writes.
Single-node ceiling vs sharding complexity. A leader is bounded by single-node IOPS. To scale beyond, you must shard, which compounds the failover problem (now you have N leader-follower groups, each with its own failover) and introduces cross-shard transaction problems.
Strong consistency from leader vs eventual from followers. Reads from the leader are linearizable; reads from followers are eventually consistent. A workload that wants both — strong consistency and read-scaling — must choose: route all reads to the leader (no scaling), or implement read-after-write affinity (sticky sessions, LSN-aware reads) to offload reads safely. The practical compromise that most teams adopt is “reads from followers for read-only flows; reads from leader for read-after-write flows,” with the routing decision made at the application or middleware layer.
Single-leader-per-cluster vs sharded leadership. A pure single-leader cluster bottlenecks on one node; sharded leadership (per-partition leaders, distributed across brokers/nodes) scales linearly. The cost of sharded leadership is cross-shard transactions becoming expensive (require Two-Phase Commit across multiple leaders) and operational tooling becoming per-shard. The choice usually depends on whether the workload has natural shard isolation: per-user, per-tenant, per-account workloads partition cleanly; cross-cutting analytics workloads do not.
Geographic distribution vs synchronous replication. Cross-region synchronous replication adds 5-50 milliseconds of WAN RTT to every write. For high-throughput workloads, this is fatal — write latency exceeds the application’s per-request budget. Most teams use cross-region async replication for disaster recovery and accept the brief data-loss window during regional failover. Cross-region sync is reserved for compliance-driven scenarios (financial regulations requiring durably-replicated writes before commit) where the latency cost is justifiable.
Failover speed vs failover safety. A 1-second timeout produces fast failover but many false-positive promotions (a long GC pause becomes “the leader is dead”). A 30-second timeout produces safe failover but extends outages. The practical middle ground depends on the workload: payment systems lean toward longer timeouts to avoid split-brain; cache-fronting systems lean toward shorter timeouts to minimize user-visible unavailability. The numerical default in many systems is 5-10 seconds for the heartbeat interval; production deployments tune this based on observed JVM/runtime pause times (full-GC pauses on a heavily-loaded JVM can exceed 1 second; database checkpoint flushes on disk-pressure can exceed 5 seconds).
Replication-stream encoding tradeoff matrix. The encoding choice — physical, logical, statement, row — is itself a multi-dimensional tradeoff that production engineers should hold mentally: physical replication is the fastest and most faithful but version-locked; logical replication is version-tolerant and downstream-flexible but pays a decoding cost and may not support all DDL. Statement-based is the most compact but breaks on non-deterministic statements. Row-based is deterministic but high-volume. The choice depends on what you need from the replication stream: a hot standby (physical), a cross-version migration (logical), a feed into Kafka for analytics (logical), a legacy MySQL setup (row-based, the modern MySQL default).
Worked Example: Choosing Replication Mode for a Payment Processor
A concrete tradeoff scenario illustrating how replication-mode choice ties to product semantics: a payment processor (think Stripe, Adyen, or a smaller analogue) must decide between async, semi-sync, and quorum replication for its core transaction database. The product constraints: transactions must never be lost (regulatory + customer trust); per-write latency budget is ~50 ms (interactive checkout flows); the system runs in three datacenters in one region for HA.
Option A: Async replication. Latency is excellent (~3 ms per write including local fsync). Data-loss window during primary crash is roughly the replication lag (~50-200 ms typical), which at the system’s transaction rate of 5K/sec equals ~250-1000 transactions of un-durable data. For a payment processor, this is unacceptable: each lost transaction is a customer charge or refund that vanishes silently. Async is rejected.
Option B: Quorum replication via embedded Raft (e.g., on top of CockroachDB or a Postgres+Patroni+sync setup). Latency is ~5-8 ms per write (local fsync + intra-DC RTT to the second-fastest follower). Tolerates one DC failure without halting writes (3 DCs, majority is 2). Zero data loss across single-DC failure because committed writes are on a majority. This is the right answer.
Option C: Pure synchronous to all followers. Latency is ~10-15 ms per write (bound by slowest follower); cluster halts on any DC failure (3 DCs all required). Zero data loss but worse availability than quorum, no benefit. Rejected.
The chosen architecture: 3-DC quorum replication with embedded consensus. Quorum gives durability across single-DC failure; latency is within budget; writes complete reliably with two healthy DCs even if the third is partitioned. The lost option that’s tempting but wrong is “async with frequent backups” — backups don’t help with the seconds-of-window between transactions and the next backup. Only quorum closes the window.
The general lesson: replication-mode choice is determined by the product’s tolerance for the data-loss window, not by raw latency budgets. For payments, the window must be zero. For ad-impressions logging, milliseconds-to-seconds is fine. For analytics events, hours of window is fine. The product team should drive this decision; the engineering team should implement whatever the product team’s tolerance demands.
Aurora’s Architectural Variation: Compute-Storage Separation
A noteworthy variation on the classical leader-follower deployment pattern is Amazon Aurora, which fundamentally changes where the WAL lives and how followers consume it. In a classical Postgres or MySQL leader-follower setup, the leader writes WAL to its own local disk and streams the same WAL bytes (or a logical decoding of them) over the network to each follower; each follower replays the WAL into its own local data files, doing its own redo work. This architecture has a cubic-ish scaling property: each follower must do the same redo work as the leader, so adding followers adds the same per-write CPU and IO load on each.
Aurora restructures this. The “leader” writes its WAL not to a local disk but to a shared, distributed storage layer (Aurora Storage). The leader synchronously replicates each write to six storage nodes spread across multiple Availability Zones within one AWS Region (two copies per AZ across three AZs), so the cluster survives the loss of an entire AZ plus one additional node without data loss (AWS, High availability for Amazon Aurora). Durability uses a quorum scheme over those six copies — a write quorum of 4 of 6 and a read quorum of 3 of 6, chosen so that the write set and read set always intersect and so that writes still succeed during an AZ outage (per the Aurora SIGMOD papers, Verbitski et al. 2017/2018). The leader’s job is only to write the WAL records; the storage layer handles redo, page generation, and replication internally. Followers (“Aurora Replicas,” up to 15) do not replay WAL — they read pages directly from the shared storage layer when needed, using asynchronous replication that does not slow the writer. The follower lag is therefore bounded only by the storage layer’s propagation latency, typically tens of milliseconds, even under heavy write load.
The architectural insight: by separating compute from storage and pushing redo into the storage layer, Aurora gets followers that don’t bottleneck on per-follower redo work. Adding a follower adds compute capacity but not IO load on the storage layer. AWS claims this gives Aurora up to 5× the throughput of vanilla MySQL/Postgres at the same hardware spec with low follower lag — but treat that multiplier as a vendor figure (see the uncertainty callout in §12): the headline number traces to AWS’s own materials and the Aurora SIGMOD papers, and independent reproductions are scarce. The cost is vendor lock-in (Aurora Storage is proprietary AWS infrastructure) and some compatibility caveats (some Postgres extensions assume direct WAL access, which Aurora doesn’t expose). For workloads that fit Aurora’s constraints, the architecture genuinely improves on classical leader-follower for the read-heavy case; for workloads with heavy write load that need maximal portability, classical replication remains the default.
Replication Lag Monitoring Patterns in Practice
Production-grade leader-follower deployments treat replication lag as a tier-1 SRE metric on par with error rate and request latency. The patterns that have emerged across the industry are worth cataloguing because they are repeatedly rediscovered the hard way:
Pattern 1: Multi-threshold alerting tied to user-visible impact. Lag thresholds are not arbitrary; they are derived from the application’s read-after-write tolerance. A typical scheme: warn at 1 second of lag (early indicator that something is degrading), page at 30 seconds (writes from 30 seconds ago will not be visible on followers — high probability of user-visible bugs), page urgently at 5 minutes (the application’s read-after-write SLO is being violated; consider failing reads to the primary). The thresholds are aligned with the application’s product semantics, not the database’s internals.
Pattern 2: Pool-eviction on lag. The read load balancer maintains a per-follower lag value (refreshed every few seconds via a cheap lag-query) and excludes followers exceeding a threshold from the read pool. Reads that would have gone to a lagging follower are routed to the primary or to a still-healthy follower. This prevents user-visible inconsistency at the cost of additional load on the primary during incidents. The exclusion is automatic; recovery is automatic when the follower catches up.
Pattern 3: Per-application lag SLOs. Different applications have different lag tolerances. A user-facing web app may tolerate 1-second lag (read-after-write feels off above this); an analytics dashboard may tolerate 60-second lag (no one notices). Sharing one large follower fleet across both produces the wrong defaults — either the analytics app gets routed to a follower that’s been lag-evicted (expensive), or the web app gets routed to a follower that’s stale by web-app standards (buggy). Mature deployments designate “tier-1 followers” (low lag, high availability) for the user-facing apps and “tier-2 followers” (laggier, cheaper) for the analytics workload.
Pattern 4: Deliberate lag introduction for backups. Some production databases keep one designated follower running with deliberate lag (often 1 hour) so that an accidental destructive write on the primary can be recovered from the lagged follower before it replays. Postgres’s recovery_min_apply_delay setting; MySQL’s MASTER_DELAY parameter on CHANGE MASTER TO. The trade-off: this follower is useless for read scaling (always stale by design) but provides “operator-error insurance” that can be cheaper than backup-restoration for fresh accidents.
Pattern 5: Synthetic write-and-read probes. A monitoring service writes a marker record to the primary every few seconds, then immediately reads it from each follower, recording the time it took the marker to propagate. This is the most direct measurement of replication lag from the application’s perspective — it bypasses any database-internal lag metric (which can be misleading if the metric reports “the latest LSN I’ve received” but not “the latest LSN I’ve actually replayed and made queryable”). The probe gives a true end-to-end lag measurement that operations teams can trust.
The common thread across all five: replication lag is a system property that is operationally visible only if you build the visibility deliberately. The default operational position is “we don’t know how lagged our followers are” because the database does not push the metric to the operator’s dashboards by default. Building the dashboards, alerts, and pool-eviction logic is a multi-week project — and not building it is the single most common reason that leader-follower setups produce user-visible read-after-write bugs in production.
8. Migration Path
From single-instance to leader-follower. A single-instance database to a leader-follower replica set is a relatively gentle migration: turn on the WAL stream, attach a follower replica, wait for it to catch up (the initial sync), turn on application-level read routing if desired. Downtime can be near-zero. The hard part is the operator discipline for failover scenarios — most teams don’t exercise failover until production demands it, and failure modes accumulate.
A key sub-question during this migration is where the new follower lives: same rack (lowest latency, no rack-failure protection), same datacenter different rack (latency similar, rack-failure protection), different datacenter same region (some latency cost, datacenter-failure protection), different region (highest latency, region-failure protection). Most teams start with same-DC followers and add cross-DC followers later. The Postgres community’s typical guidance is “two same-DC sync followers + one cross-DC async follower for disaster recovery” — durability for single-failure via the sync followers, durability for region-failure via the async follower.
From leader-follower to multi-region active-passive. Place a follower in a remote region; promote it during regional outages. Async replication is sufficient because cross-region consistency is not the goal; survivability is. The challenge is application-level: clients in the remote region must know to fail their writes over to the new primary when promotion happens.
The classic operational concern in multi-region active-passive is DNS or service-discovery propagation lag during failover. The application discovers the primary via a DNS name or service-discovery entry; on regional failover, the entry is updated to point at the new primary. DNS clients with a 60-second TTL keep using the old primary for up to 60 seconds. Service meshes (Istio, Consul) propagate faster but not instantly. Mitigations: short TTLs (5-10 seconds), client-side health checks that fail over immediately on connection refusal, or a virtual-IP-based scheme where the primary’s IP migrates atomically. Each has its own operational tax; none are free.
From leader-follower to multi-leader. A radical change. The application’s data model must tolerate concurrent writes from multiple leaders, which means conflict-resolution logic (Last-Writer-Wins, Vector Clocks, or CRDTs Basics). Most leader-follower applications cannot make this leap without redesign — the schema invariants implicitly assume single-writer ordering.
From leader-follower to leaderless. Even more radical. You give up cross-key transactions, foreign keys, and most schema constraints. Migration is rarely “in place” — usually a new system is built alongside, dual-written, and the old system retired. See Leaderless Replication Architecture.
From leader-follower to sharded leader-follower. Add a routing layer (proxy, application-side library, or middleware like Vitess for MySQL or pg_partman + pgbouncer for PostgreSQL). The routing layer maps each query to the right shard’s leader-follower group. The shard key choice is the high-stakes decision; see Sharded Architecture.
Adding a follower (online). Most operational databases support online follower addition: take a snapshot of the leader (or an existing follower) using a hot-backup tool (PostgreSQL’s pg_basebackup, MySQL’s xtrabackup, MongoDB’s mongodump --oplog); transfer the snapshot to the new follower; start replicating from the snapshot’s checkpoint. The new follower catches up over minutes-to-hours depending on data size and write rate. The leader is unaffected during the snapshot if hot-backup tools are used; legacy approaches that quiesce the leader during snapshot are increasingly rare.
Removing a follower. Trivial: stop the follower, remove from the cluster’s known-followers list, reclaim its hardware. The leader continues writing to remaining followers. The only constraint is that synchronous-replication setups must not lose so many followers that quorum cannot be met (in semi-sync, this means at least one follower must remain healthy).
Major-version upgrade with minimal downtime. Common pattern: build a follower running the new version, replicate from the old-version leader using logical replication (which is version-tolerant), promote the new-version follower to leader, decommission the old leader. Total user-visible downtime is bounded by the promotion window — often seconds. Versus the naive approach of upgrading the leader in place (which requires full downtime), this is dramatically better.
Stripe’s 2020-era public posts described doing exactly this for a multi-TB Postgres upgrade with sub-30-second user-visible downtime: build new-version followers via logical replication; let them catch up; coordinate the promotion via a tightly-scripted runbook; cut over connections; observe for issues; if issues arise, the old-version leader is still warm and can be reverted. The pattern has become the de-facto standard for major Postgres upgrades at scale.
9. Pitfalls
Pitfall 1: Split-brain on partition recovery. A network partition isolates the leader; a follower is promoted; the partition heals; the old leader rejoins and also thinks it’s the leader. Both nodes accept writes for some window before discovering the conflict. The corrupt diverged state cannot be reconciled by simple replay. Mitigation: fence the old leader (STONITH, or refuse to acknowledge it as a leader without consensus-protocol agreement). Modern systems with embedded Raft avoid this because the term-numbering scheme prevents two leaders in the same term from making progress.
Pitfall 2: Replication lag underestimation. Engineers assume followers are “almost up to date” because measurements look good in steady state. Then a long-running transaction, a long maintenance write, or a cluster restart creates seconds-to-hours of lag. Application code that reads from followers expecting near-immediate freshness produces user-visible bugs (e.g., “I just posted a comment but my profile says I have 0 comments”). Mitigation: monitor lag explicitly; add lag-aware routing; use read-after-write consistency techniques (§7.4 of this section’s source material). The standard observability stack for this is: (1) export pg_stat_replication.replay_lag (Postgres) / Seconds_Behind_Master (MySQL) / oplog_lag_seconds (MongoDB) to Prometheus or equivalent, every 10 seconds per replica; (2) alert at multiple thresholds — warn at lag > 1 second, page at lag > 30 seconds, page urgently at lag > 5 minutes (because at 5 minutes of lag a typical write workload has accumulated thousands of transactions of divergence); (3) build a lag-aware read router that automatically excludes followers whose lag exceeds a per-application SLO from the read pool, sending traffic to the primary instead; (4) instrument the application’s read-after-write code paths with explicit “saw_my_own_write” counters so you can detect rises in stale reads even when overall lag looks fine. Companies with mature operational practices (Stripe, Shopify, Twitch in their published engineering posts) treat lag monitoring as a tier-1 SRE concern equivalent to error rate monitoring.
Pitfall 3: GitHub October 2018. The canonical leader-follower-failover-gone-wrong incident. A 43-second partition between data centers triggered Orchestrator (the failover tool) to promote a US-West-Coast replica to primary while the original US-East-Coast primary was still alive on the network’s other side. When the partition healed, the system was forced to choose between divergent states; the resolution required hours of manual reconciliation. The lesson: failover decisions must be supported by consensus (the new leader must have a quorum that excludes the old leader’s view), and fencing must be enforced on the old leader before any new writes are accepted. Postmortem.
Pitfall 4: Asymmetric network partitions. A “partition” is rarely a clean cut where two halves cannot reach each other. Often it is one-way (A can send to B but B cannot send to A) or asymmetric (A reaches B and C but B and C cannot reach each other). Failover protocols must reason about this. Pre-vote (the Raft extension) and term checks help; naive heartbeat-based failover does not.
Pitfall 5: Read-after-write consistency violations. A user writes through the leader, then immediately reads — and the read goes to a follower that hasn’t replicated yet. The user sees “their” data hasn’t been saved. This is the single most common user-visible leader-follower bug, with the classic symptom being “I just posted my profile change, but my profile page shows the old data.” Mitigations: (a) route the user’s reads to the leader for a TTL after their last write (sticky routing — the standard production fix, costs a small fraction of read scaling); (b) read from leader for queries that depend on recent writes (table-level annotation: orders, profile, posts read from leader; analytics, history read from followers); (c) implement monotonic read sessions that always pick the same follower or a follower at-least-as-fresh (prevents data going backward but doesn’t fix initial freshness); (d) include the LSN/GTID of the user’s last write in their session and refuse to read from a follower behind that LSN (the most rigorous; requires LSN propagation through the application stack). Each technique handles a different facet of the consistency model: (a) gives strong read-after-write for the writer but no guarantee for other observers; (c) gives session monotonicity; (d) gives full per-session linearizability against the writer’s own causal history. Most production systems compose (a) + (c) for the common case.
Pitfall 6: Write amplification on the leader. All writes funnel through one node. If your write workload exceeds the leader’s IOPS, no number of followers helps. The cluster is write-bottlenecked. The only escape is sharding.
Pitfall 7: Long-running transactions blocking replication. Some replication mechanisms (e.g., MySQL row-based binlog) do not stream a transaction’s changes until commit. A long-running transaction holds back the entire replication stream until commit, after which followers see a burst. If the transaction is rolled back, the lag was for nothing. The pathological version is the “long-running query that takes a row-level lock” pattern: the lock is held for the duration of the query, which blocks other writes that want the same row, which themselves may block replication of unrelated transactions due to ordering constraints. Operations engineers call this the “head-of-line-blocking” of replication and watch for it via long-transaction monitoring. In Postgres, pg_stat_activity plus pg_locks exposes the offending transactions; in MySQL, INFORMATION_SCHEMA.INNODB_TRX and the slow-query log do. Killing the offending transaction is sometimes the only way out, but it’s never popular with the team that submitted it.
Pitfall 8: Slow follower brings down the cluster (sync mode). A misbehaving follower’s slow ack pace becomes the entire cluster’s write latency in sync replication. Quorum replication mitigates by requiring only a majority. Pure-sync remains hazardous.
Pitfall 9: Promotion of a stale follower. If failover picks a follower that was lagging at the moment the leader died, recently-confirmed writes are lost. Raft’s election restriction (only nodes with up-to-date logs can win) addresses this; ad-hoc failover scripts may not. The classical MySQL primary-replica setup pre-Group-Replication had no such protection — the operator had to compare LSN positions across surviving replicas before promoting, which was easy to get wrong under stress.
Pitfall 10: Replication-stream incompatibility across versions. Physical / WAL-based replication is tightly coupled to the leader’s storage-engine version. Upgrading the leader’s major version often requires a replication-format break, which means rebuilding all followers from scratch — a multi-hour or multi-day operation depending on data size. Logical replication helps because it is version-tolerant, but introduces its own complexity (e.g., cross-version replication of new features the older follower doesn’t support). PostgreSQL operators have spent considerable effort planning major-version upgrades around this constraint.
Pitfall 11: Read-only follower availability during failover. When the leader dies, followers may continue serving reads — but those reads are now strictly stale (no new writes are being applied) for the duration of the failover. Some applications can tolerate this; others cannot. A common bug pattern: a check-then-act sequence (if user.balance >= amount: deduct(amount)) reads from a follower, makes a decision, then writes to the (newly-promoted) leader. The decision was based on stale data; the write may now violate invariants.
Pitfall 12: Cascading replication topologies amplify lag. Some setups chain followers (primary → follower_A → follower_B) to reduce the primary’s network fan-out. Lag now compounds: follower B’s lag is lag(A) + lag(B). Failover from primary to B requires either replaying through A or losing whatever A had not yet shipped. Most modern setups avoid cascading in favor of fanned-out direct replication from the primary.
Pitfall 13: Failover-induced query-plan instability. After a failover, the new primary has fresh statistics and may produce different query plans than the old primary. Production engineers periodically encounter “the queries got slow after we failed over” — even though the data is identical, the new primary’s plan cache is empty and it re-plans queries with whatever heuristics happen to fire. Postgres’s pg_stat_statements and MySQL’s performance schema help diagnose this, but the underlying issue is unavoidable for engines that do per-instance plan caching. Mitigation: warm up the new primary’s plan cache during the failover process by replaying recent queries before opening it to traffic.
Pitfall 14: Sync-replication “deadlock” at three-or-more-replica configurations. A common misconfiguration: synchronous replication waiting for all followers, with three followers, two of which are simultaneously down for maintenance. The cluster halts on every write because synchronous-all cannot make progress with majority unreachable. The operator’s instinct is to disable sync replication, lose the durability guarantee, and operate in a degraded mode until maintenance completes — but this leaves a window of writes vulnerable to single-failure data loss. The correct approach: pre-plan maintenance so only one follower at a time is unavailable, or move to quorum (rather than all-followers) sync replication via embedded consensus before maintenance.
Pitfall 16: “Idempotent retries” interact badly with semi-sync timeouts. An application retries a write after a timeout, assuming the original was lost. But the original may have committed locally (and even shipped to followers) before the timeout fired; the retry produces a duplicate. Semi-sync’s auto-degrade-on-timeout can make this worse: the original write may have committed via degraded async without the operator knowing. Mitigations: idempotency keys (the application includes a unique key per logical request; duplicates with the same key are deduplicated server-side), conditional writes (CAS), or saga patterns that explicitly handle compensating actions. Naive retry-on-timeout is a foot-gun in any leader-follower system with non-trivial replication semantics.
Pitfall 15: Backup tools and replication interact subtly. A logical backup (pg_dump, mysqldump) taken while replication is running may capture an inconsistent snapshot if it doesn’t use a transaction-consistent read. Modern backup tools handle this (pg_dump --serializable-deferrable, mysqldump --single-transaction) but ad-hoc mysqldump invocations on production databases are a known source of corrupt backups. Physical-snapshot tools (pg_basebackup, xtrabackup) take a coordinated snapshot of the data files and the WAL position, producing backups that can be restored to a precise point in time — the modern best practice for any production deployment.
10. Comparison with Sibling Architectures
| Leader-Follower | Leaderless Replication Architecture | Multi-Leader Replication Architecture | |
|---|---|---|---|
| Writes accepted by | Only the leader | Any node | Multiple leaders (per region, etc.) |
| Conflict resolution | Not needed (single writer) | LWW, Vector Clocks, CRDTs Basics, app-level | LWW, Vector Clocks, CRDTs Basics |
| Write availability under partition | Halts (CP) | Available (AP, with R+W>N caveats) | Available per-leader; cross-leader merge later |
| Read consistency | Strong from leader; eventual from followers | Tunable via R+W>N | Eventually convergent |
| Write throughput | Bounded by leader | Linear in nodes | Linear in leaders |
| Cross-key transactions | Easy | Hard or impossible | Hard |
| Failover cost | Seconds to minutes; risk of split-brain | None (no leader to lose) | Per-leader same as L-F |
| Operational complexity | Medium | High (anti-entropy, gossip) | High (conflict resolution everywhere) |
| Canonical example | PostgreSQL, MongoDB, MySQL, Kafka per-partition | Cassandra, DynamoDB classic, Riak | CockroachDB multi-region, Galera, BDR |
The choice is fundamentally about the consistency-availability tradeoff (CAP in spirit, PACELC in practice). Leader-follower is CP under partition (it sacrifices write availability to maintain consistency); leaderless is AP under partition (it sacrifices consistency to remain available). Multi-leader is between — locally available, globally eventually consistent.
11. Common Interview Discussion Points
-
“Walk me through what happens when the leader dies.” The answer should hit: heartbeat failure detection, quorum-protected election, election restriction (only candidates with up-to-date logs win), promotion, fencing of old leader, client redirection. Hand-wave any step at your peril; interviewers know this is the failure mode that bites in production.
-
“Why is naive failover dangerous?” Split-brain. Two leaders for the same term, both accepting writes, producing divergent state that cannot be merged by replay. Fencing and quorum-based promotion are the prevention mechanisms.
-
“How do you handle read-after-write consistency?” Three answers acceptable: (1) read from leader for that user’s reads in a window after their write; (2) cross-replica monotonic-read sessions; (3) include the LSN/timestamp of the user’s last write in their session and refuse to read from a follower behind that LSN. The first is most common in production.
-
“What’s the difference between sync, async, and semi-sync replication?” See §5.1. Tradeoff is durability vs latency vs availability.
-
“How does Kafka use leader-follower replication?” Per-partition, with the leader handling all reads and writes; followers replicate via fetch RPCs. ISR (in-sync replicas) determines who can become leader.
acks=allmakes a write wait for all ISR replicas. -
“Compare leader-follower with Raft.” Leader-follower is the topology; Raft is a consensus protocol that implements leader-follower with formal guarantees about election safety, log matching, and leader completeness. Many leader-follower systems use Raft internally for their election; some still use ad-hoc heartbeat-based mechanisms (older MySQL setups).
-
“Why does the GitHub October 2018 incident matter for understanding leader-follower?” It is the textbook case of failover without proper fencing producing data divergence. It demonstrates that the architecture’s correctness depends entirely on the failover protocol’s correctness. The architecture failed in three reinforcing ways: heartbeat-based failure detection promoted a replica during a partition that didn’t kill the original primary; lack of fencing allowed both primaries to accept writes simultaneously; asynchronous cross-region replication meant the new primary was missing recent writes from the old primary. Modern systems with embedded Raft (CockroachDB, etcd) avoid this by requiring quorum consensus to depose a leader and by using fencing tokens to prevent any deposed leader from accepting subsequent writes.
-
“When would you choose leader-follower over leaderless?” When you need cross-row transactions, foreign keys, and strong consistency on writes. Leaderless cannot give you these without massive operational gymnastics.
-
“Why is async replication the default in MySQL and PostgreSQL?” Latency. Synchronous replication adds a network round-trip and a follower fsync to every write. For most workloads, the small data-loss window of async (milliseconds at typical write rates) is preferable to doubling write latency. Operators who need durability flip the synchronous flag explicitly.
-
“What is bounded-staleness consistency?” The follower is at most N seconds behind the leader (where N is bounded by the application). Cassandra’s
LOCAL_QUORUMand MongoDB’sreadPreference: secondaryPreferredwithmaxStalenessSecondsboth implement variations. The application accepts staleness within a known bound but rejects reads from followers more than N seconds behind. -
“How does Aurora differ from traditional PostgreSQL replication?” Aurora separates compute from storage: the WAL is written to a distributed shared-storage layer; both the leader and followers read pages from this shared storage. Followers don’t replay WAL — they read pages directly. This dramatically reduces follower lag because they don’t have to do the redo work themselves and can also serve more aggressive concurrency (the storage layer handles redo at the page level).
-
“What’s the difference between failover and switchover?” Failover is unplanned (the leader died); switchover is planned (the operator is upgrading or rebalancing). Switchover can be done cleanly: the old leader stops accepting writes, drains its replication queue to followers, then a follower is promoted with no data loss. Failover under unplanned death may lose any not-yet-replicated tail of writes.
-
“How is replication lag measured?” PostgreSQL exposes
pg_stat_replication.replay_lag(the time difference between the leader’s current LSN and the follower’s replayed LSN). MySQL exposesSeconds_Behind_Master. MongoDB exposes oplog timestamps. Kafka exposes per-partition consumer lag (which is similar but consumer-side rather than replica-side). Operators monitor these metrics aggressively; they are leading indicators of impending consistency violations. -
“Why does Kafka use leader-follower per partition rather than a cluster-wide leader?” Per-partition leadership is a sharded composition: each partition has its own leader, so the cluster has many leaders simultaneously, distributing the write load. A single cluster-wide leader would bottleneck Kafka’s per-second-millions-of-events workload. The per-partition leader is the smallest unit of replication that maintains ordering guarantees within a partition.
-
“What does the consensus group protect, exactly?” In modern leader-follower systems with embedded Raft, the consensus group protects three things: (1) the identity of the leader (only one valid leader per term); (2) the log of writes — each commit requires a majority quorum, so a committed entry survives any single failure; (3) the configuration changes — adding or removing a node is itself a consensus operation, so transient majority calculations don’t go wrong during membership changes.
-
“How is split-brain prevented in modern leader-follower systems?” Three layers: (1) consensus-protocol term numbers — a leader’s term increases on each election, and any node receiving a higher-term message demotes itself; (2) majority quorum — a write requires a majority ack, and two simultaneous leaders cannot both have majorities; (3) fencing tokens — clients carry a token issued by the current leader, and followers refuse to apply writes with stale tokens. Together these eliminate the classical split-brain corruption window.
-
“What’s a write-ahead log and why is it central to leader-follower?” The Write-Ahead Log (WAL) is a sequential, append-only record of every change before it is applied to the database’s main data structures. Three purposes: (1) crash recovery — on restart, replay the WAL since the last checkpoint to restore state; (2) replication — the WAL is exactly what gets streamed to followers; (3) durability — a write is “committed” once its WAL entry is fsynced. The WAL is the unification of these three concerns into a single primitive.
-
“Why are MongoDB replica sets typically 3 nodes rather than 5 or 7?” Three nodes tolerates one failure (majority is 2 of 3). Five nodes tolerates two failures (majority is 3 of 5). Seven nodes tolerates three failures. Most workloads value the failure tolerance of three but don’t justify the cost of five — each additional node adds replication overhead, ops cost, and election time. Five-node sets appear in higher-stakes deployments; seven is rare outside specific compliance regimes.
-
“What is logical replication and why does it matter for migrations?” Logical replication streams the logical events (this row was inserted, this row was updated to these values) rather than the storage-format-specific bytes. Cross-version replication works because both sides understand the logical event format. This makes major-version upgrades painless: build a new-version follower, replicate from the old-version leader logically, promote the new-version follower, decommission the old leader. Used at scale at Stripe, GitHub, and Heroku for PostgreSQL upgrades.
-
“What’s a fencing token?” An identifier issued by the consensus group to the current leader, monotonically increasing on each leader change. Followers and clients refuse to accept operations from leaders with stale fencing tokens. Prevents an old leader from continuing to write after a new leader has been elected. Martin Kleppmann’s “How to do distributed locking” essay popularized this term in the context of Redis Redlock; the underlying concept (epoch numbers, term numbers in Raft) is much older.
-
“How does Patroni manage PostgreSQL failover?” Patroni is a Python daemon that runs alongside each PostgreSQL instance and uses an external consensus store (etcd, Consul, or ZooKeeper) to coordinate. Each Patroni instance keeps a TTL-based lock in the consensus store; the lock-holder is the leader. On lock expiration, surviving Patronis race for the lock; the winner promotes its local PostgreSQL. The architecture cleanly separates the consensus problem (delegated to etcd) from the database mechanics. The pattern is widely adopted; pg_auto_failover is a simpler alternative; repmgr is older.
-
“What is a quorum write and why is it important?” A quorum write requires a majority of replicas to acknowledge before the leader confirms the write to the client. With 2f+1 replicas, the majority is f+1 — survives any f failures. The mathematical guarantee: any committed write is on f+1 replicas; any future leader’s election majority must overlap by at least one node; therefore the committed write propagates forward. This is the Raft / Paxos High-Level consistency primitive.
-
“Why is
acks=allin Kafka different from synchronous replication in Postgres?” Kafka’sacks=allwaits for all in-sync replicas (ISR) to ack — but the ISR set is dynamic. A slow replica gets removed from the ISR; the write proceeds without it. Withmin.insync.replicas=2and replication factor 3, the cluster requires at least 2 ISR replicas to ack but tolerates one being out. This is closer to quorum semantics than to strict synchronous-all. Postgressynchronous_commit=onwithsynchronous_standby_namesis more rigid; the listed standbys must ack or the write blocks. -
“What’s the difference between
synchronous_commit = remote_writeandremote_applyin Postgres?”remote_writewaits for the standby’s receipt and OS-level write of the WAL but not for fsync or replay.remote_applywaits for the standby to actually replay the change, making the new state queryable on the standby. The hierarchy:local(write to local WAL only, then commit) →remote_write(also wait for standby’s OS write) →remote_apply(also wait for standby’s replay).remote_applyis the strongest and the most expensive; it gives read-after-write consistency from the standby in addition to durability. Most applications don’t needremote_apply— they’re satisfied with the durability ofremote_writeplus a separate read-routing strategy for read-after-write consistency. -
“How does MySQL semi-sync’s auto-degradation differ from Postgres synchronous replication?” MySQL semi-sync auto-falls-back to async after a configurable timeout (default 10 seconds) if no follower acks. The fallback is silent — only a warning in the error log — and exposes the cluster to data loss for the duration of the degraded operation. Postgres’s behavior is to block writes when the configured synchronous standby is unreachable, prioritizing durability over availability. The MySQL design optimizes for availability; the Postgres design optimizes for durability. Both are defensible; both can produce production incidents if the operator’s expectation doesn’t match the configured behavior.
-
“What’s the difference between asynchronous, synchronous, and quorum replication numerically?” Asynchronous: leader commits locally and replies in ~300 microseconds, propagating to followers in the background; data loss window is ~milliseconds of writes if leader crashes. Synchronous-one-follower: leader commits locally then waits for one follower’s ack, total latency ~1-1.5 milliseconds intra-DC; no data loss across single-node failure. Quorum: leader commits locally then waits for majority of followers to ack; for a 5-node cluster, latency is bounded by the third-fastest follower, typically ~1-3 ms intra-DC; tolerates
(N-1)/2follower failures without halting writes. Cross-DC adds 5-50 ms WAN RTT to any of these. The numerical reality is that durability levels above asynchronous each cost a small constant-factor increase in write latency; the cluster’s availability response to follower failures is the qualitatively different concern. -
“What is GTID and why does MySQL replication need it?” Global Transaction Identifier — a unique identifier per transaction in MySQL replication, replacing the older binlog-position-based replication. With GTIDs, every transaction has a globally-unique identifier; followers track which GTIDs they’ve applied; failover doesn’t require manually finding the binlog position to resume from — the new primary just continues from its highest GTID. Critical for multi-source replication and for failover scenarios where the binlog-position chain would be broken. MySQL 5.6+ supports GTIDs; modern deployments universally use them.
-
“How does Kafka per-partition leadership generalize the leader-follower architecture?” Each topic partition has its own leader; the cluster has many leaders simultaneously, distributed across brokers. The controller deliberately spreads partition leaderships across brokers (avoiding the case where one broker is leader for too many partitions) so a single broker’s failure triggers re-election only for its share of partitions, not the whole cluster. This is the same structural pattern as sharded leader-follower in Vitess (per-keyspace tablets), CockroachDB (per-range Raft groups), MongoDB sharded clusters (per-shard primaries), and Spanner (per-tablet leaders). The pattern — sharded leader-follower with deliberate leadership balancing — is the modern default for any system that needs leader-follower’s consistency model with horizontal scale.
-
“How do you reason about the tradeoff between Postgres physical and logical replication?” Physical replication is faster, more faithful, and locked to one Postgres major version. Logical replication is slower, requires per-table publication setup, and decouples the upstream and downstream — enabling cross-version replication, partial replication, and downstream consumers (Kafka, analytics warehouses). The community’s consensus by 2023: physical for high-availability followers; logical for migrations and cross-system integration. Stripe, GitHub, Heroku publicly use logical replication for major-version Postgres upgrades on multi-TB databases.
-
“What’s the difference between failover and switchover, and why does that matter for production planning?” Failover is unplanned (the leader died unexpectedly) and may lose any not-yet-replicated tail of writes. Switchover is planned (the operator is upgrading or rebalancing) and can be done cleanly: the old leader stops accepting new writes, drains its replication queue to followers, then a follower is promoted with no data loss. Operationally, switchover should be exercised regularly (monthly or quarterly) so the team has confidence the procedure works; many production teams have discovered during their first real failover that their switchover automation had decayed because nobody ran it. Disaster-recovery exercises (“game days”) are the modern best practice for keeping switchover machinery healthy.
-
“How does etcd’s leader election work and why is etcd a good study reference?” Etcd uses Raft for both leader election and log replication; the etcd codebase is a particularly clean implementation that’s small enough to read end-to-end. Election: nodes start in follower state with randomized election timeouts; if a follower’s timeout expires without hearing from a leader, it becomes a candidate and starts a vote; if a candidate gets a majority, it becomes leader for that term. The randomized timeout prevents split votes (multiple candidates simultaneously) from being a permanent failure mode. The election restriction (only candidates with up-to-date logs can win) ensures committed writes survive elections. Worth reading:
raftexamplein the etcd repository is a minimal Raft implementation in ~500 lines of Go, intended as a study aid.
12. Open Questions
- How do modern systems quantify the “write loss window” of async replication in practice? (Few companies publish their durability budgets explicitly.)
- At what cluster scale does external-coordinator failover (Sentinel, ZK) become more reliable than embedded-consensus failover (Raft-style)? Or does it ever?
- What is the right read-after-write consistency mechanism for a read-heavy workload — sticky routing, LSN-aware reads, or session affinity?
- How does Aurora’s “compute-storage separation” (where followers don’t replay WAL but read pages from shared storage) compare on tail-latency to traditional leader-follower under bursty workloads? AWS’s published benchmarks favor Aurora but independent comparisons are scarce.
- Has the gradual standardization on Raft for failover (etcd, MongoDB, Kafka KRaft, MySQL Group Replication) effectively retired ad-hoc / heartbeat-based failover for new systems, or do simpler architectures still have merit at small scale?
- When does adopting a lease-based leadership model (Spanner, CockroachDB) outperform heartbeat-based detection in practice? The former requires bounded clocks; the latter requires careful timeout tuning.
- Has logical replication’s adoption pattern stabilized, or are there still cross-version migration scenarios where physical replication remains preferable?
- What is the practical operational threshold (terabytes? transactions per second?) above which sharded leader-follower becomes mandatory rather than optional?
Uncertain uncertain
Verify: Aurora’s “5× the read throughput of vanilla Postgres/MySQL at the same hardware spec” figure (stated in §“Aurora’s Architectural Variation” below). Reason: this multiplier originates from AWS’s own marketing and the Aurora SIGMOD papers (Verbitski et al. 2017/2018); the primary PDF could not be machine-parsed during this verification pass, and independent third-party benchmarks reproducing the multiplier under comparable conditions are scarce. The architecture itself is verified against AWS docs — storage is synchronously replicated to six storage nodes across multiple Availability Zones, up to 15 Aurora Replicas use asynchronous replication and read pages from shared storage rather than replaying redo, and failover typically completes in under 60 s, often under 30 s (AWS, High availability for Amazon Aurora). To resolve: read the throughput tables in the Aurora SIGMOD 2017/2018 papers directly and corroborate with an independent benchmark.
13. See Also
- Leaderless Replication Architecture — the AP cousin
- Multi-Leader Replication Architecture — the geo-distributed cousin
- Sharded Architecture — usually composed with leader-follower at per-shard level
- Raft — the consensus protocol used by modern leader-follower failover
- Paxos High-Level — Raft’s elder
- Two-Phase Commit — needed for cross-shard transactions on top of sharded leader-follower
- Vector Clocks — relevant when leader-follower interacts with multi-leader replication
- Distributed Key Value Store System Design — many KV stores use sharded leader-follower internally
- Distributed SQL Database System Design — Spanner, Cockroach are sharded leader-follower at the per-tablet level
- Distributed Log System Design — Kafka per-partition leader-follower
- System Architectures MOC
- SWE Interview Preparation MOC