vault

Replication Lag and Read-Your-Writes Consistency

11 min read

Replication Lag and Read-Your-Writes Consistency

When a database replicates asynchronously, a follower is always some interval behind the leader — the replication lag — and a read served from that follower reflects the past, not the present. For read scaling this is usually fine, but it breaks a family of guarantees that applications (and users) unconsciously assume, and the resulting bugs are among the most confusing in distributed data. The three that matter most are the session guarantees formalized for the Bayou system by Terry et al. (Session Guarantees for Weakly Consistent Replicated Data, 1994): read-your-writes (you must see your own just-made write), monotonic reads (you must not see time go backward), and consistent-prefix reads (you must not see an effect before its cause). None of these is provided by “async replication + read-balancing across followers”; each must be deliberately engineered on top, and this note covers the anomalies, the primary definitions, and the implementation techniques — reading from the leader, tracking log positions, and sticky routing. It is the correctness-consequence companion to Leader-Follower Replication and Synchronous versus Asynchronous Replication.

Mental Model: The Follower Lives in the Past

Every asynchronous follower has an applied position — a point in the leader’s log up to which it has replayed. The leader is at position 8000; a follower at 7980 is showing you the world as of 20 log entries ago. Load-balanced reads make it worse: consecutive reads from one client can land on different followers at different positions, so the client’s view of “now” jumps around unpredictably. Every session-guarantee violation is a story about a read landing on a follower whose applied position is behind some write the client cares about.

sequenceDiagram
    actor U as User
    participant L as Leader (pos 8000)
    participant Fa as Follower A (pos 7995)
    participant Fb as Follower B (pos 7980)
    U->>L: WRITE order #abc (commits at pos 8000)
    L-->>U: "committed"
    Note over U: user redirected to order page
    U->>Fb: READ order #abc
    Fb-->>U: "not found" (Fb is at 7980 < 8000)
    Note over U,Fb: read-your-writes VIOLATED
    U->>Fa: refresh (routed elsewhere)
    Fa-->>U: order #abc found (Fa at 7995... or has since caught up)

The canonical read-your-writes failure on an e-commerce checkout. What it shows: the user’s write commits at leader position 8000 and is acknowledged, but the immediately-following read is load-balanced to Follower B, still at position 7980, which has no record of the order and returns “not found”; a refresh routed to a more-current follower then shows it. The insight to take: the write genuinely succeeded and is genuinely durable — the bug is purely that the read consulted a replica that had not yet applied it. The user experiences this as “the system lost my order,” one of the most trust-destroying failures a product can have, and it is produced by the default combination of async replication plus naive read-balancing.

The Three Session Guarantees, Precisely

Terry et al. define the guarantees over a client session — a sequence of a client’s reads and writes — against a weakly consistent store where different reads may hit different servers (Terry 1994).

1. Read Your Writes (read-after-write). Formally: “If Read R follows Write W in a session and R is performed at server S at time t, then W is included in DB(S,t)” — the server serving your read must already contain your earlier write. The anomaly it prevents: you change your password, then log in with the new password and get “invalid password” because the authentication read hit a replica that had not yet received the password change. Or the checkout example above.

2. Monotonic Reads. Formally: “If Read R1 occurs before R2 in a session and R1 accesses server S1 at time t1 and R2 accesses server S2 at time t2, then RelevantWrites(S1,t1,R1) is a subset of DB(S2,t2)” — once you have seen a write, no later read may fail to show it, even from a different server. The anomaly it prevents: you refresh your calendar and see a newly-added meeting; you refresh again (routed to a more-lagged replica) and the meeting disappears. You have watched time run backward. This is weaker than read-your-writes — it does not require seeing your own latest write, only that you never un-see something you already saw.

3. Consistent Prefix Reads. The reader sees writes in an order consistent with the order they were actually written — no effect appears before its cause. The anomaly it prevents: an observer watching a conversation sees the answer (“about ten seconds”) arrive before the question (“how long does it take?”), because the two writes (possibly on different partitions) replicated to the reader’s replica in the wrong relative order. In Terry’s framework the closely-related formal guarantees are Writes Follow Reads (“if Read R1 precedes Write W2 in a session… any W1 in RelevantWrites… is also in DB(S2) and WriteOrder(W1,W2)”) and Monotonic Writes (“if Write W1 precedes Write W2 in a session, then… W1 is also in DB(S2)… and WriteOrder(W1,W2)”), which together preserve causal and write order. All four Terry guarantees are catalogued in Session Guarantees for Consistency; the three above are the ones application developers hit first.

A crucial point: these are session (per-client) guarantees, deliberately weaker and cheaper than global linearizability. They give a single client a coherent view without forcing the whole store to agree on a global order — which is exactly why they are the practical patch of choice for replication lag.

How Bad Is the Lag? Quantifying the Window

The size of the anomaly window is exactly the replication lag, and it is not a constant — it spikes under write bursts, network degradation, and slow-applying followers, and it is bounded only by whatever back-pressure the system has. The PBS work turns this into a measurable quantity via t-visibility, “the probability of reading a write t seconds after it returns,” and finds that in the average case eventually-consistent stores “frequently return consistent data within tens of milliseconds” — but that the window is workload- and hardware-sensitive: one production system needed 45.5 ms for 99.9%-consistent reads on spinning disks versus 1.85 ms on SSDs, purely because faster writes have lower latency variance (Bailis et al., PBS, VLDB 2012). The lesson for implementers: you cannot assume a small fixed lag; a mitigation that reads from a follower “if it’s caught up” must measure caught-up-ness, not guess a timeout.

Measuring lag is engine-specific. PostgreSQL exposes both the received and applied positions — pg_last_wal_receive_lsn() and pg_last_wal_replay_lsn() on the standby, against pg_current_wal_lsn() on the primary — and the docs name the health metric as “the amount of WAL records generated in the primary, but not yet applied in the standby” (PostgreSQL streaming replication); MySQL exposes Seconds_Behind_Source and the relay-log positions. These are the raw materials for position-aware routing below.

Implementation Techniques

There is no single fix; production systems layer several. Ordered from bluntest to most precise:

1. Read from the leader for a window after a write. The simplest read-your-writes fix: for a short interval after any write, route that user’s reads to the leader (which is always current). Implement it as a per-session marker “last_write_at = T0” and route to the leader while now < T0 + Δ, where Δ safely exceeds the maximum expected lag. Costs read-scaling for that user for the window, but eliminates the anomaly for the common “write then immediately view” journey. This is the mitigation most teams reach for first.

2. Track the write’s log position and read from any follower that has passed it. More precise: on each write, record the resulting log position in the session (PostgreSQL LSN, MySQL GTID/position, a Cassandra timestamp). On a subsequent read, pick any follower whose applied position is ≥ the recorded one, falling back to the leader if none has caught up. This preserves read-scaling for caught-up followers and gives exact read-your-writes, at the cost of plumbing log positions through every write and read path and querying per-follower lag — invasive but correct. PostgreSQL’s replay-LSN functions and MySQL’s executed-GTID sets are precisely what this needs.

3. Sticky routing (pin a session to one follower). Route a whole session’s reads to a single follower. That follower may be stale, but its position only ever advances, so the client gets monotonic reads for free — it can never see time go backward because it never hops to a more-lagged replica. Sticky routing does not by itself give read-your-writes (the pinned follower may still be behind the user’s own write), but it is the natural monotonic-reads fix and is often combined with technique 1.

4. Coordinator affinity in leaderless stores. Dynamo builds a read-your-writes bias directly into coordinator selection: “the coordinator for a write is chosen to be the node that replied fastest to the previous read operation which is stored in the context information of the request. This optimization enables us to pick the node that has the data that was read by the preceding read operation thereby increasing the chances of getting ‘read-your-writes’ consistency” (Dynamo, SOSP 2007). It is a probabilistic nudge, not a guarantee.

5. Push durability forward so the follower is already consistent. For flows where read-after-write matters, write synchronously to the level that makes a follower query-visible — PostgreSQL synchronous_commit = remote_apply guarantees the synchronous standby has applied (not merely received) the change, so a read there is immediately consistent (see Synchronous versus Asynchronous Replication). This trades write latency for eliminating the read-side problem, and is applied selectively to high-stakes tables.

6. Operational: keep lag small and eject lagging followers. Independently of application logic, monitor per-follower lag (pg_stat_replication.replay_lag, Seconds_Behind_Source) and pull any follower exceeding a threshold out of the read pool until it catches up. This does not fix a single user’s bug but prevents the anomaly window from ballooning cluster-wide when replication degrades.

Failure Modes and Diagnosis

  • “Users report their data vanishes then reappears.” Classic monotonic-reads violation from load-balanced reads across followers at different positions. Diagnosis: correlate the reports with follower lag spikes; check whether reads are sticky. Fix: sticky routing (technique 3).
  • “New signups can’t log in immediately.” Read-your-writes violation on the auth path. Fix: route post-write reads to the leader (technique 1) or make the auth write synchronous-apply.
  • “Comments appear out of order / reply before original.” Consistent-prefix violation, common when related writes span partitions replicated independently. Fix: keep causally-related data co-partitioned, or track causal dependencies.
  • Silent, growing anomaly window. The mitigation used a fixed timeout Δ that was fine at low load but is now smaller than actual lag under a write burst. Fix: switch from timeout-based (technique 1) to position-based routing (technique 2), which is self-correcting.

Alternatives and When to Choose Them

The heaviest hammer is to eliminate stale reads entirely by reading only from the leader (or a synchronous-apply standby) — full read-after-write and monotonic reads, zero read scaling from followers. The opposite extreme, plain async + read-balancing, gives maximum read scaling and none of the guarantees; it is correct only for genuinely stale-tolerant reads (analytics dashboards, recommendation feeds). The session-guarantee techniques above are the pragmatic middle, buying per-client coherence cheaply without paying for global consistency. When an application truly needs a global real-time order (not just per-session), the answer is linearizability via Consensus in Databases with Raft and Paxos or a globally-ordered store like Spanner — far more expensive, and rarely what a lag bug actually requires.

Production Notes

The dominant production pattern is “reads from followers for stale-tolerant flows; reads from the leader for the user’s own recently-written data,” with the routing decision made in the application or a middleware layer — cheap, effective, and the reason most teams never need position-tracking. The recurring failure is treating replication lag as a monitoring concern rather than a correctness concern: teams add follower read-scaling for performance, ship it, and only discover the read-your-writes and monotonic-reads bugs when users report them, because the bugs are invisible in the happy path and appear only under lag. The defensive posture — assume the follower is arbitrarily stale, engineer the guarantee you need explicitly — is the one lesson worth internalizing, and it is the same lesson whether the store is single-leader (Leader-Follower Replication) or leaderless (Leaderless Replication and Quorums, where lag manifests as a not-yet-converged replica).

See Also

Graph View