Replication Lag & Session Guarantees — Read-Your-Writes, Monotonic Reads & Consistent Prefix

Overview

Async followers are always a little behind. Usually it's milliseconds, but during a bulk load, a long query on the replica or a network hiccup it can be minutes. That's replication lag. Lag on its own is fine. The trouble is anomalies the user can see. The three that interviewers ask about come from the 1994 "session guarantees" work. Read-your-writes: after I save, I see my save. Monotonic reads: I never see time go backwards between two reads. Consistent prefix: I never see an answer before its question. Each has a cheap, targeted fix, so you rarely need to send every read to the leader.

Anomalies and fixes

flowchart TB
  w["1. Client writes avatar"]
  l["2. Leader commits at LSN 1007"]
  rq["4. Next read carries token"]
  rt["5. Router: which replica replayed at least 1007?"]
  w -->|1. Client writes avatar to 2. Leader commits at LSN 1007| l
  l -->|3. returns token lsn 1007| w
  w -->|1. Client writes avatar to 4. Next read carries token| rq
  rq -->|4. Next read carries token to 5. Router: which replica replayed at least 1007?| rt

Press Run. Snippets must be self-contained — no network, files, or native modules.

Output:

leader lsn=7, r1 replay=7, r2 replay=4
naive read  -> ('old-avatar.png', 'r2') (stale: user sees old avatar)
token read  -> ('new-avatar.png', 'r1') (read-your-writes holds)

Strategies compared, and what happens if you pick another

• If you only pin by time window: it works until the day lag spikes during a backfill. Then support tickets say "my settings don't save". Pair it with a lag circuit breaker: if a replica's lag exceeds the threshold, take it out of the read pool.
• If you send everything to the leader "to be safe": it's correct, but you've given up read scaling, and the leader fails under read load before write load.
• If you rely on sticky replicas only: monotonic reads hold, but your own writes still aren't visible until that replica catches up, so you still need RYW handling.

// Monotonic reads: random replica per request vs sticky replica per user.
// Random routing lets a user see a newer value, then an OLDER one ("time travel").
// Sandbox-runnable: tsc --strict then node. Deterministic PRNG.

let seed = 42;
const rand = () => ((seed = (seed * 1103515245 + 12345) % 2 ** 31) / 2 ** 31);

const replicaLag = [0, 2, 6];          // records behind leader for replicas 0..2
const REQUESTS = 2000;

function simulate(pick: (user: number) => number): number {
  let timeTravel = 0;
  const lastSeen = new Map<number, number>();         // user -> highest version seen
  for (let t = 10; t < REQUESTS + 10; t++) {
    const user = Math.floor(rand() * 50);
    const r = pick(user);
    const versionSeen = t - replicaLag[r];             // leader is at version t
    const prev = lastSeen.get(user) ?? -1;
    if (versionSeen < prev) timeTravel++;              // went backwards!
    lastSeen.set(user, Math.max(prev, versionSeen));
  }
  return timeTravel;
}

const randomPick = () => Math.floor(rand() * replicaLag.length);
const stickyPick = (user: number) => user % replicaLag.length;  // hash(user) -> replica

console.log(`random replica per request : ${simulate(randomPick)} backwards reads`);
console.log(`sticky replica per user    : ${simulate(stickyPick)} backwards reads`);

Output:

random replica per request : 32 backwards reads
sticky replica per user    : 0 backwards reads

Measuring lag correctly

• Postgres: pg_stat_replication on the primary shows write_lag, flush_lag, replay_lag and the LSNs. On a replica, now() - pg_last_xact_replay_timestamp() is misleading when the primary is idle, so compare LSNs instead.
• MySQL: Seconds_Behind_Source measures the relay log, not the source, and reads 0 when the IO thread is stalled. A heartbeat table (pt-heartbeat) is more reliable: the primary writes now() every second and you compare on the replica.
• Causes of lag: single-threaded apply (use parallel apply such as replica_parallel_workers), long replica queries conflicting with replay (Postgres max_standby_streaming_delay), huge transactions, under-provisioned replica IO, and cross-region bandwidth.
• Alert on lag in seconds and in bytes, and route around replicas that exceed the threshold.

Interview Q&A

Q1. How would you implement read-your-writes for a web app with read replicas? On write, capture the commit position (Postgres pg_current_wal_lsn(), MySQL GTID) and store it in the user's session. On read, the router chooses a replica whose replay position is at least that token. If none qualifies, it waits briefly (MySQL WAIT_FOR_EXECUTED_GTID_SET with a timeout), then falls back to the leader. Keep the token server-side so it works across devices.

Q2. Why is "read from leader for 5 seconds after a write" fragile? It assumes lag stays under 5 seconds. Lag is unbounded in exactly the situations where the guarantee matters: incidents, bulk jobs, failovers. It also ignores other users' writes (a comment you were notified about) and cross-device reads.

Q3. What is consistent prefix and when does it bite? Observers should see writes in an order consistent with causality. It bites in sharded or partitioned systems where a question and its answer live on different partitions that replicate at different speeds. Fix it by co-locating causally related data (same partition key, for example conversation_id) or by tracking causal dependencies.

Q4. Monotonic reads vs read-your-writes? RYW concerns your own writes being visible to you. Monotonic reads concerns any data never going backwards between your successive reads. Sticky routing gives monotonic reads. A token with your last write's LSN gives RYW. A token with the max LSN you've seen gives both.

Q5. Replica lag is 30 minutes and climbing. What do you check? Is the replica's apply single-threaded and CPU-bound? Is there a long-running transaction or huge batch on the primary? Is replay blocked by a long query on the replica (conflict wait)? Look at IO saturation and network throughput, and check whether a schema change or index build is replaying. In the meantime, pull it from the read pool.

Go Deeper

• PostgreSQL docs: Hot Standby (query conflicts, replay)
• PostgreSQL docs: pg_stat_replication view
• MySQL 8.4: WAIT_FOR_EXECUTED_GTID_SET and GTID functions
• MongoDB: Causal consistency and read/write concerns
• Werner Vogels: Eventually Consistent, Revisited

Related

• Prev: Sync vs Async vs Semi-Sync Replication - Durability, RPO & Physical vs Logical Logs (replication-sync-async-semi-sync-physical-logical)
• Next: Failover & Split Brain - Detection, Promotion, Fencing & Lost Writes (replication-failover-split-brain-fencing)

This series:

• Database Replication for Engineers - Leader-Follower, Multi-Leader & Leaderless Quorums (database-replication-leader-follower-multi-leader-leaderless)
• Sync vs Async vs Semi-Sync Replication - Durability, RPO & Physical vs Logical Logs (replication-sync-async-semi-sync-physical-logical)
• Replication Lag & Session Guarantees - Read-Your-Writes, Monotonic Reads & Consistent Prefix (replication-lag-read-your-writes-session-guarantees) - this page
• Failover & Split Brain - Detection, Promotion, Fencing & Lost Writes (replication-failover-split-brain-fencing)
• Multi-Leader & Active-Active - Conflict Detection, LWW, CRDTs & Home Regions (multi-leader-replication-conflicts-active-active)
• Leaderless Replication - N/R/W Quorums, Hinted Handoff, Read Repair & Anti-Entropy (leaderless-replication-quorums-read-repair-anti-entropy)

Existing Study pages (cross-link only, not rewritten here):

• MVCC & snapshot isolation (mvcc-snapshot-isolation)
• Database sharding - partition keys & rebalancing (db-sharding-partitioning-keys-rebalancing)