Distributed systems
Part 1 of 6 · Time, Clocks & OrderingTime, Clocks & Ordering in Distributed Systems - Physical Clocks, Lamport, Vector Clocks, HLC & TrueTime
Interview hub: why no machine knows the real time; wall vs monotonic; the ladder from NTP wall clocks to Lamport, vector clocks, HLC and TrueTime with a decision flow; runnable LWW-on-skewed-clocks data loss vs Lamport vs vector clocks; comparison table incl. timestamp oracles; Cloudflare 2017 leap second, Spanner, CockroachDB, Dynamo, Snowflake/UUIDv7 anchors.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Question ladder
L1
Why can't two machines share one exact 'now'?
Answer
Clocks skew, drift, and jump when NTP steps them or around leap seconds.
L2
When do you use a monotonic clock?
Answer
For durations on one machine: timeouts, latency, lease countdowns, and retry backoff.
L3
What does a Lamport clock guarantee?
Answer
If a happens-before b, then L(a) < L(b). The converse is false.
L4
When do you need a vector clock?
Answer
When you must notice that two writes were concurrent so you can keep both and merge.
L5
How does Spanner get external consistency?
Answer
TrueTime returns an interval. Spanner commits at the latest edge and waits until that timestamp is in the past.
L6
Why doesn't CockroachDB copy commit wait?
Answer
NTP-grade uncertainty would add tens of milliseconds to every write, so it uses HLC plus uncertainty restarts.
L7
What does a time-ordered ID inherit from this ladder?
Answer
Snowflake, UUIDv7, and ULID sort by time and break if the clock moves backwards.
Failure modes
Wall-clock last-writer-wins drops the newer write
In the example, replica B is 120 ms slow, so v2 at true time 1050 is stamped 930 and v1 wins.
A lease uses the wall clock
An NTP step backwards makes the holder believe the lease is still valid after it has expired elsewhere.
Commit wait with a wide uncertainty
Every write pays roughly twice the clock uncertainty. That is why CockroachDB chose HLC plus read restarts instead of TrueTime-style waiting on NTP.
Misconceptions
A timestamp is a safe total order across machines.
Skew, drift, NTP steps, and leap seconds can stamp two close events in the wrong order.
If L(a) is less than L(b), then a caused b.
Lamport clocks only promise one direction. A smaller timestamp can still be concurrent.
Every system should copy Spanner's commit wait.
The wait stays small only with tight uncertainty. Wider bounds put that latency on every commit.
Interviewer traps
Ordering a feed or URL shortener by wall-clock time and stopping there.
Say whether you need sortability, index locality, or no coordination, then name Snowflake, UUIDv7, ULID, or a sequence.
Treating a monotonic clock as a cross-machine order.
It measures elapsed time on one machine. It does not compare across hosts.
Design scenario
Same prompt for every reader.
Requirements
The cart must not drop a concurrent add. The ledger must respect real-time commit order. Timeouts must survive an NTP step.
Traffic / scale
Example from the hub snippet: v2 is 50 ms after v1 in true time, and replica B is 120 ms slow.
Latency
Commit wait grows with clock uncertainty. A wide bound is too expensive for every ledger write.
Consistency
Cart writes need concurrency detection. The ledger needs external consistency, not only a Lamport total order.
Availability
A single timestamp oracle is a place every transaction waits on.
Failure assumptions
- Replica clocks can disagree by more than the gap between two writes.
- NTP can step the wall clock backwards.
- A leap second can repeat or smear a second.
Constraints
- Do not use wall-clock last-writer-wins for the cart.
- Do not measure the timeout with the wall clock.
Prompt
A multi-writer cart and a global ledger share a platform. Also pick a clock for request timeouts.
API
What context does the client pass so a later cart write is not blind?
Data
Which stamp do you store on the ledger commit, and when do you acknowledge?
Architecture
Where do monotonic time, vector clocks, and HLC or TrueTime each sit?
What you are actually ordering
Prefer
The weakest mechanism that answers the question
Match the clock to the question: duration, approximate stamps, causal total order, concurrency detection, or external consistency.
- Monotonic time for timeouts and leases on one machine.
- Lamport when a total order must respect causality.
- Vector clocks or a CRDT when two writers must both survive.
Alternative
One wall-clock timestamp for every decision
Last-writer-wins, lease expiry, and 'latest' reads inherit skew, NTP steps, and leap seconds.
- A slow replica stamps a newer write as older.
- A backwards step makes a lease look unexpired.
- Spanner-style waits are too expensive if the uncertainty is wide.
From the question to the clock
The hub map. Sibling pages hold NTP math, Lamport's blind spot, siblings, commit wait, and ID schemes.
- 1
Name the question
Duration, approximate time of day, causal order, concurrency, or real-time commit order. - 2
Reject wall-clock last-writer-wins for causality
In the example, replica B is 120 ms slow, so the newer v2 is stamped earlier and discarded. - 3
Pick Lamport, a vector clock, HLC, or TrueTime
Lamport total-orders. Vector clocks detect concurrency. HLC stays near wall time. TrueTime commit-waits for external consistency. - 4
Remember IDs inherit the clock
Snowflake, UUIDv7, and ULID are time-ordered and break if the clock steps backwards.
Overview
Every distributed system eventually has to answer "which of these two things happened first?", and the honest answer is that no machine knows the real time exactly. Wall clocks drift, get stepped by NTP, repeat or smear leap seconds, and disagree across machines by milliseconds to seconds. This cluster teaches the ladder of answers engineers actually use: physical clocks when approximate is fine, Lamport clocks when you need an order consistent with causality, vector clocks when you must detect concurrent writes, hybrid logical clocks (HLC) when you want causality plus human-readable timestamps, TrueTime with commit wait when you need real-time ordering across the planet, and the ID generators (Snowflake, UUIDv7, ULID) that quietly depend on all of this.
Why this matters in interviews and in production
- "Use the timestamp to decide which write wins" is one of the most common silent data-loss bugs in replicated systems (last-writer-wins on skewed clocks).
- Leases, locks, timeouts and TTLs break when they are measured with the wall clock instead of a monotonic clock.
- Interviewers love the follow-up chain: why can't you just use timestamps? -> Lamport -> what Lamport can't tell you -> vector clocks -> why don't Dynamo-style stores keep them forever? -> how does Spanner do it? -> how does CockroachDB do it without atomic clocks?
- Every "design a URL shortener / feed / chat" question needs an ID scheme, and the answer depends on ordering, index locality and coordination trade-offs covered here.
The core vocabulary
| Term | Meaning | One-line intuition |
|---|---|---|
| Wall clock (time-of-day) | Seconds since the Unix epoch, synced by NTP/PTP | Human readable, can jump backwards |
| Monotonic clock | Counter that only moves forward at a steady rate | Good for durations, meaningless across machines |
| Clock skew / offset | Difference between two clocks at the same instant | "Node B is 120 ms behind" |
| Drift | Rate at which a clock gains or loses time | Measured in ppm (parts per million) |
| Happens-before (a -> b) | a could have influenced b (same process earlier, or via a message chain) | Causality, not wall time |
| Concurrent (a || b) | Neither happens-before the other | They could not have seen each other |
| Logical clock | Counter that respects happens-before | Lamport, vector, HLC |
| External consistency | If T1 commits before T2 starts (in real time), T1's timestamp < T2's | Spanner's guarantee, a.k.a. strict serializability |
The ladder of clocks (step-labeled)
Decisions
- 1
Step 1: What question are you asking about time?
- nextStep 2: Measuring a duration on one machine?
- ?
Step 2: Measuring a duration on one machine?
- yesUse a monotonic clock (CLOCK_MONOTONIC, performance.now, time.monotonic)
- noStep 3: Need an order consistent with causality?
- 3
Use a monotonic clock (CLOCK_MONOTONIC, performance.now, time.monotonic)
- ?
Step 3: Need an order consistent with causality?
- no, approximate is fineNTP-synced wall clock, treat as approximate (logs, dashboards)
- yesStep 4: Must you DETECT concurrent writes?
- 5
NTP-synced wall clock, treat as approximate (logs, dashboards)
- ?
Step 4: Must you DETECT concurrent writes?
- yesVector clock / dotted version vector or a CRDT
- no, a total order is enoughStep 5: Want timestamps close to real time?
- 7
Vector clock / dotted version vector or a CRDT
- ?
Step 5: Want timestamps close to real time?
- noLamport clock + node id tie-break
- yesStep 6: Need real-time order across all clients (external consistency)?
- 9
Lamport clock + node id tie-break
- ?
Step 6: Need real-time order across all clients (external consistency)?
- noHybrid logical clock (CockroachDB, MongoDB, YugabyteDB)
- yesBounded-uncertainty clock + commit wait (Spanner TrueTime) or a single timestamp oracle
- 11
Hybrid logical clock (CockroachDB, MongoDB, YugabyteDB)
- 12
Bounded-uncertainty clock + commit wait (Spanner TrueTime) or a single timestamp oracle
Lesson map
Time, Clocks & Ordering in Distributed Systems - Physical Clocks, Lamport, Vector Clocks, HLC & TrueTime
Interview hub: why no machine knows the real time; wall vs monotonic; the ladder from NTP wall clocks to Lamport, vector clocks, HLC and TrueTime with a decision flow; runnable LWW-on-skewed-clocks data loss vs Lamport vs vector clocks; comparison table incl. timestamp oracles; Cloudflare 2017 leap second, Spanner, CockroachDB, Dynamo, Snowflake/UUIDv7 anchors.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB q["Step 1: What question are you asking about time?"] d["Step 2: Measuring a duration on one machine?"] mono["Use a monotonic clock (CLOCK_MONOTONIC, performance.now, time.monotonic)"] c["Step 3: Need an order consistent with causality?"] wall["NTP-synced wall clock, treat as approximate (logs, dashboards)"] cc["Step 4: Must you DETECT concurrent writes?"] vc["Vector clock / dotted version vector or a CRDT"] h["Step 5: Want timestamps close to real time?"] lam["Lamport clock + node id tie-break"] r["Step 6: Need real-time order across all clients (external consistency)?"] hlc["Hybrid logical clock (CockroachDB, MongoDB, YugabyteDB)"] tt["Bounded-uncertainty clock + commit wait (Spanner TrueTime) or a single timestamp oracle"] q -->|continues| d d -->|yes| mono d -->|no| c c -->|no, approximate is fine| wall c -->|yes| cc cc -->|yes| vc cc -->|no, a total order is enough| h h -->|no| lam h -->|yes| r r -->|no| hlc r -->|yes| tt
Why "just use timestamps" fails (runnable)
A user writes v1 through replica A, reads it back, and 50 ms later writes v2 through replica B, whose clock is 120 ms slow. With last-writer-wins on wall clocks, the newer write loses. A Lamport-style counter carried through the client fixes the causal case, and a vector clock is the only one of the three that can also tell a causal overwrite apart from a truly concurrent one.
# Why "just use timestamps" loses data: last-writer-wins (LWW) on skewed wall clocks,
# compared with a logical clock and a vector clock on the same two writes.
# All millisecond values are EXAMPLE values chosen to make the failure visible.
SKEW_B_MS = -120 # replica B's wall clock runs 120 ms behind true time (example)
def wall(node: str, true_ms: int) -> int:
"""What each replica's wall clock *reads* at a given true time."""
return true_ms + (SKEW_B_MS if node == "B" else 0)
# The real-world sequence: a user writes v1 via replica A, reads it back,
# then 50 ms later writes v2 via replica B. v2 is causally AFTER v1.
writes = [
{"value": "v1", "node": "A", "true_ms": 1_000},
{"value": "v2", "node": "B", "true_ms": 1_050},
]
# 1) Wall-clock LWW: each write is stamped with the local wall clock, highest wins.
for w in writes:
w["wall"] = wall(w["node"], w["true_ms"])
lww = max(writes, key=lambda w: w["wall"])
print("1) wall-clock LWW")
for w in writes:
print(f" {w['value']} via {w['node']}: true={w['true_ms']} stamped={w['wall']}")
print(f" winner={lww['value']} -> the NEWER write v2 is silently discarded\n")
# 2) Lamport-style LWW: the client carries the max timestamp it has seen (from reading v1),
# and the writer stamps max(local_counter, seen) + 1. Causal order is now preserved.
seen_by_client = 0
lamport = {"A": 0, "B": 0}
stamps = {}
for w in writes:
n = w["node"]
lamport[n] = max(lamport[n], seen_by_client) + 1
stamps[w["value"]] = (lamport[n], n) # (counter, node id) tie-break gives a total order
seen_by_client = lamport[n] # client read its own write back
print("2) Lamport-stamped LWW:", stamps, "-> winner", max(stamps, key=lambda v: stamps[v]))
# 3) Vector clocks: tell 'v2 happened after v1' apart from 'v2 was concurrent with v1'.
def descends(a: dict, b: dict) -> bool:
return all(a.get(k, 0) >= v for k, v in b.items())
v1 = {"A": 1}
v2_after_read = {"A": 1, "B": 1} # client passed v1's context along with its write
v2_blind = {"B": 1} # client never saw v1
print("3) vector clocks")
print(" v2 (with context) descends v1:", descends(v2_after_read, v1), "-> safe overwrite")
print(" v2 (blind) vs v1 concurrent:", not descends(v2_blind, v1) and not descends(v1, v2_blind),
"-> keep both as siblings, let the app merge")Output:
1) wall-clock LWW
v1 via A: true=1000 stamped=1000
v2 via B: true=1050 stamped=930
winner=v1 -> the NEWER write v2 is silently discarded
2) Lamport-stamped LWW: {'v1': (1, 'A'), 'v2': (2, 'B')} -> winner v2
3) vector clocks
v2 (with context) descends v1: True -> safe overwrite
v2 (blind) vs v1 concurrent: True -> keep both as siblings, let the app mergeChoosing the mechanism (runnable)
// A tiny decision helper: which time/ordering mechanism fits a requirement?
// The mapping mirrors the comparison table on the hub page; it is a teaching aid, not a library.
type Need = {
name: string;
humanReadableTime: boolean; // do people or SLAs read this timestamp?
causalOrder: boolean; // must "B saw A" imply ts(A) < ts(B)?
detectConcurrency: boolean; // must we notice two writes that did not see each other?
externalConsistency: boolean; // must real-time order across the whole system match commit order?
measureDuration: boolean; // timeouts, latency, lease countdowns
};
function choose(n: Need): string {
if (n.measureDuration) return "monotonic clock (never wall clock) for elapsed time";
if (n.externalConsistency) return "TrueTime-style bounded clock + commit wait (or a single timestamp oracle)";
if (n.detectConcurrency) return "vector clock / dotted version vector (or a CRDT)";
if (n.causalOrder && n.humanReadableTime) return "hybrid logical clock (HLC)";
if (n.causalOrder) return "Lamport clock with node-id tie-break";
return "NTP-synced wall clock, treated as approximate";
}
const needs: Need[] = [
{ name: "request timeout / lease countdown", humanReadableTime: false, causalOrder: false, detectConcurrency: false, externalConsistency: false, measureDuration: true },
{ name: "log line timestamps for humans", humanReadableTime: true, causalOrder: false, detectConcurrency: false, externalConsistency: false, measureDuration: false },
{ name: "replicated log / total order", humanReadableTime: false, causalOrder: true, detectConcurrency: false, externalConsistency: false, measureDuration: false },
{ name: "MVCC versions in a geo-distributed DB", humanReadableTime: true, causalOrder: true, detectConcurrency: false, externalConsistency: false, measureDuration: false },
{ name: "multi-writer shopping cart", humanReadableTime: false, causalOrder: true, detectConcurrency: true, externalConsistency: false, measureDuration: false },
{ name: "global bank ledger, strict serializable", humanReadableTime: true, causalOrder: true, detectConcurrency: false, externalConsistency: true, measureDuration: false },
];
for (const n of needs) {
console.log(`${n.name.padEnd(42)} -> ${choose(n)}`);
}Output:
request timeout / lease countdown -> monotonic clock (never wall clock) for elapsed time
log line timestamps for humans -> NTP-synced wall clock, treated as approximate
replicated log / total order -> Lamport clock with node-id tie-break
MVCC versions in a geo-distributed DB -> hybrid logical clock (HLC)
multi-writer shopping cart -> vector clock / dotted version vector (or a CRDT)
global bank ledger, strict serializable -> TrueTime-style bounded clock + commit wait (or a single timestamp oracle)Expectedrequest timeout / lease countdown -> monotonic clock (never wall clock) for elapsed time log line timestamps for humans -> NTP-synced wall clock, treated as approximate replicated log / total order -> Lamport clock with node-id tie-break MVCC versions in a geo-distributed DB -> hybrid logical clock (HLC) multi-writer shopping cart -> vector clock / dotted version vector (or a CRDT) global bank ledger, strict serializable -> TrueTime-style bounded clock + commit wait (or a single timestamp oracle)
Press Run. Snippets must be self-contained — no network, files, or native modules.
Comparison: what each mechanism buys and costs
| Mechanism | Ordering guarantee | Detects concurrency? | Size per timestamp | Needs synced clocks? | Real systems |
|---|---|---|---|---|---|
| Wall clock (NTP) | None reliable across nodes | No | 8 bytes | Yes, and still lies | Logs, Cassandra LWW cells |
| Lamport clock | If a -> b then L(a) < L(b) | No | 8 bytes | No | Total order broadcast, Lamport mutex |
| Vector clock / version vector | a -> b iff V(a) < V(b) | Yes | O(nodes or actors) | No | Dynamo, Riak, Voldemort |
| Hybrid logical clock | Causal like Lamport, close to wall time | No | ~12 bytes (time + counter) | Loosely (bounded skew) | CockroachDB, MongoDB cluster time, YugabyteDB |
| TrueTime + commit wait | External consistency | No (prevents the need) | Interval [earliest, latest] | Yes, with GPS and atomic references | Google Spanner |
| Timestamp oracle (central) | Total, real-time order | No | 8 bytes | No, but one hot service | Google Percolator, TiDB PD |
What happens if you choose otherwise
- Wall-clock LWW where causality matters: lost updates you will never see in logs, because the "losing" write looked older. Kleppmann's DDIA chapter 8 walks through exactly this failure.
- Lamport clocks where you needed conflict detection: you get a deterministic winner, but two users who never saw each other's edits still lose one edit.
- Vector clocks with one entry per client: correct, but the vector grows with every user and has to be pruned, and pruning can resurrect false conflicts.
- TrueTime-style commit wait with large uncertainty: every write pays roughly twice the clock uncertainty in latency. With NTP-grade uncertainty (tens of milliseconds) that is too expensive, which is why CockroachDB chose HLC plus read restarts instead.
- Central timestamp oracle at global scale: simple and correct, but every transaction pays a round trip to one place, and that service becomes a scaling and availability bottleneck.
Real-world anchors
- Cloudflare, 1 January 2017: the leap second made a wall-clock duration negative inside their Go DNS service (RRDNS); the negative value reached
rand.Int63n, which panics, and some DNS resolutions failed. Go 1.9 later added monotonic readings totime.Timeso subtracting twotime.Now()values is safe. - Leap second of 30 June 2012: a Linux kernel timer bug triggered CPU spikes and outages at several large sites (Reddit among them).
- Google Spanner (OSDI 2012): TrueTime exposes clock uncertainty explicitly and commit-waits it out to get external consistency.
- CockroachDB: hybrid logical clocks plus a configured maximum clock offset (default 500 ms); reads restart when they hit a value inside their uncertainty window, and nodes shut themselves down if their clock drifts too far from the cluster.
- Amazon Dynamo (SOSP 2007): vector clocks with sibling versions and client-side reconciliation (the shopping cart).
- Twitter Snowflake (2010) and RFC 9562 (2024, UUIDv7): time-ordered IDs that inherit every clock problem above.
The cluster map
- Physical clocks: NTP/PTP, drift and skew, wall vs monotonic, leap seconds and smearing, why timestamps lie.
- Lamport clocks: happens-before, the clock condition, total order with tie-breaks, and the concurrency blind spot.
- Vector clocks and version vectors: detecting concurrency, siblings, dotted version vectors, pruning, LWW vs CRDTs.
- HLC and TrueTime: commit wait, uncertainty intervals, external consistency vs serializability, read restarts.
- Distributed ID generation: Snowflake, UUIDv4 vs UUIDv7, ULID, KSUID, sequences, hi-lo and ticket servers.
Interview Q&A
Why can't you order events in a distributed system by wall-clock timestamp?
Answer
Clocks on different machines disagree (skew), run at different rates (drift), and can jump backwards when NTP steps them or around leap seconds. Two events a few milliseconds apart on different machines can easily be stamped in the wrong order, so any decision based on that order (LWW, "latest" reads, lease expiry) can be wrong.
What does a Lamport clock guarantee, and what does it not?
Answer
If a happens-before b, then L(a) < L(b). The converse does not hold: L(a) < L(b) can mean a caused b or that they were concurrent. Lamport clocks give a causality-respecting total order, not concurrency detection.
When do you need vector clocks instead?
Answer
When you must notice that two writes were concurrent (neither saw the other), so you can keep both and merge, instead of silently picking one. Dynamo-style multi-writer stores are the classic case.
How does Spanner get external consistency, and why doesn't everyone copy it?
Answer
TrueTime returns an interval guaranteed to contain true time; Spanner picks a commit timestamp at the top of the interval and waits until that timestamp is definitely in the past before acknowledging. It needs tight uncertainty (GPS and atomic clocks in each datacenter) to keep that wait to a few milliseconds.
Your service uses `Date.now()` to check whether a lease expired. What's wrong?
Answer
Wall-clock time can jump. Measure elapsed time with a monotonic clock, keep a safety margin for clock rate differences between holder and grantor, and use fencing tokens so a stale holder cannot do damage even if it believes the lease is still valid.
What is external consistency?
Answer
If T1 commits before T2 starts in real time, T1's timestamp is less than T2's. That is Spanner's guarantee, also called strict serializability. A Lamport total order does not promise it.
Why is a monotonic clock useless across machines?
Answer
It counts time since an arbitrary point, often boot, and only moves forward on that machine. Another host cannot compare it, and it is meaningless after reboot.
What broke in Cloudflare DNS on 1 January 2017?
Answer
The leap second made a wall-clock duration negative inside their Go DNS service. The negative value reached rand.Int63n, which panics. Go 1.9 later put a monotonic reading inside time.Time so subtracting two time.Now values stays safe.
Check yourself
Take one write path you operate. Say whether it needs a duration, an approximate stamp, a causal total order, concurrency detection, or external consistency. Name the mechanism you would refuse, and why.
Elsewhere in the library
These pages stay as they are. This lesson only points at them: Raft Consensus — Leader Election, Log Replication & Safety, CRDTs — Conflict-Free Types, Convergence & When Consensus Wins, Database Replication — Leader-Follower, Multi-Leader & Leaderless Quorums, Distributed Locks — Correctness, Leases & Fencing Tokens, MVCC, Snapshot Isolation & Write Skew, Stream Processing — Event Time, Windows, State & Exactly-Once.
Go Deeper
- Leslie Lamport, Time, Clocks, and the Ordering of Events in a Distributed System (1978)
- Spanner: Google's Globally-Distributed Database (OSDI 2012)
- Justin Sheehy, There Is No Now (ACM Queue)
- Martin Kleppmann, Distributed Systems lecture notes (Cambridge)
- Martin Kleppmann's Distributed Systems lecture series (YouTube)