Request Collapsing / Coalescing

Overview

Caching stores a completed response. Collapsing deduplicates in-flight work before a response exists. When a hot key expires, hundreds of clients miss together. Without collapsing, each miss is an origin (or DB, or payment API) call — a thundering herd. With collapsing, waiters park on one leader.

It is not a cache: after the leader finishes, later requests HIT the store. It is not a shield: 200 POPs with perfect local collapsing still send 200 origin fetches unless a shared tier sits in front.

Why single-flight wins

1. 1

   TTL expiry → N origin fetches on one node

   No collapsing

   Fifty parallel misses on /hot become fifty origin GETs. CPU, DB, and third-party QPS spike together. Serverless cold starts multiply.

2. 2

   Winner locally: in-flight map keyed like the cache

   First waiter is the leader. Everyone else awaits the same promise. Success or error broadcasts. nginx proxy_cache_lock and Go singleflight are this idea.

3. ?

   Winner globally: collapse at edge, mid-tier, and shield

   Local single-flight times POP count is still a herd. An origin shield (Cloud CDN, CloudFront, Fastly) is the topology choke point. Skip collapsing on uncacheable private traffic or you serialize users behind one slow POST.

Core mechanics

1. Key derivation — same identity as the cache key: canonical URL plus allow-listed Vary. Collapse too broadly (ignore Authorization) and you merge distinct users. Collapse too narrowly (Vary: Cookie) and nobody shares a flight. See cache keys.
2. Dedup window — usually "while in flight," not a timed bucket. A timed window that is too long inflates waiter latency; too short and you miss the herd.
3. In-flight registry — concurrent map of key → promise/future. Go's singleflight is the canonical library.
4. Wait queue — other callers await that promise. Bound the queue or memory explodes on a stuck origin.
5. Fan-out — complete once; every waiter gets the same bytes or the same error.
6. Cleanup — delete the registry entry in finally so the next miss can lead again.

nginx proxy_cache_lock holds a lock per cache key so only one request updates the cache; others wait. Redis EVAL is how app-tier caches implement compare-and-set single-flight (set inflight:key atomically, losers wait or retry). Same pattern, different layer.

flowchart TB
  a["Client A"]
  b["Client B"]
  s["Collapser"]
  o["Origin"]
  a -->|GET key| s
  s -->|Step2 leader| o
  b -->|GET same key| s
  o -->|response| s
  s -->|body| a
  s -->|body| b

Collapsing vs shielding vs SWR

SWR without a leader flag still stampedes: every stale request might start a refresh. Soft purge plus SWR plus collapsing is the humane publish path. Hard purge-all plus no collapsing is the incident.

TLS HTTP/2 multiplexing reuses connections; it does not merge identical GETs. Do not say "we enabled HTTP/2 so we do not need collapsing."

Naming collision: connection coalescing. Browsers also "coalesce" HTTP/2 connections when certificates and origins allow (one socket to a CDN VIP serving many hostnames). That is not request collapsing. Connection coalescing saves handshakes. Request collapsing saves upstream GETs for one cache key. Say which one you mean.

Worked counts

Assume one hot HTML object expires everywhere at once. No SWR (worst case). Each POP has many clients in the same millisecond.

Those five numbers are the interview. If your collapse key includes a session cookie, you fall from the "1" rows back to the "50" row even on one node.

Cold start / serverless. The leader pays the cold-start; waiters reuse the warm result. That is a latency win only if you bound how long waiters will sit. A 30s origin timeout × 10k waiters is a memory incident.

Rate-limit protection. Third-party APIs (maps, payments, auth) often cap QPS. Collapsing at the edge of your service is how 10k page views do not become 10k vendor calls for the same public config payload. Cache the result too; collapsing alone only helps the simultaneous window.

Idempotency. Collapse GET and HEAD. Do not collapse POST, PUT, PATCH, or DELETE unless you have a dedicated idempotency-key design — that is a different lesson. A coalesced POST would apply a mutation once for N callers who each expected their own side effect. The wait-queue is for reads.

Failure modes

Error fan-out. The leader's 503 is everyone's 503. Decide: propagate, retry once (only if idempotent GET), or fall back to SIE stale. A retry storm from all waiters undoes collapsing — one retry belongs to the leader.

Uncacheable leader. If the leader's response is Cache-Control: private, no-store, do not store it, and think twice before collapsing it. A slow personalized request should not block fifty other users who happened to share a too-fat key.

Stuck origin. Waiters pile up until timeout. Set a lock TTL (proxy_cache_lock_timeout in nginx). Shed excess waiters with 503 rather than unbounded memory.

Stampede after success. The herd is collapsed during the miss. After fill, traffic is HITs. The next expiry needs jittered TTLs or SWR so the herd does not reform on a timer.

Global lock. One mutex around the whole registry serializes unrelated keys. Stripe by key (Go singleflight does).

Playground: 50 waiters, then 200 POPs

In-memory maps. No fetch. First loop is one node. Second loop is many nodes without a shield, then with a shield that has its own single-flight.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Teaching numbers: 50 waiters → 1 origin on one node; 3 POPs × 1 leader → 3 origin without a shield; with a shield → 1 origin. Error path stores nothing, so the next wave can lead again.

Interview Q&A

Pitfalls

Go Deeper

• Go — singleflight
• ACM Queue — The thundering herd
• Redis — EVAL
• Google Cloud CDN — Origin shield
• nginx — proxy_cache_lock

Sibling pages: CDN hierarchy · Cache keys · SWR / stale-if-error · Purge and generation tokens · TLS / Anycast