Single-Flight Locking for Cache Fills

Overview

Jitter stops many keys from expiring together. It does not split one key. This lesson is the mutex that makes "200 concurrent misses" become one DB hit.

By the end of this lesson you should be able to:

• Acquire with SET key token NX PX ms in one command
• Size lock TTL from loader p99, not from a guessed 1 second
• Release only if you still own the lock (Lua GET + DEL)
• Make waiters poll the cache, with a timeout policy that is not "everyone loads"
• Combine Redis with in-process singleflight, and leave Redlock on the shelf unless you have a real multi-master reason

Why coalescing exists

When a popular key expires under concurrency:

• N request handlers all see miss
• All N hit Postgres/MySQL for the same row
• DB CPU and the connection pool saturate → latency cascade → retries → worse

That is a cache stampede (thundering herd on a miss). Single-flight / request coalescing means: only the first request performs the expensive load; the rest wait for the result. Single-flight caps DB load at one loader per key, but at TTL expiry every other request still waits or polls until the fill lands, so latency spikes; a soft TTL or early recompute refreshes before expiry so readers keep getting the old value.

It does not prevent a herd when many distinct keys expire together — that is jitter. It does not prevent a herd when the lock TTL is shorter than the load — two winners. It does not make Redis the source of truth.

flowchart TB
  A["Pod A Ready"]
  B["Pod B Ready"]
  R["Redis Ready"]
  DB["Primary DB Ready"]
  fail["Failure path - load outlives PX 1500, lock expires, a second loader starts Ready"]
  A -->|Step 1 GET data:42 - miss| R
  A -->|Step 2 SET lock:42 tokenA NX PX 1500| R
  R -->|OK - A is the loader| A
  B -->|Step 3 SET lock:42 tokenB NX PX 1500| R
  R -->|nil - lock busy, B becomes a waiter| B
  A -->|Step 4 Load row 42| DB
  B -->|Step 5 GET data:42| R
  A -->|Step 6 SET data:42 value EX 30| R
  A -->|Step 7 Lua - DEL lock:42 only if it still holds tokenA| R
  B -->|Step 8 GET data:42 - hit, return| R
  A -->|load outlives PX 1500| fail

The Redis mutex

Acquire atomically:

SET lock:{key} {random_token} NX PX {lock_ttl_ms}

• NX — only if the lock key does not exist
• PX — millisecond TTL so a dead loader cannot hold it forever
• token — a random UUID per acquire, not a constant "1"

Rules interviewers love:

1. One command. SETNX then EXPIRE is a bug: a crash between them leaves an immortal lock. Always SET key val NX PX ms.
2. Lock TTL ≥ p99 loader latency, with a safety factor (1.5–2×), and a floor. Too short: the lock expires mid-load, a second loader starts, you paid for a stampede with extra code. Too long: a crashed loader blocks fills until TTL. Dynamic max(p99_recent, min_ttl) adapts when the DB slows down.
3. Re-check the data key after acquire. The previous winner may have SET and unlocked between your miss and your NX.
4. Waiters poll the data key, not the DB, not the lock key as a substitute for the value. Interval 10–50 ms with backoff; cap the wait. Optional PUBLISH on fill for extremely hot keys — extra moving parts.
5. Release only if you still own it. Release the lock with a Lua script that deletes lock_key only if it still holds your token (a plain DEL can remove another worker's lock).

-- release: delete only if the token still matches
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

Bare DEL on unlock can delete another worker's lock after your TTL expired. That is how a "safe" mutex turns back into a stampede. The playground shows it.

After wait timeout, one degraded load is still better than all N loading immediately. If a soft TTL exists, prefer serving stale over a 503 or a herd.

In-process singleflight + Redis

A Redis lock is across pods. Inside one process, 80 concurrent requests for user:1 would still issue 80 SET NX calls, and 79 waiters would still poll Redis.

Language-level coalescing (Go singleflight.Group, similar patterns in other runtimes) makes one in-process loader. Combine them:

1. In-process group keyed by the cache key
2. That one goroutine / task talks to Redis (GET, then NX lock if needed)
3. Redis lock coalesces across instances

In-process alone is not enough at 20 replicas. Redis alone works but wastes chatter. Both is the production shape.

Per-key locks (lock:<key>) maximize concurrency. Bucketed locks (lock:<hash-bucket>) shrink keyspace and create false contention. Start per-key.

Redlock, briefly

Redlock is an algorithm that takes a lock on a majority of independent Redis masters, with clock-drift assumptions. It is aimed at distributed locking where correctness of the lock is the product.

For cache fills, use single-node SET NX PX with a token; skip Redlock. A duplicate fill only costs one extra DB read, so a best-effort lock is enough; if correctness depends on the lock, use fencing tokens or a consensus store, not Redlock. You are optimizing load, not linearizing a bank transfer.

For cache fills a single-node SET NX PX is enough even across regions (a rare double load is harmless); Redlock's timing assumptions are disputed, so use fencing tokens or a consensus store when the lock guards correctness. Do not oversell Redlock in this interview. Mention it only to say: single-node SET NX PX with a token is the cache mutex; Redlock is a different tool.

Failure handling

Observability is part of the design: acquire rate, acquire failures, wait timeouts, duplicate-load count (should be ~0), rebuild delta, DB QPS on that key.

Working sketches

The TypeScript client below is a complete getOrLoad: SET NX PX with a token, a cache re-check after the lock, token-checked Lua release, and bounded backoff. It needs ioredis. The Python sketch shows the same re-check, and _publish_result stores the ttl argument. The sandbox runs that protocol on a dict.

# Sketch — redis-py shape. Not runnable here (needs a server).
import uuid

CACHE_TTL_SEC = 30
LOCK_TTL_MS = 1500

RELEASE = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end
"""

def _publish_result(r, key, value, ttl):
    # Honor ttl. Hard-coding CACHE_TTL_SEC here dropped the caller's TTL.
    r.set(key, value, ex=ttl)

def get_or_load(r, key, loader, ttl=CACHE_TTL_SEC):
    cached = r.get(key)
    if cached is not None:
        return cached
    token = str(uuid.uuid4())
    lock_key = "lock:" + key
    if r.set(lock_key, token, nx=True, px=LOCK_TTL_MS):
        try:
            again = r.get(key)  # re-check the cache after acquiring the lock
            if again is not None:
                return again
            fresh = loader()
            _publish_result(r, key, fresh, ttl)
            return fresh
        finally:
            r.eval(RELEASE, 1, lock_key, token)
    return None  # waiter: poll GET with backoff until timeout

/*
 * singleFlightRedis.ts
 * Redis-backed single-flight cache fill in Node.js.
 * Public API: getOrLoad(key, loader, ttlSec?) -> Promise<T>
 * Dependencies: npm i ioredis
 */
import Redis from "ioredis";
import { randomUUID } from "node:crypto";

const redis = new Redis({ host: "127.0.0.1", port: 6379 });

const LOCK_TTL_MS = 1500; // must exceed p99 load latency
const POLL_BASE_MS = 20;
const MAX_RETRIES = 25;
const CACHE_TTL_SEC = 30;

const RELEASE_LUA = `
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end`;

/** Try to become the loader; returns the token on success, null if the lock is busy. */
async function acquireLock(lockKey: string): Promise<string | null> {
  const token = randomUUID();
  const ok = await redis.set(lockKey, token, "PX", LOCK_TTL_MS, "NX");
  return ok === "OK" ? token : null;
}

/** Release only if we still own the lock (token check + DEL in one Lua call). */
async function releaseLock(lockKey: string, token: string): Promise<void> {
  await redis.eval(RELEASE_LUA, 1, lockKey, token);
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

export async function getOrLoad<T>(key: string, loader: () => Promise<T>, ttlSec = CACHE_TTL_SEC): Promise<T> {
  const lockKey = `lock:${key}`;
  const dataKey = `data:${key}`;
  const cached = await redis.get(dataKey);
  if (cached !== null) return JSON.parse(cached) as T;

  const token = await acquireLock(lockKey);
  if (token) {
    try {
      const again = await redis.get(dataKey); // another pod may have filled it
      if (again !== null) return JSON.parse(again) as T;
      const fresh = await loader();
      await redis.set(dataKey, JSON.stringify(fresh), "EX", ttlSec);
      return fresh;
    } finally {
      await releaseLock(lockKey, token);
    }
  }

  let backoff = POLL_BASE_MS;
  for (let i = 0; i < MAX_RETRIES; i++) {
    await sleep(backoff);
    const polled = await redis.get(dataKey);
    if (polled !== null) return JSON.parse(polled) as T;
    backoff = Math.min(backoff * 1.5, 500);
  }
  throw new Error(`cache fill timeout for ${key}`); // or serve stale / 503
}

In-memory Redis (run this)

Dict-backed GET / SET / SET NX PX / Lua-style token unlock. Eight concurrent misses. Bare DEL vs token unlock.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Read the last four lines: bare DEL stole lock; owner-3 NX True vs lua unlock expired owner 0 and third NX False. That is the whole unlock lesson.

Per-process single-flight (run this)

Combine this map with Redis SET NX in multi-instance deploys. Alone, it only coalesces inside one process. 200 pods with only this map still issue 200 origin loads.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Interview Q&A

Pitfalls

Go Deeper

Official docs and the paper

• Redis SET (NX, PX)
• Redis EVAL
• Go singleflight
• Redis — Distributed locks
• Vattani et al., VLDB 2015

In this cluster

• Cache TTL design and jitter
• Next: Negative caching
• Hub: Redis cache-aside, stampede
• HTTP cousins: request collapsing · SWR