Redis Cache-Aside, Invalidation & Stampede Prevention

Overview

Most read-heavy services (product catalogs, user profiles, feed cards, config) die the same way under load: the primary DB becomes the bottleneck, P95 climbs, connection pools exhaust, and a brief traffic spike turns into a cascade.

Redis cache-aside is the default fix — but naive cache-aside fails at the exact moment you need it most: when a hot key expires and hundreds of pods all miss at once (cache stampede / thundering herd).

By the end of this lesson you should be able to:

• Draw the cache-aside read path and the delete-on-write write path
• Pick cache-aside vs write-through vs write-behind with a reason
• Layer stampede controls: jitter, single-flight, XFetch / stale-while-revalidate
• Cache "not found" safely (negative caching)
• State an explicit Redis-down policy: fail-open vs fail-closed

Cache-aside (lazy loading)

The app owns the policy. Redis is a shared L2, not the source of truth.

Read path

1. GET the Redis key
2. On hit → return
3. On miss → load from the primary → SET with TTL → return

Write path (recommended)

1. Write the primary (source of truth)
2. DEL the cache key (invalidate)
3. The next read repopulates

Do not try to keep cache and DB perfectly in sync by writing both on every update unless you have a strong reason (that is write-through). Invalidation is simpler and safer under partial failures.

flowchart TB
  R["Step 1 GET cache:user:42 Ready"]
  H["Step 2 Hit? Ready"]
  A["Step 3a Return the cached value Ready"]
  L["Step 3b SET lock NX PX - one loader wins Ready"]
  C["Step 4a Re-check cache, then load the primary DB Ready"]
  S["Step 5 SET value EX base TTL plus jitter Ready"]
  U["Step 6 Unlock with token-checked Lua Ready"]
  P["Step 4b Poll the cache with backoff Ready"]
  D["Failure path - serve stale or degrade, never send all N to the DB Ready"]
  W["Step 7 Write path - commit DB, then DEL the key Ready"]
  T["Failure path - stale until TTL retry DEL Ready"]
  R -->|continues| H
  H -->|yes| A
  H -->|no| L
  L -->|won| C
  C -->|continues| S
  S -->|continues| U
  L -->|lost| P
  P -->|timeout| D
  W -->|DEL fails| T

Write-through vs write-behind

Interview default for user-facing reads: cache-aside + invalidate-on-write.

Read-through is cache-aside with miss loading moved into a library or proxy. Same races. Depth (who fills, ack timing, crash-before-flush): write-through vs write-behind.

Delete-on-write, not update-the-cache

Updating the cache on write races with concurrent writers and can resurrect an older value: write A → write B → set cache A. Deleting forces the next read to rebuild from the committed primary.

If you must set-on-write, use versioning / compare-and-set. Delete-on-write still has one race: a reader that missed loads the old row, the writer commits and deletes, then the reader SETs the old value. TTL bounds it; a versioned SET or a delayed second DEL closes it. Prefer delete-after-commit so readers never see an uncommitted or rolled-back row. If the DEL is lost, retry it (outbox / queue); TTL is the backstop, not the happy path.

Forget invalidate on all write paths — admin tools, batch jobs, and migrations silently leave stale data.

TTL as a staleness bound

TTL is not just memory management. It is your maximum staleness window if invalidation fails.

Typical production practice:

• Short TTL for volatile data (seconds to minutes)
• Longer TTL for slowly changing data
• TTL jitter: base_ttl + random(0, jitter) so a slab of batch-warmed keys does not expire in lockstep

Identical TTLs on a warm create synchronized expiry cliffs. Jitter spreads them. It is not enough alone for an ultra-hot key.

Cache stampede

When a popular key expires under concurrency:

• N request handlers all see miss
• All N hit Postgres/MySQL for the same row
• DB CPU / pool saturates → latency cascade

Layer the fixes — they are not rivals:

What fails if you choose wrong

• Only process-local single-flight across 200 pods → still 200 DB queries. Sell Go singleflight as a pod tool, not a cluster solution.
• Lock without TTL / fencing → deadlock after filler crash; permanent miss storm.
• Serve-stale forever without refresh success metrics → silent data rot.
• No jitter on identical TTLs → synchronized expiry of many keys. Related: TTL design & jitter. Mutex depth: single-flight locking. HTTP cousins: request collapsing and SWR.

Store {value, soft_exp, hard_exp} when you want serve-stale. After soft_exp, still return the blob but trigger async refresh (or only the lock winner refreshes). After hard_exp, treat as miss.

Single-flight mutex (Redis official pattern)

Acquire the lock atomically:

SET lock:{key} {random_token} NX PX {lock_ttl_ms}

Rules interviewers love:

• Lock TTL must be greater than p99 loader latency (else the lock expires mid-load and a second loader starts)
• Release only if you still own the lock (token-checked Lua GET + DEL) — never bare DEL
• Waiters poll the cache, not immediately fall through to the DB
• After acquiring the lock, re-check the cache (another worker may have filled it)
• SETNX then EXPIRE as two commands is a bug: a crash between them leaves an immortal lock. Always SET key val NX PX ms in one call.

-- acquire (conceptually SET NX PX)
-- return redis.call('SET', KEYS[1], ARGV[1], 'NX', 'PX', ARGV[2])

-- release: delete only if the token still matches
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

Bare DEL on unlock can delete another worker's lock after your TTL expired. That is how a "safe" mutex turns back into a stampede.

After wait timeout, loading once is still better than all N loading immediately. If a soft TTL exists, prefer serving stale over a 503 or a herd.

XFetch (VLDB 2015)

Vattani, Grosvenor, Rodriguez — Optimal Probabilistic Cache Stampede Prevention.

Store (value, delta, logical_expiry) with a physical Redis TTL slightly longer than logical expiry. On each hit, refresh early if:

now - delta * beta * ln(random()) >= expiry

• delta = measured recompute cost
• beta ≈ 1.0 (higher = earlier refresh)
• No locks; occasional redundant refreshes are OK
• Combine with the mutex for hard misses (value already gone)

Negative caching

Cache "not found" briefly to stop hammering the DB for missing IDs (bot traffic, deleted entities, scanners). Use a short TTL and a distinct sentinel / typed miss marker — never confuse a miss marker with a real empty object.

Without negative caching, miss storms become a DB DoS.

Key design

cache:{entity}:{id}          # e.g. cache:user:42
cache:{entity}:{id}:v{n}     # versioned keys for schema migrations

Prefer hashes (HSET / HGET) or RedisJSON for field-level updates without rewriting the whole blob.

Do not cache huge blobs or unbounded lists — memory eviction (LRU/LFU) plus latency. Paginate, or cache IDs and hydrate.

Architecture and operations

Client → API pods → Redis (L2 shared cache)
                 ↘ PostgreSQL (source of truth)

On write: API → Postgres COMMIT → DEL the per-user cache key (optional pub/sub or Redis Streams to other regions).

On miss under load: API → SET NX lock → one pod loads Postgres → SET cache → release lock. Other pods poll GET until warm (or timeout → degrade).

Scaling

• Shard Redis / use Cluster for memory + throughput
• Hot keys: local L1 (Caffeine) + Redis L2, or key sharding (user:42:0..N) for extreme hotspots
• Multi-region: invalidate locally; accept cross-region staleness or replicate invalidation events. Do not assume instant global invalidate.

Metrics that catch expiry cliffs: hit ratio, miss latency, lock acquire rate, lock wait timeouts, rebuild duration (delta), stale-serve count, DB QPS correlated with TTL boundaries.

Fail-open vs fail-closed

When Redis is unavailable you need an explicit policy:

Often the production mix is: short circuit breaker + serve last-known L1 stale + shed load. Never silently amplify DB QPS without limits.

Other failure modes:

• Lock timeout: waiters may stampede or return 503 — prefer degrade with stale if a soft TTL exists
• Invalidate lost: stale until TTL — TTL bounds damage
• Partial write success: DB updated, cache not deleted — delete-after-commit and retry delete

Working examples (real Redis)

These sketches need a real client. The sandbox below simulates the same protocol with a dict.

# Sketch — redis-py. Not runnable here (needs Redis).
import json, random, time, uuid

ACQUIRE_LOCK = "return redis.call('SET', KEYS[1], ARGV[1], 'NX', 'PX', ARGV[2])"
RELEASE_LOCK = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end
"""

def ttl_with_jitter(base_seconds: int, jitter: int = 30) -> int:
    return base_seconds + random.randint(0, jitter)

def cache_get_or_load(r, key, loader, *, ttl_seconds=300, lock_ttl_ms=2000, max_wait_ms=2000):
    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)
    lock_key = f"lock:{key}"
    token = str(uuid.uuid4())
    acquired = r.eval(ACQUIRE_LOCK, 1, lock_key, token, lock_ttl_ms)
    if acquired:
        try:
            cached = r.get(key)  # double-check after NX
            if cached is not None:
                return json.loads(cached)
            value = loader()
            r.set(key, json.dumps(value), ex=ttl_with_jitter(ttl_seconds))
            return value
        finally:
            r.eval(RELEASE_LOCK, 1, lock_key, token)
    deadline = time.monotonic() + max_wait_ms / 1000.0
    while time.monotonic() < deadline:
        time.sleep(0.05)
        cached = r.get(key)
        if cached is not None:
            return json.loads(cached)
    return loader()  # degraded: one load after wait, not N immediate loads

def update_user(user_id, patch):
    user = db_update_user(user_id, patch)  # commit source of truth first
    r.delete(f"cache:user:{user_id}")     # invalidate after commit
    return user

// Sketch — node-redis. Same mutex: SET NX PX + Lua token unlock.
function ttlWithJitter(baseSeconds: number, jitter = 30): number {
  return baseSeconds + Math.floor(Math.random() * (jitter + 1));
}

// acquire: SET lockKey token NX PX lockTtlMs
// release: GET lockKey == token then DEL else 0
// waiters: poll GET, do not stampede the DB

# XFetch: True => refresh now even though the value is still present
import math, random, time

def should_xfetch(expiry: float, delta: float, beta: float = 1.0) -> bool:
    return time.time() - delta * beta * math.log(random.random()) >= expiry

In-memory Redis (run this)

Dict-backed GET / SET / DEL, SET NX PX, and Lua-style token unlock. No network.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Interview Q&A

Pitfalls

Go Deeper

Official docs

• Redis cache-aside overview
• Cache-aside with Python (redis-py)
• Cache-aside with Node.js
• SET (NX PX)
• Redis — Distributed locks
• Go singleflight

Papers and talks

• Vattani et al., Optimal Probabilistic Cache Stampede Prevention (VLDB 2015)
• Preventing cache stampede with Redis and XFetch — Jim Nelson

In this cluster

• Next: Write-through vs write-behind
• TTL jitter · Single-flight locking
• HTTP cousins: request collapsing · SWR