Prefix & Prompt Caching Across Requests - Cache Keys, Routing, TTL & Hit-Rate Economics
KV block hashing (vLLM APC) vs radix tree (SGLang); what goes into cache keys (adapter, tokenizer, images, salts); prefix-aware routing across replicas; provider prompt caching write premiums, read discounts, TTL; runnable fleet hit-rate sim + economics.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Question ladder
L1
What does prefix caching store?
Answer
The per-layer keys and values for prefix tokens.
L2
Is it lossless?
Answer
Yes: a hit gives the same logits as recomputing, unlike semantic caching.
L3
What goes into a block key?
Answer
Parent hash, token ids, and extra keys such as LoRA id, image hashes and a salt.
L4
Why does routing matter?
Answer
KV lives on one replica; random routing scatters each prefix and caches thrash.
L5
What is the break-even hit rate?
Answer
(w - 1) / (w - r) for write multiplier w and read multiplier r.
L6
Why salt caches per tenant?
Answer
A fast TTFT can reveal that someone else sent the same prefix.
L7
Why did the hit rate drop to zero after a release?
Answer
Something volatile entered the prefix, such as a timestamp or reordered tool list.
Failure modes
Volatile token at the top
Every block key after it changes and the hit rate collapses to zero.
Round-robin across replicas
Each prefix is computed on every replica and caches hold the whole working set.
Write premium on rare prefixes
Low-traffic prefixes cost more than no caching.
Misconceptions
The same text always hits.
A different adapter, chat template or tokenization is a miss.
Request-level hit rate is the right metric.
Measure token-level hit rate.
Caches survive upgrades.
They are keyed per model and version.
Interviewer traps
Pure cache-affinity routing.
Cap affinity by load so one tenant does not melt a replica.
Only one breakpoint after a long growing suffix.
Add a breakpoint at the end of the static region so the lookback window reaches the previous write.
Design scenario
Same prompt for every reader.
Requirements
High token hit rate, tenant isolation, and lower cost than no caching.
Failure assumptions
- Prompts start with a timestamp.
- One tenant sends far more traffic than others.
- Some prefixes recur less than hourly.
Constraints
- Shared GPUs across tenants.
- Provider caching with a write premium on part of the traffic.
Prompt
A multi-tenant assistant has 200 tenant prefixes of 10,000 tokens and 50,000 requests per day across 8 replicas. Design caching.
API
How are prompts laid out and which cache salts or keys does each request carry?
Data
Which metrics track token hit rate and cost multiplier per prefix?
Architecture
How does the router balance prefix affinity against load, and which TTL fits which prefixes?
Overview
Most production prompts are mostly repeated: the same system prompt, tool definitions, few-shot examples, retrieved policy documents or conversation history, with a small new part at the end. Prefix caching stores the KV cache computed for a prompt prefix and reuses it for every later request that starts with exactly the same tokens, so those tokens skip prefill. Self-hosted engines do this with hashed KV blocks (vLLM automatic prefix caching) or a radix tree (SGLang RadixAttention). Hosted APIs expose it as provider prompt caching with discounted cached-input pricing, sometimes a write premium, and a TTL. This page goes beyond the single-replica mechanics covered on the RadixAttention and PagedAttention pages and the application-level prompt-ordering advice on the agent caching page. It covers how cache keys are built, why routing decides your hit rate across a fleet, how TTL and write premiums make caching cheaper or more expensive, and how to isolate tenants.
What is actually cached, and what is not
- Cached: the attention keys and values for each prefix token at every layer. That is the expensive output of prefill.
- Not cached: the output. Prefix caching is exact and lossless; a hit produces the same logits the full prefill would (up to floating-point non-determinism). Contrast this with semantic caching, which returns a previous answer for a similar question and can be wrong (see the agent caching page).
- Matching is prefix-only and token-exact. Changing one token invalidates everything after it, but nothing before it.
Cache keys: how a block is identified
In vLLM's design, the KV cache is split into fixed-size blocks (for example 16 tokens), and each full block's key is a hash of:
- the parent block's hash (so the key encodes the entire prefix, not just these tokens),
- the token ids in this block, and
- extra keys that change the KV for the same tokens: the LoRA adapter id, multimodal input hashes (an image's tokens are placeholders; the hash of the image distinguishes them), and an optional per-request cache salt for isolation.
Engines use strong hashes (vLLM defaults to SHA-256 based hashing) because a collision would silently serve another prompt's KV. SGLang's radix tree reaches the same result structurally: shared prefixes share tree nodes, and eviction is LRU over leaves.
Cache-key implications people miss:
- The same text with a different adapter is a miss, correctly, because the adapter changes the K/V projections.
- Tokenization must be identical: a different chat template, a trailing space, or a reordered JSON tool schema changes token ids.
- Isolation: a shared cache across tenants can leak information through timing (a fast TTFT reveals that someone else sent this prefix). Use per-tenant salts in shared deployments; hosted providers isolate caches per organization or workspace.
Provider prompt caching, compared (check current docs before relying on numbers)
| Aspect | Self-hosted (vLLM APC, SGLang Radix) | Anthropic prompt caching | OpenAI prompt caching | Google Gemini context caching |
|---|---|---|---|---|
| How you opt in | Engine flag (default on in recent vLLM V1 and in SGLang) | cache_control breakpoints on content blocks (up to 4), or one top-level cache_control for automatic placement; each breakpoint looks back at most 20 blocks for a prior write | Automatic on long prompts; newer models also support explicit breakpoints; prompt_cache_key improves routing for shared prefixes | Implicit caching on newer models; explicit cache objects with a TTL you set |
| Minimum cacheable prefix | One full block | 512 to 4,096 tokens depending on model | Around 1024 tokens | Model-dependent minimum |
| Lifetime | Until evicted (LRU under memory pressure) | 5 minutes (refreshed on hit) or 1 hour option | Minutes of inactivity, longer retention options on some models | Explicit TTL, billed for storage per hour |
| Pricing shape (Oct 2026 docs) | Your GPU time | Write 1.25x (5m) or 2x (1h) base input; reads about 0.1x | Reads discounted (about 0.1x on recent models); newest models add a write premium | Discounted cached tokens plus storage cost |
| Hit visibility | Engine metrics (prefix cache hit rate) | cache_creation_input_tokens, cache_read_input_tokens in usage | cached_tokens in usage details | Cached token count in usage metadata |
The common lesson: the cache key is the exact prefix, the economics depend on the write premium vs the read discount and on how often the prefix recurs within the TTL, and high request rates on one prefix can overflow onto machines that do not hold it (OpenAI documents overflow routing for very high per-prefix rates), so hit rates are never exactly 100 percent.
Random routing or prefix-aware routing?
Prefer
Prefix-aware routing, capped by load
Send each prefix to the replica that most likely holds it.
- Token hit rate rose from 83.7% to 96.8%.
- Reuse concentrates instead of thrashing four small caches.
- Watch for hot tenants overloading one replica.
Alternative
Random or round-robin routing
Spread requests evenly.
- Each tenant's prefix is recomputed on every replica.
- Every cache holds every tenant, so small caches thrash.
- A timestamp at the top still drives any layout to 0.0%.
A request through a cache-aware fleet
Diagram 1 condensed: key, route, reuse, and the failure path.
- 1
Tokenize with a stable layout
Static first, volatile last. - 2
Hash blocks
Parent hash, token ids, adapter id, salt. - 3
Route by prefix and load
Prefer the replica holding the leading blocks. - 4
Reuse cached KV
Hit blocks skip prefill; only the new suffix is computed. - 5
Volatile prefix
Every key changes and the hit rate falls to zero.
Across a fleet: routing decides the hit rate
A replica can only reuse KV it holds. With random or round-robin load balancing, each tenant's prefix is recomputed on every replica and every replica's cache holds every tenant, so small caches thrash. Prefix-aware (cache-aware) routing hashes the leading blocks and prefers the replica that most likely holds them, balanced against load so a hot tenant does not melt one replica. SGLang's router, the llm-d inference scheduler and NVIDIA Dynamo's KV-aware router all implement this idea. P/D disaggregated setups add a further dimension: the KV produced on prefill nodes must be transferred to decode nodes (see the inference parallelism page).
"""Block-hashed prefix cache + routing, the way self-hosted runtimes do automatic prefix caching.
* The prompt is split into fixed blocks (16 tokens here). Each block's key is
hash(parent_key, tokens_in_block), so a key identifies the WHOLE prefix up to that block.
One changed token invalidates that block and every block after it, never the ones before.
* Each replica has its own LRU cache of KV blocks (GPU memory is per replica).
* Router A sends requests to a random replica; router B hashes the first blocks of the
prompt so requests sharing a prefix land on the same replica (prefix-aware routing).
"""
import hashlib, random
from collections import OrderedDict
BLOCK = 16
def block_keys(tokens):
keys, parent = [], "root"
for i in range(0, len(tokens) - len(tokens) % BLOCK, BLOCK): # only full blocks are cacheable
parent = hashlib.sha256((parent + "|" + " ".join(tokens[i:i + BLOCK])).encode()).hexdigest()[:16]
keys.append(parent)
return keys
class Replica:
def __init__(self, capacity_blocks):
self.cache, self.cap = OrderedDict(), capacity_blocks
def serve(self, tokens):
hit = 0
keys = block_keys(tokens)
for k in keys: # longest cached prefix = first miss stops reuse
if k in self.cache:
self.cache.move_to_end(k); hit += 1
else:
break
for k in keys[hit:]: # computed blocks are inserted for future requests
self.cache[k] = True
if len(self.cache) > self.cap:
self.cache.popitem(last=False) # evict least recently used
return hit * BLOCK, len(tokens)
random.seed(3)
SYSTEM = [f"sys{i}" for i in range(1200)] # shared 1,200-token system prompt + tools
TENANT_DOCS = {t: [f"{t}doc{i}" for i in range(800)] for t in "ABCDEFGH"} # per-tenant context
def request(ts_at_top=False):
t = random.choice("ABCDEFGH")
user = [f"u{random.randint(0, 10**6)}" for _ in range(60)]
head = [f"time={random.randint(0, 10**6)}"] if ts_at_top else [] # the classic cache killer
return t, head + SYSTEM + TENANT_DOCS[t] + user
def run(router, ts_at_top=False, n=2000, replicas=4, cap=400):
reps = [Replica(cap) for _ in range(replicas)]
hit = tot = 0
for _ in range(n):
tenant, toks = request(ts_at_top)
if router == "random":
r = random.randrange(replicas)
else: # prefix-aware: hash of the first few blocks (here they encode tenant context)
r = int(hashlib.md5(" ".join(toks[:BLOCK * 80]).encode()).hexdigest(), 16) % replicas
h, t = reps[r].serve(toks)
hit += h; tot += t
return hit / tot
print(f"{'layout / router':<38}{'token hit rate':>15}")
print(f"{'stable prefix, random routing':<38}{run('random'):>15.1%}")
print(f"{'stable prefix, prefix-aware routing':<38}{run('prefix'):>15.1%}")
print(f"{'timestamp at top, prefix-aware':<38}{run('prefix', ts_at_top=True):>15.1%}")
print("\nRandom routing spreads each tenant's prefix over every replica, so 4 small caches thrash.")
print("Prefix-aware routing concentrates reuse (watch for hot tenants overloading one replica).")
print("Failure path: one volatile token at the top (a timestamp, request id, shuffled tool list)")
print("changes every block key after it, and the hit rate collapses to zero.")Output:
layout / router token hit rate
stable prefix, random routing 83.7%
stable prefix, prefix-aware routing 96.8%
timestamp at top, prefix-aware 0.0%
Random routing spreads each tenant's prefix over every replica, so 4 small caches thrash.
Prefix-aware routing concentrates reuse (watch for hot tenants overloading one replica).
Failure path: one volatile token at the top (a timestamp, request id, shuffled tool list)
changes every block key after it, and the hit rate collapses to zero.Hit-rate economics
With write multiplier w, read multiplier r and token hit rate h, the cost of the prefix relative to no caching is h*r + (1-h)*w. Caching pays when that is below 1, so the break-even hit rate is (w - 1) / (w - r). If requests sharing a prefix arrive at rate lambda and each hit refreshes the TTL, a rough steady-state hit probability is 1 - exp(-lambda * TTL).
// Provider prompt-cache economics: write premium, read discount, TTL, and traffic rate.
// Pricing multipliers are EXAMPLES in the style published in Oct 2026 docs (relative to the base
// input price): 5-minute write 1.25x, 1-hour write 2x, read 0.1x; some providers charge no write
// premium (1.0x). Always re-check the current pricing page for your model.
type Policy = { name: string; ttlMin: number; write: number; read: number };
const policies: Policy[] = [
{ name: "5m TTL (write 1.25x)", ttlMin: 5, write: 1.25, read: 0.1 },
{ name: "1h TTL (write 2.0x)", ttlMin: 60, write: 2.0, read: 0.1 },
{ name: "no write premium", ttlMin: 5, write: 1.0, read: 0.1 },
];
// Simplification: steady traffic (no day/night cycle). If requests sharing a prefix arrive as a Poisson process with rate lambda/min and every hit refreshes
// the TTL, a request hits when the previous one arrived within TTL: P(hit) = 1 - exp(-lambda * TTL).
const pHit = (perHour: number, ttlMin: number) => 1 - Math.exp(-(perHour / 60) * ttlMin);
// Expected cost of the prefix per request, in units of "uncached input price".
const multiplier = (h: number, p: Policy) => h * p.read + (1 - h) * p.write;
// Break-even hit rate where caching costs the same as not caching: h*r + (1-h)*w = 1.
const breakEven = (p: Policy) => Math.max(0, (p.write - 1) / (p.write - p.read));
console.log("break-even token hit rate:");
policies.forEach((p) => console.log(` ${p.name.padEnd(22)} ${(100 * breakEven(p)).toFixed(1)}%`));
console.log("\nrequests/hour per prefix -> P(hit) and cost multiplier (1.00 = no caching)");
console.log(("req/h " + policies.map((p) => p.name.padEnd(26)).join("")).trimEnd());
for (const perHour of [1, 6, 12, 60, 600]) {
const cells = policies.map((p) => {
const h = pHit(perHour, p.ttlMin);
return `hit ${(100 * h).toFixed(0).padStart(3)}% x${multiplier(h, p).toFixed(2)}`.padEnd(26);
});
console.log((String(perHour).padEnd(8) + cells.join("")).trimEnd());
}
// Worked example: 10k-token shared prefix, 50k requests/day spread over 200 tenants (separate prefixes).
const basePerMTok = 3.0; // USD per 1M uncached input tokens (example)
const prefixTok = 10_000, reqPerDay = 50_000, tenants = 200;
const perHour = reqPerDay / 24 / tenants;
console.log(`\nworked example: ${tenants} tenant prefixes x ${prefixTok} tokens, ${reqPerDay} req/day (~${perHour.toFixed(1)} req/h per prefix)`);
const uncached = (reqPerDay * prefixTok * basePerMTok) / 1e6;
console.log(` no caching: $${uncached.toFixed(0)}/day for prefix tokens`);
for (const p of policies) {
const h = pHit(perHour, p.ttlMin);
console.log(` ${p.name.padEnd(22)} hit ${(100 * h).toFixed(0)}% -> $${(uncached * multiplier(h, p)).toFixed(0)}/day`);
}
console.log("\nFailure path: low-traffic prefixes with a write premium cost MORE than no caching (multiplier > 1).");
console.log("Fixes: longer TTL only when inter-arrival is between the two TTLs, merge prefixes across tenants");
console.log("where data rules allow, or pre-warm only the hottest prefixes.");Output:
break-even token hit rate:
5m TTL (write 1.25x) 21.7%
1h TTL (write 2.0x) 52.6%
no write premium 0.0%
requests/hour per prefix -> P(hit) and cost multiplier (1.00 = no caching)
req/h 5m TTL (write 1.25x) 1h TTL (write 2.0x) no write premium
1 hit 8% x1.16 hit 63% x0.80 hit 8% x0.93
6 hit 39% x0.80 hit 100% x0.10 hit 39% x0.65
12 hit 63% x0.52 hit 100% x0.10 hit 63% x0.43
60 hit 99% x0.11 hit 100% x0.10 hit 99% x0.11
600 hit 100% x0.10 hit 100% x0.10 hit 100% x0.10
worked example: 200 tenant prefixes x 10000 tokens, 50000 req/day (~10.4 req/h per prefix)
no caching: $1500/day for prefix tokens
5m TTL (write 1.25x) hit 58% -> $874/day
1h TTL (write 2.0x) hit 100% -> $150/day
no write premium hit 58% -> $717/day
Failure path: low-traffic prefixes with a write premium cost MORE than no caching (multiplier > 1).
Fixes: longer TTL only when inter-arrival is between the two TTLs, merge prefixes across tenants
where data rules allow, or pre-warm only the hottest prefixes.Expectedbreak-even token hit rate: 5m TTL (write 1.25x) 21.7% 1h TTL (write 2.0x) 52.6% no write premium 0.0% requests/hour per prefix -> P(hit) and cost multiplier (1.00 = no caching) req/h 5m TTL (write 1.25x) 1h TTL (write 2.0x) no write premium 1 hit 8% x1.16 hit 63% x0.80 hit 8% x0.93 6 hit 39% x0.80 hit 100% x0.10 hit 39% x0.65 12 hit 63% x0.52 hit 100% x0.10 hit 63% x0.43 60 hit 99% x0.11 hit 100% x0.10 hit 99% x0.11 600 hit 100% x0.10 hit 100% x0.10 hit 100% x0.10 worked example: 200 tenant prefixes x 10000 tokens, 50000 req/day (~10.4 req/h per prefix) no caching: $1500/day for prefix tokens 5m TTL (write 1.25x) hit 58% -> $874/day 1h TTL (write 2.0x) hit 100% -> $150/day no write premium hit 58% -> $717/day Failure path: low-traffic prefixes with a write premium cost MORE than no caching (multiplier > 1). Fixes: longer TTL only when inter-arrival is between the two TTLs, merge prefixes across tenants where data rules allow, or pre-warm only the hottest prefixes.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Diagram 1: a request through a cache-aware fleet
Decisions
- 1
Step 1: build prompt in stable order - tools, system, static docs, history, then new user turn
- nextStep 2: gateway tokenizes and hashes the leading blocks (plus adapter id and tenant salt)
- 2
Step 2: gateway tokenizes and hashes the leading blocks (plus adapter id and tenant salt)
- nextStep 3: router picks the replica holding the longest matching prefix, unless it is overloaded
- 3
Step 3: router picks the replica holding the longest matching prefix, unless it is overloaded
- nextStep 4: blocks found in that replica's cache?
- ?
Step 4: blocks found in that replica's cache?
- nextStep 5a: reuse KV for the matched blocks, prefill only the suffix
- nextStep 5b: full prefill, store full blocks under their hash keys
- 5
Step 5a: reuse KV for the matched blocks, prefill only the suffix
- nextStep 6: decode, return usage with cached token counts
- 6
Step 5b: full prefill, store full blocks under their hash keys
- nextStep 6: decode, return usage with cached token counts
- 7
Step 6: decode, return usage with cached token counts
- nextStep 7: LRU or TTL evicts cold blocks under memory pressure
- nextFailure path: hit rate near zero after a deploy
- 8
Step 7: LRU or TTL evicts cold blocks under memory pressure
- 9
Failure path: hit rate near zero after a deploy
- nextDiff the token prefix - timestamp or request id at top, reordered tools, changed template - move volatile parts to the end
- 10
Diff the token prefix - timestamp or request id at top, reordered tools, changed template - move volatile parts to the end
- nextStep 1: build prompt in stable order - tools, system, static docs, history, then new user turn
Lesson map
Prefix & Prompt Caching Across Requests - Cache Keys, Routing, TTL & Hit-Rate Economics
KV block hashing (vLLM APC) vs radix tree (SGLang); what goes into cache keys (adapter, tokenizer, images, salts); prefix-aware routing across replicas; provider prompt caching write premiums, read discounts, TTL; runnable fleet hit-rate sim + economics.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB s1["Step 1: build prompt in stable order - tools, system, static docs, history, then new user turn"] s2["Step 2: gateway tokenizes and hashes the leading blocks (plus adapter id and tenant salt)"] s3["Step 3: router picks the replica holding the longest matching prefix, unless it is overloaded"] s4["Step 4: blocks found in that replica's cache?"] s5["Step 5a: reuse KV for the matched blocks, prefill only the suffix"] s6["Step 5b: full prefill, store full blocks under their hash keys"] s7["Step 6: decode, return usage with cached token counts"] s8["Step 7: LRU or TTL evicts cold blocks under memory pressure"] f1["Failure path: hit rate near zero after a deploy"] f2["Diff the token prefix - timestamp or request id at top, reordered tools, changed template - move volatile parts to the end"] s1 -->|continues| s2 s2 -->|continues| s3 s3 -->|continues| s4 s4 -->|continues| s5 s4 -->|continues| s6 s5 -->|continues| s7 s6 -->|continues| s7 s7 -->|continues| s8 s7 -->|continues| f1 f1 -->|continues| f2 f2 -->|continues| s1
Prompt layout rules that keep keys stable
- Static first, volatile last: tool definitions and system prompt, then long-lived documents, then conversation history, then the new turn.
- No timestamps, request ids, random examples or user names in the static region; pass them later or in metadata.
- Serialize tool schemas and JSON deterministically (sorted keys, fixed order).
- Pin the chat template and tokenizer version with the model.
- For conversations, append; never rewrite earlier turns (summarise into a new block instead, accepting one miss).
Decision chart: how to cache prompts
Decisions
- 1
D1
- nextPrefix caching will not help; look at queueing, chunked prefill or quantization
- nextStep 2: self-hosted or hosted API?
- 2
Prefix caching will not help; look at queueing, chunked prefill or quantization
- ?
Step 2: self-hosted or hosted API?
- nextStep 3: more than one replica?
- nextStep 4: does each prefix recur within the TTL often enough to beat break-even?
- ?
Step 3: more than one replica?
- nextEnable prefix caching plus prefix-aware routing with load balance; per-tenant cache salt
- nextEnable prefix caching; size KV memory for the working set of hot prefixes
- 5
Enable prefix caching plus prefix-aware routing with load balance; per-tenant cache salt
- 6
Enable prefix caching; size KV memory for the working set of hot prefixes
- ?
Step 4: does each prefix recur within the TTL often enough to beat break-even?
- nextShort TTL caching, breakpoints after static blocks
- nextLonger TTL option if the write premium still pays
- nextDo not pay a write premium; restructure to share prefixes across requests
- 8
Short TTL caching, breakpoints after static blocks
- 9
Longer TTL option if the write premium still pays
- 10
Do not pay a write premium; restructure to share prefixes across requests
What happens if you choose otherwise
- Round-robin across 8 replicas: each prefix is computed 8 times and caches hold 8x the working set; hit rate and TTFT look much worse than the single-replica benchmark.
- Pure cache-affinity routing: one big tenant overloads its replica while others idle; always cap affinity by load.
- 1-hour caching for a prefix that recurs once an hour or less: you pay 2x writes for few reads; caching costs more than not caching.
- Shared cache without salts in a multi-tenant deployment: a timing side channel reveals other tenants' prompts.
Pitfalls
- Measuring request-level hit rate instead of token-level hit rate.
- Assuming a hit after a model or engine upgrade; caches are keyed per model and version.
- Ignoring KV memory: a large prefix cache competes with running requests for HBM; FP8 KV doubles the capacity.
- Breaking the prefix with per-user personalisation in the system prompt.
- With breakpoint-style APIs, placing the only breakpoint after a long, growing suffix: a lookback window (20 blocks in Anthropic's design) may not reach the previous write, so add a breakpoint at the end of the static region.
Interview Q&A
What exactly does prefix caching store, and is it lossless?
Answer
The per-layer keys and values for prefix tokens. It is exact: a hit gives the same result as recomputing, unlike semantic caching which returns old answers.
How is a KV block's cache key built?
Answer
Hash of the parent block's hash, the block's token ids, and extra keys such as LoRA id, multimodal hashes and an optional tenant salt. Chaining the parent hash makes the key represent the whole prefix.
Why does load balancing matter for cache hit rates?
Answer
KV lives on one replica. Random routing scatters each prefix across replicas so each cache holds everything and thrashes; prefix-aware routing concentrates reuse, balanced against load.
When does provider caching cost more than no caching?
Answer
When the hit rate is below (w - 1)/(w - r), for example about 22 percent with a 1.25x write premium and 0.1x reads, typically because prefixes recur less often than the TTL.
Your hit rate dropped to zero after a release. What happened?
Answer
Something volatile entered the prefix (timestamp, request id, reordered tool list, changed chat template). Diff the tokenized prefixes of two consecutive requests.
How do vLLM and SGLang differ in prefix caching?
Answer
vLLM hashes fixed-size KV blocks (automatic prefix caching); SGLang shares prefixes structurally in a radix tree with LRU eviction over leaves.
Why do engines use strong hashes for block keys?
Answer
A collision would silently serve another prompt's KV.
Why are hit rates never exactly 100 percent with providers?
Answer
High request rates on one prefix can overflow onto machines that do not hold it.
Check yourself
Diff the tokenized prompts of two consecutive production requests. Mark where they first differ, move anything volatile below the static region, and estimate your break-even hit rate for your provider's write and read multipliers.
Elsewhere in the library
These pages stay as they are. This lesson only points at them: SGLang — RadixAttention, Continuous Batching & Structured Generation, PagedAttention & Continuous Batching, Caching for Agents — Prompt Cache, KV Reuse, Semantic Cache & Tool Result Cache, Prefill vs Decode & KV Cache Mechanics, Cache Invalidation — TTL vs Event-Driven vs Versioned Keys, Inference Parallelism — Tensor Parallel, Expert Parallel & Prefill/Decode Disaggregation.
Go Deeper
- vLLM automatic prefix caching design (GitHub docs)
- SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention)
- Anthropic prompt caching documentation
- OpenAI prompt caching guide
- Google Gemini API context caching
- Mooncake: a KVCache-centric disaggregated architecture for LLM serving