High-level design

Oct 2026

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Scale-up HLD: ID generation compared, Redis cache-aside, sharding, 301 vs 302, async analytics; cross-link existing pages.

1Gist
2Maps
3Q&A
4Sandbox

View

Voice readout needs Web Speech Synthesis in this browser.

Why it exists: After the Flask file works, the interviewer asks what happens at 10k redirects per second. The coding lock does not answer that.
Definition: Scale-up for a URL shortener is a read-heavy redirect path: a cache in front of a code-sharded mapping store, globally unique ids, and hit analytics off the critical path.
30-second answer: Assume create is rare and redirect is the hot path. Cache code to URL with a TTL capped by link expiry. Mint ids with a counter, Snowflake, or a key service, then place the row by hash of the code. Return 302 when counts must stay honest. Enqueue hit events. Rate-limit create. On a cache miss that is truly absent, negative-cache briefly.
Mental model: The request should die at the edge or at Redis. The shard is the miss path. Analytics is a queue, not a write in the redirect handler.
Key tradeoff: 302 revalidates and keeps counts movable. 301 lets a CDN absorb traffic and then your hit counter freezes unless a separate beacon exists.
Interview cue: Say read-heavy, shard by code, and async hits. Point at the existing pages for consistent hashing, sharding, cache-aside, CDNs, negative caching, and rate limits instead of re-deriving them.

Question ladder

L1
What is the capacity sketch?
Answer
About 100 million new URLs a year is a few creates per second. The design target is 10k redirects per second, with viral keys far hotter than that.
L2
Why shard by code?
Answer
Redirects only carry the code. Sharding by owner or by the long URL makes the hot path a scatter.
L3
How do ids stay unique across shards?
Answer
Mint them globally with Snowflake, a key service, or a central counter, then store the row at hash of the code.
L4
What TTL do you put in Redis?
Answer
The minimum of the cache policy and the remaining link TTL. Do not outlive the short link.
L5
When is 301 acceptable?
Answer
When a CDN or browser cache is the point, and analytics arrive from a beacon rather than from the redirect itself.
L6
What if Redis is down?
Answer
Fail open to the store behind bulkheads and rate limits, or fail closed if errors are cheaper than overload. Say which.
L7
How do billions of keys expire?
Answer
Lazy expiry on read plus a compaction job. Redis TTLs cover the cache, not the system of record.

Failure modes

Cache stampede
A hot code expires and every redirect hits one shard. Single-flight or a soft TTL belongs on the cache-aside page.
Synchronous analytics
A hit write on the redirect path adds a dependency the user waits on. Enqueue instead.
301 with a live counter
Intermediaries cache the redirect and the counter stops moving.

Misconceptions

Each shard can run its own auto-increment.
Without a namespace, codes collide. Mint globally, then place by code.
Shard by the long URL because that is the value.
The redirect request does not have the long URL yet. The key you have is the code.
The coding-round process lock is the production design.
It is correct for one SQLite file. It is not a 10k redirect design.

Interviewer traps

Re-teach consistent hashing from scratch.
Name the ring and virtual nodes, then link the hashing study.
Put a Kafka write in the latency budget as if it were free.
The budget is cache or shard read plus an enqueue, not a synchronous aggregate.

Design scenario

Same prompt for every reader.

Requirements

Global ids, a cache on GET /r/code, shards keyed by code, and analytics off the redirect.

Traffic / scale

A few creates per second. 10k redirects per second, with some codes wildly hotter.

Latency

A cache hit plus an async enqueue. A miss may read one shard.

Consistency

A code maps to one long URL. Hit counts may lag. Uniqueness may not.

Availability

Say whether a dead cache fails open to the store or fails closed.

Failure assumptions

One viral code can stampede.
A shard can be hot if the hash ignores a celebrity code.

Constraints

Do not shard by owner.
Do not write analytics synchronously on redirect.

Prompt

Take the working single-node shortener to 10k redirects per second without losing unique codes.

API

When do you return 302, and when would you accept 301?

Data

What is the key and what is the value in the mapping store?

Architecture

Where do Redis, the shards, and the analytics queue sit on a miss?

Redirect status at scale

Prefer

302 while the counter still matters

Clients come back, so a delete, a TTL, and a hit event stay attached to the request you serve.

Cache the mapping yourself, with a TTL you control.
Analytics is a queue.
A miss can negative-cache.

Alternative

301 into a CDN

The edge absorbs repeats. Your hit row stops moving unless something else reports the click.

Right for a permanent marketing link.
Wrong if the interview still wants accurate counts.
Pair it with a beacon before you celebrate the offload.

Overview

After the Flask+SQLite coding solution works, interviewers ask how it survives 10k+ redirects per second. This page compares ID generation strategies, places a cache on the read path, shards the mapping store, and splits analytics from the redirect hot path. It deepens existing pages on consistent hashing, sharding, Redis cache-aside, CDN, rate limiting and load balancing instead of rewriting them.

Capacity sketch

Assume 100M new URLs/year (~3/s create), 10k redirects/s peak, 100B mapping reads/day possible with viral links. Redirect is read-heavy; create is write-light. Optimize the GET /r/<code> path.

Step-by-step scale-up

Put a cache (Redis) in front of KV/DB for code -> url with TTL aligned to link expiry.
Choose distributed ID generation (Snowflake, Redis INCR+base62, or KGS).
Shard the mapping table by code hash / consistent hashing.
Prefer 302 for counted links; 301 only when caching at CDN is intentional.
Push hit events to a queue/stream; aggregate async (do not write analytics on the redirect critical path).
Rate-limit create and redirect per IP/token to blunt abuse.
Plan failure: cache stampede, shard imbalance, TTL clock skew, poisoned vanity codes.

Redirect at scale

Diagram 1. A cache hit never touches a shard. A miss fills Redis only when the row is alive. Hits are an event, not a synchronous write.

1
GET /r/code
The client sends only the code. An edge or an app load balancer receives it.
2
Redis lookup
Cache-aside: the hot mapping should already be code to long URL.
3
Cache hit
Enqueue a hit event and return 302. Do not write analytics inline.
4
Miss: read the shard
The shard is hash of the code. The long URL is not a shard key.
5
Missing or expired
Return 404 and store a short negative cache entry so the shard is not stampeded.
6
Fill and redirect
Set Redis TTL to the minimum of policy and remaining link life, emit the hit, then 302.

Zoom

Decisions

1
1. Client GET /r/code
- next2. Edge or app load balancer
2
2. Edge or app load balancer
- next3. Lookup code in Redis cache
3
3. Lookup code in Redis cache
- next4. Cache hit?
?
4. Cache hit?
- Yes5. Emit hit event async
- No6. Read shard for code
5
5. Emit hit event async
- next10. 302 redirect to long URL
6
6. Read shard for code
- next7. Found and alive?
?
7. Found and alive?
- No8. Failure: 404 and negative cache
- Yes9. Fill cache with TTL
8
8. Failure: 404 and negative cache
9
9. Fill cache with TTL
- next5. Emit hit event async
10
10. 302 redirect to long URL

Lesson map

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Scale-up HLD: ID generation compared, Redis cache-aside, sharding, 301 vs 302, async analytics; cross-link existing pages.

1. Client GET /r/code2. Edge or app load balancer3. Lookup code in Redis cache4. Cache hit?5. Emit hit event async6. Read shard for code7. Found and alive?8. Failure: 404 and negative cache9. Fill cache with TTL10. 302 redirect to long URL

Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.

Mermaid export

flowchart TB
  a["1. Client GET /r/code"]
  b["2. Edge or app load balancer"]
  c["3. Lookup code in Redis cache"]
  d["4. Cache hit?"]
  e["5. Emit hit event async"]
  f["6. Read shard for code"]
  g["7. Found and alive?"]
  h["8. Failure: 404 and negative cache"]
  i["9. Fill cache with TTL"]
  j["10. 302 redirect to long URL"]
  a -->|continues| b
  b -->|continues| c
  c -->|continues| d
  d -->|Yes| e
  d -->|No| f
  f -->|continues| g
  g -->|No| h
  g -->|Yes| i
  i -->|continues| e
  e -->|continues| j

Zoom

Decisions

1
ID generator choice
- nextCounter base62
- nextSnowflake
- nextKey Generation Service
- nextHash URL
2
Counter base62
- nextSingle writer OK?
3
Snowflake
4
Key Generation Service
5
Hash URL
?
Single writer OK?
- YesUse counter
- NoSnowflake
7
Use counter

ID generation compared

Strategy	Pros	Cons	Ops notes
DB auto-increment + base62	Short, simple	Hot page / failover hard	Fine early; pin primary
Redis INCR + base62	Fast, central	Redis availability becomes critical	Multi-AZ Redis
Snowflake-style	Distributed, time-sortable	Clock sync; longer raw ids	Re-encode to base62
Pre-generated KGS	No runtime coordination	Key server; unused key waste; security if keys leak	Two-tier unused/used
Hash(long_url)	Deterministic reuse	Collisions; foresability	Add salt + collision list

Caching and 301 vs 302

Cache-aside: on miss load from shard, set Redis TTL = min(policy, remaining link TTL).
Negative cache short TTL for 404s to protect shards.
302: browsers revalidate; better for accurate counts.
301: CDN/browser may cache permanently; pair with separate analytics beacon if used.
Cross-link: Redis cache-aside, CDN cache hierarchy, negative caching, cache invalidation.

Sharding and data model

KV shape: code -> {url, owner, expires_at, created_at}. Shard by hash(code) % N or consistent hashing with vnodes. Avoid sharding by owner (redirects only have code). Range scans for "my links" use a secondary index store. Cross-link: database sharding and consistent hashing.

Analytics

Redirect handler enqueues {code, ts, ip_hash, ua_hash} to Kafka/PubSub; workers update counts and rollups. Redirect latency stays a cache/DB read + enqueue. Cross-link rate limiting for create abuse: rate limiting and distributed rate limits.

Concepts used, learn more

Read the underlying idea on its own study page. This lesson applies it. It does not replace those pages.

Interview Q&A

How do you keep codes unique across shards?

Show

Answer

Allocate IDs globally (Snowflake/KGS/Redis), then place the row on hash(code). Do not generate independent counters per shard without a namespace prefix.

What if Redis is down?

Show

Answer

Fail open to the data store with bulkheads and rate limits, or fail closed on redirect if SLA prefers errors over overload. State the choice.

How do you expire billions of keys?

Show

Answer

Lazy expiry on read + periodic compaction job; Redis TTLs for cache only.

What is a negative cache here?

Show

Answer

A short TTL for a code you just learned is missing, so the next thousand clients do not all read the shard. The full pattern is negative caching.

How do you invalidate when a link is deleted?

Show

Answer

Delete or overwrite the cache key. TTL still bounds a missed event. Compare TTL, events, and versioned keys on cache invalidation.

Where do rate limits sit?

Show

Answer

On create, and on redirect per IP or token, so a client cannot fill the key space or melt a shard. Start with rate limiting and the gateway case on distributed rate limits.

Pitfalls

Putting synchronous analytics writes on the redirect path.
301 everything then wondering why hit counts freeze.
Sharding by long_url (redirects never have it).

Go Deeper

The series pager also walks these pages.

High-level design

Part 3 of 3 · URL Shortener

Oct 2026

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Scale-up HLD: ID generation compared, Redis cache-aside, sharding, 301 vs 302, async analytics; cross-link existing pages.

1Gist
2Maps
3Q&A
4Sandbox

View

Voice readout needs Web Speech Synthesis in this browser.

Why it exists: After the Flask file works, the interviewer asks what happens at 10k redirects per second. The coding lock does not answer that.
Definition: Scale-up for a URL shortener is a read-heavy redirect path: a cache in front of a code-sharded mapping store, globally unique ids, and hit analytics off the critical path.
30-second answer: Assume create is rare and redirect is the hot path. Cache code to URL with a TTL capped by link expiry. Mint ids with a counter, Snowflake, or a key service, then place the row by hash of the code. Return 302 when counts must stay honest. Enqueue hit events. Rate-limit create. On a cache miss that is truly absent, negative-cache briefly.
Mental model: The request should die at the edge or at Redis. The shard is the miss path. Analytics is a queue, not a write in the redirect handler.
Key tradeoff: 302 revalidates and keeps counts movable. 301 lets a CDN absorb traffic and then your hit counter freezes unless a separate beacon exists.
Interview cue: Say read-heavy, shard by code, and async hits. Point at the existing pages for consistent hashing, sharding, cache-aside, CDNs, negative caching, and rate limits instead of re-deriving them.

Question ladder

L1
What is the capacity sketch?
Answer
About 100 million new URLs a year is a few creates per second. The design target is 10k redirects per second, with viral keys far hotter than that.
L2
Why shard by code?
Answer
Redirects only carry the code. Sharding by owner or by the long URL makes the hot path a scatter.
L3
How do ids stay unique across shards?
Answer
Mint them globally with Snowflake, a key service, or a central counter, then store the row at hash of the code.
L4
What TTL do you put in Redis?
Answer
The minimum of the cache policy and the remaining link TTL. Do not outlive the short link.
L5
When is 301 acceptable?
Answer
When a CDN or browser cache is the point, and analytics arrive from a beacon rather than from the redirect itself.
L6
What if Redis is down?
Answer
Fail open to the store behind bulkheads and rate limits, or fail closed if errors are cheaper than overload. Say which.
L7
How do billions of keys expire?
Answer
Lazy expiry on read plus a compaction job. Redis TTLs cover the cache, not the system of record.

Failure modes

Cache stampede
A hot code expires and every redirect hits one shard. Single-flight or a soft TTL belongs on the cache-aside page.
Synchronous analytics
A hit write on the redirect path adds a dependency the user waits on. Enqueue instead.
301 with a live counter
Intermediaries cache the redirect and the counter stops moving.

Misconceptions

Each shard can run its own auto-increment.
Without a namespace, codes collide. Mint globally, then place by code.
Shard by the long URL because that is the value.
The redirect request does not have the long URL yet. The key you have is the code.
The coding-round process lock is the production design.
It is correct for one SQLite file. It is not a 10k redirect design.

Interviewer traps

Re-teach consistent hashing from scratch.
Name the ring and virtual nodes, then link the hashing study.
Put a Kafka write in the latency budget as if it were free.
The budget is cache or shard read plus an enqueue, not a synchronous aggregate.

Design scenario

Same prompt for every reader.

Requirements

Global ids, a cache on GET /r/code, shards keyed by code, and analytics off the redirect.

Traffic / scale

A few creates per second. 10k redirects per second, with some codes wildly hotter.

Latency

A cache hit plus an async enqueue. A miss may read one shard.

Consistency

A code maps to one long URL. Hit counts may lag. Uniqueness may not.

Availability

Say whether a dead cache fails open to the store or fails closed.

Failure assumptions

One viral code can stampede.
A shard can be hot if the hash ignores a celebrity code.

Constraints

Do not shard by owner.
Do not write analytics synchronously on redirect.

Prompt

Take the working single-node shortener to 10k redirects per second without losing unique codes.

API

When do you return 302, and when would you accept 301?

Data

What is the key and what is the value in the mapping store?

Architecture

Where do Redis, the shards, and the analytics queue sit on a miss?

Redirect status at scale

Prefer

302 while the counter still matters

Clients come back, so a delete, a TTL, and a hit event stay attached to the request you serve.

Cache the mapping yourself, with a TTL you control.
Analytics is a queue.
A miss can negative-cache.

Alternative

301 into a CDN

The edge absorbs repeats. Your hit row stops moving unless something else reports the click.

Right for a permanent marketing link.
Wrong if the interview still wants accurate counts.
Pair it with a beacon before you celebrate the offload.

Overview

Capacity sketch

Assume 100M new URLs/year (~3/s create), 10k redirects/s peak, 100B mapping reads/day possible with viral links. Redirect is read-heavy; create is write-light. Optimize the GET /r/<code> path.

Step-by-step scale-up

Put a cache (Redis) in front of KV/DB for code -> url with TTL aligned to link expiry.
Choose distributed ID generation (Snowflake, Redis INCR+base62, or KGS).
Shard the mapping table by code hash / consistent hashing.
Prefer 302 for counted links; 301 only when caching at CDN is intentional.
Push hit events to a queue/stream; aggregate async (do not write analytics on the redirect critical path).
Rate-limit create and redirect per IP/token to blunt abuse.
Plan failure: cache stampede, shard imbalance, TTL clock skew, poisoned vanity codes.

Redirect at scale

Diagram 1. A cache hit never touches a shard. A miss fills Redis only when the row is alive. Hits are an event, not a synchronous write.

1
GET /r/code
The client sends only the code. An edge or an app load balancer receives it.
2
Redis lookup
Cache-aside: the hot mapping should already be code to long URL.
3
Cache hit
Enqueue a hit event and return 302. Do not write analytics inline.
4
Miss: read the shard
The shard is hash of the code. The long URL is not a shard key.
5
Missing or expired
Return 404 and store a short negative cache entry so the shard is not stampeded.
6
Fill and redirect
Set Redis TTL to the minimum of policy and remaining link life, emit the hit, then 302.

Zoom

Decisions

1
1. Client GET /r/code
- next2. Edge or app load balancer
2
2. Edge or app load balancer
- next3. Lookup code in Redis cache
3
3. Lookup code in Redis cache
- next4. Cache hit?
?
4. Cache hit?
- Yes5. Emit hit event async
- No6. Read shard for code
5
5. Emit hit event async
- next10. 302 redirect to long URL
6
6. Read shard for code
- next7. Found and alive?
?
7. Found and alive?
- No8. Failure: 404 and negative cache
- Yes9. Fill cache with TTL
8
8. Failure: 404 and negative cache
9
9. Fill cache with TTL
- next5. Emit hit event async
10
10. 302 redirect to long URL

Lesson map

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Scale-up HLD: ID generation compared, Redis cache-aside, sharding, 301 vs 302, async analytics; cross-link existing pages.

Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.

Mermaid export

flowchart TB
  a["1. Client GET /r/code"]
  b["2. Edge or app load balancer"]
  c["3. Lookup code in Redis cache"]
  d["4. Cache hit?"]
  e["5. Emit hit event async"]
  f["6. Read shard for code"]
  g["7. Found and alive?"]
  h["8. Failure: 404 and negative cache"]
  i["9. Fill cache with TTL"]
  j["10. 302 redirect to long URL"]
  a -->|continues| b
  b -->|continues| c
  c -->|continues| d
  d -->|Yes| e
  d -->|No| f
  f -->|continues| g
  g -->|No| h
  g -->|Yes| i
  i -->|continues| e
  e -->|continues| j

Zoom

Decisions

1
ID generator choice
- nextCounter base62
- nextSnowflake
- nextKey Generation Service
- nextHash URL
2
Counter base62
- nextSingle writer OK?
3
Snowflake
4
Key Generation Service
5
Hash URL
?
Single writer OK?
- YesUse counter
- NoSnowflake
7
Use counter

ID generation compared

Strategy	Pros	Cons	Ops notes
DB auto-increment + base62	Short, simple	Hot page / failover hard	Fine early; pin primary
Redis INCR + base62	Fast, central	Redis availability becomes critical	Multi-AZ Redis
Snowflake-style	Distributed, time-sortable	Clock sync; longer raw ids	Re-encode to base62
Pre-generated KGS	No runtime coordination	Key server; unused key waste; security if keys leak	Two-tier unused/used
Hash(long_url)	Deterministic reuse	Collisions; foresability	Add salt + collision list

Caching and 301 vs 302

Cache-aside: on miss load from shard, set Redis TTL = min(policy, remaining link TTL).
Negative cache short TTL for 404s to protect shards.
302: browsers revalidate; better for accurate counts.
301: CDN/browser may cache permanently; pair with separate analytics beacon if used.
Cross-link: Redis cache-aside, CDN cache hierarchy, negative caching, cache invalidation.

Sharding and data model

Analytics

Concepts used, learn more

Read the underlying idea on its own study page. This lesson applies it. It does not replace those pages.

Interview Q&A

How do you keep codes unique across shards?

Show

Answer

Allocate IDs globally (Snowflake/KGS/Redis), then place the row on hash(code). Do not generate independent counters per shard without a namespace prefix.

What if Redis is down?

Show

Answer

Fail open to the data store with bulkheads and rate limits, or fail closed on redirect if SLA prefers errors over overload. State the choice.

How do you expire billions of keys?

Show

Answer

Lazy expiry on read + periodic compaction job; Redis TTLs for cache only.

What is a negative cache here?

Show

Answer

A short TTL for a code you just learned is missing, so the next thousand clients do not all read the shard. The full pattern is negative caching.

How do you invalidate when a link is deleted?

Show

Answer

Delete or overwrite the cache key. TTL still bounds a missed event. Compare TTL, events, and versioned keys on cache invalidation.

Where do rate limits sit?

Show

Answer

On create, and on redirect per IP or token, so a client cannot fill the key space or melt a shard. Start with rate limiting and the gateway case on distributed rate limits.

Pitfalls

Putting synchronous analytics writes on the redirect path.
301 everything then wondering why hit counts freeze.
Sharding by long_url (redirects never have it).

Go Deeper

The series pager also walks these pages.

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Question ladder

Failure modes

Misconceptions

Interviewer traps

Design scenario

302 while the counter still matters

301 into a CDN

Overview

Capacity sketch

Step-by-step scale-up

ID generation compared

Caching and 301 vs 302

Sharding and data model

Analytics

Concepts used, learn more

Interview Q&A

Pitfalls

Go Deeper

Related

URL Shortener Scale-up HLD - ID Generation, Caching, Sharding & Analytics

Question ladder

Failure modes

Misconceptions

Interviewer traps

Design scenario

302 while the counter still matters

301 into a CDN

Overview

Capacity sketch

Step-by-step scale-up

ID generation compared

Caching and 301 vs 302

Sharding and data model

Analytics

Concepts used, learn more

Interview Q&A

Pitfalls

Go Deeper

Related