Multi-AZ vs Multi-Region - Blast Radius, Cell Architecture & Static Stability

Overview

Cloud providers give you two nested failure domains. An availability zone (AZ) is one or more data centers with independent power, cooling and networking, a few milliseconds from the other zones in the same region. Spreading across AZs is cheap (synchronous replication is practical at that latency) and protects against the most common large failures: a data center power or network event. Spreading across regions protects against region-wide failures, such as a regional control plane or network problem, at the cost of tens to hundreds of milliseconds of latency, asynchronous data, and much more complexity. Inside either, cells and shuffle sharding shrink the blast radius of the failure you cause yourself most often: a bad deploy or a poison request.

Failure domains compared

Availability math, and where it lies (runnable)

# Availability math for zones and regions, and why correlated failures break the "multiply" rule.
def parallel(*avail: float) -> float:
    # Service is up if ANY copy is up (assumes independent failures).
    down = 1.0
    for a in avail:
        down *= (1 - a)
    return 1 - down

def serial(*avail: float) -> float:
    # Request needs EVERY dependency to be up.
    up = 1.0
    for a in avail:
        up *= a
    return up

az = 0.999            # one availability zone (example)
minutes_per_year = 525_600
for label, a in [("1 AZ", az), ("3 AZs (independent)", parallel(az, az, az))]:
    print(f"{label:24} {a:.9f} -> {(1 - a) * minutes_per_year:8.2f} min down/yr")

# Reality check: a shared regional dependency (control plane, IAM, DNS) is in SERIES with the AZs.
regional_dep = 0.9995
multi_az = serial(parallel(az, az, az), regional_dep)
print(f"{'3 AZs + regional dep':24} {multi_az:.9f} -> {(1 - multi_az) * minutes_per_year:8.2f} min down/yr")
two_regions = parallel(multi_az, multi_az)
print(f"{'2 regions of that':24} {two_regions:.9f} -> {(1 - two_regions) * minutes_per_year:8.2f} min down/yr")

Output:

1 AZ                     0.999000000 ->   525.60 min down/yr
3 AZs (independent)      0.999999999 ->     0.00 min down/yr
3 AZs + regional dep     0.999499999 ->   262.80 min down/yr
2 regions of that        0.999999750 ->     0.13 min down/yr

Multiplying independent AZ availabilities gives absurd numbers like nine nines. Real failures are correlated: a shared regional dependency is in series with every zone, so it caps you. That is why multi-region exists, and also why multi-region only helps if the regions truly share nothing in the request path.

Static stability

Static stability (an AWS Builders' Library term) means the system keeps working during a failure without needing to make any changes. The classic example: if you run in three AZs and need 6 instances of capacity, a statically stable design runs 9 (3 per zone) so that losing a zone leaves 6 still running. A design that runs 6 and relies on auto-scaling to replace the lost zone's capacity depends on the control plane, which may be the thing that is impaired.

flowchart TB
  s1["1. AZ-a fails"]
  r1["1. AZ-a fails"]
  s2["2. AZ-b and AZ-c already hold full capacity"]
  r2["2. Auto-scaling must launch replacements"]
  s3["3. No API calls needed, users unaffected"]
  r3["3. Control plane healthy?"]
  r4["4a. Recovers in minutes"]
  r5["4b. Stuck at 66% capacity"]
  s1 -->|continues| s2
  s2 -->|continues| s3
  r1 -->|continues| r2
  r2 -->|continues| r3
  r3 -->|yes| r4
  r3 -->|no, regional event| r5

The same idea applies to data planes versus control planes: the data plane (serving requests, routing packets) should keep running on its last known good configuration even when the control plane (APIs that change configuration) is down.

Cell-based architecture and shuffle sharding (runnable)

A cell is a complete, independent copy of the service stack serving a fixed subset of customers. A thin routing layer maps each customer to a cell. Deploys roll out one cell at a time, so a bad change hurts one cell's customers before it is caught.

// Cell-based architecture: customers are pinned to one of N independent cells.
// A bad deploy or poison request takes out one cell, not the whole service.
// Shuffle sharding goes further: each customer gets a random PAIR of nodes, so two customers
// rarely share their whole failure domain.
function hash(s: string): number {
  let h = 2166136261;                     // FNV-1a, small and deterministic
  for (const c of s) { h ^= c.charCodeAt(0); h = Math.imul(h, 16777619) >>> 0; }
  return h;
}

const CELLS = 8;
const customers = Array.from({ length: 1000 }, (_, i) => `cust-${i}`);
const cellOf = (c: string) => hash(c) % CELLS;

const badCell = 3;                         // a bad deploy lands on cell 3 first
const hit = customers.filter((c) => cellOf(c) === badCell).length;
console.log(`cells=${CELLS}: bad cell hits ${hit}/${customers.length} customers (${(100 * hit / customers.length).toFixed(1)}%)`);

// Shuffle sharding: 8 nodes, each customer assigned 2 distinct nodes -> C(8,2)=28 possible shards.
const shardOf = (c: string): [number, number] => {
  const h = hash(c);
  const a = h % 8;
  const b = (a + 1 + ((h >>> 8) % 7)) % 8;  // second node: different from a, picked from other hash bits
  return [Math.min(a, b), Math.max(a, b)];
};
const noisy = shardOf("cust-42");          // a noisy customer overloads both of its nodes
const fullyHit = customers.filter((c) => {
  const [x, y] = shardOf(c);
  return x === noisy[0] && y === noisy[1];
}).length;
const partlyHit = customers.filter((c) => {
  const [x, y] = shardOf(c);
  return (x === noisy[0] || x === noisy[1] || y === noisy[0] || y === noisy[1]) && !(x === noisy[0] && y === noisy[1]);
}).length;
console.log(`shuffle sharding: ${fullyHit}/${customers.length} share BOTH nodes with the noisy customer (fully impacted)`);
console.log(`${partlyHit} share one node and still have a healthy second node to retry on`);

Output:

cells=8: bad cell hits 124/1000 customers (12.4%)
shuffle sharding: 34/1000 share BOTH nodes with the noisy customer (fully impacted)
432 share one node and still have a healthy second node to retry on

Press Run. Snippets must be self-contained — no network, files, or native modules.

With 8 cells, a poisoned cell hits about 1/8 of customers. With shuffle sharding, each customer gets a random combination of nodes, so a noisy or poisonous customer fully impacts only those few who happen to share its exact combination; everyone else who overlaps on one node can retry on their other node.

Where each pattern is used

• Multi-AZ is the default for every production database and stateless tier. Managed databases offer it as a checkbox (synchronous standby or quorum storage across zones).
• Multi-region is used for tier 0 journeys, latency to global users, and regulatory data residency.
• Cells are used by large SaaS and cloud providers themselves to bound the impact of deploys and noisy tenants.
• Shuffle sharding is used for multi-tenant front ends such as DNS and API gateways.

What happens if you choose otherwise

• Single AZ to save cross-AZ data transfer cost: a single data center event takes you fully down.
• Multi-region without isolating dependencies: both regions call the same global auth or config service, so you pay double and still fail together.
• No cells: a bad config push or poison message reaches 100% of customers before the alarm fires.
• Too many tiny cells: the router, migrations and capacity planning become their own operational burden, and cross-cell features (search across all customers) get hard.

Pros and cons

Interview Q&A

Check yourself

Elsewhere in the library

These pages stay as they are. This lesson only points at them: Resilience patterns, Consistent hashing, Private networking, Feature flags.

Go Deeper

• AWS Builders' Library: Static stability using Availability Zones
• AWS Builders' Library: Workload isolation using shuffle sharding
• AWS whitepaper: Reducing the scope of impact with cell-based architecture
• AWS summary of the December 2021 us-east-1 event