Disaster Recovery & Multi-Region - RTO/RPO, Backups, Pilot Light to Active-Active

Overview

Disaster recovery (DR) is the plan for getting a service back when something bigger than a single server fails: a whole zone, a whole region, a corrupted database, a deleted bucket, ransomware, or a bad deploy that poisons data everywhere at once. Two numbers drive every DR decision. RTO (Recovery Time Objective) is how long the business can tolerate the service being down. RPO (Recovery Point Objective) is how much recent data the business can tolerate losing, measured in time ("we can lose at most the last 5 minutes of writes"). Lower RTO and RPO cost more, because they need more infrastructure running before the disaster happens.

The industry vocabulary comes in four standard strategies, ordered from cheapest and slowest to most expensive and fastest: backup and restore, pilot light, warm standby, and multi-site active-active. A senior engineer's job is not to pick the fanciest one. It is to tier services by business impact, pick the cheapest strategy that meets each tier's RTO and RPO, and then prove with drills that the plan actually works. A DR plan that has never been exercised is a hope, not a plan.

This cluster covers the whole picture:

1. RTO, RPO and the four DR strategies, with the cost math behind them.
2. Backups that actually restore: snapshots, point-in-time recovery, immutable copies and restore drills.
3. Multi-AZ vs multi-region: blast radius, cell-based architecture and static stability.
4. Active-active vs active-passive data: home regions, write routing, conflicts.
5. Region failover in practice: runbooks, DNS and global load balancer cutover, failback and game days.

High availability vs disaster recovery vs backup

These three get mixed up constantly in interviews. They protect against different failures.

The key insight: replication is not backup. If someone runs DELETE FROM orders without a WHERE, a synchronous replica faithfully deletes everything within milliseconds. Only a backup taken before the delete, or a point-in-time log you can stop replaying before the bad statement, saves you.

flowchart TB
  a["1. Failure happens"]
  b["2. What broke?"]
  c["3a. HA: replicas in other AZs take over in seconds"]
  d["3b. DR: shift traffic to another region"]
  e["3c. Backup: restore to a point before the damage"]
  f["4. Users served again"]
  g["4b. Reconcile writes made after the damage"]
  a -->|continues| b
  b -->|one node or AZ| c
  b -->|whole region| d
  b -->|data itself is wrong| e
  c -->|continues| f
  d -->|continues| f
  e -->|continues| g
  g -->|continues| f

The four DR strategies compared

The ranges are planning guides, not guarantees. Your real RTO depends on detection time, decision time and how much of the runbook is automated.

Choosing the cheapest strategy that meets the targets (runnable)

# Pick the cheapest DR strategy that still meets a service's RTO and RPO targets.
# Numbers are illustrative planning ranges, not vendor quotes.
from dataclasses import dataclass

@dataclass
class Strategy:
    name: str
    rto_min: float          # typical worst-case recovery time, minutes
    rpo_min: float          # typical worst-case data loss window, minutes
    standing_cost: float    # monthly cost as a multiple of one production region (1.0 = a full copy)

STRATEGIES = [
    Strategy("backup-and-restore", rto_min=12 * 60, rpo_min=24 * 60, standing_cost=0.05),
    Strategy("pilot-light",        rto_min=60,      rpo_min=5,       standing_cost=0.15),
    Strategy("warm-standby",       rto_min=15,      rpo_min=1,       standing_cost=0.40),
    Strategy("active-active",      rto_min=1,       rpo_min=0.1,     standing_cost=1.00),
]

def pick(rto_target_min: float, rpo_target_min: float) -> Strategy:
    # Keep only strategies that meet BOTH targets, then take the cheapest one.
    ok = [s for s in STRATEGIES if s.rto_min <= rto_target_min and s.rpo_min <= rpo_target_min]
    if not ok:
        raise ValueError("no strategy meets these targets; revisit the requirement or the design")
    return min(ok, key=lambda s: s.standing_cost)

services = {
    "internal-wiki":   (24 * 60, 24 * 60),  # a day of downtime and a day of lost edits is tolerable
    "order-history":   (4 * 60, 15),        # hours of downtime OK, but only minutes of data loss
    "checkout":        (15, 1),             # revenue path: minutes, near-zero loss
    "payments-ledger": (2, 0.5),            # must survive a region loss almost transparently
}
for svc, (rto, rpo) in services.items():
    s = pick(rto, rpo)
    print(f"{svc:16} RTO<={rto:>6.0f}m RPO<={rpo:>6.1f}m -> {s.name:18} (~{s.standing_cost:.2f}x region cost)")

Output:

internal-wiki    RTO<=  1440m RPO<=1440.0m -> backup-and-restore (~0.05x region cost)
order-history    RTO<=   240m RPO<=  15.0m -> pilot-light        (~0.15x region cost)
checkout         RTO<=    15m RPO<=   1.0m -> warm-standby       (~0.40x region cost)
payments-ledger  RTO<=     2m RPO<=   0.5m -> active-active      (~1.00x region cost)

Notice that most services in a typical company do not need active-active. Paying a full extra region for an internal wiki is waste; under-protecting the payments ledger is a business risk.

RTO is a budget, not a script runtime (runnable)

// RTO is not "how fast the failover script runs". It is the sum of every phase
// from the moment users are hurt until they are served again.
type Phase = { name: string; minutes: number; automatable: boolean };

const manualPlan: Phase[] = [
  { name: "detect (alerts fire, on-call paged)", minutes: 5, automatable: true },
  { name: "triage + decide to fail over",        minutes: 20, automatable: false },
  { name: "promote DB replica in region B",      minutes: 5, automatable: true },
  { name: "scale app tier in region B",          minutes: 15, automatable: true },
  { name: "shift traffic (DNS TTL drain)",       minutes: 5, automatable: true },
  { name: "validate (smoke tests, dashboards)",  minutes: 10, automatable: false },
];

function total(plan: Phase[]): number {
  return plan.reduce((sum, p) => sum + p.minutes, 0);
}

const rtoTarget = 30;
console.log(`manual plan total: ${total(manualPlan)} min (target ${rtoTarget})`);

// Warm standby removes the scale-up phase; a pre-approved runbook shrinks the decision.
const improved = manualPlan.map((p) =>
  p.name.startsWith("scale") ? { ...p, minutes: 0 } :
  p.name.startsWith("triage") ? { ...p, minutes: 5 } : p,
);
console.log(`warm standby + pre-approved decision: ${total(improved)} min`);
const biggest = [...improved].sort((a, b) => b.minutes - a.minutes)[0];
console.log(`largest remaining phase: ${biggest.name} (${biggest.minutes} min)`);

Output:

manual plan total: 60 min (target 30)
warm standby + pre-approved decision: 30 min
largest remaining phase: validate (smoke tests, dashboards) (10 min)

Press Run. Snippets must be self-contained — no network, files, or native modules.

The biggest win usually comes from cutting decision time: a pre-agreed rule like "if region health checks fail for 10 minutes and the provider confirms a regional event, the incident commander fails over without further approval".

Why tiering beats one strategy for everything, and what happens if you choose otherwise

• Everything active-active: you pay for double capacity everywhere, every team must solve multi-region data conflicts, and the complexity itself causes outages. Most companies that try this end up with a few truly active-active paths and a lot of accidental single-region dependencies.
• Everything backup and restore: cheap, but a region outage becomes a day-long outage for revenue-critical paths, and restoring dozens of services at once overwhelms the recovery region and the on-call team.
• Tiered (for example tier 0 active-active, tier 1 warm standby, tier 2 pilot light, tier 3 backups only): spend follows business impact. The cost is discipline: every service needs an owner who knows its tier and its dependencies, because a tier 0 service that synchronously calls a tier 3 service inherits tier 3 recovery.

The dependency trap

A service's real RTO is the worst RTO of anything it needs in the request path. Common hidden single-region dependencies:

1. Identity and secrets: the login service, IAM, KMS keys or a secrets manager that only exists in the primary region.
2. Control planes: the DR plan needs to launch instances or change DNS, but those APIs are degraded during the very regional event you are escaping (AWS us-east-1, December 2021, is the classic example).
3. CI/CD and container registries: you cannot deploy into the recovery region if images only live in the primary.
4. Observability: dashboards and alerting hosted in the failed region go dark exactly when you need them.
5. Third parties: a payment provider or SaaS API that is itself single-region.

Pros and cons at a glance

Interview Q&A

Check yourself

Elsewhere in the library

These pages stay as they are. This lesson only points at them: Database replication, Global load balancing, Resilience patterns, SLOs and error budgets, Load, chaos, and production validation.

Go Deeper

• AWS whitepaper: Disaster recovery options in the cloud
• AWS Well-Architected Reliability Pillar
• Google SRE book: Data Integrity
• AWS summary of the December 2021 us-east-1 event