Region Failover in Practice - Runbooks, DNS/GLB Cutover, Failback & Game Days

Overview

A region failover is an incident procedure, not a button. It has a trigger (who decides, based on what signals), a sequence (fence, promote data, scale, shift traffic, validate), a traffic mechanism (DNS, anycast, a global load balancer, or client-side routing), and a return path (failback, which is usually riskier than failover because data has to be resynchronized in the opposite direction). The only way to know it works is to rehearse it: scheduled DR drills, game days and, at the extreme, deliberately evacuating a region with real traffic the way Netflix's Chaos Kong exercises did.

Automatic vs manual failover

A common pattern: traffic shifting for stateless tiers is automatic, database promotion needs one human approval from a pre-authorized incident commander, and failback is always manual.

A failover controller with hysteresis (runnable)

# Automated region failover with hysteresis: fail over only after N consecutive bad checks,
# fail back only after M consecutive good checks AND a human approval. Prevents flapping.
FAIL_AFTER, RECOVER_AFTER = 3, 5

def run(checks, approve_failback_at=None):
    state, bad, good, log = "PRIMARY_ACTIVE", 0, 0, []
    for t, healthy in enumerate(checks):
        if state == "PRIMARY_ACTIVE":
            bad = 0 if healthy else bad + 1
            if bad >= FAIL_AFTER:
                state, good = "FAILED_OVER", 0
                log.append(f"t={t}: {bad} bad checks -> FAIL OVER to secondary")
        elif state == "FAILED_OVER":
            good = good + 1 if healthy else 0
            if good >= RECOVER_AFTER:
                state = "READY_TO_FAILBACK"
                log.append(f"t={t}: primary healthy {good}x -> ready, waiting for approval")
        elif state == "READY_TO_FAILBACK":
            if not healthy:
                state, good = "FAILED_OVER", 0
                log.append(f"t={t}: primary flapped -> stay on secondary")
            elif approve_failback_at is not None and t >= approve_failback_at:
                state, bad = "PRIMARY_ACTIVE", 0
                log.append(f"t={t}: approved -> FAIL BACK (after data resync)")
    return state, log

# primary health over time: blip, real outage, flapping recovery, stable recovery
checks = [1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1]
final, log = run([bool(c) for c in checks], approve_failback_at=21)
print("\n".join(log))
print("final state:", final)

Output:

t=6: 3 bad checks -> FAIL OVER to secondary
t=12: primary healthy 5x -> ready, waiting for approval
t=13: primary flapped -> stay on secondary
t=18: primary healthy 5x -> ready, waiting for approval
t=21: approved -> FAIL BACK (after data resync)
final state: PRIMARY_ACTIVE

Requiring several consecutive bad checks avoids failing over on a single blip, and requiring a longer run of good checks plus approval before failback avoids bouncing back into a region that is still unstable.

flowchart TB
  primaryactive["PrimaryActive"]
  failedover["FailedOver"]
  readytofailback["ReadyToFailback"]
  resyncing["Resyncing"]
  primaryactive -->|1. N consecutive failed health checks| failedover
  failedover -->|2. M consecutive healthy checks| readytofailback
  readytofailback -->|3. Primary flaps again| failedover
  readytofailback -->|4. Human approves failback| resyncing
  resyncing -->|5. Data caught up, traffic shifted back| primaryactive

The failover runbook, step by step

1. Declare the incident and name an incident commander. Confirm the scope (provider status page, your own synthetic checks from outside the region).
2. Freeze deploys and risky batch jobs everywhere.
3. Fence the failing primary so it cannot keep accepting writes: revoke credentials, block its network path, or bump the fencing epoch.
4. Promote data stores in the recovery region (database replicas, queue consumers, caches warmed from the database).
5. Scale the recovery region to full capacity if it is not already statically provisioned.
6. Shift traffic with the global load balancer, DNS or anycast change.
7. Validate with synthetic transactions and business metrics (orders per minute), not just green health checks.
8. Communicate status to customers and internal teams.
9. Plan failback only after the primary is stable, data is resynchronized and there is a quiet window.

How traffic actually moves (runnable)

// DNS-based failover is not instant: resolvers and clients cache the old answer for up to the TTL,
// and some clients ignore TTLs entirely. Estimate how much traffic still hits the dead region.
function stillOnOld(minutesAfterChange: number, ttlMin: number, stickyFraction: number): number {
  // Caches expire roughly uniformly over the TTL window; sticky clients (JVMs with infinite
  // DNS cache, long-lived connections) stay until restarted or the connection breaks.
  const caching = Math.max(0, 1 - minutesAfterChange / ttlMin) * (1 - stickyFraction);
  return caching + stickyFraction;
}

for (const ttl of [1, 5, 60]) {
  const row = [0, 1, 5, 15, 60].map((m) => `${m}m:${(100 * stillOnOld(m, ttl, 0.03)).toFixed(0)}%`);
  console.log(`TTL ${String(ttl).padStart(2)} min -> ${row.join("  ")}`);
}
console.log("anycast / global load balancer health checks move traffic without waiting on client DNS caches");

Output:

TTL  1 min -> 0m:100%  1m:3%  5m:3%  15m:3%  60m:3%
TTL  5 min -> 0m:100%  1m:81%  5m:3%  15m:3%  60m:3%
TTL 60 min -> 0m:100%  1m:98%  5m:92%  15m:76%  60m:3%
anycast / global load balancer health checks move traffic without waiting on client DNS caches

Press Run. Snippets must be self-contained — no network, files, or native modules.

Prefer mechanisms whose data plane keeps working when a provider's control plane is impaired. Some providers offer dedicated recovery controls for this (for example AWS Route 53 Application Recovery Controller), designed so that flipping traffic does not depend on the region you are leaving.

Failback is the dangerous half

During failover you knew exactly what you wanted: get out. During failback you are moving traffic back to a region that just failed, and data written in the recovery region must flow back first. Common mistakes:

• Reversing replication without a clean cutover point, so writes from both sides interleave.
• Failing back during peak traffic because the team is eager to return to normal.
• Forgetting the writes stranded in the old primary during the lag window; they must be reviewed and replayed or discarded deliberately.

Many teams simply make the recovery region the new primary and fail back days later as a planned change.

Drills and game days

Google's DiRT program and Netflix's region evacuations are the well-known examples. Start small: one service, one dependency, a staging environment, a written hypothesis ("checkout keeps working when the primary DB is promoted"), and a scheduled time with everyone watching.

Lessons from real incidents

1. Facebook, October 2021: a maintenance command accidentally disconnected their backbone network, their DNS servers then withdrew their BGP routes, and all services went offline for hours. Internal tools and even physical access depended on the same network, which slowed recovery. Lesson: keep out-of-band access and tooling that does not depend on production.
2. AWS us-east-1, December 2021: an internal network event impaired control plane APIs and monitoring in the region for hours. Teams whose failover required calling those APIs struggled. Lesson: static stability and recovery paths outside the failing region.
3. GitLab, January 2017: during an incident, an engineer ran a delete on the wrong database host, and several backup paths were broken. Lesson: guardrails on destructive commands, and tested restores.

What happens if you skip this

• The first real failover is also the first attempt, under stress, at 3 a.m.
• Runbooks reference dashboards, scripts and people that no longer exist.
• Capacity in the recovery region is not there when needed.
• Failback causes a second outage.

Pros and cons

Interview Q&A

Check yourself

Elsewhere in the library

These pages stay as they are. This lesson only points at them: Global load balancing, L4 and L7 load balancing, Failover and fencing, Load, chaos, and production validation.

Go Deeper

• AWS Route 53 Application Recovery Controller
• ACM Queue: Weathering the Unexpected (Google DiRT)
• Meta engineering: More details about the October 4 outage
• Principles of Chaos Engineering