Security
Part 4 of 6 · Secrets & KMSCredential Rotation — Dual-Read, Overlap Windows & Break-Glass
Rotation without downtime needs a dual-read overlap, version stages, and a rehearsed break-glass path. This lesson covers overlap windows, consumer lag, database password patterns, API key versioning, and what fails when only half the fleet has the new secret.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Overlap versus a flip
Prefer
Mint v2, accept both, then retire v1
Rolling deploys, clock skew, and cached clients all still succeed. You watch the v1 authentication rate hit zero before you expire it.
- Verifiers dual-accept for the whole window.
- Clients prefer the newest non-expired version.
- Cache TTL is at most half the overlap.
- Break-glass is a state in the same machine, not a shared password.
Alternative
Replace the secret at one timestamp
Clear on a whiteboard. Pods that have not restarted, and clients that cached the old value, fail closed in production and open a flood of retries.
- Half the fleet still sends version 1.
- SDK caches that never expire outlive the cut.
- External Secrets sync lag shows the old value after you think you flipped.
- The rollback is another hard cut.
Overview
Rotation without downtime needs an overlap window, version stages, and a break-glass path you have rehearsed. A single cutover timestamp fails because rolling deploys, clock skew, and cached clients do not share one clock. Prefer four steps: mint version 2 while version 1 is still valid, dual-accept on verifiers and dual-read on clients, watch success metrics, and expire version 1 after the longest deploy, the longest cache TTL, and a safety margin.
Store version stages (AWSCURRENT and AWSPENDING) are how a cloud Secrets Manager encodes this. See secret stores. CMK version overlap is the same idea applied to wrapped DEKs on the envelope page.
How the caller proved identity before presenting a key is the OAuth 2.1 & OIDC hub. Who may start a rotation or open break-glass is the Authorization hub. This page does not re-teach grant flows or policy engines.
Why a naive flip fails
Healthy pods 401 because they still hold version 1 after you deleted it. The fix is temporal, not a faster deploy:
- Mint version 2 while version 1 is still valid.
- Dual-accept on verifiers, and dual-read on clients.
- Observe success metrics, including authentications that still present version 1.
- Expire version 1 after max(deploy time, daemon cache TTL, async worker lag) plus a safety margin.
Overlap state machine
States
- 1
Start → Stable v1
Start → Stable v1
- 2
Stable v1 → Minting v2
Stable v1 → Minting v2
rotate start
- 3
Minting v2 → DualAccept
Minting v2 → DualAccept
v2 published
- 4
DualAccept → Prefer v2
DualAccept → Prefer v2
clients flipped
- 5
Prefer v2 → Retire v1
Prefer v2 → Retire v1
lag below SLO
- 6
Retire v1 → Stable v2
Retire v1 → Stable v2
v1 expired
Lesson map
Credential Rotation — Dual-Read, Overlap Windows & Break-Glass
Rotation without downtime needs a dual-read overlap, version stages, and a rehearsed break-glass path. This lesson covers overlap windows, consumer lag, database password patterns, API key versioning, and what fails when only half the fleet has the new secret.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB stable_v1["Stable v1"] minting_v2["Minting v2"] dualaccept["DualAccept"] prefer_v2["Prefer v2"] stable_v1 -->|rotate start| minting_v2 minting_v2 -->|v2 published| dualaccept dualaccept -->|clients flipped| prefer_v2
Break-glass returns to dual-accept. It does not jump to "only the emergency password works." When the human unblocks automation, both versions are valid again until you deliberately retire version 1.
States
- 1
DualAccept → BreakGlass
DualAccept → BreakGlass
automation stuck
- 2
BreakGlass → DualAccept
BreakGlass → DualAccept
human unblocks
Rotation patterns
| Pattern | How | Pros | Cons |
|---|---|---|---|
| Dual-read client | Try v2, then v1 | Simple for pull clients | Needs ordered versions |
| Dual-accept server | Validate either | Fits API keys and key ids | Attack surface is wider during the overlap |
| Blue/green secret | Swap env on cutover | Clear moment | Needs a coordinated deploy |
| Dynamic lease (Vault) | Short TTL and renew | Continuous rotation | The app must handle renew failures |
| DB dual-user | User A and user B alternate | Pools can switch without a hard drop | App and pool must change user |
Break-glass
- Dual control: two humans, or a human plus a ticket.
- Time-boxed IAM or Vault policy escalation.
- Mandatory audit, and a page to security.
- Automatic expiry of the break-glass grant.
- After the incident, rotate everything that path could touch.
Break-glass left open is an incident of its own. An audit job has to catch grants past their expiry. Detection of that class of leak is the threat model.
Runnable overlap
Press Run. Snippets must be self-contained — no network, files, or native modules.
API key id rotation
Present the key id with the secret. A forged pair that mixes the new secret with the old id fails. During overlap both ids still verify against their own secret.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Failure modes during rotation
- Half the fleet is stale. Clients still send version 1 after you retire it, and 401s spike. Keep the overlap longer.
- Cache TTL longer than the overlap. Config SDKs that cache forever will miss the cut. Force max TTL to at most half the overlap.
- One-way sync lag. External Secrets delay means pods still see the old value. Monitor sync age.
- Break-glass left open. The emergency policy is never revoked. An audit job must catch it.
Interview Q&A
What is an overlap window?
Answer
The period when both the old and the new credential are valid, so rolling systems never hit a hard cut. Length is measured from the slowest consumer, not from the moment you clicked rotate.
Dual-read versus dual-accept?
Answer
Dual-read: consumers fetch the newest version and fall back to the older one. Dual-accept: verifiers accept either presented secret or key id. You usually need both. Clients pull. Servers check what was presented.
How do you rotate database passwords when pools hold connections?
Answer
Prefer two database users. Point the pool at user B, drain user A, then retire A. Vault dynamic credentials with renew are the other pattern. Flipping one password under a live pool drops connections that still present the old value.
How long should the overlap last?
Answer
At least the max of deploy time, daemon cache TTL, and async worker lag, plus a safety margin. Measure it with canaries. A guess that is shorter than the External Secrets sync period will 401 the pods that synced late.
What is break-glass?
Answer
Controlled emergency access when automation is stuck. Dual-control, audited, short-lived, and paged. Afterward, rotate every credential that path could read.
How do cloud Secrets Manager version stages help?
Answer
Stages such as AWSPENDING and AWSCURRENT encode the dual-read handshake for rotation functions. The app and the verifier can see which value is current and which value is still accepted without inventing a second store.
Can you rotate a CMK the same way?
Answer
Yes, conceptually. Publish a new key version, re-wrap DEKs, and keep the old version for decrypt until the backlog is done. The data plane dual-accepts key versions. That procedure is the envelope lesson. This page is the credential form of the same overlap.
What signal means retirement is safe?
Answer
Zero authentications with version 1 for N minutes across all regions, and no lagging sync ages. A green deploy is not that signal. The old credential can still be in a queue consumer that has not restarted.
Pitfalls
Ten minutes after you disabled version 1, a single region still 401s. Name the three lag sources you check first, and the overlap change you make before the next rotation.