Security
Part 1 of 6 · Secrets & KMSSecrets & KMS — Envelope Encryption, Rotation & Blast Radius
Credentials, API keys, and data-encryption keys are high-blast-radius assets. This hub is the senior-SWE decision map for where secrets live, how KMS wraps data keys (envelope encryption), how you rotate without downtime, and how apps fetch secrets at runtime — without re-teaching OAuth/OIDC token flows or RBAC/ABAC policy engines. Focus: Vault, cloud Secrets Manager, and Kubernetes Secrets tradeoffs, the DEK/KEK/CMK hierarchy, dual-read rotation, injection paths, and leakage threat models.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Where a production secret is allowed to live
Prefer
Secret store + workload identity + short TTL
The value is minted or fetched for one identity, audited on read, and rotated with an overlap window. Local DX uses a personal IAM or agent session.
- KMS-backed envelope encryption for data at rest.
- Short-lived, identity-bound credentials for service calls.
- Kubernetes Secrets only as a projection of a real store.
- Break-glass is time-boxed, dual-control, and paged.
Alternative
A .env checked in and shipped
Fast for a throwaway prototype. The same file becomes git history, an image layer, a crash dump, and a support ticket.
- No rotation, no read audit, no per-identity binding.
- Process lists, child processes, and forks can see it.
- One leaked laptop is the whole environment.
- Fine for a prototype you will throw away the same day.
Pick the control from the asset
Same question in the diagram below.
- 1
Name the asset
Long-lived data at rest wants envelope encryption. A runtime credential or API key wants a store and a TTL. - 2
Name the consumers
Many services want a central store and dynamic leases. One Kubernetes workload asks whether the pod must see a synced file. - 3
Choose injection
Sync into the pod via CSI or an agent when the app only reads files. Otherwise fetch at runtime with a short cache TTL. - 4
Rotate with overlap
Mint the next version while the previous version is still accepted. Retire the old version after deploy lag and cache TTL. - 5
Audit every read and decrypt
Alert on anomalous decrypt volume, a new principal, or a break-glass grant that outlives its window.
Overview
Credentials, API keys, and data-encryption keys are high-blast-radius assets. Senior interviews ask where the secret lives, how a KMS wraps the data key, how you rotate without a 401 storm, and how the process actually receives the value. The interview question underneath all of those is: if this secret leaks tonight, how many systems can an attacker reach, and how fast can we rotate?
This cluster is that decision map. It does not re-teach how a client proves identity or how a policy engine allows a path. Those are the OAuth and Authorization clusters, linked below.
You should be able to:
- Separate data-encryption keys from runtime credentials.
- Say why a plaintext DEK never rests on disk.
- Walk a dual-read overlap without dropping healthy pods.
Why this is a product and ops decision
Interviewers probe five moves:
- Envelope encryption: a DEK encrypts the payload, a KEK or CMK wraps the DEK, and the plaintext DEK is not stored.
- Vault versus AWS, GCP, or Azure Secrets Manager versus Kubernetes Secrets, and the outage or leak each one fails open into.
- Credential rotation with dual-read, an overlap window, and break-glass.
- Injection: environment variables, sidecars or agents, CSI drivers, or a runtime fetch, and the blast radius of each.
- Audit trails and side channels: logs, core dumps, CI output, crash dumps.
Ask the blast-radius question before you name a product. A database password copied into twelve deployments is a different system than a per-tenant DEK wrapped by a customer CMK.
Decision matrix
Decisions
- 1
Need to protect a secret or encrypt data?
- Long-lived data at restEnvelope encryption via KMS
- Runtime credential or API keyWho consumes it?
- 2
Envelope encryption via KMS
- nextCMK in KMS, DEK per object or tenant
- ?
Who consumes it?
- Many services, dynamic credsCentral store and short TTL
- Single Kubernetes workloadNeed a file in the pod?
- 4
Central store and short TTL
- nextRotate with dual-read overlap
- ?
Need a file in the pod?
- YesCSI or agent inject, encrypt at rest
- No runtime mountRuntime fetch SDK and cache TTL
- 6
CSI or agent inject, encrypt at rest
- nextRotate with dual-read overlap
- 7
Runtime fetch SDK and cache TTL
- nextRotate with dual-read overlap
- 8
CMK in KMS, DEK per object or tenant
- 9
Rotate with dual-read overlap
- nextAudit every read and decrypt
- 10
Audit every read and decrypt
Lesson map
Secrets & KMS — Envelope Encryption, Rotation & Blast Radius
Credentials, API keys, and data-encryption keys are high-blast-radius assets. This hub is the senior-SWE decision map for where secrets live, how KMS wraps data keys (envelope encryption), how you rotate without downtime, and how apps fetch secrets at runtime — without re-teaching OAuth/OIDC token flows or RBAC/ABAC policy engines. Focus: Vault, cloud Secrets Manager, and Kubernetes Secrets tradeoffs, the DEK/KEK/CMK hierarchy, dual-read rotation, injection paths, and leakage threat models.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB a["Need to protect a secret or encrypt data?"] b["Envelope encryption via KMS"] c["Who consumes it?"] d["Central store and short TTL"] a -->|Long-lived data| b a -->|Runtime| c c -->|Many services,| d
Rule of thumb: never bake long-lived secrets into images or git. Prefer KMS-backed envelope encryption for data. Prefer short-lived, identity-bound credentials for service-to-service calls — how that identity is proven lives on the OAuth 2.1 & OIDC hub. Treat a Kubernetes Secret as a distribution mechanism, not a vault.
Where secrets live
| Approach | Strengths | Weaknesses | Prefer when |
|---|---|---|---|
| HashiCorp Vault | Dynamic secrets, leases, policies, Transit | Ops cost, HA and unseal complexity | Multi-cloud, dynamic database or cloud credentials |
| AWS, GCP, or Azure Secrets Manager | Managed, IAM-native, audit | Less dynamic than Vault Transit; vendor lock-in | Single-cloud apps already on that IAM |
| Cloud KMS plus envelope encryption | CMK control, HSM options, crypto ops | Not a full secret store by itself | Data-encryption keys and envelopes |
| Kubernetes Secrets | Native to pods, CSI sync | etcd has historically stored base64; needs RBAC and encryption at rest | Injecting into Kubernetes workloads only |
| Env vars or dotenv in git | Simple | Leaks via process lists, logs, and forks | Never for production secrets |
| CI OIDC short-lived tokens | No long-lived CI keys | Needs provider setup | CI/CD supply chain; issuance stays in the OAuth cluster |
Depth on each row lives on the sibling pages: key hierarchy, store choice, rotation, injection, and leakage.
Checked-in dotenv
Pros. Fast local DX. Works for a throwaway prototype.
Cons. The file ends up in git history, Docker layers, crash dumps, and support tickets. There is no rotation, no audit of who read what, and no binding to an identity. One leaked laptop is the entire environment.
Better alternative: a secret store, workload identity, and a short TTL. Local DX is a Vault agent or a cloud Secrets Manager CLI with personal IAM. The checked-in file never holds a production value.
Map of the cluster
| Part | Lesson | Focus | Neighbors |
|---|---|---|---|
| 1 | This hub | Strategy and blast radius | Prev: threat model. Next: envelope |
| 2 | Envelope encryption | DEK, KEK, CMK and the encrypt path | Prev: hub. Next: stores |
| 3 | Secret stores | Vault, cloud SM, Kubernetes | Prev: envelope. Next: rotation |
| 4 | Credential rotation | Dual-read, overlap, break-glass | Prev: stores. Next: injection |
| 5 | App secret injection | Env, sidecar, CSI, runtime fetch | Prev: rotation. Next: threat model |
| 6 | Secrets threat model | Leakage, side channels, audit | Prev: injection. Next: hub |
Identity and authorization stay in their clusters
This series stores and rotates secrets and keys. It does not re-teach grant flows or policy engines.
- OAuth 2.1 & OIDC — Authorization Code + PKCE covers how clients prove identity and receive short-lived tokens, including workload identity and CI federation. Pipeline signing and SBOMs sit next to that identity story. Do not duplicate them here.
- Authorization — RBAC, ABAC, ReBAC & Policy covers who may read a secret path. Policy engines decide. Secret stores enforce.
Minimal envelope sketch
Production bulk encryption is AES-GCM inside a boundary you control. The KMS holds the CMK and returns a wrapped DEK. This sandbox uses XOR so it runs with the standard library. The shape is the interview shape: ciphertext, wrapped DEK, and no plaintext DEK in the stored object. AAD binding is the next lesson.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Minimal dual-read sketch
During the overlap window the app prefers the new version and the verifier still accepts the old one. Retire version 1 only after the fleet, the caches, and the safety margin have moved. The full state machine is on the rotation page.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
What is envelope encryption?
Answer
Generate a data key (DEK) and encrypt the payload with it. Wrap that DEK with a KMS master key (KEK or CMK). Persist ciphertext plus the wrapped DEK. On decrypt, the KMS unwraps the DEK and the app decrypts locally. The CMK never leaves the KMS or HSM, so a stolen disk is ciphertext plus an opaque wrapped key.
Why not encrypt every byte directly with the CMK?
Answer
The CMK has request quotas, network round trips, and a huge blast radius if the decrypt grant is stolen. Per-object DEKs keep AES local and let you re-wrap DEKs on CMK rotation without rereading every plaintext byte in some designs. The hierarchy lesson walks re-wrap versus re-encrypt.
Vault or a cloud Secrets Manager?
Answer
Vault is the better fit for dynamic secrets, leases, and Transit across clouds. A cloud Secrets Manager is the simpler IAM-native store and rotator for static secrets when you already live in one cloud IAM. Kubernetes Secrets are neither of those; they project a value into a pod.
Are Kubernetes Secrets safe?
Answer
Only with etcd encryption at rest, tight RBAC, no log leakage, and preferably a CSI or external-secrets sync from a real store. A default Secret object is base64 in etcd. Base64 is encoding, not encryption.
What is dual-read rotation?
Answer
During an overlap window, writers mint version 2 while readers and verifiers still accept version 1 and version 2. Then you retire version 1. The overlap absorbs clock skew, rolling deploys, and cached clients. A single cutover timestamp 401s the pods that have not restarted yet.
What is the blast radius of env-var injection?
Answer
The value shows up in process listings, child processes, crash reports, and some APM agents. Prefer a tmpfs file mode 0600, a memory-only agent, or a runtime fetch with a short cache. Injection tradeoffs are the fifth lesson in this cluster.
What is break-glass?
Answer
An emergency path to read or rotate when automation is stuck. It is dual-control, time-bounded, audited, and alerted. It is not a daily backdoor. After use, rotate everything that path could touch.
Name one metric for secrets health.
Answer
Use several together: percent of secrets past max age, failed rotation rate, anomalous read spikes by identity, and time-to-revoke after an incident. A dashboard that only shows "secret exists" will not page you when a new principal starts decrypting.
Pitfalls
A laptop with a production database password is stolen. List the systems that password can reach, the audit query that shows who else read it, and the overlap steps you run before you disable version 1.