Security
Part 6 of 6 · Secrets & KMSSecrets Threat Model — Leakage Paths, Side Channels & Audit Trails
Even perfect KMS math fails if secrets leak through logs, CI, cores, tickets, or over-broad IAM. This lesson catalogs leakage paths, side channels, detection via audit trails, and incident response for credential exposure — closing the Secrets and KMS cluster.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Prevent and detect are both required
Prefer
A control that blocks the leak, plus a trail that shows the miss
Workload identity, short TTLs, redaction, and CMK deny stop classes of leaks. Decrypt spikes, new principals, and secret scanners catch what prevention missed.
- Short TTL limits how long a stolen value works.
- Log redaction and pre-commit scans stop the easy copies.
- KMS Decrypt by principal is the unwrap signal.
- Audit logs ship to an account the workload cannot delete.
Alternative
Correct AES and no telemetry
The envelope is sound. The secret still leaves through a debug print, a plan file, or a role that can GetSecretValue on production.
- TLS protects the wire and stops at the process.
- High-entropy scanners miss structured passwords.
- An admin can delete the only copy of the trail.
- Break-glass on a shared screen is a leak with a ticket id.
Overview
Perfect KMS math fails when the secret leaves through logs, CI, core dumps, tickets, or an IAM grant that was wider than the design. This lesson closes the cluster: leakage paths, side channels seniors forget, the audit events that show a DEK unwrap, and the first minutes of credential exposure.
Envelope structure is the hierarchy lesson. Where the value was injected is injection. How you mint version 2 while version 1 still works is rotation. OAuth token-theft and login CSRF are a different threat model on the OAuth hub. Who may call Get is the Authorization hub.
Threat model
| Threat | Example | Mitigation |
|---|---|---|
| Spoofing | A stolen cloud key impersonates the service | Short TTL, workload identity, MFA for humans |
| Tampering | An attacker edits wrapped DEK metadata | AAD or encryption context, integrity checks |
| Repudiation | No record of who decrypted | KMS, Secrets Manager, and Vault audit logs in an immutable sink |
| Information disclosure | The secret is in an exception message | Redaction and structured-logging allowlists |
| Denial of service | KMS disabled, or a mass revoke | Quotas, a brief DEK cache, multi-region CMKs |
| Elevation | A dev role can Get production secrets | Path policies, separate accounts, break-glass |
Leakage paths
Flow
- 1
Secret value
- nextApp logs and APM
- nextCI logs and artifacts
- nextCrash dumps and profiles
- nextShell history and tickets
- nextContainer images and layers
- nextOver-broad IAM read
- nextCompromised admin laptop
- 2
App logs and APM
- nextExfil
- 3
CI logs and artifacts
- nextExfil
- 4
Crash dumps and profiles
- nextExfil
- 5
Shell history and tickets
- nextExfil
- 6
Container images and layers
- nextExfil
- 7
Over-broad IAM read
- nextExfil
- 8
Compromised admin laptop
- nextExfil
- 9
Exfil
Lesson map
Secrets Threat Model — Leakage Paths, Side Channels & Audit Trails
Even perfect KMS math fails if secrets leak through logs, CI, cores, tickets, or over-broad IAM. This lesson catalogs leakage paths, side channels, detection via audit trails, and incident response for credential exposure — closing the Secrets and KMS cluster.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB s["Secret value"] l1["App logs and APM"] l2["CI logs and artifacts"] l3["Crash dumps and profiles"] s -->|Secret value to App logs and APM| l1 s -->|Secret value to CI logs and artifacts| l2 s -->|Secret value to Crash dumps and profiles| l3
Any one of those edges is enough. The diagram is a map of copies, not a procedure for making one.
Side channels seniors forget
- Debug flags that print config structs, including secrets.
- Verbose HTTP clients that log Authorization headers.
- Core dumps on OOM that contain heap secrets.
- Browser extensions or a screen share during break-glass.
- Terraform state that stores secret resource values.
- Spark or ETL explain plans that embed connection strings.
TLS protects bytes in transit. The endpoints still hold the secret in memory, logs, and backups. A scanner on git history does not see the explain plan.
Audit trails that matter
- KMS Decrypt events by principal, key id, and count spikes. This is the DEK unwrap signal. Vault Transit decrypt is the same idea.
- Secret Get or Vault read by path and identity.
- IAM policy changes that grant secret read.
- CI job ids that assumed a cloud role through OIDC. The federation design lives on the OAuth hub. Here you only ask which job id received the role.
- Break-glass approvals and their expiry.
Alert on decrypt volume anomalies, a read from a new geography or principal, and a Get that follows a burst of failed auth.
Detect versus prevent
| Control | Prevent | Detect |
|---|---|---|
| Workload identity | Yes, removes long-lived keys | IAM anomalies |
| Short TTL | Limits blast radius | Renew failures |
| Log redaction | Yes, if the allowlist holds | Secret scanning in log pipelines |
| KMS CMK deny | Yes, cuts decrypt | CloudTrail Decrypt denies and spikes |
| Pre-commit secret scan | Yes, on the commit path | Still scan git history |
High-entropy detectors miss structured passwords. Pair scanners with the prevent column. A green scanner is not a reason to skip redaction.
Incident response
- Contain. Revoke the IAM principal, disable the secret version, or block the CI role.
- Rotate. Mint version 2, dual-accept, and expire version 1 as fast as the overlap rules allow. Those rules are the rotation lesson.
- Scope. Audit who read since the start time. Treat those identities as burned.
- Eradicate. Remove the leak source: the log line, the image tag, the ticket.
- Recover. Redeploy with runtime fetch. Write the postmortem around the detection gap, not only the rotated value.
Crypto-shred a tenant by deleting or disabling their CMK or wrapped DEKs so ciphertext is unrecoverable. Do that only when destruction is the intent. Backups of wrapped keys undo it.
Runnable redaction
Deny-list field names and a JWT-shaped pattern. This does not make logging safe by itself. It shows the allowlist mindset: known secret keys become a mask, and token-shaped strings in other fields get masked too. The sample values are placeholders.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Deny-list logger
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
Name five secret leakage paths.
Answer
App logs, CI artifacts, container images, Terraform state, over-broad IAM, and crash dumps. Any five of those will do. Each one copies the value after the KMS has done its job.
What audit event shows a DEK unwrap?
Answer
KMS Decrypt, or Vault Transit decrypt, attributed to a principal. Alert when the count or the principal set leaves the baseline. A storage Get of ciphertext is not the same event.
How do you crypto-shred a tenant?
Answer
Delete or disable that tenant's CMK, or delete the wrapped DEKs, including backups. The ciphertext stays and cannot be unwrapped. Do this when offboarding requires destruction, and confirm replicas before you call it done.
Why do audit sinks have to be immutable?
Answer
An attacker with admin rights often deletes logs first. Write audit events to an append-only store in a separate account the workload cannot change.
What are the limits of secret scanning?
Answer
High-entropy detectors miss structured passwords and short tokens. Scan anyway, including git history, and keep prevent controls: redaction, short TTLs, and no secrets in state files.
What are the first minutes after a cloud key leak?
Answer
Disable the key. Read the trail for who used it and which resources they created. Rotate dependent secrets. Assume identities that read the secret since the start time are burned.
How do side channels beat TLS?
Answer
TLS ends at the process. The secret still sits in memory, logs, core dumps, tickets, and backups. Transit encryption does not redact a debug print.
What metric catches exfiltration?
Answer
A sudden multi-region secret Get by one role, or Decrypt QPS far above the historical baseline. Pair it with "read from a principal that has never decrypted this key."
Pitfalls
CloudTrail shows Decrypt QPS on one CMK jumped tenfold, from a role that has never used that key, in a second region. Order the contain, scope, and rotate steps. Say which identities you treat as burned and why a storage read of ciphertext is a different event.