Security
Part 3 of 6 · Secrets & KMSSecret Stores Compared — Vault, Cloud Secrets Manager & Kubernetes Secrets
Picking a secret store is an architecture choice: dynamic leases in Vault, a managed cloud Secrets Manager, or Kubernetes-native Secrets plus CSI. This lesson compares trust boundaries, auth to the store, encryption at rest, HA, and anti-patterns — without rehashing OAuth grant types or full RBAC engines.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Source of truth versus a pod cache
Prefer
Vault or cloud Secrets Manager, projected in
The store is the system of record. Kubernetes receives a short-lived projection. Dynamic credentials expire on purpose.
- Audit lives on the store, not only in etcd.
- Version stages support dual-read.
- Identity to the store is a ServiceAccount or cloud IAM.
- A deleted pod does not delete the source secret.
Alternative
A Kubernetes Secret as the vault
Native and easy to kubectl. Default storage is base64 in etcd unless you turn encryption on.
- list, get, and watch on Secrets is production-admin power.
- No dynamic database users.
- The same object copied across namespaces crosses environments.
- kubectl output pasted into a ticket is a leak.
Overview
Picking a secret store is an architecture choice. Vault gives dynamic leases and Transit. A cloud Secrets Manager stores and rotates static secrets behind the cloud IAM you already operate. Kubernetes Secrets distribute bytes into pods. Senior interviews want the trust boundary, how the workload authenticates, encryption at rest, and the anti-pattern that made the last incident boring.
This page does not re-teach OAuth grant types. How a pod proves who it is belongs on the OAuth 2.1 & OIDC hub. Who may read a path belongs on the Authorization hub.
Comparison matrix
| Dimension | HashiCorp Vault | AWS SM or GCP Secret Manager | Kubernetes Secrets |
|---|---|---|---|
| Primary job | Dynamic secrets, Transit, policies | Store and rotate static secrets | Distribute to pods |
| AuthN to the store | AppRole, Kubernetes auth, cloud IAM, JWT | Cloud IAM or workload identity federation | ServiceAccount plus RBAC |
| Dynamic DB or cloud creds | First-class leases | Limited, often a rotation function | No, unless an external store provides them |
| Encryption at rest | Seal/unseal and the storage backend | Provider-managed, with CMK options | etcd encryption config required |
| Ops burden | High: HA, unseal, upgrades | Low | Medium: cluster plus CSI |
| Multi-cloud | Strong | Weak (per cloud) | Cluster-local |
When to pick what
Decisions
- 1
Secret store choice
- nextNeed dynamic short-lived creds?
- ?
Need dynamic short-lived creds?
- YesVault, or cloud IAM roles via workload identity
- No, static config secretsSingle cloud and mature IAM?
- 3
Vault, or cloud IAM roles via workload identity
- nextInject via agent, CSI, or runtime SDK
- ?
Single cloud and mature IAM?
- YesCloud Secrets Manager
- Multi-cloud or strict policyVault
- 5
Cloud Secrets Manager
- nextInject via agent, CSI, or runtime SDK
- 6
Vault
- nextInject via agent, CSI, or runtime SDK
- 7
Inject via agent, CSI, or runtime SDK
- nextKubernetes Secret only as a cache projection
- 8
Kubernetes Secret only as a cache projection
Lesson map
Secret Stores Compared — Vault, Cloud Secrets Manager & Kubernetes Secrets
Picking a secret store is an architecture choice: dynamic leases in Vault, a managed cloud Secrets Manager, or Kubernetes-native Secrets plus CSI. This lesson compares trust boundaries, auth to the store, encryption at rest, HA, and anti-patterns — without rehashing OAuth grant types or full RBAC engines.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB a["Secret store choice"] b["Need dynamic short-lived creds?"] c["Vault, or cloud IAM roles via workload identity"] d["Single cloud and mature IAM?"] a -->|Secret store choice| b b -->|Yes| c b -->|No, static| d
Envelope encryption for data keys is the previous lesson. How the pod receives the value is injection.
Vault
- Transit engine. Encryption as a service. The app never sees DEKs. Pair it with the envelope pattern when you want the KMS-shaped API inside Vault.
- Leases and renewal. Database credentials expire. A missed renew is an intentional failure, not a surprise outage. Design the app to renew or fail closed.
- Policies. Path-based capabilities. You still need an identity mapped onto a policy. That mapping decision is authorization; the store enforces the path.
Cloud Secrets Manager
- Native rotation hooks (Lambda or Cloud Functions).
- Version stages such as AWSCURRENT and AWSPENDING encode the dual-read handshake. The rotation lesson uses that handshake.
- IAM conditions (source VPC, MFA for humans) reduce exfiltration. Those conditions are not a substitute for a short TTL.
Kubernetes Secrets
- A default Secret is base64 in etcd, not encrypted, unless EncryptionConfiguration uses a KMS provider.
- Prefer the Secrets Store CSI Driver or External Secrets Operator syncing from Vault or a cloud Secrets Manager.
- RBAC on list, get, and watch of Secrets is powerful. Treat it like production admin. Developers in prod do not get a standing get.
Anti-patterns
- Long-lived cloud access keys inside Kubernetes Secrets. Use workload identity (IRSA or equivalent) instead. Identity issuance stays on the OAuth hub.
- The same Secret object across prod and stage namespaces.
- Logging
kubectl get secret -o yamlinto tickets. - Using Vault only as a static KV with no leases. A cloud Secrets Manager may be enough, and it will cost less to operate.
Runnable store facade
In-memory stand-ins. The source of truth versions the secret. The projection is a cache, not the system of record.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Version stages
AWSCURRENT and AWSPENDING are the cloud Secrets Manager form of dual-read. The app may prefer pending during a rollout. The verifier accepts either stage until you drop the old value.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
Is a Kubernetes Secret a vault?
Answer
No. It is a distribution cache. The source of truth should be Vault or a cloud Secrets Manager, with encryption and audit. The Secret object is how the pod gets a copy.
When is Vault worth the ops cost?
Answer
Dynamic secrets, more than one cloud, Transit, fine-grained path policies, and one audit trail across clouds. If you only need static key-value in a single cloud IAM, the managed Secrets Manager is usually enough.
How does a pod authenticate to Vault?
Answer
The Kubernetes auth method trades a ServiceAccount JWT for a Vault role and a short-lived token. Cloud IAM auth is the other common path. The token that comes back is leased. How workload identity is issued in general is the OAuth cluster, not this page.
What is the External Secrets Operator?
Answer
A controller that syncs external store values into Kubernetes Secret objects on a schedule. Those objects still need etcd encryption, RBAC, and a sync-age alert. The operator does not make the Secret the system of record.
Can a cloud Secrets Manager mint dynamic database users?
Answer
Partly, through rotation functions. Vault's database engine is the more native lease-based user. If the requirement is "a database user that dies in an hour," start with Vault or an equivalent dynamic engine.
What is seal and unseal?
Answer
Vault splits a master key. After a restart the server stays sealed until enough shares, or an auto-unseal KMS, restore it. Auto-unseal via cloud KMS is the common production setup. A sealed Vault is a planned outage if you forgot that step.
Why avoid secrets in container environment variables?
Answer
Children inherit them. docker inspect and core dumps can capture them. Prefer files or a runtime API. The injection lesson compares the options.
What does the Transit engine change?
Answer
The app sends plaintext to Vault and gets ciphertext back, or the reverse. It never handles the DEK. That is encryption as a service, closest to "KMS is the only thing that sees key material."
Pitfalls
The platform team runs Vault and every secret is a permanent KV entry with no lease. Argue for staying, or for moving static secrets to the cloud Secrets Manager and keeping Vault for database leases and Transit. Name the ops cost you are paying either way.