vault

Secrets Best Practices

14 min read

Secrets Best Practices

A practices note — a synthesis of the secrets-handling guidance scattered across the Kubernetes vault into a single prioritized checklist with rationale. The starting premise, stated bluntly in the Secret note and in the Kubernetes documentation, is that a Kubernetes Secret by itself protects almost nothing: “Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd” (k8s.io — Good practices for Kubernetes Secrets). A Secret is base64 encoding (trivially reversible), an immutable flag, a tmpfs mount, and a convention that says “treat this carefully.” Real protection is a layered architecture: encryption at rest, tight RBAC, an external source of truth, careful consumption modes, and least-privilege service accounts. This note ranks those layers by leverage — the items at the top stop the most attackers per unit of effort — and explains the why behind each so the checklist is not cargo-culted. It deliberately does not re-derive the mechanics of any one tool; each item cross-links to the dedicated note. Treat this as the page you re-read before a security review of a cluster.

Mental Model

flowchart TB
    subgraph THREATS["Who can read your secret?"]
        T1["Someone with<br/>'get secrets' RBAC"]
        T2["Someone with a<br/>stolen etcd snapshot"]
        T3["Someone reading<br/>/proc/PID/environ"]
        T4["Someone reading<br/>your Git history"]
        T5["A compromised<br/>node's kubelet"]
    end
    subgraph DEFENSES["The layer that stops them"]
        D1["RBAC audit +<br/>least-privilege SAs"]
        D2["etcd encryption<br/>at rest (KMS)"]
        D3["File mounts, not<br/>env vars"]
        D4["External secret<br/>manager / Sealed Secrets"]
        D5["Node isolation +<br/>scoped SA tokens"]
    end
    T1 --> D1
    T2 --> D2
    T3 --> D3
    T4 --> D4
    T5 --> D5
    D1 -.->|"highest leverage"| RANK[Prioritized checklist]
    D2 -.-> RANK
    D3 -.-> RANK
    D4 -.-> RANK
    D5 -.-> RANK

What this diagram shows. Each threat against a Secret is stopped by a specific defensive layer — there is no single control that addresses all of them. A cluster with perfect etcd encryption but sloppy RBAC is wide open to threat T1; a cluster with locked-down RBAC but plaintext etcd is wide open to T2. The insight to extract: secrets security is a checklist, not a switch. You must reason about each threat independently and confirm the corresponding layer is actually present. The rest of this note is that checklist, ordered by leverage — RBAC first because it is the threat most clusters fail and the cheapest to fix.

The Prioritized Checklist

1. RBAC is the real boundary — audit who has get secrets

The single most important fact about a Kubernetes Secret: it is exactly as protected as the Kubernetes RBAC rules governing the secrets resource in its namespace. Anyone with get, list, or watch on secrets can read every Secret in scope; list is especially dangerous because it returns all Secrets’ data in one call. The Kubernetes good-practices doc puts RBAC first for the same reason: “Restrict Secret access to specific containers … Enable encryption at rest … Configure least-privilege access to Secrets” (k8s.io).

The common production audit finding is a developer or CI ServiceAccount with get secrets cluster-wide — usually inherited from an over-broad ClusterRole like the built-in edit or admin role bound at cluster scope, or from a Helm chart that requested more than it needs. The remediation:

# Enumerate every subject that can read Secrets, namespace by namespace.
kubectl auth can-i list secrets --as=system:serviceaccount:payments:ci-deployer -n payments
# Or sweep all subjects with a tool such as `rbac-lookup` / `kubectl-who-can`:
kubectl who-can get secret --all-namespaces

Rules of thumb: prefer namespaced Role/RoleBinding over ClusterRole/ClusterRoleBinding; never grant list on secrets when get of a named Secret will do (RBAC supports resourceNames to scope a rule to specific Secret names); and remember that anyone who can create a Pod in a namespace can mount any Secret in that namespace — Pod-create is effectively Secret-read. This last point means namespace boundaries, not just RBAC verbs, are part of the secrets boundary.

2. etcd encryption at rest is not optional — use a KMS provider

By default the API server writes Secret values into etcd in their raw decoded form — not even base64-encoded. A stolen etcd snapshot, a compromised etcd node, or an unencrypted backup volume hands the attacker every credential in the cluster as plaintext. For any cluster holding real credentials, etcd Encryption at Rest is mandatory, and it must be configured with a KMS provider, not a local aescbc/aesgcm key.

The reason “KMS, not local keys” is non-negotiable: a local-key EncryptionConfiguration stores the AES key in a file on the control-plane node, usually right next to the etcd data it protects. An attacker who can read the etcd snapshot can almost always read that file too, so local-key encryption mostly defends against the lost-disk scenario and little else. A KMS provider (AWS KMS, GCP Cloud KMS, Azure Key Vault, or HashiCorp Vault’s KMS plugin) does envelope encryption: a per-Secret data-encryption-key (DEK) is itself encrypted by a key-encryption-key (KEK) that never leaves the KMS. Compromising etcd alone is then insufficient — the attacker also needs live KMS access, which is independently audited and revocable.

Managed-Kubernetes services make this a one-line setting: EKS, GKE, and AKS all offer “encrypt Secrets with a customer-managed KMS key” as a cluster-creation option. Turn it on. Cross-link etcd Encryption at Rest for the provider-chain mechanics and the re-encrypt-all-Secrets migration procedure.

3. Prefer external secret managers for high-value credentials

For database master passwords, cloud API keys, signing keys, and anything whose compromise is a company-level incident, the in-cluster Secret object should not be the source of truth. The mature pattern is an external secret manager — HashiCorp Vault on Kubernetes, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — with the in-cluster Secret acting only as a cache, synced by the External Secrets Operator (which materializes external secrets into ordinary K8s Secrets) or injected directly by the Vault Agent sidecar (which bypasses the K8s Secret object entirely).

Why this is worth the operational cost:

  • Rotation is handled upstream, with overlapping validity windows, instead of being a manual kubectl rollout restart ritual (see Secret §rotation — the K8s API gives you no rotation help at all).
  • Dynamic secrets become possible: Vault can mint a per-Pod, short-lived database credential or AWS STS token that expires in minutes, shrinking the blast radius of any leak to near zero.
  • Audit is centralized: the secret manager logs every read, with identity, independently of the K8s audit log.
  • Blast radius: a compromised cluster does not compromise the secrets — they live elsewhere, behind a separate authentication boundary.

The in-cluster Secret API is fine for low-stakes credentials (image-pull tokens, dev-environment passwords). It is the wrong system of record for crown-jewel credentials.

4. Avoid env-var injection of secrets — prefer file mounts

When a Secret is consumed via env/envFrom, its value lands in the container process’s environment. Environment variables leak through an alarming number of channels:

  • /proc/<pid>/environ — readable by any process running as the same user, and by anything that can enter the Pod’s PID namespace.
  • Crash dumps and core files — a segfaulting process commonly serializes its full environment into the dump.
  • Child processes — every subprocess inherits the parent’s environment by default, so a shelled-out curl or a debugging tool sees the credential.
  • kubectl describe pod — the Pod spec’s env block is visible to anyone with get pods; even though secretKeyRef hides the value, many apps and Helm charts inject plaintext defaults.
  • Application logging frameworks — a surprising number dump process.env on startup or on error.

File mounts via a Secret volume avoid all of these: the kubelet writes the Secret into a tmpfs filesystem (memory-backed, never on node disk), the file is readable only by the configured defaultMode/fsGroup, and it does not propagate into child processes or crash dumps. The Kubernetes good-practices doc recommends volume mounts over env vars for exactly this reason. As a bonus, volume-mounted Secrets live-update when the Secret changes (within the kubelet sync period), whereas env vars are frozen at container start. Use secretKeyRef env vars only for legacy applications that genuinely cannot read a file, and even then prefer a small init wrapper that reads the file and execs the app.

5. Use projected bound service-account tokens, not static token Secrets

Before Kubernetes 1.24, every ServiceAccount automatically generated a long-lived kubernetes.io/service-account-token Secret: a JWT with no expiry, no audience binding, and no tie to a Pod’s lifetime. If exfiltrated, it was valid forever and usable from anywhere. KEP-driven changes (the LegacyServiceAccountTokenNoAutoGeneration feature, stable by 1.24 — k8s.io 1.24 release notes) stopped auto-generating these.

The modern replacement is the projected bound service-account token (Projected Volume), mounted automatically into Pods by the kubelet:

  • Time-bound — short TTL (default ~1 hour), auto-refreshed in place by the kubelet.
  • Audience-scoped — the token’s aud claim names the intended recipient (e.g. the API server, or a specific aggregated API), so a token meant for service A is rejected by service B.
  • Object-bound — the token carries the Pod’s UID; when the Pod dies, the token is invalid even before its TTL expires.

Audit your cluster for lingering legacy long-lived token Secrets (kubectl get secrets --all-namespaces --field-selector type=kubernetes.io/service-account-token) and delete the orphans. They are auditable RBAC time bombs.

6. Never commit Secrets to Git — use Sealed Secrets or SOPS

Putting data: { password: c2VjcmV0MTIz } into a Git-tracked manifest is not encryption — it is base64, decodable by anyone with repo read access, and it persists in Git history forever even after you “delete” it. GitOps (ArgoCD, Flux) makes this trap easy to fall into because GitOps wants everything in Git.

Two safe ways to keep secrets in Git:

  • Sealed Secrets (Bitnami) — encrypt a Secret with the cluster’s public key into a SealedSecret custom resource. Only the in-cluster controller, holding the private key, can decrypt it into a real Secret. The encrypted blob is safe to commit; even a full repo compromise yields nothing without the cluster’s private key.
  • SOPS (Mozilla) — encrypt the values of a YAML/JSON file (leaving keys readable for diffs) using a KMS key, age, or PGP. Flux and Helmfile decrypt SOPS files at apply time. Good when you want encrypted manifests rather than a separate CR type.

The cleanest option of all is to not commit secrets in any form — point GitOps at an External Secrets Operator ExternalSecret manifest, which is itself non-sensitive (it only names where the real secret lives) and which the operator resolves against Vault/AWS SM at runtime.

7. Immutable Secrets, least-privilege SAs, automountServiceAccountToken: false

Three hardening defaults that cost nothing once set:

  • Immutable Secrets (immutable: true, GA since 1.21). Prevents accidental edits, and lets the kubelet stop watching the Secret — reducing API-server watch-stream load at scale. Pair with the hash-into-name pattern (the Secret’s name embeds a content hash; changing content means a new Secret name and an automatic Pod re-roll). The trade-off: immutable Secrets cannot be edited, only deleted and recreated — use deliberately.
  • Least-privilege ServiceAccounts. Do not run workloads under the namespace’s default ServiceAccount, and do not bind broad ClusterRoles to it. Create a dedicated ServiceAccount per workload with exactly the RBAC it needs. The default SA is shared by everything in the namespace; any RBAC granted to it is granted to all of them.
  • automountServiceAccountToken: false by default. Unless a Pod actually calls the Kubernetes API, it has no reason to carry a service-account token. Set automountServiceAccountToken: false on the ServiceAccount (or on the Pod spec) so the kubelet does not mount a token at all. A token that is never mounted cannot be stolen. Opt in for the minority of Pods (operators, controllers, in-cluster API clients) that genuinely need it.

8. Image-pull-secret hygiene

imagePullSecrets (kubernetes.io/dockerconfigjson Secrets) carry registry credentials. They are real secrets and deserve the same treatment:

  • Scope them narrowly. A pull secret attached to a ServiceAccount is usable by every Pod under that SA. Prefer a dedicated SA per team/namespace rather than one cluster-wide pull credential.
  • Prefer keyless / short-lived registry auth. On managed clouds, bind the node or the workload to a cloud IAM identity (Workload Identity (Kubernetes), IRSA, GKE Workload Identity) so the kubelet pulls from the cloud’s own registry (ECR, Artifact Registry, ACR) using short-lived tokens — no static pull Secret at all. Tools like the ECR credential helper or kubelet’s credential-provider plugins do this transparently.
  • Rotate registry credentials like any other; a leaked pull secret lets an attacker pull (and sometimes push) images, enabling supply-chain attacks.
  • Do not reuse a human’s registry login as the cluster pull secret — use a dedicated robot account scoped to read-only pull on exactly the needed repositories.

Failure Modes

  • “We enabled etcd encryption but old Secrets are still plaintext.” Encryption-at-rest only encrypts on write. After enabling it you must force-rewrite every existing Secret (kubectl get secrets -A -o json | kubectl replace -f -) so they re-persist encrypted. See etcd Encryption at Rest.
  • Over-broad list secrets granted via a ClusterRoleBinding to a CI tool. The CI token, if leaked, reads every Secret in the cluster. Symptom found only by RBAC audit; no runtime alarm.
  • Secret leaked through a crash dump uploaded to an error-tracking SaaS. Env-var consumption + a crash reporter that serializes the environment = credential shipped to a third party. Mitigation: file mounts (item 4).
  • GitOps drift on a hand-edited Secret. If a Secret is managed by GitOps but also hand-edited with kubectl, the GitOps controller either reverts the edit or flags permanent drift. Decide on one owner; do not split.
  • Legacy long-lived SA token never revoked after an employee offboards. Pre-1.24 token Secrets have no expiry; they survive forever. Audit and delete (item 5).
  • Sealed Secret unsealable after a cluster rebuild. The Sealed Secrets controller’s private key is cluster-specific; rebuild the cluster without backing up that key and every committed SealedSecret becomes undecryptable. Back up the controller’s key, or treat the encrypted secrets as disposable.

Alternatives and When to Choose Them

ApproachSource of truthBest for
Plain Secret + RBAC + etcd encryptionetcdLow-stakes credentials; dev clusters
External Secrets OperatorVault / AWS SM / GCP SM / Azure KVHigh-value credentials needing central audit + rotation
HashiCorp Vault on Kubernetes (Agent Injector)VaultDynamic, per-Pod, short-lived secrets; strongest blast-radius reduction
Sealed SecretsGit (encrypted)GitOps shops wanting secrets versioned alongside manifests
SOPSGit (encrypted)GitOps with encrypted manifest files rather than a separate CR
Workload Identity (Kubernetes)Cloud IAMCloud-API access — eliminates the secret entirely

The decision rule: the higher the value of the credential, the further its source of truth should be from the cluster. Crown-jewel credentials belong in Vault or a cloud secret manager; cloud-API access should use workload identity and carry no secret at all; only low-stakes data belongs directly in the Secret API.

Production Notes

  • The dominant 2026 production architecture, as documented across Spotify, Shopify, and Airbnb infrastructure write-ups, is consistent in shape even when the tooling differs: an external source of truth (Vault or a cloud secret manager), an in-cluster sync layer (External Secrets Operator or the Vault Agent), etcd encryption at rest with a cloud KMS as defense-in-depth, workload identity for cloud-API access, and Sealed Secrets / SOPS only for the residue that genuinely must live in Git.
  • RBAC audits catch more real exposure than encryption audits. In practice, far more clusters are compromised through over-broad get secrets grants than through stolen etcd snapshots. Spend the first hour of any secrets review on kubectl who-can get secret and Pod-create permissions, not on encryption config.
  • automountServiceAccountToken: false as a cluster-wide default is increasingly recommended baseline hardening — most workloads never touch the API and gain nothing from a mounted token. Pod Security and admission policies (Kyverno, OPA Gatekeeper) can enforce it.
  • Treat the Secret API as an intent signal, not a vault. Its real value over a ConfigMap is the convention it enforces: stricter RBAC, encryption-at-rest config, audit-log scrutiny, and CI pipelines that redact Secret values from logs. The technical hardening (base64, tmpfs) is secondary; the discipline the convention triggers is the point.

See Also

Graph View