HashiCorp Vault is the canonical enterprise secrets-management system, and Vault on Kubernetes is the umbrella for the three patterns by which K8s workloads consume secrets from a Vault cluster (developer.hashicorp.com/vault/docs/platform/k8s). Vault’s distinctive value beyond simple key-value secret storage is its dynamic-secrets capability — generating short-lived per-Pod database credentials, cloud IAM credentials, X.509 certificates, and SSH keys on demand, with automatic revocation when the lease expires or the Pod terminates. The three K8s integrations differ in where secrets land: (1) the Vault Agent Injector (vault-k8s) injects a sidecar that writes secrets to a Pod-local tmpfs volume — no K8s Secret object is ever created; (2) the Vault Secrets Operator (VSO) (vault-secrets-operator) reconciles Vault content into native K8s Secrets via CRDs, mirroring the External Secrets Operator design; (3) the Vault CSI Provider mounts secrets at Pod-volume time via the Secrets Store CSI driver. All three rest on the Vault Kubernetes auth method (developer.hashicorp.com/vault/docs/auth/kubernetes), which authenticates Pods using their projected ServiceAccount JWT validated via the kube-apiserver’s TokenReview API.
Mental Model
Vault is a separate cluster (not deployed inside K8s by necessity, though it commonly is). Pods consume secrets from Vault. The two architectural questions are (a) how does the Pod authenticate to Vault and (b) how does the secret get from Vault into the Pod’s environment.
For (a), the Kubernetes auth method is the answer: Pods authenticate as their ServiceAccount. The ServiceAccount’s projected token (a JWT signed by the apiserver) is presented to Vault; Vault calls back to kube-apiserver’s TokenReview API to verify the token; on success, Vault issues a Vault token scoped to the role bound to that ServiceAccount.
For (b), Vault offers three patterns with different trade-offs:
flowchart TB
subgraph VAULT[Vault cluster]
V[Vault server<br/>KV, DB engines, PKI, Transit]
end
subgraph K8S[Kubernetes cluster]
APISERVER[kube-apiserver]
subgraph POD1[Pod (Agent Injector)]
APP1[app container] -.read file.-> TMPFS[tmpfs /vault/secrets]
INIT[init: vault-agent<br/>fetch on start]
SIDE[sidecar: vault-agent<br/>renew leases]
end
subgraph POD2[Pod (VSO)]
APP2[app container] -.envFrom/volume.-> SECRET[K8s Secret]
end
VSO[Vault Secrets Operator]
subgraph POD3[Pod (CSI)]
APP3[app container] -.CSI mount.-> CSIVOL[CSI volume]
end
CSIDRIVER[Secrets Store CSI Driver<br/>+ Vault provider]
end
POD1 -->|auth + read| V
VSO -->|auth + read| V
VSO -->|upsert| SECRET
CSIDRIVER -->|auth + read| V
APISERVER -.TokenReview.-> V
What this diagram shows. All three integrations authenticate to Vault using the cluster’s apiserver as the trust anchor (the dashed TokenReview link). The data path differs: Agent Injector keeps secrets out of etcd entirely — they live only in the Pod’s tmpfs; VSO materialises secrets as ordinary K8s Secret objects in etcd; CSI mounts them through a CSI volume without a K8s Secret object (though optionally syncing to one). The insight to extract: the choice among the three is a confidentiality vs. compatibility trade-off. Agent Injector has the best confidentiality (no etcd footprint, no K8s Secret to leak) but requires app code to read from a file; VSO has the worst confidentiality (a Secret in etcd, subject to etcd Encryption at Rest limits) but supports envFrom and full app compatibility.
Mechanical Walk-through
Pattern 1: Vault Agent Injector
The injector (vault-k8s) is a MutatingAdmissionWebhook registered with the apiserver. When a Pod is created with the annotation vault.hashicorp.com/agent-inject: "true", the webhook rewrites the PodSpec to:
Add an init container running vault-agent with the configuration derived from the Pod’s annotations. The init container authenticates to Vault using the Pod’s ServiceAccount token, fetches the requested secrets, and writes them to a shared emptyDir (typically tmpfs-backed, so they never hit disk) at /vault/secrets/<name>.
Add a sidecar container running another vault-agent instance. The sidecar continues running for the Pod’s lifetime, renewing leases before they expire and refreshing rendered files when the underlying Vault data or lease state changes.
Add a shared emptyDir: { medium: Memory } volume mounted at /vault/secrets on all containers, so the application container can read the rendered files.
vault.hashicorp.com/role: "payments-app" — the Vault role this Pod authenticates as (bound to the ServiceAccount on the Vault side).
vault.hashicorp.com/agent-inject-secret-database: "database/creds/payments-db" — fetch from the path database/creds/payments-db (a dynamic DB credential), render to /vault/secrets/database.
vault.hashicorp.com/agent-inject-template-database: | — optional Consul Template / Go template to customise the rendered output (e.g., produce a JDBC connection string, a pg_hba.conf snippet, or env-var lines for source /vault/secrets/database).
For dynamic database secrets, every Pod that starts gets a freshly-generated DB user with a short TTL (commonly 1 hour). The sidecar renews the lease while the Pod runs; when the Pod terminates, the sidecar’s revocation hook tells Vault to revoke the DB user. This is the defining feature of Vault — even if the secret leaks, its blast radius is bounded to the lease TTL.
Pattern 2: Vault Secrets Operator (VSO)
VSO (vault-secrets-operator) is the newer, CRD-based approach mirroring the External Secrets Operator design. It progressed through v0.x betas during 2023–2025 and reached its first generally-available release, v1.0.0, on 24 September 2025 (VSO releases); the line has continued (v1.4.0 shipped 5 May 2026). Notably, the CRD API group version remained secrets.hashicorp.com/v1beta1 through the v1.0 GA and into the v1.4 series — the operator binary reaching 1.0 did not promote the CRD schema to a v1 API version, so manifests written against v1beta1 continue to apply unchanged (per the VSO API reference, as of May 2026). Its CRDs are:
VaultConnection — defines a connection to a Vault server (address, tlsServerName, caCertSecretRef).
VaultAuth — defines an authentication method bound to a VaultConnection; the method field selects kubernetes, appRole, jwt, aws, or gcp. The kubernetes method references a K8s ServiceAccount whose token is exchanged for a Vault token.
VaultStaticSecret — reads from a Vault KV-v1 or KV-v2 engine and reconciles a K8s Secret. Fields: vaultAuthRef, mount, path, type (kv-v1/kv-v2), version (specific KV-v2 version), refreshAfter (polling interval), destination (target K8s Secret), rolloutRestartTargets (Deployments to restart on update), hmacSecretData (drift detection via HMAC).
VaultDynamicSecret — reads from a dynamic engine (database, AWS, etc.) and reconciles a K8s Secret with automatic lease management. Fields: requestHTTPMethod, renewalPercent (when to renew; default 67% of TTL), revoke (revoke on delete), allowStaticCreds.
VaultPKISecret — generates X.509 certificates from a Vault PKI engine. Fields: mount, role, commonName, altNames, ttl, format (pem/der/pem_bundle), expiryOffset (renewal threshold), revoke.
HCPAuth, HCPVaultSecretsApp — integrate with the SaaS HCP Vault Secrets product rather than self-hosted Vault.
The reconciliation pattern is identical in shape to ESO: the operator polls Vault, materialises the result as a K8s Secret, and (optionally) triggers a rolling restart of consuming Deployments. The differentiation from ESO is first-class dynamic-secret support with proper lease lifecycle — VSO renews leases at renewalPercent of TTL, regenerates entirely on lease expiry, and revokes on VaultDynamicSecret deletion. ESO can technically pull from Vault but does not natively manage dynamic-secret leases.
Pattern 3: Vault CSI Provider
The Secrets Store CSI Driver is a generic CSI driver that mounts secrets from external stores; the Vault provider plugin plugs into it. Pods declare a SecretProviderClass and mount it as a volume; the CSI driver fetches secrets from Vault at mount time. Optional syncSecret populates a K8s Secret alongside the volume mount. Choose CSI when you have other secret stores (AWS Secrets Manager, GCP Secret Manager) and want one driver for all of them.
The token_reviewer_jwt is the ServiceAccount JWT that Vault uses to call back to kube-apiserver’s TokenReview API. That ServiceAccount needs system:auth-delegator ClusterRoleBinding:
This says “any Pod running as ServiceAccount payments-app in namespace payments is allowed to log in as Vault role payments-app, which grants payments-db-policy, with a Vault token TTL of 1 hour.”
Authentication flow from a Pod:
Pod reads its projected ServiceAccount token from /var/run/secrets/kubernetes.io/serviceaccount/token (or, since K8s 1.21, from the bound, audience-scoped projected token volume mounted by the apiserver).
Pod sends POST /v1/auth/kubernetes/login to Vault with {"role": "payments-app", "jwt": "<the SA token>"}.
Vault calls apiserver:/apis/authentication.k8s.io/v1/tokenreviews with the JWT. The apiserver validates the signature against the cluster’s signing key and confirms the ServiceAccount still exists (since K8s 1.7 with --service-account-lookup=true, the default).
On success, Vault checks that the ServiceAccount name and namespace match the role’s bound_service_account_names and bound_service_account_namespaces.
Vault issues a Vault token scoped to the role’s policies and TTL, returning it to the Pod.
Two version facts matter here, both from the official auth-method docs (developer.hashicorp.com/vault/docs/auth/kubernetes). First, support for short-lived, bound service-account tokens was added in Vault 1.9.0 — necessary because Kubernetes 1.21 made the BoundServiceAccountTokenVolume feature default-on, so the SA JWT now carries an exp and is bound to the Pod and ServiceAccount lifetimes rather than being a never-expiring static token. Second, the recommended in-cluster configuration is to omit token_reviewer_jwt entirely: when it is unset, Vault uses the client Pod’s own JWT as the bearer token for the TokenReview call, and — as of Vault 1.9.3+ — Vault periodically re-reads its own mounted token file /var/run/secrets/kubernetes.io/serviceaccount/token so that it keeps working as that token rotates. Pinning a long-lived token_reviewer_jwt is the older pattern and is fragile against rotation; the modern guidance is to let each Pod present its own token and let Vault refresh its reviewer token from disk.
Configuration / API Surface
Agent Injector — Pod annotations
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-api namespace: paymentsspec: replicas: 3 selector: matchLabels: app: payments-api template: metadata: labels: app: payments-api annotations: # Enable injection vault.hashicorp.com/agent-inject: "true" # Vault role bound to this Pod's ServiceAccount vault.hashicorp.com/role: "payments-app" # Fetch a dynamic DB credential, write to /vault/secrets/db-creds vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/payments-db" # Custom render template — produce env-var-friendly output vault.hashicorp.com/agent-inject-template-db-creds: | {{- with secret "database/creds/payments-db" -}} export DB_USERNAME="{{ .Data.username }}" export DB_PASSWORD="{{ .Data.password }}" {{- end }} # Sidecar keeps lease alive; alternatively pre-populate only via init container vault.hashicorp.com/agent-pre-populate-only: "false" spec: serviceAccountName: payments-app containers: - name: api image: payments-api:1.4.2 # Source the rendered file at startup command: ["/bin/sh", "-c"] args: ["source /vault/secrets/db-creds && exec /usr/local/bin/api"]
Line-by-line. agent-inject: "true" is the trigger that causes the webhook to inject the init and sidecar containers; without it the webhook is a no-op for this Pod. role: payments-app references a Vault role created on the Vault side, bound to the K8s ServiceAccount payments-app. agent-inject-secret-<name> declares a secret to fetch; the <name> becomes the filename under /vault/secrets/. agent-inject-template-<name> overrides the default render format (which is “key=value lines”) with a custom Consul Template — here producing shell-source-compatible export lines.
agent-pre-populate-only: "false" (the default) keeps the sidecar running for lease renewal; "true" would only run the init container and skip the sidecar — appropriate for short-lived Pods (Jobs) where lease renewal isn’t worth the sidecar overhead.
Line-by-line. VaultConnection is a one-time configuration of the Vault server endpoint and TLS trust. VaultAuth references it and declares “authenticate using the kubernetes method at mount kubernetes, presenting the SA token for payments-app audience-scoped to vault.” VaultDynamicSecret references the auth, declares the secret to fetch (a dynamic DB credential at database/creds/payments-db), and configures lease management: renew at 67% of TTL, revoke on deletion of the CR, restart the payments-api Deployment when the underlying credential rotates (so Pods don’t continue using stale leases that may have been revoked at TTL expiry).
The reconciled K8s Secret (payments-db-secret) has data.username and data.password keys; the Pod consumes it via envFrom or volume mount.
Failure Modes
Vault unreachable. Agent Injector init container fails to fetch secrets; Pod stays in Init:0/1 until Vault recovers. Symptoms: kubectl describe pod shows the init container’s logs reporting connection refused to vault.vault.svc.cluster.local:8200. Diagnosis: kubectl run --image=curlimages/curl -i --rm vault-test --command -- curl -v https://vault.vault.svc:8200/v1/sys/health. Fix: restore Vault availability, restart the Pod.
Wrong ServiceAccount or role binding. Vault returns 403 permission denied on auth attempt. The Pod’s logs show failed to login with kubernetes auth. Cause: bound_service_account_names on the Vault role doesn’t match the Pod’s actual ServiceAccount, or bound_service_account_namespaces doesn’t include the Pod’s namespace. Diagnosis: vault read auth/kubernetes/role/<role> and compare to kubectl get pod -o yaml’s spec.serviceAccountName and metadata.namespace.
TokenReview returns user not found. Vault calls TokenReview with the Pod’s SA token but the apiserver rejects it. Causes: the SA token was issued by a different cluster (multi-cluster confusion), the SA was deleted and recreated (new UID), or --service-account-lookup is disabled. Fix: re-issue tokens (delete and recreate Pods), or verify apiserver flag.
Lease expiry without renewal. Agent Injector sidecar’s renewal failed (Vault down during a renewal window) and the dynamic DB credential expired. The Pod’s connections to Postgres start returning role does not exist. Symptom: app errors after exactly one TTL window from Pod start. Fix: restart Pod (which re-fetches via init container) and investigate why the renewal failed.
vault-agent-init ran but produced empty file. Common when the Vault path returns null. Templates that reference fields not present in the secret data render as empty strings without erroring. Diagnose with kubectl logs <pod> -c vault-agent-init.
KV-v1 vs KV-v2 path confusion. Vault’s KV v2 prepends data/ to the path: secret/myapp in v1 becomes secret/data/myapp in v2 for reads. Configuring VSO with type: kv-v1 against a KV-v2 mount returns 404. Fix: set type: kv-v2 and use the un-prefixed path; the operator inserts data/ internally.
Vault token revoked while Pod still running. Admin revokes a Vault token (e.g., during incident response) that the sidecar holds. Renewals fail; rendered secrets become stale. Symptom: Vault token-revocation event in the audit log followed by Pod-side renewal failures. Mitigation: VSO’s rolloutRestartTargets automatically restarts the Deployment on next reconcile, picking up a fresh token via the SA login flow.
Slow TokenReview under apiserver load. Vault’s TokenReview round-trip is on the apiserver’s critical path. Under heavy LIST traffic (API Priority and Fairness starving the auth-delegator flow), Vault auth latency spikes. Symptom: Pod startup latency climbs without obvious cause; Vault’s vault.auth.kubernetes.login.duration metric spikes.
Sidecar resource cost. A 3-replica Deployment with Agent Injector becomes a 3-replica Deployment with 6 containers (1 main + 1 sidecar each) plus 3 init containers. The sidecar’s CPU and memory footprint (~10m CPU / 64Mi RAM each at idle) adds up across thousands of Pods. The Sidecar Sprawl Anti-Pattern is real with Agent Injector at scale; VSO avoids it by centralising reconciliation in one Deployment.
Alternatives and When to Choose Them
Among the three Vault-on-K8s patterns:
Agent Injector when: (a) you need dynamic secrets with per-Pod identity; (b) no K8s Secret object is acceptable (highest confidentiality); (c) app code can read from a mounted file; (d) sidecar overhead per Pod is tolerable.
Vault Secrets Operator (VSO) when: (a) you want envFrom semantics in apps; (b) sidecar-per-Pod cost is unacceptable at scale; (c) you can tolerate K8s Secret objects in etcd (use etcd Encryption at Rest to mitigate); (d) the slightly delayed rotation reflected via Deployment rolling restart is acceptable.
Vault CSI Provider when: (a) you also use AWS Secrets Manager / GCP Secret Manager / Azure Key Vault via CSI and want one driver; (b) you want file-only mounting without sidecars.
Among broader alternatives:
External Secrets Operator — multi-provider CRD-based reconciliation. Better choice when secrets span multiple stores (Vault + AWS Secrets Manager + GCP). VSO is Vault-only.
Sealed Secrets — no Vault at all; encrypted CRDs in Git. Better for air-gapped or no-KMS environments.
SOPS — file-level encryption in Git. Better when secrets rotate infrequently and a true KMS isn’t deployed.
Native K8s Secret + etcd Encryption at Rest — simplest, but no dynamic secrets, no per-Pod identity, no centralized audit.
Decision frame: Use Vault when secrets must be dynamic and short-lived (DB creds, AWS IAM creds, certs). The PKI engine alone — issuing 1-hour TLS certs to every Pod — is justification enough for Vault in many security-conscious shops. If your secrets are mostly static (API keys, third-party tokens), External Secrets Operator or Sealed Secrets is simpler and adequate.
Production Notes
Vault deployment topology: most production Vault-on-K8s installations run Vault outside the K8s cluster it serves, on dedicated VMs with HSM-backed master keys. This gives Vault its own blast-radius isolation — a K8s control-plane compromise doesn’t yield Vault. The official Helm chart at github.com/hashicorp/vault-helm supports in-cluster deployment too, with Raft storage and auto-unseal via cloud KMS for the master key.
PKI engine for mTLS: the canonical “real” use case — every Pod requests a short-lived (1-hour) X.509 cert from Vault’s PKI engine via VaultPKISecret or the Agent Injector’s agent-inject-pki family of annotations. Combined with a service mesh like Istio or Linkerd (which also issues short-lived certs for mTLS), Vault becomes the org’s root CA. SPIFFE/SPIRE is an alternative identity provider for mesh-style mTLS.
Multi-cluster Vault auth: each cluster’s apiserver is a separate trust domain. Setup involves running vault write auth/kubernetes/config once per cluster (or once per mount with different paths: auth/kubernetes-prod-us-east/, auth/kubernetes-prod-eu-west/). Pod roles are namespaced under the per-cluster mount.
Audit logging: Vault’s audit-log device captures every secret read, every auth, every lease renewal — much richer than Kubernetes Audit Logging alone. This is a major regulatory benefit (PCI, SOC 2, HIPAA frequently mandate “who read this secret when”). Stream audit logs to SIEM (Splunk, Sumo, ELK).
HCP Vault: HashiCorp’s SaaS-hosted Vault. The VSO CRDs HCPAuth and HCPVaultSecretsApp integrate against HCP rather than self-hosted Vault, eliminating Vault operational burden for teams that don’t want it.
Pet shop case studies: HashiCorp’s own customer case studies (Adobe, Comcast, Roblox) cite Vault on K8s for dynamic database credentials and short-lived cloud IAM credentials as the primary value driver, not static secret storage. Static secrets are “table stakes”; dynamic secrets are why Vault.
Kubernetes-version compatibility: the Vault Helm chart (latest v0.32.0 as of May 2026) is tested against Kubernetes 1.31 through 1.35, with 1.29 the earliest version it has historically been tested against; earlier versions may work but are untested. Independently of the chart, Kubernetes ≥1.21’s default bound-projected-SA-token behaviour is the floor for the modern rotation-safe auth flow above — older K8s lacks it, complicating long-running Pod auth.
Lease-renewal timing across the two paths
The two integrations renew on slightly different machinery, and a common confusion is to assume they share one renewal constant. They do not; they happen to converge on a similar number.
VSO exposes renewalPercent on VaultDynamicSecret, and the schema’s default is 67 — i.e. VSO renews when 67% of the lease TTL has elapsed (the kubebuilder default annotation on the CRD field is literally 67, with a small jitter applied to avoid synchronised renewal storms across many secrets; verified against the upstream VaultDynamicSecret CRD definition, as of May 2026). The valid range is 0–90.
The Agent Injector’s sidecar runs a full Vault Agent, whose lease cache renews each lease independently on Vault Agent’s own schedule (Vault Agent renews well before expiry rather than at a single fixed published percentage you set per-secret). The injector does not surface a renewalPercent-style knob the way VSO does; renewal is Vault-Agent-internal.
The practical lesson holds regardless of the exact constant: leases shorter than a few minutes are dangerous — the renewal window is so tight that a brief Vault outage or a slow TokenReview (see Failure Modes) can miss it, expiring the credential and breaking the Pod. Tune dynamic-secret TTLs to comfortably exceed your worst-case renewal latency, or accept that very short TTLs require the Pod to tolerate credential rotation gracefully.
See Also
Kubernetes MOC — parent index, §8 Configuration and Secrets
Secret — what VSO and CSI materialise into; what Agent Injector deliberately avoids
etcd Encryption at Rest — complementary protection when secrets do land in etcd (VSO/CSI paths)