vault

Sealed Secrets

15 min read

Sealed Secrets

Sealed Secrets is Bitnami’s controller-based pattern for storing Kubernetes secret material safely in Git: the secret is encrypted asymmetrically to a public key whose corresponding private key never leaves the cluster, then committed as a SealedSecret custom resource (bitnami-labs/sealed-secrets). At runtime, an in-cluster controller decrypts the SealedSecret and produces an ordinary Kubernetes Secret. The encryption uses a hybrid of RSA-OAEP (with SHA-256) and AES-256-GCM — the actual secret payload is AES-256-GCM-encrypted with a randomly generated single-use 32-byte session key, and that session key is RSA-OAEP-wrapped to the controller’s public key, giving “encrypt arbitrarily large payloads against a public key” semantics without RSA’s size limits (per the crypto design doc). The architectural value is GitOps-native, no-runtime-dependency secret distribution: the SealedSecret YAML is committed alongside every other manifest, ArgoCD/Flux sync it like any other resource, and there is no external KMS or sidecar to operate. The trade-offs are that the ciphertext sits in Git forever (re-keyability becomes important if the controller’s private key is ever exposed) and there is no built-in audit trail of “who read this secret when.”

Mental Model

Sealed Secrets is asymmetric public-key encryption applied at the per-Secret granularity. The controller, deployed once per cluster, owns an RSA keypair (default 4096-bit — if no keypair is found and none is provided, the controller generates a fresh 4096-bit RSA key pair, per the crypto design doc). The public key is freely distributable — engineers kubeseal --fetch-cert it to their laptops. Encryption happens offline with the public key (no cluster contact needed); decryption happens only inside the cluster by the controller, which holds the private key.

sequenceDiagram
    participant Dev as Developer<br/>(kubeseal CLI)
    participant Pub as Public key<br/>(in Git or cached)
    participant Git as Git repo
    participant ArgoCD as ArgoCD / Flux
    participant K8s as kube-apiserver
    participant Ctrl as sealed-secrets-controller
    participant Priv as Private key<br/>(in cluster)
    Dev->>Pub: fetch public cert
    Dev->>Dev: kubeseal &lt; secret.yaml &gt; sealed.yaml<br/>(AES + RSA-OAEP locally)
    Dev->>Git: commit SealedSecret CRD
    Git->>ArgoCD: PR merged
    ArgoCD->>K8s: apply SealedSecret
    Ctrl->>K8s: watch SealedSecret events
    K8s-->>Ctrl: SealedSecret created
    Ctrl->>Priv: load private key
    Ctrl->>Ctrl: decrypt payload
    Ctrl->>K8s: create owned Secret
    Note over K8s: Pod consumes Secret normally

What this diagram shows. The path forks: secret recipes travel through Git, ArgoCD, kube-apiserver to the controller; secret plaintext only exists momentarily in the controller process’s memory before becoming a regular K8s Secret. The insight to extract: this design has no external runtime dependency — even with the network entirely partitioned, the cluster can deploy and decrypt SealedSecrets as long as the controller is healthy. That makes it appealing for air-gapped, edge, and disaster-recovery scenarios where External Secrets Operator or HashiCorp Vault on Kubernetes would fail because the external store is unreachable.

Mechanical Walk-through

Sealing a Secret

The developer workflow starts with a normal K8s Secret manifest. Because kubectl create secret is the canonical way to build one (handling base64 encoding correctly), most workflows shape like:

kubectl create secret generic db-creds \
  --from-literal=username=postgres \
  --from-literal=password='s3cret!' \
  --dry-run=client -o yaml \
  | kubeseal --controller-namespace=sealed-secrets \
             --controller-name=sealed-secrets \
             --format=yaml \
             > db-creds-sealed.yaml

kubeseal fetches the controller’s public cert (over kubectl, by default — or --cert <file> for offline) and runs the hybrid encryption: a random 32-byte AES-256 key is generated, the Secret’s data and stringData fields are AES-GCM-encrypted, and the AES key is RSA-OAEP-encrypted to the controller’s public key. The result is a SealedSecret CRD:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
  namespace: payments
spec:
  encryptedData:
    username: AgB7K9pq...   # opaque ciphertext
    password: AgC8M3vrL...
  template:
    metadata:
      name: db-creds
      namespace: payments
    type: Opaque

This file is safe to commit. The template block carries non-secret metadata that will appear on the resulting Secret — labels, annotations, type — so consumers can match by label and the controller does not have to invent them.

Decryption inside the cluster

The controller runs as a Deployment in kube-system (or sealed-secrets) namespace. It watches SealedSecret events cluster-wide. On each event:

  1. Looks up the active private key from its in-cluster key store (a set of K8s Secrets labeled sealedsecrets.bitnami.com/sealed-secrets-key=active).
  2. Tries to decrypt the encrypted AES key with the active private key first; on failure tries each historical key (kept around for backward compatibility — see Rotation below). If all keys fail, marks the SealedSecret status with an error.
  3. Decrypts the data fields with the recovered AES key.
  4. Constructs a Kubernetes Secret from the template plus the decrypted data, sets the SealedSecret as the owner reference (so cascading delete works), and applies it via the apiserver.

The resulting Secret is a perfectly normal K8s Secret — consumers cannot tell it was decrypted from a SealedSecret.

Encryption scopes (the binding rules)

By default, a SealedSecret is strictly scoped to a specific name AND namespace. The binding is implemented by feeding a label into the RSA-OAEP key-wrapping step as its optional label parameter (RSA-OAEP takes an optional label that becomes part of the integrity check — not the AES-GCM AAD, a detail the crypto doc is explicit about). The label is the concatenation of the Secret’s namespace and name in strict scope, the namespace alone in namespace-wide scope, and empty in cluster-wide scope (crypto.md). Because the label is bound into the RSA-OAEP wrap, unwrapping the session key only succeeds when the same label is supplied at decrypt time — so a SealedSecret moved to a different namespace fails to decrypt: the controller computes a different label and RSA-OAEP unwrapping fails. Three scopes exist (per README):

  • strict (default) — bound to name + namespace. Renaming or moving requires re-sealing.
  • namespace-wide (kubeseal --scope namespace-wide) — bound to namespace only. Can be renamed within a namespace without re-sealing.
  • cluster-wide (kubeseal --scope cluster-wide) — no location binding. Can be moved anywhere in the cluster.

The scope is a deliberate security control: strict scope prevents a tenant in namespace frontend from copying a SealedSecret targeted at namespace payments, even if they happen to obtain the YAML — the controller will refuse to decrypt. This matters because anyone can produce a SealedSecret (the public key is, well, public) but only the right namespace can use it. Without the scope binding, an attacker could exfiltrate ciphertext from a public Git repo and try to apply it in a namespace they control.

Key rotation

The controller renews its keypair every 30 days by default — the README states verbatim that “Sealing keys are automatically renewed every 30 days,” and the cadence is configurable via the --key-renew-period flag (720h). A new keypair is generated, marked active (it becomes the default for new SealedSecret encryptions that fetch the cert), and the previous keypair stays as a historical key — used to decrypt SealedSecrets that were sealed against it. Old keys are never deleted, because that would brick old SealedSecrets in Git that have not been re-sealed.

This means the controller’s set of decryption keys grows monotonically — by ~12 keys per year. For most clusters this is fine; for very long-lived clusters or those with high-rotation-tempo paranoia, the operator may want to forcibly re-encrypt every SealedSecret periodically:

kubeseal --re-encrypt < old-sealed.yaml > new-sealed.yaml

After re-encrypting everything in Git, the historical key can be removed from the controller’s key store and the SealedSecrets are bound only to the active key. This is operationally onerous and rarely done outside compliance-driven environments.

The single most important operational concern for Sealed Secrets is: back up the controller’s private keys. They live in K8s Secrets in the controller namespace. If lost (cluster destroyed without backup), every SealedSecret in Git is permanently undecryptable. Standard practice is to export them to encrypted offline storage:

kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-keys-backup.yaml
# Then encrypt this backup file with another mechanism (age, GPG, hardware key)

The Bring-Your-Own-Certificate (docs/bring-your-own-certificates.md) variation lets the operator generate the controller’s keypair externally (e.g., in an HSM-backed CA) and inject it into the cluster, avoiding the bootstrapping concern that the keypair was generated inside a possibly-compromised cluster.

Configuration / API Surface

A complete deployment using the official Helm chart:

# values.yaml for the sealed-secrets-controller Helm chart
keyrenewperiod: 720h           # 30 days, the default
secretName: sealed-secrets-key  # name pattern for the stored keypair Secret
fullnameOverride: sealed-secrets
namespace: sealed-secrets
 
# Network policy: lock down the controller — it only needs to talk to apiserver
networkPolicy:
  enabled: true
 
# Resource limits — controller is lightweight, 50m CPU / 64Mi RAM typical
resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 256Mi
 
# Run as non-root, drop all capabilities — security baseline
securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: [ALL]
 
# Use the v2 metrics endpoint for Prometheus scraping
serviceMonitor:
  enabled: true

Line-by-line. keyrenewperiod: 720h is the default 30-day cadence; for compliance regimes requiring 7-day rotation set 168h. The secretName prefix is the convention the controller uses to discover historical keys (it lists Secrets matching sealedsecrets.bitnami.com/sealed-secrets-key label and orders them by age). The networkPolicy.enabled: true is important: the controller’s only API consumer is the kube-apiserver, so an ingress-deny + egress-only-to-apiserver policy is appropriate and significantly reduces blast radius if the controller is ever compromised.

A SealedSecret manifest, ready to commit to Git:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: stripe-api-key
  namespace: payments
  annotations:
    # Optional: cluster-wide scope (rare; prefer strict)
    # sealedsecrets.bitnami.com/cluster-wide: "true"
spec:
  encryptedData:
    api_key: AgBy3i4OJSWK+...
  template:
    metadata:
      name: stripe-api-key
      labels:
        app: payments-api
    type: Opaque

The encryptedData map is opaque — values are base64-encoded ciphertext. Pretty diff output is hopeless; treat these files as binary blobs that change atomically when the underlying secret rotates.

Once applied, kubectl get sealedsecret -n payments shows status:

NAME              STATUS   SYNCED   AGE
stripe-api-key             True     2m

SYNCED=True means the controller successfully produced the corresponding Secret.

Failure Modes

Lost private key, no backup. Cluster rebuilt without restoring the keypair from backup. Every SealedSecret in Git is now ciphertext-without-a-key — permanently undecryptable. Recovery: re-seal every secret from source (which means re-deriving the plaintext from wherever the source-of-truth is). This is the catastrophic failure mode that justifies the keypair backup discipline.

Wrong-namespace or wrong-name apply. Strict-scoped SealedSecret applied to the wrong namespace. Controller emits an event “no key could decrypt secret” and the SealedSecret status shows synced=False. Fix: re-seal in the correct namespace, or use kubeseal --re-encrypt to migrate.

Public key drift between developers and cluster. Developer’s local --cert file is stale (controller rotated and the developer is sealing against the old cert). The seal will still work — old keys are kept — but it’s a leading indicator that the developer’s local cache is out of sync. Operational hygiene: always kubeseal --fetch-cert against the live cluster before sealing.

Controller down during SealedSecret apply. ArgoCD applies a SealedSecret but the controller is not running. The SealedSecret object lands in etcd; no Secret is produced; Pods consuming the Secret crashloop on missing-Secret errors. The fix is automatic once the controller recovers (it reconciles all SealedSecrets on startup), but ordering matters during cluster bootstrap — the controller must be up before any SealedSecret-dependent Pod is scheduled.

Re-key drift after key rotation. A SealedSecret in Git was sealed 2 years ago against a now-retired key (if the operator deleted historical keys). Decryption fails. Recovery: regenerate from source plaintext. Mitigation: do not delete historical keys without a --re-encrypt sweep first.

Insecure use of cluster-wide scope. Cluster-wide scope removes the namespace binding; an attacker who exfiltrates the SealedSecret YAML from Git can apply it in their own tenant namespace. Audit: SealedSecrets in Git should be strict by default; cluster-wide is reserved for legitimately-cluster-scoped secrets (e.g., a webhook TLS cert used by an admission controller in every namespace).

Key-rotation Job race. During the 30-day key renewal, brief windows exist where the controller has just generated a new key but ArgoCD has not refreshed its cached public cert. New seals during this minute or two work against the old cert; the controller decrypts them via the now-historical key. No data loss, but it’s a subtle race that confuses logs.

Alternatives and When to Choose Them

  • External Secrets Operator — fetches from a real KMS at runtime. Better audit, better rotation story, but adds runtime dependency on the KMS. Choose ESO when you have a KMS and care about rotation; choose Sealed Secrets when you don’t.

  • SOPS — the closest sibling. SOPS encrypts the file (any structured file — YAML, JSON, ENV, INI, even non-K8s like Terraform .tfvars), using a per-file symmetric AES-256-GCM data key wrapped to a recipient list (AWS/GCP/Azure/HuaweiCloud KMS, HashiCorp Vault Transit, age, or PGP). The contrasts that decide between them:

    • Symmetric-with-recipient-list (SOPS) vs asymmetric-to-one-public-key (Sealed Secrets). SOPS can grant decrypt to an arbitrary set — several humans plus CI plus a break-glass key — each independently. Sealed Secrets has exactly one decryptor: the cluster controller. Multi-recipient and offline-plus-CI workflows are native to SOPS, awkward with Sealed Secrets.
    • Where the plaintext appears. With Sealed Secrets the plaintext only ever exists inside the cluster (the controller decrypts); the GitOps engine and CI never see it. With SOPS the decrypt happens at deploy time — on an operator’s machine or in Flux’s decryption provider — so whoever runs that step handles plaintext. If you want decryption confined to the workload cluster, Sealed Secrets is stronger; if you want no in-cluster decryption component at all, SOPS is stronger.
    • Scope of applicability. Sealed Secrets only encrypts a Kubernetes Secret. SOPS encrypts arbitrary config files, so a platform with non-K8s secrets consolidates on SOPS.
    • Operational surface. Sealed Secrets adds one in-cluster controller plus the catastrophic back-up-the-private-key discipline. SOPS adds no in-cluster component but requires maintaining .sops.yaml recipient rules and distributing/rotating age (or KMS) access per developer.

    Choose SOPS when you want zero in-cluster components, multiple independent recipients, or need to encrypt non-K8s files, and are happy decrypting at the GitOps engine (it is the de-facto standard in the Flux ecosystem). Choose Sealed Secrets when you want decryption to happen inside the cluster (e.g. Flux/ArgoCD runs in a less-trusted environment), are strictly Kubernetes-only, and prefer a single in-cluster decryptor over per-developer key distribution.

  • HashiCorp Vault on Kubernetes Agent Injector — no K8s Secret object exists; Vault renders secrets to a Pod’s filesystem. Best confidentiality (no etcd footprint), but adds sidecar overhead.

  • CSI Secrets Store Driver — mounts from external store via CSI; optionally syncs to K8s Secret. Choose when you want file-only mounts.

  • age-encrypted secrets in Git, decrypted at deploy time by a custom script — simplest, but no Kubernetes integration. Useful for bootstrap.

  • Raw K8s Secret in Git — base64 is not encryption. Never do this; even private repos eventually leak.

The decision frame: Sealed Secrets is the right answer when you want GitOps-pure secrets (everything in Git, no out-of-band rotation), no runtime dependency on an external KMS, and operational simplicity above all. Its biggest weakness is the immutable-Git-history problem: a sealed secret committed today is preserved in git log forever, and if the controller’s private key is ever exposed historically all those ciphertexts become decryptable.

Production Notes

  • GitOps integration: ArgoCD/Flux apply SealedSecrets just like any other manifest; no special handling. ArgoCD’s argocd-vault-plugin is not used (that’s a different model). The diff in PRs shows opaque ciphertext changes — review for which keys changed, not what values changed.
  • CI/CD note: never let CI runners hold the controller’s private key. CI’s job is to seal (which only needs the public cert) — sealing always happens with the public cert pulled at sealing time. Decryption happens only in the target cluster.
  • Helm chart distribution: most platform teams ship the sealed-secrets-controller via the official Helm chart at oci://registry-1.docker.io/bitnamicharts/sealed-secrets. The chart handles the controller Deployment, RBAC, and the optional ServiceMonitor for Prometheus.
  • HA: the controller can run with replicas: 2+, using leader election. Since decryption is a low-frequency operation (only on SealedSecret events) the secondary is essentially idle, but HA insulates against rolling restarts blocking secret sync.
  • Compliance angle: SOC 2 / PCI auditors sometimes balk at “ciphertext in Git” because it complicates the “where are the secrets?” question. A defensible answer: the secrets are in the cluster’s encrypted etcd (post-decryption) plus the controller’s private key; the Git ciphertexts are not exploitable without the private key, which lives only in the cluster.
  • CNCF status: Sealed Secrets is not a CNCF project — it’s a Bitnami / VMware Tanzu (now Broadcom) open source project. Active development continues at bitnami-labs/sealed-secrets; see the CONTRIBUTORS file. Compare to External Secrets Operator which is CNCF Sandbox.

See Also

Graph View

Backlinks