vault

etcd Backup and Restore

14 min read

etcd Backup and Restore

etcd is the single source of truth for a Kubernetes cluster — every Pod, Service, Secret, ConfigMap, and arbitrary CRD instance lives in its keyspace (Kubernetes — Components). Every other control-plane component (apiserver, scheduler, controller-manager) is stateless and can be killed and restarted without data loss; etcd cannot. Protecting etcd is therefore the most consequential backup task in any self-managed cluster. The mechanism is the snapshot: etcdctl snapshot save produces a point-in-time copy of the entire etcd keyspace as a single file (etcd — Disaster recovery, v3.6). The mechanism is asymmetric: taking a snapshot is an online, low-impact operation, but restoring one is not — restore materializes a new data directory offline, and applying it to a cluster is effectively a control-plane stop → restore → restart operation, never an injection into a live etcd. Two boundaries define what an etcd snapshot is for: it captures all Kubernetes API objects and nothing else — it is not persistent-volume data (that needs Velero or volume snapshots), and a snapshot of an encrypted-at-rest etcd is still ciphertext that needs the KMS keys to be readable.

Mental Model

The mental model is photograph the cluster’s memory, store the photograph somewhere the cluster can’t take down with it. An etcd snapshot is a consistent copy of the bbolt database file at a known revision. Because it is the whole keyspace, restoring it restores the cluster to exactly the state it was in when the snapshot was taken — every object, every revision boundary, the lot. The asymmetry is the key idea: you snapshot a live cluster, but you restore into a stopped one.

flowchart LR
    subgraph LIVE["Live cluster (online)"]
        ETCD[(etcd<br/>bbolt keyspace)]
    end
    ETCD -->|"etcdctl snapshot save<br/>(online, low-impact)"| SNAP["snapshot.db<br/>point-in-time copy"]
    SNAP -->|"aws s3 cp / gsutil cp<br/>(store OFF-cluster)"| OBJ["Object storage<br/>(survives cluster loss)"]

    OBJ -.->|"disaster: cluster lost"| RESTORE

    subgraph RESTORE["Restore path (OFFLINE)"]
        direction TB
        R1["1. STOP kube-apiserver + etcd"]
        R2["2. etcdutl snapshot restore<br/>→ fresh data-dir"]
        R3["3. point etcd at new data-dir"]
        R4["4. START etcd, then apiserver"]
        R1 --> R2 --> R3 --> R4
    end

What this diagram shows. Left-to-right is the backup path — snapshot a live etcd, ship the file off-cluster. The dashed arrow is disaster: the cluster (and its etcd) is gone. The bottom box is restore, and every step is labelled OFFLINE for a reason. The insight to extract: you never restore into a running etcd. Restore builds a new data directory from scratch; you then stop the dead/wrong etcd, swap in the new directory, and start fresh. Treating restore as a live operation is the classic way to corrupt a cluster.

Mechanical Walk-through

Taking a snapshot

etcdctl snapshot save connects to a single etcd endpoint over gRPC and streams the entire keyspace into a file (Kubernetes — Operating etcd clusters). It is a consistent snapshot — it captures a single revision boundary, so the file is internally coherent (no half-written transaction). It is low-impact: it does not block writes; etcd serves the snapshot from its MVCC store. Because the snapshot is the whole keyspace, its size tracks the etcd database size (the recommended ceiling is ~8 GiB — see etcd).

The discipline:

  1. Schedule it. A snapshot is only as good as its freshness. Common practice is a snapshot every 15–30 minutes via a CronJob or a systemd timer.
  2. Store it off-cluster. A snapshot sitting on the etcd node’s local disk dies with the node. The canonical runbook is etcdctl snapshot save followed immediately by aws s3 cp / gsutil cp / equivalent to object storage in a different failure domain.
  3. Verify it. etcdutl snapshot status <file> reports the snapshot’s revision count, total keys, and a hash — a quick integrity check that the file is not truncated. A snapshot taken with etcdctl snapshot save carries an integrity hash that etcdutl snapshot restore checks automatically at restore time; a file copied directly out of a member’s data directory has no such hash and can only be restored with --skip-hash-check (etcd — Disaster recovery, v3.6).

Restoring a snapshot

Restore is fundamentally different (etcd — Disaster recovery, v3.6). etcdutl snapshot restore does not talk to a running etcd. It reads the snapshot file and materializes a brand-new etcd data directory on disk — a fresh write-ahead log (WAL) and a fresh bbolt database, with new cluster and member UUIDs. The new UUIDs are deliberate: they stop the restored member from accidentally rejoining a surviving cluster and causing split-brain.

The binary the restore lives in changed across etcd minor versions, and the change is now permanent. etcd ships two command-line tools: etcdctl, the online client that talks to a running cluster over gRPC, and etcdutl, the offline utility that operates directly on data files. In etcd v3.5 the offline operations were moved into etcdutl but etcdctl snapshot restore/etcdctl snapshot status were retained as deprecated aliases. etcd v3.6.0 (released 15 May 2025 — the first minor release since v3.5.0 in June 2021) removed those aliases outright: etcdctl snapshot restore, etcdctl snapshot status, and etcdctl defrag --data-dir no longer exist (Announcing etcd v3.6.0). As the v3.6 disaster-recovery guide states, “starting from etcd v3.6, users can only use etcdctl to take the data to a snapshot, but use etcdutl to restore data from a snapshot” (etcd — Disaster recovery, v3.6). The split is clean: etcdctl snapshot save (online, talks to the live keyspace) versus etcdutl snapshot status / etcdutl snapshot restore (offline, operate on the file). Older clusters and older docs still show etcdctl snapshot restore; on any v3.6+ cluster it is etcdutl only.

Applying that restored data directory to a cluster is a control-plane reboot:

  1. Stop the kube-apiserver on every control-plane node, and stop etcd on every member.
  2. Restore the snapshot into a new data directory on each member (or restore on one and re-add the others — see below).
  3. Point each etcd’s --data-dir at the restored directory and --initial-cluster at the intended membership.
  4. Start etcd, wait for quorum, then start the apiservers.

The cluster comes back up at exactly the snapshot’s state. Everything that happened after the snapshot — every object created, scaled, or deleted in the gap — is lost.

Multi-member restore

For a 3- or 5-member etcd cluster there are two valid approaches:

  • Restore all members from the same snapshot. Each member runs etcdutl snapshot restore against the same snapshot file, each with its own --name and --initial-advertise-peer-urls, all sharing one --initial-cluster and --initial-cluster-token. The v3.6 guide is explicit that “all members should restore using the same snapshot” — restoring different members from different snapshots produces an inconsistent cluster. They form a new cluster with consistent data. This is the cleanest path for a full disaster.
  • Restore one member, re-add the others. Restore a single member into a new single-member cluster, start it, then use the etcd membership API (etcdctl member add) to add the remaining members; each new member catches up by streaming state from the leader.

The first approach is preferred for a true cluster-wide disaster; the second is closer to “rebuild a quorum” and is more error-prone.

The --bump-revision consideration

A subtle gotcha: after a restore, etcd’s global revision counter resets to a value at-or-near the snapshot’s. Kubernetes informer caches and watch clients track resourceVersion, which is derived from etcd’s revision. If the restored revision is lower than what clients last saw, watches can behave oddly. etcdutl snapshot restore --bump-revision (and --mark-compacted) advances the restored revision past the old high-water mark to avoid this — recommended when restoring into an environment where clients may remember higher revisions.

Configuration / API Surface

A scheduled snapshot and a restore, on a TLS-secured etcd:

# --- BACKUP: take + verify + ship off-cluster (run on a schedule) ---
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db
 
etcdutl snapshot status /backup/etcd-snapshot-20260516-0300.db -w table   # integrity check
aws s3 cp /backup/etcd-snapshot-20260516-0300.db s3://cluster-dr-bucket/  # OFF-cluster, different failure domain
 
# --- RESTORE: offline, into a NEW data directory (control plane already stopped) ---
etcdutl snapshot restore /backup/etcd-snapshot-20260516-0300.db \
  --name=etcd-0 \
  --data-dir=/var/lib/etcd-restored \
  --initial-cluster=etcd-0=https://10.0.1.10:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380 \
  --bump-revision=1000000000 --mark-compacted
# then: point etcd's --data-dir at /var/lib/etcd-restored, start etcd, then start kube-apiserver

Line-by-line: the three TLS flags (--cacert, --cert, --key) are mandatory because a production etcd enforces mutual TLS on the client port — anyone who can take a snapshot can read every Secret in the cluster, so the snapshot path is itself a sensitive credential boundary. snapshot save writes a timestamped file so successive snapshots don’t overwrite each other. etcdutl snapshot status is the cheap integrity gate before you trust the file. The aws s3 cp is the single most important line — a snapshot on local disk is not a backup. In the restore, --data-dir points at a new, empty path (/var/lib/etcd-restored), never the live data dir; --initial-cluster declares the intended membership; --bump-revision/--mark-compacted advance the revision counter past the old high-water mark so informer caches across the cluster don’t see a resourceVersion rollback.

Interaction with encryption at rest

If the cluster encrypts Secrets at rest via an EncryptionConfiguration (see etcd Encryption at Rest), the snapshot file contains the encrypted Secret values. Restoring the snapshot restores the ciphertext — but the apiserver can only decrypt those Secrets if it still has access to the same KMS / encryption keys that were active when the data was written. A disaster-recovery plan that backs up etcd but loses the KMS key has backed up unreadable data. The KMS key (or the static encryption key file) must be part of the DR plan, stored and protected independently.

Failure Modes

Snapshot stored on the etcd node. The single most common mistake. The node fails, taking both the live etcd and its only backup. A snapshot must land in a different failure domain — ideally object storage in another region.

Restoring into a live etcd. Treating snapshot restore as something you do to a running cluster. It is not — restore builds a new data directory; you must stop etcd and the apiservers first. Done wrong, it corrupts the cluster.

Stale snapshot = large data loss. Snapshot-based recovery has an RPO equal to the snapshot interval: restore from a 30-minute-old snapshot and you lose up to 30 minutes of cluster changes. If the cluster is GitOps-driven this is mostly recoverable (the controller re-reconciles from Git), but imperative changes and any in-cluster-only state are gone.

Untested snapshots. A snapshot file that has never been restored is an unverified assumption. Truncated files, wrong TLS material, missing KMS keys — all surface only at restore time. The fix is to drill restores (into a throwaway cluster) at least quarterly.

Lost KMS key for an encrypted etcd. Backing up etcd but not the encryption key. The Secrets in the snapshot are permanently unreadable. Cross-link etcd Encryption at Rest.

Revision rollback breaking informers. Restoring without --bump-revision into an environment where clients remember higher resourceVersions can cause watch caches to misbehave. Use --bump-revision/--mark-compacted.

Assuming the snapshot covers PV data. An etcd snapshot is only API objects. A PersistentVolumeClaim object is in the snapshot; the data on the underlying disk is not. Restoring etcd brings back the PVC/PV objects pointing at volumes whose contents were never backed up. Cross-link Velero, Volume Snapshot.

Alternatives and When to Choose Them

etcd snapshot — the right tool for whole-cluster control-plane state. When you need to recover the entire cluster’s memory — every namespace, every object, every CRD instance, RBAC, the works — an etcd snapshot is the canonical and only complete mechanism. It is the foundation of Disaster Recovery for Kubernetes for self-managed clusters.

Velero — the right tool for application-scoped backup and migration. Velero backs up API objects filtered by namespace/label/resource by querying the apiserver, plus persistent-volume data via CSI snapshots or its file-level node-agent. Choose Velero when you want to back up one application’s namespace, restore it selectively, or migrate workloads from cluster A to cluster B. Velero does not capture the whole control-plane state the way an etcd snapshot does (it would not faithfully restore an entire cluster’s etcd internals), and an etcd snapshot does not capture PV data the way Velero does. They are complementary, not substitutes.

Managed control planes (EKS / GKE / AKS). On managed Kubernetes the provider runs and backs up etcd invisibly; you cannot — and need not — etcdctl snapshot save it. Your DR responsibility shifts entirely to application state (Velero) and the cluster definition (GitOps / Cluster API). Choosing a managed service is, in part, choosing to outsource etcd backup.

GitOps as a complement. If the cluster’s desired state lives in Git and is reconciled by ArgoCD/Flux, a large fraction of cluster state is reproducible without any etcd snapshot — re-point a fresh cluster at the Git repo and it rebuilds. The etcd snapshot still matters for state that is not in Git (imperatively-created objects, generated Secrets, operator-managed status), but GitOps dramatically shrinks how much an etcd restore has to recover.

Production Notes

The operational consensus from the etcd disaster-recovery docs (v3.6) and Kubernetes etcd operations guide:

  • Snapshot every 15–30 minutes, retain at least 7 days, store off-cluster. The interval is your RPO. Retention guards against discovering a corruption that started days ago.
  • Drill the restore quarterly. The backup you have never restored is not a backup. The drill should be a full restore into a throwaway cluster, validated by kubectl get against the expected objects.
  • Snapshot immediately before any risky operation — a control-plane upgrade, a CRD migration, a mass kubectl delete. The fresh snapshot is the rollback point.
  • Treat the snapshot as a secret. It contains every Secret in the cluster. The S3 bucket holding snapshots needs the same access controls as the cluster’s most sensitive credential, plus its own encryption at rest.
  • RTO of a snapshot restore is dominated by the control-plane reboot, not the file copy — stopping apiservers, restoring on each member, re-forming quorum, restarting. For a small cluster this is minutes; for a large one with a multi-GiB database it can be tens of minutes. Practising the runbook is what keeps RTO predictable.
  • Tools like etcd-backup-operator, Velero’s etcd plugin, and managed-cluster vendors automate the snapshot+ship loop — but the restore still requires a human following a drilled runbook.
  • On a kubeadm cluster, etcd runs as a static Pod by default (kubeadm writes a static-pod manifest to /etc/kubernetes/manifests/etcd.yaml and the kubelet runs it), with TLS material under /etc/kubernetes/pki/etcd/ — exactly the --cacert/--cert/--key paths used above (Kubernetes — Operating etcd clusters). Because it is a static Pod, the “stop etcd” step of a restore is done by moving the static-pod manifest out of the manifests directory (the kubelet tears the Pod down), restoring into a new data directory, then editing etcd.yaml’s --data-dir (and the hostPath volume) to point at the restored directory before moving the manifest back. The kube-apiserver static Pod is stopped the same way.

The binary split between etcdctl (online) and etcdutl (offline) is covered in the Mechanical Walk-through above: as of etcd v3.6.0 (May 2025) the restore lives only in etcdutl — etcdctl snapshot restore was removed and is not an alias any longer (Announcing etcd v3.6.0). On a pre-v3.6 cluster you may still see etcdctl snapshot restore; on v3.6+ it does not exist.

See Also

Graph View

Backlinks