vault

Disaster Recovery for Kubernetes

13 min read

Disaster Recovery for Kubernetes

Disaster recovery (DR) for Kubernetes is the discipline of recovering from cluster-level loss — a corrupted or destroyed etcd, a deleted cluster, a region outage, a control-plane catastrophe — and bringing workloads back within agreed time and data-loss bounds. It is fundamentally a planning discipline, anchored on two numbers borrowed from general business-continuity practice (AWS — DR options): the Recovery Time Objective (RTO) — how long recovery may take — and the Recovery Point Objective (RPO) — how much recent data may be lost. Every DR design is a trade-off against those two targets, and against cost. The defining mistake in Kubernetes DR is believing that “back up etcd” is sufficient: an etcd snapshot captures every API object but no persistent-volume data, no external dependencies, no DNS, no cloud load balancers, and not the cluster definition itself. A real plan protects four distinct layers — control-plane state, application API objects, persistent-volume data, and the cluster definition — each with its own mechanism. And the iron rule of the discipline: the backup you have never restored is not a backup (Kubernetes Failure Stories).

Mental Model

The mental model is four layers, each lost independently, each restored by a different mechanism — all measured against RTO and RPO. Kubernetes does not have one thing to back up; it has a stack, and a DR plan that protects one layer and forgets the others fails at exactly the layer it forgot.

flowchart TB
    subgraph TARGETS["The two numbers every DR design serves"]
        RTO["RTO — how LONG recovery may take"]
        RPO["RPO — how much recent DATA may be lost"]
    end

    subgraph LAYERS["The four layers a Kubernetes DR plan must protect"]
        direction TB
        L1["1. Control-plane state<br/>→ etcd Backup and Restore"]
        L2["2. Application API objects<br/>→ Velero  OR  GitOps repo"]
        L3["3. Persistent-volume DATA<br/>→ Volume Snapshot / Velero node-agent /<br/>storage-level replication"]
        L4["4. The cluster definition itself<br/>→ Cluster API / IaC / GitOps<br/>(so a cluster can be REBUILT)"]
    end

    TARGETS -.->|"design to meet"| LAYERS

    subgraph STRATEGY["DR strategies (increasing cost, decreasing RTO/RPO)"]
        direction LR
        D1["Rebuild-from-Git<br/>(slow RTO, cheap)"]
        D2["Cluster-pair / standby<br/>(medium)"]
        D3["Multi-region active-active<br/>(near-zero RTO, expensive)"]
        D1 --> D2 --> D3
    end

What this diagram shows. Top: the two objectives every DR decision serves. Middle: the four layers — note that each has a different recovery mechanism, and missing any one leaves a gap. Bottom: the strategy spectrum, ordered by cost and by how aggressively it meets RTO/RPO. The insight to extract: “Kubernetes DR” is not a single backup job — it is a per-layer plan, and the strategy you pick (rebuild-from-Git vs standby vs active-active) is chosen by how small your RTO/RPO must be and how much you will pay for it.

Mechanical Walk-through

RTO and RPO — the two numbers

RTO (Recovery Time Objective) is the maximum acceptable time between disaster and restored service. RPO (Recovery Point Objective) is the maximum acceptable data loss, expressed as a time window — an RPO of 30 minutes means “we can tolerate losing the last 30 minutes of changes.” These two numbers are set by the business, not by engineering, and they drive every downstream choice. A 4-hour RTO permits a rebuild-from-backup approach; a 5-minute RTO forces a hot standby. An RPO of 30 minutes is met by a 30-minute snapshot interval; an RPO near zero forces continuous replication. DR design is the work of meeting those numbers at the lowest cost.

Layer 1 — control-plane state

The cluster’s “memory” lives in etcd. Protect it with etcd snapshots — etcdctl snapshot save on a schedule, stored off-cluster. The snapshot interval is the RPO for control-plane state; restore is a control-plane-stop-restore-restart operation whose duration is the RTO contribution. On managed Kubernetes (EKS/GKE/AKS) this layer is the provider’s responsibility — you neither back up nor restore etcd, which removes one layer from your plan entirely.

Layer 2 — application API objects

The Deployments, Services, ConfigMaps, Secrets, CRDs that define your applications. Two protection mechanisms:

  • Velero — backs up API objects (filtered by namespace/label/kind) to object storage; supports selective and cross-cluster restore.
  • GitOps repository — if desired-state lives in Git and is reconciled by ArgoCD/Flux, the Git repo is the backup of your application objects. Re-point a fresh cluster at the repo and it rebuilds itself.

GitOps is the stronger model for declared state: it is versioned, auditable, and reproducible. But it does not capture imperatively-created objects, generated Secrets, or operator-managed runtime status — Velero (or accepting that those are recreated) covers that gap.

Layer 3 — persistent-volume data

The actual bytes on disk — database files, uploaded content. No etcd snapshot and no GitOps repo contains this. A PersistentVolumeClaim object is in those backups; its data is not. Protect it with:

  • Volume Snapshots — storage-level point-in-time snapshots via the CSI driver.
  • Velero’s node-agent (kopia/restic) — file-level copy of volume contents to object storage.
  • Storage-level replication — cross-region disk replication provided by the storage backend, independent of Kubernetes.

This is the layer most often forgotten, and the one whose loss is least recoverable — a deleted Deployment can be re-applied; deleted database files cannot be re-applied.

Layer 4 — the cluster definition itself

After a disaster you may need a new cluster to restore into. If the cluster is a hand-built snowflake, rebuilding it is slow, error-prone, and adds enormously to RTO. The fix is to make the cluster reproducible:

  • Cluster API — declares clusters as Kubernetes resources; a management cluster can recreate a workload cluster from its declarative spec (Cluster API — Introduction).
  • Infrastructure-as-Code (Terraform, Pulumi, CloudFormation) — the cluster’s VPC, node pools, IAM, and managed control plane are all code.
  • GitOps — once the empty cluster exists, GitOps repopulates it.

With layer 4 covered, “recover the cluster” becomes “run the IaC, then let GitOps reconcile” — a reproducible, drillable procedure rather than heroics.

Why “back up etcd” alone is insufficient

A plan that snapshots etcd and stops there fails on contact with a real disaster:

  • PV data is not in etcd. Restore etcd and your PVC objects come back pointing at volumes whose contents were never backed up.
  • External dependencies are not in etcd. Managed databases (RDS, Cloud SQL), external message queues, third-party APIs — the cluster references them but does not contain them.
  • DNS is not in etcd. External DNS records, certificate-manager-issued certs, and the mapping from public hostnames to the cluster’s ingress are outside the cluster.
  • Cloud load balancers are not in etcd. A LoadBalancer Service object is in etcd; the actual cloud LB is provisioned by the cloud-controller-manager. Restore etcd into a new cluster and the LBs (and their public IPs) are different.
  • The cluster definition is not in etcd. etcd is inside the cluster — if the cluster is gone you need somewhere to restore the snapshot to.

A complete plan addresses all four layers and these out-of-cluster dependencies explicitly.

Configuration / API Surface

DR is mostly procedure, not YAML — but the layers are concrete. A worked plan for a self-managed cluster with a 1-hour RTO and 30-minute RPO:

LAYER 1 — control-plane state (etcd)
  mechanism : etcdctl snapshot save, every 30 min, → S3 bucket in another region
  RPO       : 30 min  (= snapshot interval)
  restore   : etcdutl snapshot restore → new data-dir; stop+restart control plane
  see       : [[etcd Backup and Restore]]
 
LAYER 2 — application API objects
  mechanism : (a) GitOps repo = source of truth, reconciled by ArgoCD
              (b) Velero nightly + 6-hourly Backup → object storage
  RPO       : ~0 for declared state (Git);  ≤6h for runtime/generated objects (Velero)
  see       : [[GitOps]], [[Velero]]
 
LAYER 3 — persistent-volume data
  mechanism : CSI VolumeSnapshot every 30 min  +  cross-region snapshot copy
  RPO       : 30 min
  see       : [[Volume Snapshot]], [[Velero]]
 
LAYER 4 — cluster definition
  mechanism : Terraform (VPC, node pools, IAM) + Cluster API spec, both in Git
  RTO impact: empty cluster reproducible in ~15 min, unattended
  see       : [[Cluster API]]
 
OUT-OF-CLUSTER
  DNS       : external-dns config in Git; failover record TTL kept low (60s)
  certs     : cert-manager re-issues on the new cluster
  cloud LBs : new LBs provisioned on restore; DNS repointed to new ingress IP
 
DR DRILL   : full restore into a throwaway cluster, quarterly, RTO/RPO measured

Reading the plan: each layer names its mechanism, its RPO contribution, and its cross-link to the note that covers it. Layer 1’s RPO equals the etcd snapshot interval. Layer 2 splits — declared state has near-zero RPO because Git holds every commit, while generated/runtime state falls back to the Velero interval. Layer 4 is what keeps RTO low: a reproducible empty cluster means recovery is “run Terraform, apply the Cluster API spec, let GitOps reconcile,” not “rebuild a snowflake by hand.” The out-of-cluster section is the part etcd-only plans forget. The final line — the quarterly drill — is non-negotiable: a plan whose restore has never been executed is a hypothesis.

Failure Modes

“We backed up etcd” — and nothing else. The canonical failure. The etcd restore succeeds and the database files are still gone, the cloud LBs are new, DNS still points at the dead cluster. Recovery stalls on the layers the plan never covered.

Backups never tested. A snapshot that has never been restored hides truncated files, wrong credentials, missing KMS keys (etcd Encryption at Rest), storage-class mismatches. All of these surface for the first time during a real disaster, when there is no time to debug them. Quarterly DR drills are the only defense.

RPO/RTO unstated. Without explicit targets, “DR” is undefined — you cannot tell whether a 6-hour-old snapshot is acceptable or a catastrophe. Set the numbers with the business first; design to them second.

Backups in the same failure domain as the cluster. etcd snapshots on the etcd node, Velero backups in a bucket in the same region as the cluster. A regional outage takes the cluster and the backups. Store backups cross-region.

Lost encryption keys. etcd encrypted at rest, snapshot dutifully taken — but the KMS key is gone or inaccessible. The snapshot is unreadable ciphertext. The encryption key is part of the DR plan and must be backed up independently.

Restore into a mismatched cluster. Velero restore or etcd restore into a cluster whose Kubernetes version, storage classes, CSI drivers, or availability zones differ from the source — restore fails or produces broken volumes. The target must be compatible; layer 4’s reproducible definition helps guarantee that.

DNS / external dependency drift. The cluster recovers but public traffic still routes to the dead cluster because DNS TTLs were long or failover was manual. Keep failover-record TTLs low and script the repoint.

Operator-managed state assumed to be in Git. A database operator’s CRs may be in Git, but the database’s data and the operator’s reconstructed runtime status are not. Stateful workloads need explicit data-layer DR (Database Operators).

Alternatives and When to Choose Them

The DR strategy spectrum, ordered by cost and by how aggressively it meets RTO/RPO:

Rebuild-from-Git (backup-and-restore). No standby cluster. On disaster, provision a new cluster (layer 4), restore etcd / let GitOps reconcile (layer 2), restore PV data (layer 3). The manifests in Git are the backup — the cluster is fully reproducible. RTO is in the tens-of-minutes to hours range; cost is minimal (you pay only for backups and the IaC, not for idle infrastructure). Choose when RTO of an hour or more is acceptable — which is most internal and many production systems. This is the strategy GitOps makes cheap and reliable.

Cluster-pair / standby cluster (pilot light / warm standby). A second cluster exists in another region, kept current via continuous backup replication or GitOps reconciliation. On disaster, traffic fails over to the standby. RTO is minutes (the standby already exists; you mostly repoint traffic); RPO depends on replication lag. Cost is the standby’s infrastructure — “pilot light” keeps it minimal and scales it up on failover; “warm standby” keeps it running smaller-scale. Choose when an hours-long RTO is unacceptable but you can tolerate a brief failover window and a few minutes of data loss.

Multi-region active-active. Two or more clusters in different regions both serve live traffic, behind a global load balancer, with data replicated continuously. There is no “failover” — losing a region just removes capacity. RTO and RPO are near zero; cost is the highest (full duplicate infrastructure, plus the hard problem of multi-region data consistency). Choose only for systems where downtime is genuinely intolerable. Cross-link Multi-Region Active-Active Architecture.

Managed Kubernetes shifts the layer split. On EKS/GKE/AKS, layer 1 (etcd) is the provider’s problem and layer 4 (control plane) is partly theirs. Your DR plan narrows to layers 2 and 3 plus the node/IaC definition — a meaningful reduction in DR scope and a strong argument for managed clusters.

Production Notes

The lessons from the Kubernetes Failure Stories catalog and DR practice:

  • The backup you never restored is not a backup. This is the single most-repeated lesson. Teams that drill restores quarterly recover; teams that assume their backups work discover otherwise during the disaster. Make the drill a calendar event with a measured RTO/RPO outcome.
  • GitOps is the highest-leverage DR investment. A cluster whose entire desired-state is in Git and reconciled by ArgoCD/Flux turns “rebuild the cluster” into a near-mechanical procedure — and the same repo is your audit log and your change-management system. The manifests are the backup.
  • Stateful workloads are where DR plans fail. Stateless services rebuild trivially from Git. Databases and anything with PV data need a deliberate data-layer plan — volume snapshots, replication, application-consistent Velero hooks. Cross-link Stateful Workloads on Kubernetes.
  • Don’t forget the out-of-cluster surface. DNS, certificates, cloud load balancers, external managed databases, IAM — these live outside etcd and outside Git and are routinely the thing that stalls a recovery that “should have worked.”
  • State the numbers. A DR plan without an explicit RTO and RPO is not a plan. Get the business to commit the numbers, then prove the design meets them in a drill.
  • Disaster recovery and the cluster upgrade discipline converge. A blue-green cluster-replacement upgrade is a DR rehearsal — building a fresh cluster and migrating workloads is exactly the recovery procedure, run on a good day. Clusters that upgrade by replacement get DR practice for free.

See Also

Graph View

Backlinks