vault

Velero

13 min read

Velero

Velero (formerly Heptio Ark) is the canonical open-source tool for backing up, restoring, and migrating Kubernetes clusters (velero.io). It originated at Heptio, came to VMware (and then Broadcom) with that acquisition, and as of early 2026 was accepted into the Cloud Native Computing Foundation at the Sandbox maturity level — the most nascent of the three CNCF tiers (Sandbox → Incubating → Graduated), reflecting that CNCF stewardship is recent rather than the long-graduated status some secondary write-ups assume (CNCF Sandbox application #457; the velero.io footer declares “Cloud Native Computing Foundation sandbox project”). The current release line is v1.18 (v1.18.1, May 2026) (Velero GitHub releases). It operates at the API-object level: an in-cluster controller queries the kube-apiserver for the resources you ask for — filtered by namespace, label, or resource kind — serializes them into a tarball, and uploads that tarball to object storage (S3, GCS, Azure Blob) (Velero — How Velero works). For workloads with state, Velero also backs up persistent-volume data, via two distinct paths: CSI volume snapshots (a storage-level point-in-time snapshot of the disk) or its node-agent file-level path (formerly restic, now kopia), which copies the volume’s files. Velero supports scheduled backups (cron), selective restore, and backup hooks that run commands inside Pods before and after the backup — e.g., to quiesce a database for a consistent copy. Its two headline use cases are disaster recovery of an application’s namespaces and cluster migration — back up cluster A, restore into cluster B. Crucially, Velero is complementary to etcd snapshots, not a replacement: Velero excels at namespace-scoped application backup and migration; an etcd snapshot is the tool for whole-cluster control-plane state.

Mental Model

The mental model is back up the cluster the way you’d back up a database: query the canonical store, serialize the result, store it elsewhere. Velero treats the apiserver as the source of truth for what objects exist and treats the volume layer as a separate concern for what data lives in those objects’ disks. It deliberately does not touch etcd directly — it goes through the apiserver, which means it sees the same logical objects a user sees, with the same RBAC and the same API semantics.

flowchart TB
    subgraph CLUSTER["Source cluster"]
        APIS[kube-apiserver]
        subgraph VEL["Velero (in-cluster)"]
            BC[Backup Controller]
            NA["node-agent DaemonSet<br/>(kopia / restic uploader)"]
        end
        PV[(PersistentVolumes)]
        APIS <--> BC
    end

    BC -->|"1. query API objects<br/>(filtered by ns / label / kind)"| APIS
    BC -->|"2. tarball of objects"| OBJ["Object storage<br/>(BackupStorageLocation:<br/>S3 / GCS / Azure Blob)"]
    BC -->|"3a. CSI snapshot request"| CSI["CSI driver<br/>→ storage-level VolumeSnapshot"]
    NA -->|"3b. file-level copy of volume"| OBJ
    PV --- CSI
    PV --- NA

    OBJ -.->|"restore (same or DIFFERENT cluster)"| TARGET["Target cluster<br/>(DR / migration)"]

What this diagram shows. The Backup Controller does three things: query the apiserver for objects (1), write them as a tarball to object storage (2), and back up volume data either via a CSI snapshot (3a, storage-level) or via the node-agent’s file copy (3b, file-level). The dashed arrow is the payoff — the same backup in object storage can be restored into the same cluster (DR) or a different cluster (migration). The insight to extract: Velero cleanly separates “the objects” (always backed up) from “the volume data” (backed up by one of two interchangeable mechanisms), and both land in portable object storage — which is what makes cross-cluster restore possible.

Mechanical Walk-through

Backing up API objects

When a Backup custom resource is created (via velero backup create or a Schedule), the Backup Controller validates it and queries the apiserver for the resources in scope (Velero — How Velero works). Scope is set by filters: include/exclude by namespace, by label selector, by resource kind. The controller collects the matching objects, serializes them, and uploads the tarball to a BackupStorageLocation — a configured bucket in S3, GCS, or Azure Blob.

One honest limitation: a Velero backup is not strictly atomic. The controller iterates over resources; an object created or edited during the backup’s iteration may be captured inconsistently or missed. In practice this is rare, but for a workload that must be backed up at a single consistent instant, backup hooks (below) are the answer.

Backing up persistent-volume data — two paths

A PersistentVolumeClaim object is just API state and is captured in the tarball like any other object. But the data on the underlying disk is not — that needs one of two mechanisms (Velero — CSI, Velero — File System Backup):

  1. CSI volume snapshots. Velero asks the CSI driver to take a storage-level Volume Snapshot of the volume — a point-in-time snapshot managed by the storage system (EBS snapshot, GCP PD snapshot, etc.). Mechanically, Velero’s BackupItemAction plugins fire first against each PersistentVolumeClaim; when a PVC is backed by a CSI driver, the action creates a VolumeSnapshot object against an appropriate VolumeSnapshotClass, and the upstream CSI external-snapshotter controller then produces the VolumeSnapshotContent and drives the storage system’s snapshot API (Velero — CSI). The CSI logic was folded into Velero core in v1.14 (no longer a separately-installed velero-plugin-for-csi), but it is still gated behind the EnableCSI feature flag — velero install --features=EnableCSI — so a cluster that “should” snapshot via CSI but never enabled the flag silently falls back to file-system backup or no volume data at all. This path is storage-platform-dependent and more consistent (the storage layer guarantees a coherent point-in-time image), but only works where the storage provider supports CSI snapshots.

  2. File-system backup via the node-agent. Velero runs a node-agent DaemonSet — one Pod per node — which copies the volume’s files directly. The original implementation used restic; the current default and only file-level data mover is kopia. The restic path follows a staged deprecation (Velero — File System Backup): in v1.15–v1.16 a restic-path backup still creates and succeeds but emits warnings; in v1.17–v1.18 creating new restic-path backups is disabled while restoring from previously-created restic backups is still allowed; from v1.19 onward both new restic backups and restores of old restic backups are prohibited. The node-agent reconciles PodVolumeBackup / PodVolumeRestore custom resources. File-system backup works with any volume type — including NFS, EFS, emptyDir, and local volumes that have no CSI snapshot capability — and is portable across storage platforms, but it is less consistent (it copies a live filesystem) and requires the node-agent to have root access.

The two are mutually exclusive per volume: if file-system backup is enabled for a volume, Velero skips the CSI snapshot for it. Selection is by annotation — opt-in (annotate backup.velero.io/backup-volumes=<vol> on the Pod for each volume to file-back-up) or opt-out (back up all volumes with file-system backup except those in backup.velero.io/backup-volumes-excludes, enabled cluster-wide with --default-volumes-to-fs-backup).

Backup hooks

For workloads that need a consistent backup — a database that must flush and quiesce before its volume is copied — Velero supports backup hooks: commands run inside a Pod’s container before (pre) and after (post) the volume backup (Velero — Backup Hooks). A typical pattern: a pre-hook runs fsfreeze or a database’s FLUSH TABLES WITH READ LOCK / checkpoint command so the on-disk state is coherent at backup time, and a post-hook releases it. Hooks turn Velero’s non-atomic, file-level backup into an application-consistent one.

Scheduled backups

A Schedule resource carries a cron expression; Velero creates a Backup on that cadence (Velero — Schedule). Each scheduled backup is named <schedule-name>-<timestamp>. A schedule also typically sets a TTL so old backups are garbage-collected from object storage.

Restore

A Restore resource references an existing backup and recreates its objects in a cluster (Velero — Restore Reference). Restore can be full or selective (the same namespace/label/resource filters apply), supports namespace remapping (restore namespace prod into prod-recovered), and can run in a read-only / dry-run mode. Restoring into a different cluster is what makes Velero a migration tool — and a disaster-recovery tool, since “a different cluster” can be a freshly built replacement.

Configuration / API Surface

A Schedule with a hook, and a Restore:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: nightly-prod-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"                 # cron — every day at 02:00
  template:                            # the Backup spec produced on each tick
    includedNamespaces:
      - prod                           # scope: only the prod namespace
    excludedResources:
      - events                         # don't bother backing up Events (high-churn, low-value)
    snapshotVolumes: true              # back up PV data via CSI snapshots
    defaultVolumesToFsBackup: false    # not the file-level path for this backup
    ttl: 168h0m0s                      # retain each backup 7 days, then GC from object storage
    hooks:
      resources:
        - name: quiesce-postgres
          includedNamespaces: [prod]
          labelSelector:
            matchLabels: {app: postgres}
          pre:                         # run BEFORE the volume backup
            - exec:
                container: postgres
                command: ["/bin/sh", "-c", "psql -c 'CHECKPOINT;'"]
                onError: Fail          # abort the backup if the checkpoint fails
---
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-prod-into-dr
  namespace: velero
spec:
  backupName: nightly-prod-backup-20260516020000   # which backup to restore
  includedNamespaces: [prod]
  namespaceMapping:
    prod: prod-dr                                  # remap: restore into a different namespace / cluster
  restorePVs: true                                 # also restore the persistent-volume data

Line-by-line: spec.schedule is a standard cron string; spec.template is literally a Backup spec stamped out on each tick. includedNamespaces: [prod] is the scoping filter — Velero shines at namespace-scoped backup, which is exactly this. excludedResources: [events] drops the high-churn, low-value Events resource. snapshotVolumes: true selects the CSI-snapshot path for PV data; defaultVolumesToFsBackup: false means the node-agent file path is not the default here. ttl: 168h is the retention window. The hooks block runs a CHECKPOINT inside the Postgres container before the volume is captured, with onError: Fail so a non-consistent backup is never produced silently. In the Restore, backupName points at one timestamped backup; namespaceMapping remaps prod → prod-dr, which is the migration/DR primitive; restorePVs: true brings the volume data along.

Failure Modes

Backup is not atomic. Objects mutated during the backup’s iteration can be captured inconsistently. For databases and any consistency-critical workload, use a pre/post hook to quiesce — without it, a Velero backup of a live database can be a crash-consistent-at-best, torn-at-worst image.

Node-agent not installed, PV data silently not backed up. File-system backup requires the node-agent DaemonSet (--use-node-agent at install time). If it is absent and CSI snapshots are also unavailable, the Backup succeeds — but it captured only the objects, not the volume data. The restored PVC points at an empty/wrong volume. Always verify the backup actually includes volume data.

restic deprecation. Velero began deprecating the restic uploader in v1.15 (backups still succeed with warnings through v1.16); v1.17–v1.18 disable creating new restic backups but still permit restoring from previously-created ones; from v1.19 both new restic backups and restic restores are prohibited. Clusters still on restic must migrate to the kopia data mover and re-take their backups under kopia before upgrading Velero to v1.19, or those old backups become unrestorable (Velero — File System Backup).

CSI snapshot unsupported by the storage backend. CSI volume snapshots only work where the CSI driver implements snapshot capability. NFS, EFS, emptyDir, and local volumes have no CSI snapshot — for those you must use the node-agent file-level path or the data is not captured.

Cross-cluster restore breaks on storage-class / topology mismatch. Restoring a backup into a different cluster can fail if the target cluster’s StorageClass names, CSI drivers, or availability zones differ from the source. Migration requires aligning storage classes (or using velero storage-class mappings) and ensuring the target has compatible volume infrastructure.

Restoring cluster-scoped resources can clobber the target. A full restore that includes cluster-scoped resources (CRDs, ClusterRoles, PriorityClasses) into a live cluster can overwrite or conflict with what is already there. Selective restore filters mitigate this.

Object-storage credentials / location misconfigured. A BackupStorageLocation pointing at an unreachable or mis-permissioned bucket produces Backup objects stuck in a failed/partial phase. Monitor backup phase, not just the existence of Schedule objects.

Alternatives and When to Choose Them

Velero vs etcd snapshots — the central comparison. They solve overlapping but distinct problems:

  • Use an etcd snapshot when you need to recover the entire cluster’s control-plane state — every namespace, every object, RBAC, CRD instances, the lot — as one coherent image. An etcd snapshot is the complete whole-cluster backup; Velero’s object-by-object tarball does not faithfully reconstruct an entire etcd’s internal state.
  • Use Velero when you need namespace-scoped application backup, selective restore, persistent-volume data, or cluster migration. An etcd snapshot captures no PV data and cannot be selectively restored or moved to a different cluster.
  • They are complementary. A mature DR plan for a self-managed cluster uses both: etcd snapshots for the control plane, Velero for application namespaces and volume data.

Storage-level replication (cloud-native). Provider snapshot/replication features (cross-region EBS snapshot copy, etc.) protect volume data at the storage layer, independent of Kubernetes. Choose alongside Velero when you want volume DR decoupled from the cluster lifecycle.

GitOps. If application desired-state lives in Git and is reconciled by ArgoCD/Flux, the manifests are already a backup of the API objects — re-point a new cluster at the repo and it rebuilds. GitOps does not back up PV data or imperatively-created/generated objects, so Velero (or volume snapshots) still covers that gap. GitOps + Velero is a common pairing: Git for desired-state, Velero for data and runtime state.

Commercial backup platforms (Kasten K10, Portworx PX-Backup, Trilio). Vendor products with policy management, app-aware backups, and broader UIs, often built on or alongside Velero’s primitives. Choose when you need enterprise policy/compliance features and vendor support; Velero itself is the open-source baseline.

Production Notes

The recurring lessons from Velero’s docs and DR practice:

  • Hooks for stateful workloads, always. A Velero backup of a database without a quiescing pre-hook is a gamble. Database operators (Database Operators) increasingly ship Velero-hook annotations or their own backup CRs precisely because file-level volume copy is not application-consistent by itself.
  • Verify backups include what you think. A Backup in phase Completed can still be missing PV data if the node-agent wasn’t installed or CSI snapshots weren’t configured. Periodically restore a backup into a scratch namespace/cluster and check the data.
  • Test cross-cluster restore before you need it. Migration and DR both rely on restore-into-a-different-cluster working — and it is exactly where storage-class and topology mismatches bite. Drill it.
  • TTL and retention. Without a ttl on Schedules, backups accumulate in object storage indefinitely, raising cost and clutter. Set TTL to match the recovery-window requirement.
  • Velero is the canonical migration tool, used widely to move workloads between clusters during cluster upgrades-by-replacement (see Kubernetes Cluster Upgrade blue-green strategy), cloud migrations, and cluster consolidations — back up namespace by namespace from the old cluster, restore into the new one.

See Also

Graph View

Backlinks