vault

PersistentVolume

13 min read

PersistentVolume

A PersistentVolume (PV) is a cluster-scoped API resource that represents a piece of storage that has been provisioned by an administrator or dynamically created by a StorageClass. The Kubernetes documentation defines it: “A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV.” (kubernetes.io — Persistent Volumes). The point of the PV resource is to give cluster operators a stable handle on storage that lives outside the lifetime of any Pod. A Pod might be deleted, evicted, or migrated; the PV (and its backing disk, NFS export, Ceph image, etc.) persists. A separate, namespaced PersistentVolumeClaim (PVC) is the user-facing request that binds to a PV — the PV/PVC indirection separates “what storage exists in the cluster” (admin’s concern) from “what storage does my application need” (user’s concern), in the same way that Node/Pod separates “what compute exists” from “what compute does my application need”. This note covers the PV side; see PersistentVolumeClaim for the user-facing claim side and StorageClass for the template that turns claims into PVs dynamically.

Mental Model

stateDiagram-v2
    [*] --> Available: PV created<br/>(static or dynamic)
    Available --> Bound: PVC matches and binds (1:1, exclusive)
    Bound --> Released: PVC deleted
    Released --> Available: reclaim=Recycle<br/>(deprecated, removed)
    Released --> [*]: reclaim=Delete<br/>(backing storage destroyed)
    Released --> Failed: reclaim action failed
    Failed --> [*]: admin manual recovery
    Released --> Released: reclaim=Retain<br/>(admin must clean up manually)

What this diagram shows. A PV’s status.phase walks a small state machine. It starts Available when it’s first created (statically by an admin, or dynamically by the external-provisioner sidecar in response to a PVC). It transitions to Bound when a matching PVC claims it — this is an exclusive 1:1 binding, recorded in the PV’s spec.claimRef field. When the PVC is later deleted, the PV moves to Released and what happens next is governed by the PV’s persistentVolumeReclaimPolicy: Delete tears down the backing storage and removes the PV (the most common case for dynamically provisioned PVs in cloud); Retain leaves the PV in Released forever, requiring an admin to manually clear claimRef to make it Available again (the safe default for irreplaceable data); Recycle used to do an rm -rf /thevolume/* and put the PV back to Available, but is removed in current K8s. The insight to extract: the PV is a lifecycle envelope around a piece of storage, not the storage itself, and the reclaim policy is what decides what happens to the real disk when the K8s object representing it goes away.

Mechanical Walk-through

What a PV literally contains

A PV is a K8s API object whose spec has these slots:

  • capacity.storage — the size, e.g. 100Gi. Not enforced by K8s on the data path (the file system enforces it); it’s the matching criterion against PVC size requests.
  • accessModes — a set of strings from {ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod}, abbreviated RWO, ROX, RWX, RWOP. Declares what the PV is capable of; the PVC’s accessModes declare what the user wants. The PVC must request a subset of what the PV offers. See Access Modes (PV) for the full treatment.
  • persistentVolumeReclaimPolicy — Delete, Retain, or (historically) Recycle. The state machine arrow that fires when the PVC is deleted.
  • storageClassName — names the StorageClass this PV belongs to (or empty string for “no class”). PVCs must request the same class to bind.
  • volumeMode — Filesystem (default) or Block. Determines whether the volume is mounted as a directory or attached as a raw block device.
  • mountOptions — list of strings passed to the underlying mount call (nfsvers=4.1, hard, noatime, etc.).
  • nodeAffinity — constrains which nodes can mount the PV. Essential for local PVs (the disk physically lives on one node) and zone-aware cloud volumes (an EBS volume in us-east-1a can only mount to nodes in us-east-1a).
  • The volume source — exactly one of the volume-type fields from Volume Types (Kubernetes): csi, nfs, hostPath, local, or any of the deprecated in-tree cloud fields. This is the actual pointer to the storage.

A PV’s status.phase is one of Available, Bound, Released, Failed (the state machine in the diagram).

Static vs dynamic provisioning

Static provisioning (admin pre-creates PVs):

  1. Admin manually creates a PV object pointing to pre-provisioned storage (an existing EBS volume, an NFS export, a pre-formatted local disk).
  2. PV enters Available.
  3. User creates a PVC; the controller (kube-controller-manager’s persistentvolume-controller) finds an Available PV that satisfies the claim (size ≥ requested, accessModes ⊇ requested, storageClassName matches, selector matches).
  4. Controller sets PV.spec.claimRef = PVC, PVC.spec.volumeName = PV. Both become Bound.
  5. Pod references the PVC; kubelet calls CSI driver (or in-tree plugin) to attach and mount the volume.

Dynamic provisioning (PVC triggers PV creation on demand):

  1. User creates a PVC with storageClassName: fast-ssd.
  2. The external-provisioner sidecar (which runs alongside the CSI driver in the cluster) watches PVCs. It sees the new claim, looks up the StorageClass, and calls CreateVolume on the CSI driver.
  3. CSI driver creates the actual cloud disk (e.g., calls EBS CreateVolume against the AWS API).
  4. external-provisioner creates a corresponding PV object pointing at the new disk’s CSI handle.
  5. PV is born in Bound state, directly attached to the triggering PVC. (No intermediate Available state — the PV is created for this PVC.)
  6. Pod references the PVC; kubelet mounts.

The mechanics of step 2–4 are documented in Dynamic Volume Provisioning. The key point: dynamically provisioned PVs are born bound — they never enter the matching loop because they’re created for a specific PVC. Statically provisioned PVs go through the matching loop.

Binding rules

The persistentvolume-controller runs a matching loop on every PVC create/update. A PV matches a PVC if all of these hold:

  • pv.spec.capacity.storage >= pvc.spec.resources.requests.storage (size sufficient).
  • pv.spec.accessModes ⊇ pvc.spec.accessModes (capabilities sufficient).
  • pv.spec.storageClassName == pvc.spec.storageClassName (same class, including empty == empty).
  • If PVC has spec.selector, the PV’s labels must satisfy it.
  • If PVC has spec.volumeName: pv-foo, only pv-foo is considered (pre-binding).
  • PV is Available (not already bound to a different PVC).

Among matching PVs, the controller picks the smallest one that still satisfies the request — a 50Gi PVC won’t grab a 1Ti PV if a 50Gi PV is available, minimizing waste. If no match exists, the PVC stays Pending; the controller retries on every event.

Reclaim policies and what they do

When a Pod stops using a PVC and the PVC is deleted (a manual kubectl delete pvc, or namespace deletion), the bound PV moves to Released and reclaim runs:

  • Delete (default for dynamically provisioned PVs from a StorageClass with reclaimPolicy: Delete, which is itself the default StorageClass setting). The external-provisioner sidecar’s csi-snapshotter partner calls DeleteVolume on the CSI driver, which tells the cloud API to destroy the disk. Then the PV object is deleted. Irreversible — the data is gone.
  • Retain (the safe default for statically provisioned PVs). The PV stays in Released indefinitely. The backing storage is untouched. An admin can later kubectl edit pv to clear spec.claimRef, which puts the PV back to Available, ready to bind to a new PVC. This is how you do storage hand-off across namespaces: data PVC in team-a ns, PVC deleted (e.g., namespace migration), admin re-binds the same PV to a new PVC in team-b ns.
  • Recycle (REMOVED). Historically: rm -rf the volume contents, then PV → Available. This was a footgun (silent data destruction) and only ever worked for hostPath and NFS. Deprecated long ago, removed entirely; PVs with reclaimPolicy: Recycle now error.

volumeMode: Filesystem vs Block

  • Filesystem (default). The kubelet formats the volume (if needed), mounts it at a path inside the Pod’s container. Application reads/writes files. This is what 99% of workloads want.
  • Block (stable since K8s 1.18). The volume is exposed to the container as a raw block device at a devicePath (e.g. /dev/xvda). No filesystem in between. Use cases: databases that manage their own block storage (Oracle, some Cassandra deployments), workloads that need direct block I/O for performance (DPDK-based packet processors that mmap raw devices), dd-style block-level backup tools. The Pod consumes a block PVC via containers[].volumeDevices[], not containers[].volumeMounts[].

Access modes

The four modes:

  • ReadWriteOnce (RWO): mountable read-write by a single node at a time. Multiple Pods on the same node can share. The most common mode; what most cloud block storage (EBS, GCE PD, Azure Disk) actually supports physically. (Pre-K8s 1.22 docs said “single Pod”; the precise definition is “single node”.)
  • ReadOnlyMany (ROX): mountable read-only by many nodes. Useful for static content (training data, static-asset bundles).
  • ReadWriteMany (RWX): mountable read-write by many nodes simultaneously. Requires the backing storage to support concurrent multi-node writes — NFS, CephFS, EFS, Azure Files, GlusterFS (deprecated). Not supported by block-storage backends (EBS, GCE PD, Azure Disk).
  • ReadWriteOncePod (RWOP) — GA in K8s 1.29 (Kubernetes blog — RWOP GA, KEP-2485). Mountable read-write by exactly one Pod cluster-wide at a time. Stronger than RWO (which allows multiple Pods on the same node). Use case: stateful workloads like databases where two Pods accidentally writing to the same volume would corrupt data. RWOP is CSI-only — requires CSI sidecars at certain versions (csi-provisioner:v3.0.0+, csi-attacher:v3.3.0+). See Access Modes (PV).

The PV declares what it can do; the PVC declares what the user wants. The matching loop checks subset relation.

PV protection finalizers

Two finalizers prevent accidental destruction:

  • kubernetes.io/pv-protection: prevents deletion of a PV while it’s still bound to a PVC. You’d have to delete the PVC first (or use --force --grace-period=0).
  • kubernetes.io/pvc-protection: prevents deletion of a PVC while a Pod is still using it. Combined, these protect against the “delete the volume out from under a running database” footgun.

When you see a kubectl delete pv hang with status Terminating, the finalizer is still set — find what’s holding it.

Configuration / API Surface

A statically provisioned NFS PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: shared-data
spec:
  capacity:
    storage: 1Ti                                      # matched against PVC size requests
  accessModes:
    - ReadWriteMany                                   # NFS supports concurrent writers
  persistentVolumeReclaimPolicy: Retain               # admin manages this PV's lifecycle
  storageClassName: ""                                # no dynamic class; manual provisioning
  mountOptions:
    - nfsvers=4.1
    - hard                                            # block on server outage rather than fail
    - timeo=600
  nfs:                                                # the volume source
    server: nfs.internal.example.com
    path: /exports/shared

A PVC requesting storageClassName: "", accessMode RWX, size ≤ 1Ti, in any namespace, would bind to this PV. After binding, deleting the PVC moves the PV to Released and leaves the NFS export untouched — exactly the behavior wanted for shared team data.

A dynamically provisioned PV (as it appears after creation)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-abc123-7890-...                          # auto-generated by external-provisioner
  finalizers:
    - kubernetes.io/pv-protection
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete               # inherited from StorageClass
  storageClassName: gp3
  volumeMode: Filesystem
  claimRef:                                            # bound to this PVC
    namespace: app
    name: postgres-data
    uid: ...
  csi:                                                 # CSI source — the modern path
    driver: ebs.csi.aws.com
    volumeHandle: vol-0abc1234...                      # the AWS EBS volume ID
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: "..."
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.ebs.csi.aws.com/zone
              operator: In
              values:
                - us-east-1a                           # EBS volume in zone 1a
status:
  phase: Bound

Note nodeAffinity — the EBS volume is zone-local; the PV’s node affinity prevents the scheduler from placing the consuming Pod in any zone other than us-east-1a.

A raw block PV

spec:
  capacity:
    storage: 100Gi
  volumeMode: Block                                    # raw block device, no filesystem
  accessModes:
    - ReadWriteOnce
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-0xyz...

The Pod consumes this with volumeDevices instead of volumeMounts:

containers:
  - name: db
    volumeDevices:
      - name: data
        devicePath: /dev/xvda                          # raw block device appears here

Failure Modes

  • PVC stays Pending forever. No matching PV exists and the named StorageClass either doesn’t exist or has no provisioner. Diagnose: kubectl describe pvc shows no persistent volumes available for this claim and no storage class is set or failed to provision volume with StorageClass. Fix: create a matching PV, or fix the StorageClass.
  • PV stuck in Released after PVC deletion. Reclaim policy is Retain and admin hasn’t cleared claimRef. Fix: kubectl edit pv <name>, delete spec.claimRef, PV returns to Available.
  • PV stuck in Failed after reclaim. The CSI driver’s DeleteVolume call failed (cloud API throttling, permissions issue, volume already deleted by hand). The PV is marked Failed and reclaim doesn’t retry automatically. Fix: investigate the CSI driver logs, manually delete the backing storage if needed, force-delete the PV.
  • Zone-mismatch unschedulability. A gp3 StorageClass with volumeBindingMode: Immediate provisions a PV in us-east-1b; the consuming Pod is constrained (by affinity, taints, or capacity) to us-east-1c. The Pod cannot start; the PV cannot move. Fix: set volumeBindingMode: WaitForFirstConsumer on the StorageClass (the canonical mitigation — see StorageClass).
  • accessModes mismatch. PVC requests RWX but the matched-or-provisioned PV only offers RWO (e.g., the StorageClass is gp3, which is block storage). Binding fails or, more confusingly, succeeds (because RWO is a subset of RWX requested) but the second Pod can’t mount. Fix: pick a backend that supports the access mode you need (EFS for RWX, not EBS).
  • Recycle reclaim policy on a current cluster. Errors at PV creation time — the policy is removed. Fix: use Delete or Retain.
  • Resize doesn’t propagate to filesystem. Editing PVC’s resources.requests.storage triggers the CSI driver to grow the backing disk, but for filesystems that require an online resize (xfs_growfs, resize2fs), the kubelet must coordinate. Some CSI drivers/filesystems require a Pod restart. Diagnose: kubectl describe pvc shows FileSystemResizePending.

Alternatives and When to Choose Them

NeedUseWhy not the others
Pod-lifetime ephemeral storageemptyDir or Generic Ephemeral VolumeNo need for a long-lived PV
Persistent storage, ad-hocPVC with dynamic provisioningAvoid manual PV creation; StorageClass handles it
Persistent storage, hand-curatedStatic PV with Retain reclaimCluster operator controls the disks directly
Shared volume across nodesPV with RWX accessMode, NFS/EFS/CephFS backendBlock-storage PVs can’t do this
Single-writer guaranteePV with RWOP accessModeRWO permits multiple Pods on the same node
Raw block I/OPV with volumeMode: BlockFilesystem mode adds a layer the app may not want

Production Notes

  • Reclaim policy choice is a one-way risk decision. A PV with Delete is fast to clean up but destroys data on PVC deletion (and PVC deletion can be triggered by namespace deletion). The 2024 Buoyant Linkerd-deprovision incident and various public k8s.af writeups feature a “deleted the wrong namespace, lost the database” story. Production guidance: Retain for any PV holding state you can’t reconstruct from elsewhere; Delete for ephemeral state, dev clusters, and pipelines.
  • Static provisioning is rare today. With CSI everywhere, dynamic provisioning is the default. Static PVs survive mostly for: pre-existing storage being adopted into K8s, NFS exports managed outside K8s, on-prem clusters without a CSI driver.
  • Bound-but-orphaned PVs. Cluster-deletion-and-recreate scenarios (cluster recreation, cluster migration) leave the cloud-provider disks behind. These are “zombie PVs” — the K8s objects are gone but the underlying disks exist. Recoverable by hand-importing them into the new cluster as static PVs with Retain policy.
  • Use kubectl get pv -A -o wide to audit. Sort by reclaim policy. Anything Delete-policy holding stateful data is a footgun waiting to fire.

See Also

Graph View

Backlinks