vault

Dynamic Volume Provisioning

13 min read

Dynamic Volume Provisioning

Dynamic Volume Provisioning is the Kubernetes feature that materializes a PersistentVolume (PV) on-demand when a user creates a PersistentVolumeClaim (PVC) — no cluster administrator pre-provisioning required. The PVC names a StorageClass; the StorageClass names a provisioner (now invariably a CSI driver); the provisioner’s external-provisioner sidecar watches PVCs that need a PV and calls the driver’s CreateVolume gRPC to actually create storage on the backend (an AWS EBS volume, a Google Persistent Disk, a Ceph RBD image, etc.). The corresponding PV object is then created, bound to the PVC, attached to the node the Pod gets scheduled to, and mounted into the Pod’s filesystem. This dance is the one feature that makes Kubernetes storage self-service — without it, every PVC would require a cluster admin to first create a matching PV by hand. Documented authoritatively at kubernetes.io/docs/concepts/storage/dynamic-provisioning/.

Mental Model

sequenceDiagram
    autonumber
    actor User
    participant API as kube-apiserver
    participant Sched as kube-scheduler
    participant Prov as external-provisioner<br/>(sidecar)
    participant Driver as CSI driver<br/>(Controller service)
    participant Att as external-attacher
    participant Kubelet as kubelet (on chosen node)
    participant Backend as Storage backend<br/>(EBS / PD / Ceph)

    User->>API: kubectl apply Pod + PVC<br/>(PVC names StorageClass "fast")
    API->>API: store PVC in etcd (status: Pending)
    Note over Prov: external-provisioner watches PVCs<br/>with matching StorageClass

    alt volumeBindingMode == Immediate (default)
        Prov->>Driver: CreateVolume(name, capacity, params)
        Driver->>Backend: create EBS volume (in some AZ)
        Backend-->>Driver: volumeId, accessibleTopology
        Driver-->>Prov: CreateVolumeResponse
        Prov->>API: create PV (with topology constraint)
        API->>API: bind PV<->PVC
    else volumeBindingMode == WaitForFirstConsumer
        Note over Prov: external-provisioner DOES NOTHING yet
        User->>API: (Pod is also queued because PVC unbound)
        Sched->>API: pick node N for Pod (considers PVC's SC topology)
        API->>API: update PVC with "selected-node: N" annotation
        Prov->>Driver: CreateVolume(name, ..., accessibilityRequirements: zone(N))
        Driver->>Backend: create volume in N's AZ
        Driver-->>Prov: response
        Prov->>API: create PV bound to PVC
    end

    Sched->>API: schedule Pod to node N (now PVC is bound)
    API->>Att: VolumeAttachment{volume, node N}
    Att->>Driver: ControllerPublishVolume(volumeId, nodeId)
    Driver->>Backend: attach EBS to EC2 instance N
    Att->>API: VolumeAttachment.status.attached=true

    API->>Kubelet: Pod assigned to you (and volume attached)
    Kubelet->>Driver: NodeStageVolume(volumeId, stagingPath)
    Driver->>Driver: mkfs (if first time) + mount to staging
    Kubelet->>Driver: NodePublishVolume(volumeId, podMountPath)
    Driver->>Driver: bind-mount staging -> Pod's mount namespace
    Kubelet->>Kubelet: start Pod's containers

What this diagram shows. Dynamic provisioning is an end-to-end orchestration involving five distinct actors (apiserver, scheduler, provisioner sidecar, CSI driver, attacher sidecar, kubelet) interacting via the K8s API. The control plane doesn’t actually mediate the storage operations — it mediates intent: the PVC says “I want 30 GiB of fast storage,” the StorageClass says “fast means EBS gp3,” and the sidecars + driver carry that intent down to the EBS API. The single most important branch is volumeBindingMode: Immediate provisions the volume the moment the PVC is created (risk: the volume lands in a zone the Pod can’t be scheduled to); WaitForFirstConsumer (WFFC) delays provisioning until the scheduler picks a Pod’s node, then provisions in that node’s zone. For zonal block storage (EBS, GCE PD, Azure Disk), WFFC is effectively mandatory.

Mechanical Walk-through

Step 0 — the StorageClass

A cluster administrator (or the cloud provider’s bootstrap) creates one or more StorageClass objects. Each StorageClass picks one CSI driver via the provisioner field, and supplies opaque vendor-specific parameters. The StorageClass also sets cluster-wide policy: reclaim policy, expansion permission, volume binding mode, allowed topologies.

Step 1 — the user creates a PVC

The PVC is a namespaced request for storage: “give me 30 GiB, ReadWriteOnce, of StorageClass fast.” If the user omits storageClassName, the cluster’s default StorageClass (the one annotated storageclass.kubernetes.io/is-default-class: "true") is filled in by the DefaultStorageClass admission controller. If there’s no default and the user didn’t name one, the PVC will sit Pending forever — this is the most common cause of “my Pod won’t start in a fresh cluster.”

Step 2 — the external-provisioner reacts

Each CSI driver’s deployment includes the K8s-maintained external-provisioner sidecar (per kubernetes-csi.github.io/docs/external-provisioner.html). It informer-watches every PVC and PV in the cluster. When it sees a PVC that (a) names a StorageClass whose provisioner matches the driver’s name and (b) is not yet bound to a PV, it kicks off provisioning. The provisioner picks a globally-unique PV name (typically pvc-<UUID>), then calls the CSI driver’s controller service:

CreateVolume(
    name = "pvc-72d9a349-aacd-42d2-a240-d775650d2455",
    capacity_range = {required_bytes: 30 * 2^30},
    volume_capabilities = [{access_mode: SINGLE_NODE_WRITER, mount: {fs_type: "ext4"}}],
    parameters = {... StorageClass.parameters verbatim ...},
    accessibility_requirements = {... topology hints, if WFFC ...},
)

Step 3 — the CSI driver creates the backing volume

The driver translates this into a call to the storage backend’s management API. For EBS, that’s ec2:CreateVolume (with VolumeType: gp3, Iops: 3000, etc., from parameters). For Ceph RBD, rbd create. For NetApp, ontap volume create. The driver returns a CSI Volume object with a volume_id (the backend’s opaque handle) and optionally accessible_topology (e.g. “this volume exists in AZ us-east-1a only”).

Step 4 — the provisioner creates the PV

The external-provisioner takes the driver’s response and writes a PV K8s object:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-72d9a349-aacd-42d2-a240-d775650d2455
spec:
  capacity:
    storage: 30Gi
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Delete
  storageClassName: fast
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-09abcdef                # from CreateVolumeResponse
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:                       # zonal pinning from accessible_topology
        - matchExpressions:
            - key: topology.ebs.csi.aws.com/zone
              operator: In
              values: [us-east-1a]
  claimRef:                                    # back-pointer to the PVC that triggered this
    name: my-data
    namespace: default
    uid: 72d9a349-aacd-42d2-a240-d775650d2455

The claimRef and the pv.kubernetes.io/bound-by-controller annotation on the PVC together represent the binding: K8s’s PV controller (in kube-controller-manager) sees the two pointers agree and sets PVC.status.phase = Bound, PV.status.phase = Bound.

Step 5 — scheduling and attach

The Pod referencing the PVC is now scheduleable. The scheduler picks a node respecting the PV’s nodeAffinity (e.g. only nodes in us-east-1a). Once the Pod is bound to a node, the PV controller creates a VolumeAttachment object. The external-attacher sidecar sees the new VolumeAttachment, calls the driver’s ControllerPublishVolume(volume_id, node_id), which translates to (for EBS) ec2:AttachVolume. The driver returns when the volume is attached at the OS level (visible as a block device).

Step 6 — mount

The kubelet on the chosen node sees the Pod has been assigned to it, sees the PVC and its bound PV, and calls the driver’s node service: NodeStageVolume (format and mount to staging path) followed by NodePublishVolume (bind-mount into the Pod’s mount namespace). Containers in the Pod now see the filesystem at the path specified by their volumeMounts.

Step 7 — reclaim

When the PVC is deleted, the chain runs in reverse:

  1. Kubelet NodeUnpublishVolume + NodeUnstageVolume (when the Pod is gone).
  2. external-attacher calls ControllerUnpublishVolume (ec2:DetachVolume).
  3. The PV’s persistentVolumeReclaimPolicy decides what happens next: Delete (the default for dynamically-provisioned PVs) triggers external-provisioner to call the driver’s DeleteVolume (ec2:DeleteVolume — the data is gone forever). Retain leaves the PV in Released state and the backing volume untouched; an admin must manually clean up or re-bind.

The default Delete policy is convenient for stateless tests but dangerous for production data — many platform teams change the default to Retain cluster-wide via mutating webhook to prevent accidental data loss.

WaitForFirstConsumer — Topology-Aware Binding

The volumeBindingMode field on a StorageClass is the most subtle but most consequential setting in this whole machine. Originating from KEP-1432 / the 2018 topology-aware blog post:

  • Immediate (the legacy default): the moment a PVC is created, the external-provisioner calls CreateVolume immediately. The driver picks some zone (its accessible_topology); the PV is pinned to that zone via nodeAffinity. Then the Pod is scheduled. If the cluster has nodes in zones A, B, C and the volume lands in zone B but only zone A has capacity, the Pod is unschedulable.

  • WaitForFirstConsumer (the modern default in most clouds): the external-provisioner does nothing when the PVC is created. The PVC’s status stays Pending. The scheduler — using the VolumeBinding scheduler plugin — knows the PVC’s StorageClass and its (multi-zone) accessible topology, considers the Pod’s other constraints (other PVCs, NodeAffinity, etc.), picks a single optimal node, and writes volume.kubernetes.io/selected-node: <node-name> onto the PVC. The external-provisioner sees the annotation and now calls CreateVolume with accessibility_requirements constraining the volume to the chosen node’s zone.

WFFC is mandatory for zonal block storage (EBS, GCE PD, Azure Disk) and for any cluster with topology-aware scheduling. The cost is one extra round trip before the Pod can start: the volume creation no longer overlaps with scheduling. The benefit is that Pods are never unschedulable due to mismatched volume zones, and the scheduler can co-locate multiple PVCs on the same zone in one decision.

Configuration / API Surface

A complete StorageClass for AWS EBS gp3 with all the dials set:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-fast
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"   # cluster default
provisioner: ebs.csi.aws.com                              # CSI driver name
parameters:
  type: gp3                                                # EBS volume type
  iops: "3000"                                             # per-volume IOPS
  throughput: "125"                                        # MiB/s
  encrypted: "true"                                        # at-rest encryption
  kmsKeyId: "alias/aws/ebs"                                # KMS key
  csi.storage.k8s.io/fstype: ext4                          # filesystem type
reclaimPolicy: Delete                                      # default; PV deletes on PVC delete
allowVolumeExpansion: true                                 # online resize via PVC edit
volumeBindingMode: WaitForFirstConsumer                    # mandatory for EBS (zonal)
allowedTopologies:                                          # optional: restrict to a subset of AZs
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values: [us-east-1a, us-east-1b]
mountOptions:                                              # passed verbatim to mount(8)
  - discard                                                # TRIM on delete (good for SSDs)

A PVC consuming it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: default
spec:
  accessModes: [ReadWriteOnce]                             # RWO; EBS doesn't support RWX
  storageClassName: gp3-fast                               # name of the StorageClass above
  resources:
    requests:
      storage: 100Gi                                       # what gets passed to CreateVolume

If storageClassName is omitted entirely, the default StorageClass is filled in. If storageClassName: "" is explicitly set, dynamic provisioning is disabled and the PVC will only bind to a manually-created PV — useful when you want to ensure no auto-provisioning happens for a particular workload.

Failure Modes

PVC stuck in Pending forever. The single most common storage problem. Causes (run kubectl describe pvc and read the events):

  • No matching StorageClass. Failed to provision volume with StorageClass "fast": ... not found. Fix: create the StorageClass.
  • No default StorageClass and PVC didn’t name one. Event: no persistent volumes available for this claim and no storage class is set. Fix: set a default StorageClass or annotate the PVC.
  • Driver not installed. Event: external-provisioner not running. Fix: install the corresponding CSI driver Helm chart.
  • Cloud quota. Event: failed to provision volume ... VolumeLimitExceeded. Fix: raise the cloud-side quota.
  • WFFC and no Pod consuming the PVC. Event: waiting for first consumer to be created before binding. This is normal — the PVC is intentionally waiting. Create a Pod that mounts it.

Pod stuck in ContainerCreating after PVC bound. The PVC is bound but the volume can’t attach or mount. Causes:

  • Node in wrong zone. The PV is in us-east-1a but the scheduler put the Pod in us-east-1b (this should not happen with WFFC; it happens with Immediate binding when the scheduler ignored or couldn’t honor the PV’s nodeAffinity).
  • IAM / RBAC. The driver lacks permission to call ec2:AttachVolume (or equivalent). Driver logs show AccessDenied.
  • Multi-attach. Two Pods on different nodes both reference an RWO PVC. The first one attaches; the second one’s VolumeAttachment fails. Symptom: Multi-Attach error for volume. Fix: use a StatefulSet (which guarantees one Pod per identity), or change access mode to RWX (and switch to a backend that supports it).

Orphaned PV after PVC deletion. If the reclaim policy is Retain, deleting the PVC leaves the PV in Released state forever. The backing storage continues to bill. Cleanup: delete the PV and the underlying volume manually, or change reclaim policy to Delete before deleting the PVC.

DeleteVolume failures. When a PVC is deleted with reclaimPolicy: Delete, the provisioner calls DeleteVolume. If the call fails (e.g. backend reports the volume is still attached, or the driver has lost cloud credentials), the PV stays in Released with a finalizer. The external-provisioner retries with exponential backoff. If you want to give up and delete the PV anyway, remove the finalizer manually — but the backing storage is now leaked.

Volume expansion stuck. Edit a PVC’s spec.resources.requests.storage upward; the PVC gains a Resizing condition. If the StorageClass doesn’t have allowVolumeExpansion: true, the request is rejected. If the driver doesn’t support online expansion, the volume expansion completes on the backend but the filesystem inside the Pod still shows the old size — kill the Pod to trigger NodeExpandVolume on Pod restart, or use a driver that supports online FS expansion.

Alternatives and When to Choose Them

  • Static provisioning — an admin creates the PV manually, the user creates a matching PVC. Useful when (a) the backing storage was created by some other process (e.g. an existing NFS export, a pre-existing EBS volume restored from backup), or (b) you want explicit, audited control over what gets provisioned. The trade-off is operational toil: every new workload requires admin intervention.
  • Generic ephemeral volumes (Generic Ephemeral Volume) — embed a PVC template directly in the Pod spec; the ephemeral-volume controller creates a PVC at Pod-create time and deletes it at Pod-delete time. Use when you want CSI features (snapshots, specific StorageClass, encryption) on data that’s tied to the Pod’s lifetime, not stored long-term.
  • emptyDir — Pod-lifetime scratch; no driver involvement; backed by node-local disk or tmpfs. Use for caches and temporary workspaces; not for anything you’d cry about losing. Compare with Generic Ephemeral Volume for the richer Pod-lifetime story.

For any persistent storage in a managed cluster, dynamic provisioning is the default and the right choice. The combination StorageClass + PVC + CSI is the Kubernetes-native idiom; static provisioning is the escape hatch.

Production Notes

  • Default to Retain, not Delete, for critical data classes. Many production platforms set the default StorageClass’s reclaim policy to Retain and rely on a separate cleanup process (a backup-and-then-delete sweep) to actually release storage. The cost of a stuck-forever PV is operational toil; the cost of an accidental delete is irrecoverable data loss.
  • Always WaitForFirstConsumer on zonal storage. EKS, GKE, and AKS managed StorageClasses use WFFC by default. Custom StorageClasses on top of zonal drivers should match.
  • Storage capacity tracking (storageCapacity: true) lets the scheduler avoid binding a PVC to a node whose zone has run out of capacity. CSI 1.4+, K8s 1.21+. Newer drivers all enable it; some custom drivers don’t.
  • kubectl get events is the right tool for PVC debugging. Almost every dynamic-provisioning problem shows up as a clear-text event on the PVC or Pod. kubectl describe pvc aggregates them; kubectl get events --sort-by=.lastTimestamp orders them.
  • Provisioner versioning. Each driver’s external-provisioner sidecar is pinned; mismatched versions across DaemonSet pods can lead to confusing intermittent failures. Use the driver’s bundled Helm chart rather than hand-rolling manifests.

See Also

Graph View

Backlinks