vault

Container Storage Interface

15 min read

Container Storage Interface

The Container Storage Interface (CSI) is the gRPC contract that decouples Kubernetes from any specific storage backend — block, file, object, or anything else a vendor can expose through three small gRPC services. It is the storage-layer sibling of the Container Runtime Interface: same out-of-tree plugin philosophy, same UNIX-domain-socket transport, same “let third parties ship and version their plugins independently of Kubernetes” design goal. CSI was promoted to GA in Kubernetes 1.13 (January 2019) per the Kubernetes 2019-01-15 announcement, having entered alpha in 1.9 and beta in 1.10. The corresponding CSI spec hit v1.0 alongside the K8s GA. Every modern storage driver in the Kubernetes ecosystem — AWS EBS, Google Persistent Disk, Azure Disk/File, Ceph RBD/CephFS, Portworx, Pure Storage, NetApp Trident, Longhorn, OpenEBS, Rook, vSphere CSI — is a CSI driver. The historical in-tree volume plugins (the kubernetes.io/aws-ebs, kubernetes.io/gce-pd, etc. provisioners) have been migrated to CSI under KEP-625; the core migration framework reached GA in 1.25, and in-tree code is being removed in subsequent releases.

Mental Model

flowchart TB
    subgraph "Kubernetes control plane"
        APISVR[kube-apiserver]
        CTRLMGR[kube-controller-manager<br/>PV/PVC binder]
    end

    subgraph "CSI driver Deployment (controller-side, runs on any node)"
        PROV[external-provisioner<br/>watches PVCs<br/>calls CreateVolume]
        ATT[external-attacher<br/>watches VolumeAttachments<br/>calls ControllerPublishVolume]
        RES[external-resizer<br/>watches PVC resize<br/>calls ControllerExpandVolume]
        SNAP[external-snapshotter<br/>watches VolumeSnapshot<br/>calls CreateSnapshot]
        DRIVER1[CSI driver container<br/>Identity + Controller services]
    end

    subgraph "CSI driver DaemonSet (node-side, runs on every node)"
        REG[node-driver-registrar<br/>registers driver socket with kubelet]
        DRIVER2[CSI driver container<br/>Identity + Node services]
        KUBELET[kubelet<br/>calls NodeStageVolume<br/>and NodePublishVolume]
    end

    BACKEND[(Storage backend<br/>AWS EBS / GCE PD / Ceph / NetApp / ...)]

    APISVR <--> PROV
    APISVR <--> ATT
    APISVR <--> RES
    APISVR <--> SNAP
    APISVR <--> REG
    PROV -- "gRPC over UDS" --> DRIVER1
    ATT --> DRIVER1
    RES --> DRIVER1
    SNAP --> DRIVER1
    KUBELET -- "gRPC over UDS" --> DRIVER2
    REG --> KUBELET
    DRIVER1 -- "vendor API" --> BACKEND
    DRIVER2 -- "vendor API + mount syscalls" --> BACKEND

What this diagram shows. A CSI driver in Kubernetes is not a single binary — it is a driver container packaged alongside several K8s-maintained sidecar containers that translate between the K8s API (PVCs, VolumeAttachments, VolumeSnapshots) and the driver’s CSI gRPC calls. The split is deliberate: the K8s community owns the API watchers (so the K8s ↔ gRPC translation logic isn’t re-implemented per vendor); the vendor owns only the actual storage-backend interaction. The driver typically ships as two workloads: a controller-side Deployment (one replica is enough; runs the CreateVolume / CreateSnapshot / Expand calls that don’t need to be on a specific node) and a node-side DaemonSet (one pod per node; runs the mount/unmount calls that must be on the node where the pod will run). The insight to extract: K8s does not itself know how to talk to any specific storage. The K8s control plane only knows the CSI sidecars and the kubelet only knows the node-side driver socket. Every actual attach, mount, or snapshot is a vendor-supplied gRPC handler.

Why CSI Exists

Before CSI, Kubernetes shipped storage support as in-tree volume plugins: one Go package per backend (pkg/volume/aws_ebs, pkg/volume/gce_pd, pkg/volume/azure_disk, pkg/volume/cinder, pkg/volume/vsphere, …), each compiled into the kubelet and kube-controller-manager binaries. The CSI GA announcement lists three concrete pains that drove the rewrite:

  1. Lifecycle coupling. Every storage driver shipped on the K8s release cadence — vendors couldn’t fix bugs or add features without waiting for a K8s minor release, and K8s couldn’t refactor internal volume interfaces without breaking every driver.
  2. Maintenance burden. Vendor code lived in the K8s repository; K8s maintainers had to test, build, and ship code they couldn’t fully understand. Many drivers were under-maintained because the original vendor employees had moved on.
  3. Reliability and security. Bugs (or worse, vulnerabilities) in a third-party storage driver crashed or compromised the kubelet itself. There was no isolation boundary.

CSI was the FlexVolume-then-CSI generation answer: define a versioned gRPC contract (the CSI spec), require every driver to implement that contract, and let the driver run out-of-tree in its own pods. Vendors gained release independence; K8s gained a stable seam; users gained the ability to install a CSI driver as just-another-Helm-chart instead of waiting for the next K8s release. The architectural rhyme with Container Network Interface (CNI, networking) and Container Runtime Interface (CRI, runtimes) is intentional: K8s’s three biggest plug points (network, storage, runtime) all use the same pattern — versioned external contract, vendor-supplied implementation, lifecycle independent of K8s itself.

The Three CSI Services

The CSI spec defines three gRPC services. A driver must implement Identity; it may implement Controller and/or Node depending on its capabilities. Per the spec:

Identity Service

Tiny — three RPCs:

service Identity {
    rpc GetPluginInfo(GetPluginInfoRequest) returns (GetPluginInfoResponse) {}
    rpc GetPluginCapabilities(GetPluginCapabilitiesRequest) returns (GetPluginCapabilitiesResponse) {}
    rpc Probe(ProbeRequest) returns (ProbeResponse) {}
}

GetPluginInfo returns the driver name (e.g. ebs.csi.aws.com, pd.csi.storage.gke.io) and version. GetPluginCapabilities reports which capabilities the driver supports — CONTROLLER_SERVICE (does it have a controller service?), VOLUME_ACCESSIBILITY_CONSTRAINTS (does it understand topology?), etc. Probe is a health check.

Controller Service

The volume-lifecycle API — called by the K8s-side sidecars, not by the kubelet:

service Controller {
    // Volume lifecycle
    rpc CreateVolume(...) returns (...);     // called by external-provisioner on PVC create
    rpc DeleteVolume(...) returns (...);     // called by external-provisioner on PV delete
 
    // Volume attach/detach (only for drivers that support PUBLISH_UNPUBLISH_VOLUME)
    rpc ControllerPublishVolume(...) returns (...);   // "attach": called by external-attacher
    rpc ControllerUnpublishVolume(...) returns (...); // "detach"
 
    // Volume expansion
    rpc ControllerExpandVolume(...) returns (...);    // called by external-resizer
 
    // Snapshots (only for drivers that support CREATE_DELETE_SNAPSHOT)
    rpc CreateSnapshot(...) returns (...);
    rpc DeleteSnapshot(...) returns (...);
 
    // Cloning is implicit in CreateVolume with a content source
 
    // Listing
    rpc ListVolumes(...) returns (...);
    rpc ListSnapshots(...) returns (...);
    rpc GetCapacity(...) returns (...);
 
    // Capability reporting
    rpc ControllerGetCapabilities(...) returns (...);
}

The controller service is centralized: it doesn’t care which node will eventually mount the volume; it talks to the storage backend’s management API (the EBS API, the Ceph monitor, the NetApp ONTAP API). This is why it can run as a Deployment with a single replica — there’s no per-node work here.

Node Service

The per-node operations — called exclusively by the kubelet on the same node:

service Node {
    // The two-stage mount sequence
    rpc NodeStageVolume(...) returns (...);     // mount to a node-global staging path
    rpc NodeUnstageVolume(...) returns (...);
    rpc NodePublishVolume(...) returns (...);   // bind-mount into the Pod's mount namespace
    rpc NodeUnpublishVolume(...) returns (...);
 
    // Online expansion (the file-system half)
    rpc NodeExpandVolume(...) returns (...);
 
    // Stats
    rpc NodeGetVolumeStats(...) returns (...);
 
    // Self-identification
    rpc NodeGetCapabilities(...) returns (...);
    rpc NodeGetInfo(...) returns (...);          // returns the node ID the backend uses
}

Stage versus Publish — the canonical CSI distinction

The two-RPC mount sequence is the most-tested-in-CSI-interviews detail. NodeStageVolume is called once per node: it formats the device (if needed) and mounts it to a global staging path (something like /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<volume-id>/globalmount). NodePublishVolume is called once per Pod consuming the volume on that node: it bind-mounts the staging path into the Pod’s specific mount directory (/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<name>/mount).

Why the split? ReadWriteMany (RWX) and ReadOnlyMany (ROX) access modes — a single backing volume mounted once per node and then bind-mounted into many Pods on that node. Without the stage step you’d be re-mounting the underlying device once per Pod, which would either fail (for filesystems that don’t tolerate multi-mount) or churn the kernel mount table unnecessarily. The spec also allows drivers to advertise STAGE_UNSTAGE_VOLUME as a capability they don’t support (typical for block-mode volumes that go straight to the Pod); in that case the kubelet skips the stage step.

Spec evolution since 1.0

The CSI spec at v1.0 (January 2019) defined the surface area above and has accrued capabilities since then in a backwards-compatible way. Recent milestones from the container-storage-interface/spec releases: v1.5 (2021) added VolumeExpansion online/offline distinctions; v1.7–v1.9 added ControllerModifyVolume for in-place storage-class parameter changes; v1.10 broadened that and clarified topology semantics; v1.11 (November 2024) moved VolumeGroupSnapshot to GA — the long-awaited consistency-group-snapshot RPCs that take a coordinated, point-in-time snapshot across multiple volumes in one shot. New capabilities are advertised via ControllerGetCapabilities / NodeGetCapabilities; the kubelet and sidecars feature-gate their use, so a driver pinned to an older spec version still interoperates with newer Kubernetes.

The Sidecar Pattern

Per kubernetes-csi.github.io/docs/sidecar-containers.html, the K8s community ships seven sidecar containers that a vendor packages alongside the driver:

SidecarWatches K8s resourceCalls CSI RPC
external-provisionerPVCs needing a PVCreateVolume, DeleteVolume
external-attacherVolumeAttachment objectsControllerPublishVolume, ControllerUnpublishVolume
external-resizerPVCs with grown spec.resources.requests.storageControllerExpandVolume
external-snapshotterVolumeSnapshot / VolumeSnapshotContent CRDsCreateSnapshot, DeleteSnapshot
node-driver-registrar(none — local)registers the driver’s UDS path with kubelet via the plugin-registration directory
livenessprobe(none — local)calls Identity Probe and exposes an HTTP endpoint kubelet liveness-probes
external-health-monitor-controllerPVsListVolumes (with volume condition) or ControllerGetVolume; surfaces unhealthy volumes as Events on PVCs/Pods

Each sidecar is maintained by SIG Storage. The vendor’s container only handles the actual backend interaction. This decomposition is what makes CSI drivers writable in a few hundred lines for simple cases (the hostPath driver is the reference example).

The volume-health story is split in two: the external-health-monitor-controller above runs alongside the controller-side driver and reports backend-detected volume problems. The complementary node-side check is no longer a separate sidecar — per the external-health-monitor README, the node-side agent was folded into kubelet itself in Kubernetes 1.21 behind the CSIVolumeHealth feature gate, which periodically calls NodeGetVolumeStats and emits Events when the driver reports volume_condition.abnormal = true. Older write-ups that show a separate external-health-monitor-agent DaemonSet are out of date.

The kubelet discovers a driver via the plugin registration mechanism (GA in 1.13 alongside CSI): the node-driver-registrar sidecar writes a socket and a registration JSON into /var/lib/kubelet/plugins_registry/, kubelet sees the new file, opens the socket, asks the driver “who are you?” via NodeGetInfo, and registers the driver name in its internal table.

CSI Migration: The Death of In-Tree Volumes

The other side of CSI’s success is getting rid of the old in-tree code. KEP-625 defines the migration framework: for each in-tree plugin (aws-ebs, gce-pd, azure-disk, azure-file, cinder, vsphere, …), provide a feature gate (CSIMigration<X>) that, when on, redirects all in-tree volume API calls to the corresponding CSI driver. Users continue to write kubernetes.io/aws-ebs in their StorageClasses; under the hood, the kubelet and controller-manager call the AWS EBS CSI driver. The combined timeline per the Kubernetes 1.25 migration update and the CSI sidecar changelog:

DriverBeta-on-by-defaultMigration GAIn-tree plugin status
Core CSIMigration framework1.17 (beta)1.25n/a
AWS EBS1.231.25In-tree code removed (1.27+)
GCE PD1.231.25In-tree code removed (1.28); gcePersistentDisk volume type marked deprecated in current docs
Azure Disk1.231.24Deprecated; removal in 1.26+
OpenStack Cinder1.211.24Deprecated; removal in 1.26+
Azure File1.241.26Removal staged for 1.28+
vSphere1.25 (beta)1.26Removal staged for 1.28+
Portworxn/a1.25Deprecated in current volume-types docs

The user-facing impact is minimal — old YAML still works — but operators must install the corresponding CSI driver (typically a Helm chart) on any cluster they upgrade to a release where migration is on by default, otherwise PVCs will hang in Pending. The managed K8s services (EKS, GKE, AKS) install the corresponding drivers automatically — AWS enabled CSI migration by default in EKS 1.23 (EBS CSI migration FAQ); GKE 1.22+ enables it transparently; AKS ships AzureDisk and AzureFile CSI drivers as managed addons. The migration’s user-visible “all old YAML still works” property is mediated by the kube-controller-manager and kubelet translating the legacy in-tree volume types into CSI calls at runtime; once the in-tree code is fully removed in a release, those legacy kubernetes.io/aws-ebs PVs still resolve correctly only because the translation shim remains until the deprecation policy fully retires it.

Uncertain

Verify: exact Kubernetes minor versions in which the in-tree volume plugin code (not just deprecation) was finally removed from kubernetes/kubernetes. The 1.25 status blog and changelog give targets through 1.28, but later releases may have shifted dates by one or two minor versions; for any specific plugin and current cluster version, cross-check the corresponding kubernetes.io/blog/.../storage-in-tree-to-csi-migration-status-update-<ver> post for that release.

Configuration / API Surface

A CSI driver is shipped as YAML. The minimum cluster-facing object is the CSIDriver resource (in storage.k8s.io/v1):

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: ebs.csi.aws.com           # MUST match what GetPluginInfo returns
spec:
  attachRequired: true            # does kubelet wait for VolumeAttachment to be Attached?
  podInfoOnMount: false           # inject Pod metadata into NodePublishVolume's context?
  fsGroupPolicy: File             # how to apply fsGroup on mount: None / File / ReadWriteOnceWithFSType
  volumeLifecycleModes:
    - Persistent                  # PVC-backed (the normal case)
    - Ephemeral                   # CSI inline ephemeral (rare; see [[Generic Ephemeral Volume]])
  storageCapacity: true           # advertise per-topology capacity so scheduler can pre-filter
  requiresRepublish: false        # does kubelet need to periodically re-call NodePublishVolume?
  seLinuxMount: true              # use mount-time SELinux relabeling for performance

A StorageClass referencing this driver (see StorageClass and Dynamic Volume Provisioning):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com      # driver name, matches the CSIDriver.metadata.name
parameters:                       # vendor-specific; opaque to K8s, forwarded to CreateVolume
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

The parameters block is passed verbatim to the driver’s CreateVolume RPC; the driver’s docs are the source of truth for what keys are accepted. (type here is an AWS EBS volume type; for a Ceph CSI driver it would be pool, clusterID, etc.)

Failure Modes

Driver pod not ready, PVCs stuck in Pending. Most common cause of “storage doesn’t work in my new cluster.” kubectl describe pvc shows waiting for a volume to be created, either by external provisioner ... indefinitely. Check: is the driver Deployment and DaemonSet healthy (kubectl -n kube-system get pods -l app=ebs-csi-controller)? Is the external-provisioner sidecar logging anything? Often the driver lacks IAM permissions (it can’t call the cloud API to create the volume).

Sidecar version skew. Each sidecar has a K8s version-compatibility matrix in its README. A too-old external-provisioner against a newer driver, or vice versa, manifests as silently-not-creating-volumes. The sidecar compatibility tables are mandatory reading before pinning versions.

Node-driver-registrar can’t reach kubelet. Driver DaemonSet pods crash-loop with failed to register with kubelet. Cause: /var/lib/kubelet/plugins_registry/ is not mounted into the pod (the DaemonSet manifest forgot the hostPath). Fix: copy the manifest exactly from the driver’s deployment instructions; the hostPath mounts are non-optional.

RPC timeouts on NodeStageVolume. The kubelet uses a fixed timeout for CSI RPCs (default 2 minutes); if NodeStageVolume takes longer (slow mkfs on a huge volume), the kubelet retries and crictl ps shows the same volume being stage-mounted twice. Long-term fix is in the driver (do the format asynchronously); short-term fix is to pre-format the volumes.

Stuck VolumeAttachment. After a node dies, its VolumeAttachment objects can sit in attached: true indefinitely because the attacher can’t reach the dead node to detach. The volume can’t move to another node until the attachment is gone. The node’s out-of-service taint (since 1.26) is the mechanism to tell the attacher “force-detach this node’s attachments.”

Driver migration breakage. On clusters where CSI migration was newly enabled (1.23–1.25), Pods using in-tree storage broke if the CSI driver wasn’t installed. Symptom: kubectl describe pod shows Volume not attached according to node status. Fix: install the corresponding CSI driver (the migration only redirects to it, it doesn’t ship it).

Alternatives and When to Choose Them

  • FlexVolume: the predecessor of CSI; a script-on-disk interface (kubelet exec’s a binary in a known path with subcommands init / attach / mount / ...). Still in K8s but deprecated since 1.23, scheduled for removal. New drivers must be CSI; existing FlexVolume drivers should be migrated.
  • In-tree plugins: still present but soft-deprecated by CSI migration. New StorageClass usage should reference CSI driver names directly (e.g. ebs.csi.aws.com), not the legacy kubernetes.io/aws-ebs.
  • HostPath / local volumes: a degenerate alternative for single-node or fixed-mapping use cases. No CSI involvement; the kubelet mounts a path on the node directly. Useful for development clusters and for the local-volume static provisioner on bare metal.

For any new storage integration in K8s, CSI is the only correct choice. The cost of writing a CSI driver (versus other plugin systems) is mostly amortised: the csi-driver-host-path reference, the csi-driver-nfs generic NFS driver, and the csi-test sanity test suite mean a new driver is weeks of work, not months.

Production Notes

  • AWS EKS: the AWS EBS CSI driver is the only supported block storage backend; the in-tree plugin is fully migrated. EFS uses a separate efs.csi.aws.com driver for RWX. EBS volumes are zonal, so volumeBindingMode: WaitForFirstConsumer is mandatory or pods will be unschedulable when their PV is in a different AZ.
  • GKE: the Compute Engine Persistent Disk CSI Driver (pd.csi.storage.gke.io) ships preinstalled. GKE Autopilot enforces CSI exclusively — no in-tree volumes at all.
  • Azure AKS: AzureDisk and AzureFile CSI drivers ship preinstalled; migration to CSI was completed via the AKS managed addon.
  • On-prem (Ceph, VMware, NetApp): Rook-Ceph operator deploys both ceph-csi-rbd (block) and ceph-csi-cephfs (file) drivers; vSphere CSI Driver (vSphere 7.0+) replaces the legacy vSphere in-tree plugin; NetApp’s Trident is itself a CSI driver.
  • CSI driver as a security boundary. A misbehaving driver can mount arbitrary host paths into Pods via NodePublishVolume. Treat CSI driver IAM and admission policy as carefully as you’d treat the kubelet’s. The CSIDriver.spec.volumeLifecycleModes field limits whether the driver can be used inline-ephemeral (without cluster-admin PVC creation rights) — leave it set to Persistent only unless you specifically need ephemeral.

See Also

Graph View

Backlinks