vault

StatefulSet

18 min read

StatefulSet

A StatefulSet is the Kubernetes API object that manages the deployment and scaling of a set of Pods with stable identities, stable persistent storage, and ordered, predictable lifecycle. Where a Deployment treats its replicas as interchangeable cattle, a StatefulSet treats them as named, ordinally-indexed individuals: web-0, web-1, web-2. Each Pod is created in order, gets its own PersistentVolumeClaim cloned from a template, retains a stable DNS name backed by a companion Headless Service, and is replaced on update in reverse ordinal order — the new Pod inheriting the old Pod’s identity and storage. The Kubernetes documentation captures the guarantee concisely: “manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods” (kubernetes.io — StatefulSet). This guarantee is the entire reason StatefulSet exists. It is what makes Kubernetes capable of running distributed databases (Postgres operators, MySQL/Vitess, MongoDB, Elasticsearch), consensus systems (etcd, Zookeeper, Consul), and partitioned services (Kafka, Cassandra, RabbitMQ clusters) without giving up the orchestration benefits of the platform. Conversely, StatefulSet is the wrong answer for nearly anything stateless — its ordered semantics are slower, more brittle, and more operationally demanding than Deployment’s; reach for it only when you genuinely need identity.

Mental Model

flowchart TB
    SS[StatefulSet 'web'<br/>replicas=3<br/>serviceName='nginx'<br/>volumeClaimTemplates]
    HS["Headless Service 'nginx'<br/>clusterIP: None<br/>selector: app=nginx"]
    POD0["Pod web-0<br/>identity: web-0<br/>PVC: data-web-0<br/>DNS: web-0.nginx.ns.svc.cluster.local"]
    POD1["Pod web-1<br/>identity: web-1<br/>PVC: data-web-1<br/>DNS: web-1.nginx.ns.svc.cluster.local"]
    POD2["Pod web-2<br/>identity: web-2<br/>PVC: data-web-2<br/>DNS: web-2.nginx.ns.svc.cluster.local"]
    PV0[(PV data-web-0<br/>retained on Pod delete)]
    PV1[(PV data-web-1)]
    PV2[(PV data-web-2)]

    SS -- "creates in order:<br/>0 → 1 → 2" --> POD0
    SS --> POD1
    SS --> POD2
    SS -- "binds via<br/>volumeClaimTemplates" --> PV0
    SS --> PV1
    SS --> PV2
    POD0 -. "PVC reused across restarts" .-> PV0
    POD1 -.-> PV1
    POD2 -.-> PV2
    HS -- "DNS A records per Pod" --> POD0
    HS --> POD1
    HS --> POD2

    POD1 -. "peer discovery via DNS<br/>web-0.nginx... web-2.nginx..." .-> POD0
    POD2 -.-> POD0
    POD2 -.-> POD1

What this diagram shows. A StatefulSet named web with replicas: 3 and serviceName: nginx produces three Pods named web-0, web-1, web-2 — created strictly in that order, each waiting for the previous to be Running and Ready before the next is created (the default OrderedReady policy). Each Pod has a dedicated PersistentVolumeClaim minted from the StatefulSet’s volumeClaimTemplates, with predictable PVC names like data-web-0 that survive Pod deletion: when web-1 is rescheduled or restarted, the new Pod gets the same PVC bound to the same underlying PV with the same data. The companion headless Service nginx (with clusterIP: None) instructs CoreDNS to expose per-Pod DNS records of the form <pod-name>.<service-name>.<namespace>.svc.cluster.local so peer Pods can discover each other by name — the foundation for clustered apps that need to establish quorum, elect a leader, or shard data based on stable membership. The insight to extract is that every part of a Pod’s identity that matters for distributed-systems coordination — name, DNS name, storage — is stable across restarts. This is what Deployments cannot give you, and what makes StatefulSet the right primitive for databases.

Mechanical Walk-through

The four guarantees

The StatefulSet contract, as enumerated in the documentation (kubernetes.io):

  1. Stable, unique network identifiers. Each Pod’s hostname is <statefulset-name>-<ordinal> (web-0, web-1, …). The companion headless Service publishes DNS A records keyed on this name so peers can resolve each other. The hostname survives Pod restarts and reschedules — a new web-1 Pod after the old one died has the same name.
  2. Stable, persistent storage. Each Pod gets a PVC instantiated from spec.volumeClaimTemplates, with name <template-name>-<statefulset-name>-<ordinal> (e.g., data-web-0). The PVC is not deleted when the Pod is deleted, scaled down, or the StatefulSet itself is deleted — so a rescheduled Pod re-attaches to the same data.
  3. Ordered, graceful deployment and scaling. Pods are created/scaled-up in order 0, 1, 2, …, N-1, each only after the previous is Running and Ready. Scale-down proceeds in reverse: N-1, N-2, …, 0, each only after the previous has fully terminated. This is the OrderedReady policy (default); Parallel policy lifts the ordering for both creation and deletion (but still preserves the identity guarantees).
  4. Ordered, automated rolling updates. On spec.template change, Pods are replaced in reverse ordinal order (N-1, N-2, …, 0), one at a time, each waiting for the previous replacement to be Ready before proceeding. The partition field allows phased rollouts (canary by ordinal). The OnDelete strategy turns this off — replacements happen only when the user manually deletes a Pod.

The companion headless Service

A StatefulSet requires (by convention; not enforced by the API server) a Headless Service referenced via spec.serviceName. A headless Service is a Service with spec.clusterIP: None — it gets no virtual IP, but CoreDNS publishes per-Pod A records for it. With a normal Service, nginx.ns.svc.cluster.local resolves to a single virtual IP that load-balances over Pods; with a headless Service, it resolves to the set of all Pod IPs, and additionally <pod-name>.<service-name>.<namespace>.svc.cluster.local resolves to the specific Pod’s IP. This is what gives peers stable, name-based addressing — see Kubernetes DNS Specification.

The most common operational mistake is forgetting to create the headless Service. The StatefulSet starts, Pods get ordinal names, but DNS doesn’t resolve them — and any clustering protocol that relies on peer addresses by name fails to bootstrap. Diagnose with nslookup web-0.nginx.default.svc.cluster.local from inside a Pod.

Pod identity is immutable in a way Pod IPs aren’t

The Pod IP changes on restart (Pod IP is allocated by the CNI plugin at Pod start; see Pod Networking). The Pod hostname does not — kubelet sets the Pod’s hostname to <statefulset-name>-<ordinal> at start. The headless Service’s DNS records are updated to point at the new Pod IP. So:

  • Cassandra Pod cass-0 restarts; gets a new IP 10.244.0.42 → 10.244.0.51.
  • DNS cass-0.cassandra.cass.svc.cluster.local now resolves to 10.244.0.51.
  • Peer Pod cass-1 re-resolves the name; reconnects.

This is the magic that makes clustered apps survive on Kubernetes: the names are the stable address surface, not the IPs.

PVC lifecycle

Each volumeClaimTemplate becomes one PVC per ordinal. When the StatefulSet is scaled from 3 to 5, two new PVCs are created (data-web-3, data-web-4). When scaled from 5 back to 3, the two extra PVCs are not deleted by default — they remain bound to their PVs, holding the data, ready for a future scale-up to reuse them.

The PVC-retention-on-scale-down behavior is configurable via spec.persistentVolumeClaimRetentionPolicy (with whenDeleted and whenScaled fields, each accepting Retain or Delete). This is the StatefulSetAutoDeletePVC feature: it was alpha in v1.23 (k8s.io blog, Dec 2021), beta in v1.27 (k8s.io blog, May 2023), and graduated to GA in v1.32 — the feature gate now shows Stage: GA, Since: 1.32, Default: true (feature-gates reference, verified 2026-05-22; corroborated by the KEP-1847 implementation history: “1.23, alpha … 1.27, graduation to beta … 1.32, graduation to GA”). Because it is GA-and-default-on, the persistentVolumeClaimRetentionPolicy field is honored on any cluster from v1.32 onward without enabling anything.

The default behavior is retain on all delete and scale-down events (both fields default to Retain), which is the safe choice for stateful data and matches pre-feature behavior. The trade-off: PVCs (and underlying PVs, depending on the StorageClass’s reclaimPolicy) accumulate over the life of the cluster. Backups, snapshots, and explicit cleanup are the user’s responsibility — see Volume Snapshot, Velero.

Pod management policies

Two options (kubernetes.io — pod-management-policies):

  • OrderedReady (default). Create 0 → wait for Ready → create 1 → wait for Ready → … On scale-down, terminate N-1 → wait for full termination → terminate N-2 → … This is the conservative choice for cluster bootstrapping where each node must register/sync with previous nodes (Zookeeper, Cassandra, Galera MySQL).
  • Parallel. Create all Pods concurrently; delete all concurrently. The ordinal-name and PVC-template guarantees are still in force, but the sequencing guarantee is dropped. Useful for apps that have their own clustering bootstrap that doesn’t depend on K8s-level ordering — e.g., a sharded Postgres where each shard is independent.

podManagementPolicy is immutable after creation. Switching requires deleting the StatefulSet (without cascade-deleting Pods, via --cascade=orphan) and recreating.

Update strategies

Two options (kubernetes.io — update-strategies):

  • RollingUpdate (default). On spec.template change, Pods are replaced in reverse ordinal order (N-1, N-2, …, 0), one at a time, each replacement waiting for the previous to be Ready (and, since K8s 1.25 stable, additionally for minReadySeconds to elapse — see Kubernetes 1.25 blog post). The reverse order is deliberate: in many distributed apps, the lowest-ordinal Pod is the “primary” or “seed” — updating it last reduces total disruption.

    The partition field gates which Pods get updated: partition: 2 means only Pods with ordinal ≥ 2 get the new template; ordinals 0 and 1 keep the old template until partition is decreased. This is the native canary mechanism for StatefulSets — bump the partition down one Pod at a time, observe, proceed.

  • OnDelete. The StatefulSet controller does not update Pods on template change. Users must manually kubectl delete pod web-2 to have it recreated with the new template. This is the escape hatch for apps where the right rolling-update order is application-specific (e.g., update the leader last, after demoting it) — a custom operator or manual procedure can drive the deletion order.

MaxUnavailable for StatefulSet rolling updates

spec.updateStrategy.rollingUpdate.maxUnavailable is gated by the MaxUnavailableStatefulSet feature gate (KEP-961). It was introduced as Alpha (off by default) in v1.24, where it sat for ten release cycles, and was finally promoted to Beta (on by default) in v1.35 — the feature-gates reference shows the rows MaxUnavailableStatefulSet | false | Alpha | 1.24 | 1.34 followed by MaxUnavailableStatefulSet | true | Beta | 1.35 (feature-gates reference, verified 2026-05-22). So on a cluster older than v1.35 you must explicitly enable the gate on both the kube-apiserver and the kube-controller-manager (a skew between them either silently ignores the field or rejects it at admission); on v1.35+ the field is accepted by default, though the default value of maxUnavailable remains 1, so behavior is unchanged until you raise it.

maxUnavailable allows a StatefulSet rolling update to replace multiple Pods at once (subject to the bound), at the cost of giving up the strict reverse-ordinal-one-at-a-time guarantee. Most production users do not need this — the whole point of StatefulSet is conservative serial rollouts. Choose this only when you have empirically measured that your app tolerates parallel restarts.

minReadySeconds (stable in 1.25)

The Kubernetes 1.25 release graduated spec.minReadySeconds for StatefulSet to stable (kubernetes.io blog). The semantics match the Deployment field: a Pod must be Ready and minReadySeconds must have elapsed since it became Ready before the controller counts it Available and proceeds with the next Pod in the rollout. This is critical for stateful apps that require post-Ready warmup (cache hydration, secondary index build, log replay) before being safe to put under load.

Configuration / API Surface

A StatefulSet running a Postgres-flavored peer-discoverable cluster, with line-by-line commentary:

apiVersion: v1
kind: Service                              # the REQUIRED companion headless service
metadata:
  name: postgres
  namespace: db
spec:
  clusterIP: None                          # 'None' = headless; CoreDNS publishes per-Pod A records
  selector:
    app: postgres
  ports:
    - port: 5432
      name: pg
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres                    # MUST match the headless Service name
  replicas: 3                              # postgres-0 (primary), postgres-1, postgres-2 (replicas)
  podManagementPolicy: OrderedReady        # default; switch to Parallel if your app prefers
  minReadySeconds: 30                      # wait 30s after Ready before counting Available
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0                         # 0 = update all Pods; set higher for canary by ordinal
  selector:
    matchLabels:
      app: postgres                        # MUST match template labels; immutable
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 60    # databases want time to flush
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
              name: pg
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg-secret
                  key: password
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name # 'postgres-0' etc — app uses this for replication setup
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
          volumeMounts:
            - name: data                   # MUST match a volumeClaimTemplate name below
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:                    # per-Pod PVCs created from this template
    - metadata:
        name: data                         # PVCs named: data-postgres-0, data-postgres-1, ...
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3-encrypted    # the CSI driver behind this provisions actual PV
        resources:
          requests:
            storage: 100Gi
  persistentVolumeClaimRetentionPolicy:    # PVC lifecycle on scale-down / StatefulSet deletion
    whenDeleted: Retain                    # don't auto-delete PVCs when StatefulSet is deleted
    whenScaled: Retain                     # don't auto-delete PVCs on scale-down

Key surface points:

  • serviceName is mandatory but not validated. The API server accepts a StatefulSet with a serviceName referring to a nonexistent Service. The StatefulSet starts; Pods are created; but DNS lookups fail. Always create the Service before (or together with) the StatefulSet.
  • volumeClaimTemplates is append-only-ish. Adding new templates works; modifying existing ones is rejected; removing them strands PVCs. Plan carefully.
  • The DNS pattern is fixed. <pod-name>.<service-name>.<namespace>.svc.cluster.local is wired into CoreDNS. The full FQDN — and the shorter <pod-name>.<service-name> if the resolver is in the same namespace — is what cluster apps should use as their peer addresses.
  • apps.kubernetes.io/pod-index is automatically added as a label on each StatefulSet Pod, exposing the ordinal so operators can select Pods by index without parsing the name. This is the PodIndexLabel feature — alpha in v1.28, stable (gate locked on) in v1.31 (feature-gates reference, verified 2026-05-22).
  • The start ordinal field (spec.ordinals.start) reached stable in v1.31 (per the StatefulSet concept page, FEATURE STATE: Kubernetes v1.31 [stable]); it lets ordinal numbering start from a value other than 0 — useful for migrations where you want to start fresh from myapp-7 without renaming. Pods are then assigned ordinals from .spec.ordinals.start through .spec.ordinals.start + .spec.replicas - 1.

Failure Modes

  1. Missing headless Service. The most common StatefulSet error. Pods come up with correct names, but nslookup web-0.nginx.default.svc.cluster.local returns NXDOMAIN. Apps fail to bootstrap because peer discovery doesn’t resolve. Diagnose: kubectl get svc nginx; ensure it exists with clusterIP: None.

  2. Stuck rollout because Pod N-1 never becomes Ready. Reverse-ordinal rollout means the highest-ordinal Pod is updated first. If it never becomes Ready, the rollout stalls there forever. Common causes: a new image with a bug; a new readiness probe that’s stricter; a missing dependency. Diagnose: kubectl get pods; the highest-ordinal Pod will be in Pending/CrashLoopBackOff. Either fix the template and let it retry, or set updateStrategy.partition high enough to skip the broken Pod while you investigate.

  3. OrderedReady deadlock at scale-down. If Pod web-1 is stuck (not terminating cleanly, perhaps because a finalizer is blocking it or a volume detach is hung), web-0 will not be deleted on scale-down to 0 — the controller is waiting for web-1 to fully terminate first. Diagnose: kubectl get pods shows web-1 in Terminating; check finalizers and volume status. Force-delete with kubectl delete pod web-1 --force --grace-period=0 if necessary, accepting the data-corruption risk.

  4. PVC and PV out of sync after deletion. Deleting a StatefulSet (without --cascade=orphan) deletes the Pods but the default whenDeleted: Retain retains PVCs. If you then recreate the StatefulSet with the same name, the new Pods adopt the existing PVCs — the data is preserved. But if the new PodSpec mounts a different path or expects a different schema, you get silent inconsistencies. The defensive operator: explicitly kubectl delete pvc -l app=postgres after deleting the StatefulSet, when you actually want fresh storage.

  5. OrderedReady paired with slow-starting apps. A 100-replica Cassandra cluster with OrderedReady and 90-second minReadySeconds takes at least 150 minutes to deploy from scratch. Often a misjudgment — use Parallel for apps whose bootstrap is genuinely independent of ordering.

  6. Forgetting that spec.replicas change is not a rolling update. Scaling a StatefulSet from 3 to 5 creates web-3 and web-4 with the current template; scaling back to 3 deletes them. Template changes are what trigger rolling updates. A user expecting “rolling update + scale” in one apply will get only the scale; the template change needs a subsequent rollout.

  7. Headless Service selector mismatch. If the Service’s selector doesn’t match the StatefulSet’s Pod labels, no DNS records are published. Pods are running with stable names but unreachable by name. Diagnose: kubectl get endpoints <service> shows empty endpoints despite Pods running. Fix the selector.

  8. hostNetwork: true in a StatefulSet. Combining hostNetwork with StatefulSet hostname semantics is fragile — the Pod’s hostname is set to <sts>-<ordinal> but the system hostname on the node (which hostNetwork Pods inherit) may not match. Some apps that hit gethostname(3) directly get the wrong name. Avoid unless you specifically know your app handles it.

  9. CSI driver doesn’t support volumeClaimTemplates zone-pinning. If your StorageClass provisions zonal disks (AWS EBS, GCP PD), the first PV for data-web-0 is created in some zone (e.g., us-east-1a). If the Pod is later rescheduled to a node in us-east-1b, the disk can’t attach — RWO disks are zonal. Symptom: web-0 stuck in ContainerCreating with FailedAttachVolume. Fix: use a zonal node affinity, or a regional StorageClass (regional PD on GCP), or topology-aware provisioning (WaitForFirstConsumer binding mode on the StorageClass).

  10. Quorum-loss during rolling update with too-low replica count. A 3-Pod etcd or Zookeeper cluster: rolling update kills etcd-2, the quorum is (2 of 2 remaining alive). If etcd-2 doesn’t come back Ready quickly, and etcd-1 is now next to be updated, killing etcd-1 drops to (1 of 1 remaining) — quorum lost. Stock RollingUpdate doesn’t model this; you need either custom operators (Zalando Patroni operator, etcd-operator) or careful partition management. This is the most common reason “we lost the cluster during a Kubernetes update.”

Alternatives and When to Choose Them

  • Deployment — the default for everything that doesn’t need identity. If a workload’s Pods are interchangeable, use Deployment. Even if you mount a persistent volume for cache, Deployment + a single shared PVC (RWX) is usually preferable to StatefulSet.
  • Database operator with custom CRD (CNPG, Vitess, Zalando Postgres-Operator) — for serious production database use, the StatefulSet alone is rarely enough. An Operator Pattern CRD wraps the StatefulSet with failover orchestration, backups, point-in-time recovery, and topology-aware rolling updates that respect application-level invariants (don’t update the primary while a backup is running, demote leaders before replacing them). See Database Operators.
  • Managed database services (RDS, Cloud SQL, MongoDB Atlas) — for many teams, the right answer is “don’t run the database on Kubernetes at all.” StatefulSet exists and works; managed services exist and work better operationally. Choose K8s-hosted databases only when you have operator expertise, regulatory requirements, or a specific cost driver.
  • Job with persistent volume — for batch processing of stateful data that’s run-to-completion rather than long-running. Different semantic profile.
  • DaemonSet with persistent volume — if the storage is genuinely per-node (e.g., a local-storage cache, a per-node telemetry buffer), DaemonSet plus hostPath or a local PV is the right shape, not StatefulSet.

Production Notes

  • Confluent’s Kafka-on-Kubernetes deployment guide uses a StatefulSet per Kafka broker pool, with serviceName set to a headless kafka-brokers service. Brokers discover each other via <broker-name>.kafka-brokers.<ns>.svc.cluster.local. Confluent’s blog explicitly warns against using a clusterIP Service for Kafka, since the load balancing would route producer/consumer traffic to random brokers — disastrous for partition assignment.
  • CloudNativePG (CNPG) for Postgres layers a Cluster CRD on top of a StatefulSet, but with a critical departure: CNPG uses a Deployment for each instance for finer control over rolling updates and avoids StatefulSet’s reverse-ordinal-only update order. This is a recurring theme in mature operators — the StatefulSet semantics are too rigid for orderly failover, so operators replicate parts of the StatefulSet behavior in their own controllers.
  • Cassandra-on-K8s (the official cass-operator) uses StatefulSet directly, with a custom rolling-update procedure that pauses between Pod restarts to wait for the Cassandra cluster’s own gossip-driven Ready state — going beyond just K8s readiness probes.
  • k8s.af incidents involving StatefulSets typically cluster around: (a) PVC retention misunderstandings (deleting a StatefulSet, losing the data); (b) zone-pinning of EBS volumes blocking reschedules; (c) OrderedReady deadlocks during outages where Pod N-1 is stuck and the whole cluster can’t be scaled or updated.
  • Spotify’s general guidance internally is that StatefulSets are a “platform-team-approved-only” workload type — application teams default to Deployment + managed cloud database. The operational overhead of StatefulSet is real and ongoing; not every team needs to learn it.

See Also

Graph View

Backlinks