vault

Kubernetes Cluster Upgrade

13 min read

Kubernetes Cluster Upgrade

Upgrading a Kubernetes cluster’s version is the most consequential recurring Day-2 operation: Kubernetes ships a new minor release roughly every quarter, supports only the three most recent minor versions for patches, and a cluster left behind eventually runs unsupported software with known CVEs (Kubernetes — Releases). A cluster upgrade is not a single atomic action — it is an orchestrated sequence governed by the version skew policy (Kubernetes — Version Skew Policy), which dictates which components may temporarily run at different versions. The two non-negotiable rules: the control plane is upgraded before the nodes, and the cluster moves exactly one minor version at a time — skipping a minor version (e.g., 1.32 straight to 1.34) is unsupported (Kubernetes — Upgrading kubeadm clusters). Node upgrades follow a per-node rhythm of cordon → drain → upgrade kubelet/kube-proxy → uncordon, the same disruption-aware dance that Node Lifecycle Management uses for any maintenance, and the most common upgrade incident is an API deprecation removing a resource version that workloads still depend on.

Mental Model

The mental model for a cluster upgrade is a controlled, version-skewed crawl. Kubernetes is explicitly designed to tolerate a window of mixed versions — an apiserver at 1.34 can serve a kubelet at 1.31 — and the version skew policy defines the boundaries of that tolerance. The upgrade exploits this: it deliberately puts the cluster into a temporary skewed state (control plane ahead, nodes behind), holds it there long enough to verify health, then closes the gap.

flowchart TB
    subgraph SKEW["Version Skew Policy (newest kube-apiserver = N)"]
        APIS["kube-apiserver<br/>oldest within 1 minor of newest"]
        KCM["kube-controller-manager / kube-scheduler<br/>N or N-1 (≤ 1 minor older)"]
        KL["kubelet<br/>N down to N-3 (≤ 3 minors older)"]
        KP["kube-proxy<br/>N down to N-3; also ±3 of its node's kubelet"]
        APIS --> KCM
        APIS --> KL
        APIS --> KP
    end

    subgraph ORDER["Upgrade Order (one minor at a time)"]
        S1["1. kube-apiserver (all instances)"]
        S2["2. controller-manager, scheduler,<br/>cloud-controller-manager"]
        S3["3. kubelet (per node)"]
        S4["4. kube-proxy (per node)"]
        S1 --> S2 --> S3 --> S4
    end

What this diagram shows. Top: the allowed skew — the apiserver is the reference point, everything else trails it by a bounded amount. Bottom: the upgrade order, which is simply “upgrade the components in order of how far ahead they’re allowed to be.” Control-plane peers (controller-manager, scheduler) may only trail the apiserver by one minor, so they go right after it; kubelets may trail by three, so they can lag for two more release cycles. The insight to extract: the upgrade order is a consequence of the skew policy, not an independent rule — you upgrade apiserver first precisely because everything else is allowed to be older than it but nothing is allowed to be newer.

Mechanical Walk-through

The version skew policy

The current policy (Kubernetes — Version Skew Policy), with the apiserver’s newest minor as the reference:

  • kube-apiserver — in an HA control plane with multiple apiservers, the newest and oldest must be within one minor version of each other. This is what permits a rolling control-plane upgrade: one apiserver can be at 1.35 while its peers are still 1.34.
  • kubelet — may be up to three minor versions older than the apiserver, and must never be newer. So an apiserver at 1.35 supports kubelets at 1.35, 1.34, 1.33, and 1.32. (For components older than 1.25 the window was only two minors — the policy widened to three at 1.25.)
  • kube-proxy — may be up to three minor versions older than the apiserver (and never newer), and additionally may be up to three minor versions older or newer than the kubelet instance it runs alongside — verbatim from the policy: “kube-proxy may be up to three minor versions older or newer than the kubelet instance it runs alongside” (Version Skew Policy). (The older “kube-proxy matches its node’s kubelet” intuition is obsolete; the supported window is explicitly ±3 minors.)
  • kube-controller-manager, kube-scheduler, cloud-controller-manager — must not be newer than the apiservers they talk to, and may be up to one minor version older.

All four bullets are the current policy as published (Kubernetes — Version Skew Policy, verified 2026-05-24, whose worked examples use 1.36 as the reference minor). The window is anchored on the newest apiserver minor in the cluster, which is why a rolling HA upgrade — where one apiserver is briefly a minor ahead of its peers — stays inside the policy throughout.

The upgrade order

Because nothing may be newer than the apiserver, the apiserver goes first; because controller-manager and scheduler may only trail by one minor, they go next; kubelets and kube-proxy may trail by three, so they go last and can even be deferred to a later maintenance window (Version Skew Policy):

  1. Upgrade all kube-apiserver instances to the target minor (rolling, one HA node at a time).
  2. Upgrade kube-controller-manager, kube-scheduler, cloud-controller-manager — the policy imposes no required order among these three; they may be upgraded together.
  3. Upgrade kubelet on each node — optional in the sense that kubelets may legitimately remain at an earlier minor within the skew window.
  4. Upgrade kube-proxy on each node — likewise may lag.

And the cluster moves one minor at a time: the deprecation policy and API-change guidelines “require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters” (Version Skew Policy). To go from 1.32 to 1.35 you must land on 1.33, then 1.34, then 1.35 — each a complete upgrade cycle. This is enforced because the skew policy and the apiserver’s stored-object conversion logic are only tested for adjacent-minor transitions; a two-minor jump can leave objects in etcd that the new apiserver cannot decode.

The per-node rhythm

For each node (control-plane node or worker), the upgrade follows the drain dance (Kubernetes — Safely Drain a Node):

  1. Cordon — kubectl cordon <node> sets spec.unschedulable=true; the scheduler stops placing new Pods here.
  2. Drain — kubectl drain <node> --ignore-daemonsets --delete-emptydir-data evicts the node’s Pods via the Eviction API, respecting PDBs: if a PDB would be violated, drain blocks until enough replicas exist elsewhere. DaemonSet Pods are skipped.
  3. Upgrade the node components — kubelet and kube-proxy to the new version, restart kubelet.
  4. Uncordon — kubectl uncordon <node> clears unschedulable; the scheduler may place Pods here again.

Two strategies for the node phase:

  • In-place upgrade — upgrade the kubelet binary on the existing node and restart it. The node keeps its identity. Cheaper, but the node’s OS image is not refreshed.
  • Surge upgrade (rolling replacement) — provision a new node at the new version, drain a Pod’s worth of capacity onto it, then delete the old node. This is what managed node groups and Karpenter-style provisioners do; it also refreshes the OS image and avoids any in-place mutation. It needs surge capacity (extra nodes briefly) and is the default on EKS/GKE/AKS managed node pools.

kubeadm specifics

On a kubeadm-built cluster (Kubernetes — Upgrading kubeadm clusters):

  • kubeadm upgrade plan — checks feasibility and prints which versions you can move to.
  • kubeadm upgrade apply v1.X.Y — run on the first control-plane node; upgrades the static-Pod manifests for kube-apiserver, kube-controller-manager, kube-scheduler, and etcd, and renews kubeadm-managed certificates.
  • kubeadm upgrade node — run on every other control-plane node and every worker node; upgrades that node’s component config without re-running the cluster-wide apply.

The kubelet and kubectl packages are upgraded separately on each node (apt/dnf install of the pinned version) after kubeadm upgrade node, followed by systemctl restart kubelet.

Configuration / API Surface

A worker-node upgrade on a kubeadm cluster, end to end:

# --- on the first control-plane node ---
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm='1.34.1-*'   # upgrade kubeadm itself
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan                                            # preview feasible targets
sudo kubeadm upgrade apply v1.34.1                                   # upgrades static-Pod control plane + etcd
 
# --- then, for EACH worker node ---
kubectl drain worker-7 --ignore-daemonsets --delete-emptydir-data    # cordon + PDB-aware eviction
# (ssh to worker-7)
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='1.34.1-*' && sudo apt-mark hold kubeadm
sudo kubeadm upgrade node                                            # upgrade this node's component config
sudo apt-mark unhold kubelet kube-proxy
sudo apt-get install -y kubelet='1.34.1-*'                           # upgrade the node agent
sudo apt-mark hold kubelet
sudo systemctl daemon-reload && sudo systemctl restart kubelet       # pick up the new kubelet
# (back on a control-plane node)
kubectl uncordon worker-7                                            # node is schedulable again
kubectl get nodes                                                    # verify VERSION column shows v1.34.1

Line-by-line: the apt-mark unhold / hold pair is essential — Kubernetes packages are normally pinned (hold) so an unrelated apt upgrade cannot bump them out of band; you unpin only to upgrade deliberately. kubeadm upgrade plan is a dry run that refuses to proceed if the jump is more than one minor or if the cluster is unhealthy. kubeadm upgrade apply mutates the static-Pod manifests in /etc/kubernetes/manifests, and the kubelet — watching that directory — restarts the control-plane containers; it also upgrades the bundled etcd, which is why an in-flight-request stall can occur (the docs suggest SIGTERM-ing kube-apiserver first to drain it gracefully). kubeadm upgrade node is the lighter per-node counterpart that does not touch cluster-wide state. The final kubectl get nodes confirms the VERSION column — the authoritative check that the kubelet actually picked up the new binary.

Failure Modes

API deprecation breakage. The single most common upgrade incident. Kubernetes’ deprecation policy removes old API versions on a schedule (the canonical example: extensions/v1beta1 and apps/v1beta1 Deployments/Ingress removed in 1.16; policy/v1beta1 PodDisruptionBudget removed in 1.25; networking.k8s.io/v1beta1 Ingress removed in 1.22). After the upgrade, manifests, Helm charts, or operators still referencing a removed apiVersion fail with no matches for kind. Defense: run kubectl convert, pluto, or kubent before upgrading to find every deprecated API in use. Cross-link Kubernetes API Groups and Versions.

Drain stuck on an unsatisfiable PDB. If a Pod Disruption Budget requires minAvailable: 3 but the workload has only 3 replicas, draining any node violates it and kubectl drain blocks forever. Fix: scale the workload up, relax the PDB, or (dangerously) --disable-eviction.

Skipping a minor version. Attempting 1.32 → 1.34 directly. kubeadm upgrade plan refuses it; on managed services the API rejects it. The hidden danger is etcd: objects stored by the old apiserver in a now-removed storage version cannot be decoded by an apiserver two minors ahead.

Forgetting to upgrade kubelets within the skew window. If kubelets are left at N-3 and the control plane upgrades again to N+1, the kubelets are now N-4 — outside the supported skew. They may keep working but are unsupported and can break unpredictably. Upgrade nodes within two release cycles, not “eventually.”

etcd upgrade stalling the apiserver. kubeadm upgrade apply upgrades the bundled etcd; in-flight apiserver requests can stall during the etcd restart. Mitigate by gracefully shutting down kube-apiserver first (the docs’ killall -s SIGTERM kube-apiserver; sleep 20 workaround). Cross-link etcd.

Webhook / admission incompatibility. A validating or mutating admission webhook (OPA Gatekeeper, Kyverno, a service-mesh injector) compiled against an old API can reject or corrupt objects after the apiserver upgrades. Check webhook compatibility as part of upgrade pre-flight.

Alternatives and When to Choose Them

Self-managed in-place upgrade (kubeadm). Full control, full responsibility. You run kubeadm upgrade on every node yourself. Choose when you must — on-prem, air-gapped, regulated.

Managed-service upgrade (EKS / GKE / AKS). The provider upgrades the control plane for you with a single API call or console action (EKS, GKE, AKS). The control-plane upgrade is hidden — you do not see apiserver/etcd/scheduler, you just request “upgrade to 1.34.” The node upgrade is exposed: you still trigger managed-node-group upgrades, which use surge replacement under the hood. GKE goes furthest with release channels (Rapid/Regular/Stable) and auto-upgrade, where the cluster moves itself forward on a schedule. Choose managed unless you have a specific reason not to — the version-skew choreography is exactly the kind of toil managed services exist to absorb.

Blue-green cluster replacement. Instead of upgrading a cluster in place, stand up a new cluster at the target version, deploy workloads to it (via GitOps this is nearly free — the manifests are the source of truth), shift traffic, and tear down the old cluster. Choose for major jumps, for clusters where in-place upgrade risk is unacceptable, or when the cluster definition is fully reproducible via Cluster API / IaC. This makes the “upgrade” a DR-style rebuild and sidesteps the one-minor-at-a-time crawl entirely. The cost is needing double the infrastructure briefly and a robust traffic-shift mechanism.

Surge vs in-place node upgrade. Discussed above — surge replacement refreshes the OS image and avoids in-place mutation but needs spare capacity; in-place is cheaper but leaves a stale OS. Managed node groups default to surge.

Production Notes

The recurring lessons from production upgrade write-ups and the k8s.af failure stories:

  • Pre-flight the API deprecations. Teams that run pluto/kubent against their manifests and Helm releases before every upgrade avoid the no matches for kind class of incident entirely. Teams that don’t, discover removed APIs the hard way mid-upgrade.
  • Upgrade staging clusters first. A cluster topology of dev → staging → prod where each environment is upgraded a week apart catches admission-webhook and operator incompatibilities before they reach production.
  • Watch the skew window, not the calendar. The discipline is “nodes must stay within three minors of the control plane,” which in practice means upgrading nodes within ~two release cycles. Clusters that treat node upgrades as optional drift into unsupported skew.
  • Managed services move the risk, they don’t remove it. A GKE auto-upgrade still drains your nodes; a workload with a bad PDB or a misconfigured liveness probe will still cause an availability incident — the provider just won’t be the one paged. Test PDBs and probes before enabling auto-upgrade.
  • Capacity headroom matters. Surge node upgrades and drains need spare scheduling capacity; a cluster running at 95% utilization cannot drain a node without evicted Pods going Pending. Cross-link Capacity Planning on Kubernetes.
  • Back up etcd before the control-plane upgrade. kubeadm upgrade apply touches etcd; a snapshot taken immediately before gives a rollback point. Cross-link etcd Backup and Restore.

See Also

Graph View

Backlinks