vault

Cluster Autoscaler

12 min read

Cluster Autoscaler

The Cluster Autoscaler (CA) — distributed from the kubernetes/autoscaler repository as cluster-autoscaler — is the classic node-count autoscaler for Kubernetes: a controller that watches the cluster’s unschedulable Pod backlog and, when it grows, asks the cloud provider to add machines; when nodes sit idle, it drains and removes them (Kubernetes — Node Autoscaling). It is the vertical complement to the pod-replica autoscalers: where the Horizontal Pod Autoscaler decides how many Pods and the Vertical Pod Autoscaler decides how big each Pod, the Cluster Autoscaler decides how many machines exist to host those Pods. The CA does not place Pods itself — that is the kube-scheduler’s job — it only ensures the scheduler has somewhere to place them. Architecturally it is simulation-based: it never blindly adds capacity; it asks “if I added one node from node group X, would the currently-Pending Pods become schedulable?” by running the scheduler’s own predicate logic against a hypothetical node, and only scales the group whose hypothetical node would actually help. The CA’s central limitation — it can only scale pre-defined, homogeneous node groups — is precisely the constraint that motivated the creation of Karpenter.

Mental Model

The Cluster Autoscaler is a feedback loop bolted onto the output of the scheduler. The scheduler produces a stream of FailedScheduling events for Pods it cannot place. The CA consumes that stream, reasons about which node group would absorb the pressure, and calls a cloud API. After the cloud creates the machine and it registers with the API server, the scheduler — entirely independently — places the Pending Pods on it. The CA and the scheduler never talk directly; they communicate only through the shared state in the API server, exactly as Kubernetes Control Loop Pattern prescribes.

flowchart TB
    PODS[Pending Pods<br/>status: Unschedulable] --> CA{Cluster Autoscaler<br/>scan loop every ~10 s}
    CA -->|"simulate: would a node from<br/>group X make these fit?"| SIM[Scheduler-predicate<br/>simulation]
    SIM -->|"group X helps"| EXPAND[Expander<br/>pick best group if<br/>multiple help]
    EXPAND -->|"increase desired size"| CLOUD[(Cloud API:<br/>ASG / MIG / VMSS)]
    CLOUD -->|"new VM boots,<br/>kubelet registers"| NODE[New Node<br/>Ready]
    NODE --> SCHED[kube-scheduler binds<br/>the Pending Pods]
    CA -. "scale-down scan every 10 min" .-> IDLE[Underutilized node<br/>< 50% requests, sustained]
    IDLE -->|"simulate: can its Pods<br/>reschedule elsewhere?"| DRAIN[Cordon + drain node]
    DRAIN -->|"decrease desired size"| CLOUD

What this diagram shows. Two loops sharing one controller. The scale-up loop (top) is event-pressure-driven: Pending Pods → simulation → expander → cloud API → new node → scheduler binds. The scale-down loop (bottom) is utilization-driven and deliberately slower (a 10-minute default scan vs. ~10-second scale-up scan) — the asymmetry is intentional, because adding a node late costs latency while removing a node too eagerly costs a reschedule storm. The insight to extract: the CA’s only input is the scheduler’s verdict. It does not look at CPU graphs, queue depth, or request rate — it looks exclusively at whether Pods are stuck Pending. Everything the CA does is downstream of status.conditions[type=PodScheduled]=False.

Mechanical Walk-through

Scale-up: the unschedulable-Pod trigger

Every scan interval (default ~10 seconds) the CA lists all Pods and filters for those that are unschedulable — Pending with a PodScheduled condition of False and reason Unschedulable. A Pod is unschedulable because no existing node passed the scheduler’s Filter plugins: not enough allocatable CPU/memory, an unsatisfied nodeAffinity, an untolerated taint, a topology-spread violation, or a PVC bound to a zone with no capacity.

For each node group the CA maintains a node template — a synthetic representation of what a node from that group looks like (its allocatable resources, labels, taints). The CA runs the scheduler’s predicate logic against that template: would this Pending Pod pass Filter on a fresh node from group X? If yes, group X is a candidate. The CA then computes how many nodes from the candidate group it would take to clear the backlog (it bin-packs the Pending Pods onto hypothetical nodes). This is the simulation that makes the CA conservative: it never scales a group that wouldn’t actually help. If a Pod requests 32 vCPU and every group’s largest instance has 16, no group is a candidate and the Pod stays Pending forever — the CA emits no scale-up and logs that the Pod is unschedulable regardless of capacity.

When multiple node groups could absorb the pressure, the expander breaks the tie (see Configuration below).

Node groups: the cloud abstraction

The CA does not create individual VMs. It manipulates the desired size of a node group — a cloud-provider construct that the cloud then realizes:

CloudNode group construct
AWSAuto Scaling Group (ASG)
GCPManaged Instance Group (MIG)
AzureVirtual Machine Scale Set (VMSS)

The CA’s interaction is minimal: it calls the cloud API to set desiredCapacity = current + N. The cloud provider boots the VMs from the group’s launch template; each VM’s kubelet registers with the API server; the scheduler then binds the Pending Pods. The CA’s job ends at “set the desired size” — it does not own the VM boot, the OS image, or the bootstrap. Every node group has a configured min and max size; the CA never scales a group below its min or above its max.

The “homogeneous node group” assumption is load-bearing and is the CA’s defining limitation: every node in a group is identical (same instance type, same labels, same taints). To offer a workload a choice of instance sizes, the operator must pre-create one node group per instance type — leading to dozens of ASGs in large clusters, each of which the CA scans every loop.

Scale-down: the underutilization trigger

A separate, slower loop (default scan ~10 minutes) hunts for removable nodes. A node is a scale-down candidate when both conditions hold for a sustained period (default 10 minutes of continuous low utilization):

  1. Underutilized — the sum of resource requests (not actual usage) of all Pods on the node is below a threshold (default 50% of allocatable). Note this is requests, not live CPU: a node packed with Pods that request a lot but use little is not a candidate.
  2. Pods are reschedulable — the CA simulates draining the node and verifies that every Pod it hosts could be placed on some other existing node, respecting affinity, taints, topology spread, and PodDisruptionBudgets (Pod Disruption Budget).

If both hold, the CA cordons the node (marks it unschedulable), drains it (evicts every Pod, which their controllers recreate elsewhere), and then sets the node group’s desired size down by one. The cloud terminates the VM.

The safe-to-evict annotation

Some Pods cannot be safely moved by scale-down — a Pod with emptyDir data that isn’t backed up, a Pod the operator pins to a node, a system DaemonSet-like singleton. The CA respects the annotation:

cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

A node hosting any Pod with safe-to-evict: "false" is never removed by scale-down. Conversely, certain Pods the CA refuses to move by default — Pods not backed by a controller (bare Pods), Pods with restrictive PDBs, kube-system Pods without a PDB, Pods with local storage — can be made movable by setting the annotation to "true". There is also a node-level annotation cluster-autoscaler.kubernetes.io/scale-down-disabled: "true" to exempt a whole node.

Configuration / API Surface

The Cluster Autoscaler is deployed as a Deployment (usually in kube-system) with flags, not a CRD. A representative AWS deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: cluster-autoscaler
          image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.0
          command:
            - ./cluster-autoscaler
            - --cloud-provider=aws
            - --namespace=kube-system
            # Auto-discover ASGs tagged for this cluster, instead of listing each:
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
            - --expander=least-waste,price          # tie-break chain
            - --scale-down-enabled=true
            - --scale-down-utilization-threshold=0.5  # 50% requests → underutilized
            - --scale-down-unneeded-time=10m          # must be idle this long
            - --scale-down-delay-after-add=10m        # don't scale down right after scale-up
            - --max-node-provision-time=15m           # give up on a stuck node group
            - --balance-similar-node-groups=true      # spread across same-shape groups/AZs
            - --skip-nodes-with-local-storage=false   # allow draining emptyDir-backed Pods
            - --skip-nodes-with-system-pods=false

Line-by-line:

  • --cloud-provider=aws selects the cloud integration; the CA ships providers for AWS, GCP, Azure, and ~30 others.
  • --node-group-auto-discovery lets the CA find ASGs by tag rather than enumerating each — the scalable alternative to one --nodes=min:max:asg-name flag per group.
  • --expander=least-waste,price is the tie-break chain: when several groups could satisfy the Pending Pods, apply least-waste first; if still tied, apply price; if still tied, choose randomly.
  • --scale-down-utilization-threshold=0.5 — a node whose summed requests fall below 50% of allocatable becomes a scale-down candidate.
  • --scale-down-unneeded-time=10m — the node must stay underutilized for 10 continuous minutes; transient dips don’t trigger removal.
  • --scale-down-delay-after-add=10m — prevents the thrash of removing a node minutes after adding one.
  • --balance-similar-node-groups=true — when groups are equivalent (e.g., one ASG per Availability Zone), spread scale-up evenly across them for HA.

Expander strategies

When more than one node group would satisfy the Pending Pods, the expander chooses (Kubernetes autoscaler FAQ):

ExpanderBehavior
randomPick a candidate group at random. The default fallback.
most-podsPick the group whose scale-up would schedule the most Pending Pods.
least-wastePick the group leaving the least idle CPU (then memory) after scale-up — the bin-packing choice.
pricePick the group whose nodes cost the least while matching cluster size (cloud-provider price-aware).
priorityPick by an operator-defined priority list (a ConfigMap); fall through to lower priority if a group can’t scale.
grpcDelegate the decision to an external gRPC service the operator runs.

Expanders chain with commas: least-waste,price applies least-waste, and only consults price if least-waste left a tie.

Failure Modes

Pods stay Pending despite the CA running. The most common cause is that no node group’s template would satisfy the Pod. Diagnose with kubectl describe pod — the CA emits a pod didn't trigger scale-up event listing why each group was rejected (“max node group size reached”, “Insufficient cpu”, “node(s) had untolerated taint”). A Pod requesting more than the largest instance type can hold will never trigger scale-up.

Scale-up is slow. The cloud API call is fast, but VM boot + OS init + image pull + kubelet registration takes minutes (commonly 3–8 min on AWS ASGs). For bursty workloads this latency is the headline complaint that motivated Karpenter. --max-node-provision-time bounds how long the CA waits before declaring a group’s scale-up failed and trying another.

Scale-down won’t remove an obviously empty node. Usually a single un-movable Pod is pinning it: a Pod with safe-to-evict: "false", a bare Pod (no controller), a kube-system Pod with no PDB, or a Pod with local storage when --skip-nodes-with-local-storage=true. List the node’s Pods and check annotations.

Thrashing — nodes added and removed in a loop. Caused by --scale-down-delay-after-add being too short relative to workload oscillation, or by an HPA and the CA fighting. The fix is widening the scale-down delays and the unneeded-time window.

Stale node templates. The CA’s synthetic node template can diverge from reality if a node group’s launch template changes (e.g., a label added). Until a real node from the group registers, the CA reasons about a node shape that no longer exists, causing wrong scale-up decisions.

The CA stops scaling at the group max. --nodes max or the ASG MaxSize is a hard ceiling. Pods beyond it stay Pending; the CA emits max node group size reached.

Alternatives and When to Choose Them

  • Karpenter — the modern replacement. Instead of fixed node groups, Karpenter provisions right-sized nodes directly by reading the Pending Pods’ actual requirements and selecting an optimal instance type/zone/purchase-option per scale-up. It is faster (no ASG round-trip), bin-packs more tightly, and actively consolidates. On AWS EKS it is largely displacing the Cluster Autoscaler. Choose Karpenter for cost-sensitive, bursty, heterogeneous workloads on a supported cloud.
  • Cluster Autoscaler — choose CA when you need broad multi-cloud support (it has ~30 providers vs. Karpenter’s smaller set), when your organization mandates fixed, audited node groups, or when you run on a cloud Karpenter does not yet support.
  • Managed cluster autoscaling — GKE, EKS, and AKS all wrap the CA (or, on EKS, offer Karpenter) as a managed feature; GKE Autopilot removes node management entirely. Choose managed unless you have a reason to run the autoscaler yourself.
  • No node autoscaling — for fixed-capacity clusters (on-prem bare metal, regulatory fixed-footprint), the CA is simply not applicable; capacity is planned statically.

Production Notes

  • Pinterest’s Karpenter migration (2023) is the canonical CA-limitation case study: replacing the CA with Karpenter cut node-provisioning time from ~5 minutes to ~60 seconds for bursty workloads, primarily because Karpenter chooses an instance type that exactly fits the Pending Pods rather than scaling a uniform group.
  • The AWS EKS Best Practices guide (aws.github.io/aws-eks-best-practices) recommends --balance-similar-node-groups=true with one ASG per Availability Zone so scale-up spreads across zones, and warns that mixing instance types within a single ASG breaks the CA’s homogeneity assumption (the CA picks one instance’s template arbitrarily).
  • CA version must track the cluster version. The CA embeds a snapshot of the scheduler’s predicate logic; running a CA minor version that lags or leads the cluster causes simulation to disagree with the real scheduler. The project ships a CA minor release per Kubernetes minor release for this reason.
  • --scale-down-utilization-threshold is a requests metric, not a usage metric. Operators are repeatedly surprised that a node showing 5% CPU on a dashboard isn’t removed — because the Pods on it request 60% of allocatable. The CA never sees live CPU; this is by design (it must reason about whether Pods would fit elsewhere, which is a request-based question).

See Also

Graph View

Backlinks