vault

Vertical Pod Autoscaler

12 min read

Vertical Pod Autoscaler

The Vertical Pod Autoscaler (VPA) adjusts a workload’s per-Pod resource requests and limits — making each Pod bigger or smaller — rather than changing the number of Pods. It is the orthogonal counterpart to the Horizontal Pod Autoscaler: where the HPA answers “how many Pods?”, the VPA answers “how much CPU and memory should each Pod request?”. The VPA is not part of stock Kubernetes — it is an out-of-tree project living in kubernetes/autoscaler and must be installed separately. It consists of three independently-deployed components — the Recommender, the Updater, and the Admission Controller — coordinated through a VerticalPodAutoscaler CRD. The VPA exists because operators are notoriously bad at sizing requests by hand: requests set too high waste money and starve the cluster; set too low, Pods get OOM-killed or CPU-throttled. The VPA observes real usage history and continuously corrects the numbers. Its historical Achilles’ heel — applying a new recommendation required killing and recreating the Pod — is now being resolved by in-place Pod resize, which graduated to stable (GA) in Kubernetes v1.35 (Kubernetes v1.35 blog).

Mental Model

The VPA is a right-sizing feedback loop. It watches what a workload’s Pods actually consume over time, computes what their requests should be, and applies the correction. The subtlety is how it applies it — and that is what the three components divide between them.

flowchart TD
    HIST["Pod usage history<br/>(from Metrics Server / Prometheus)"]
    REC["Recommender<br/>observes usage → computes<br/>target/lowerBound/upperBound"]
    VPAOBJ["VerticalPodAutoscaler object<br/>.status.recommendation"]
    UPD["Updater<br/>compares running Pods'<br/>requests vs recommendation"]
    EVICT["evict Pod<br/>(Recreate) — or — in-place resize"]
    NEWPOD["Pod recreated by its<br/>workload controller"]
    ADM["Admission Controller<br/>(mutating webhook)"]
    APPLIED["Pod admitted with<br/>requests rewritten to recommendation"]
    HIST --> REC
    REC -->|writes| VPAOBJ
    VPAOBJ --> UPD
    UPD --> EVICT
    EVICT --> NEWPOD
    NEWPOD -->|intercepted at admission| ADM
    VPAOBJ --> ADM
    ADM --> APPLIED

What this diagram shows. The Recommender is advice-only — it only writes numbers into the VPA object’s status. The Updater is the enforcer — it decides which running Pods are far enough off to be disturbed. The Admission Controller is the applier — it rewrites requests on Pod creation (the recommendation cannot be applied to a Pod that does not exist yet, so the loop is: Updater evicts → controller recreates → Admission Controller stamps the new value). The insight to extract is that the classic VPA does not change a running Pod’s requests in place — it achieves the change by destroying and recreating the Pod, which is exactly the operational cost in-place resize removes.

Mechanical Walk-through

The Recommender

The Recommender continuously consumes per-container usage samples (CPU and memory, sourced from the Metrics Server or a Prometheus backend) and maintains a decaying histogram of usage per workload. From that histogram it computes a recommendation with three numbers per resource: a target (the request it recommends), a lowerBound (below this the Pod is under-provisioned), and an upperBound (above this the Pod is over-provisioned). CPU uses a high percentile of observed usage; memory uses a peak-oriented estimate (memory cannot be reclaimed gracefully, so the VPA is conservative). The Recommender writes all of this into the VerticalPodAutoscaler object’s status.recommendation. It changes nothing about running Pods — in updateMode: Off this is the only component that does anything, making the VPA a pure “what should my requests be” advisor.

The Updater

The Updater watches running Pods against their VPA’s recommendation. When a Pod’s current requests fall outside the [lowerBound, upperBound] window, the Updater decides it needs correction. How it corrects depends on updateMode:

  • In Recreate mode it evicts the Pod via the eviction subresource. The Pod’s workload controller (Deployment, StatefulSet) recreates it; on recreation the Admission Controller stamps the new requests. The Updater respects PodDisruptionBudgets so it cannot evict more of a workload than the PDB allows at once.
  • In InPlaceOrRecreate mode it first attempts an in-place resize — issuing the change through the Pod’s dedicated resize subresource (the mechanism that went GA in Kubernetes v1.35, v1.35 blog) so the container’s CPU/memory requests and limits change without recreating the Pod — and falls back to eviction only if the in-place path cannot satisfy the change in time. Mechanically, the resize request mutates spec.containers[*].resources; the kubelet then actuates it and reports the live values in status.containerStatuses[*].resources. Since v1.35, even memory-limit decreases are permitted (the kubelet allows the shrink only when current usage is already below the new limit — a best-effort guard against an immediate OOM-kill, not a guarantee).

The Admission Controller

The Admission Controller is a mutating admission webhook. Every time a Pod is created, the webhook checks whether a VPA targets that Pod’s workload; if so, it reads the VPA’s current status.recommendation and rewrites the Pod’s resources.requests (and limits, proportionally) before the Pod is persisted. This is the only point at which a recommendation is applied to a brand-new Pod — and it is why the Recreate loop works: eviction alone would just recreate the Pod with its old requests; the Admission Controller is what makes the recreated Pod actually larger or smaller.

updateMode

spec.updatePolicy.updateMode selects the VPA’s aggressiveness:

ModeBehavior
OffRecommender computes recommendations; nothing is applied. Pure advisor — read status.recommendation and size manually. The safe starting point.
InitialThe Admission Controller applies the recommendation only at Pod creation. Running Pods are never disturbed; they get the new size whenever they happen to be recreated for other reasons.
RecreateFull loop with eviction: the Updater evicts running Pods whose requests are too far off, and the Admission Controller stamps the new value on recreation. Disruptive.
InPlaceOrRecreateLike Recreate but the Updater tries an in-place /resize first, falling back to eviction. Zero-restart right-sizing when the resize succeeds. GA as of VPA v1.6.0 (alpha v1.4.0 → beta v1.5.0 → GA v1.6.0, per the VPA features doc); requires Kubernetes 1.33+ with the InPlacePodVerticalScaling feature gate.
InPlaceNewer mode (alpha as of VPA v1.7.0, per the features doc): attempt in-place resize only, with no eviction fallback — for workloads where a restart is worse than a deferred resize. Still alpha; gate-protected.
AutoAn alias for Recreate; deprecated since VPA v1.5 (correctly flagged as deprecated in the v1.6.0 release) — use Recreate or InPlaceOrRecreate explicitly. Slated for removal in a future API version.

The maturity picture, verified against primary sources (resolving the earlier uncertainty): the underlying Kubernetes mechanism, in-place Pod resize (InPlacePodVerticalScaling, the /resize subresource, KEP-1287), went alpha in v1.27 → beta in v1.33 → GA/stable in v1.35 (v1.35 release blog). On the VPA side, the InPlaceOrRecreate mode that consumes that mechanism graduated alpha (v1.4.0) → beta (v1.5.0) → GA (v1.6.0) in the kubernetes/autoscaler project; at GA it no longer needs the VPA-side InPlaceOrRecreate feature gate, though it still depends on a Kubernetes cluster at 1.33+ with InPlacePodVerticalScaling enabled (now the default, since that gate is GA). Early open issues about the in-place path not always firing (autoscaler #8609, #8805) date from the alpha period; the still-evolving frontier is now the eviction-free InPlace mode (alpha in v1.7.0), not InPlaceOrRecreate.

Configuration / API Surface

A VerticalPodAutoscaler targeting a Deployment, with bounds and per-container control:

apiVersion: autoscaling.k8s.io/v1   # the VPA CRD's group — installed by the VPA, NOT core k8s
kind: VerticalPodAutoscaler
metadata:
  name: web-vpa
spec:
  targetRef:                        # WHICH workload's Pods to right-size
    apiVersion: apps/v1
    kind: Deployment
    name: web
  updatePolicy:
    updateMode: "InPlaceOrRecreate" # try /resize first, fall back to eviction
    minReplicas: 2                  # don't evict if it would drop ready replicas below this
  resourcePolicy:
    containerPolicies:
      - containerName: "web"        # per-container tuning ("*" matches all)
        minAllowed:                 # the Recommender will never recommend BELOW these
          cpu: "100m"
          memory: "128Mi"
        maxAllowed:                 # ...nor ABOVE these — caps a runaway recommendation
          cpu: "2"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]   # which resources the VPA manages
        controlledValues: RequestsAndLimits      # adjust both requests AND limits...
                                                 # (or RequestsOnly to leave limits alone)
      - containerName: "log-sidecar"
        mode: "Off"                 # exempt this container — sidecar stays hand-sized

Line-by-line. apiVersion: autoscaling.k8s.io/v1 is the VPA’s own CRD group — note it is not the core autoscaling/v2 of the HPA; the VPA being out-of-tree means it ships its own API. targetRef names the workload (the VPA reads its Pods’ usage and writes their requests). updatePolicy.minReplicas is a safety floor for the eviction path — the Updater will not evict a Pod if doing so drops the workload below this many ready replicas. The resourcePolicy.containerPolicies block is where production VPA usage lives: minAllowed/maxAllowed bound the Recommender so a metric anomaly cannot recommend an absurd 50-core request; controlledValues: RequestsAndLimits keeps the request/limit ratio intact, while RequestsOnly adjusts requests but leaves limits as the manifest set them; and per-container mode: "Off" exempts containers (a common need — let the VPA size the app container but leave a fixed-cost sidecar alone).

Failure Modes

Disruptive restarts (Recreate, and the deprecated Auto alias). In the classic eviction modes, every recommendation change restarts Pods. For a singleton or a stateful workload this is real downtime. Mitigations: a PDB plus minReplicas to bound concurrent evictions; or — the modern fix — InPlaceOrRecreate (GA in VPA v1.6.0) with in-place resize, which avoids the restart entirely when the node has headroom.

HPA + VPA on the same metric. The cardinal VPA sin. If the HPA scales on CPU utilization and the VPA also manages CPU requests, they fight: VPA raises the CPU request → measured usage/request utilization drops → HPA scales replica count down → load per Pod rises → VPA raises the request again. They never converge. The supported pattern is VPA owns CPU/memory; HPA owns a different signal (requests-per-second, queue depth via KEDA). The Kubernetes MOC states this as a hard decision rule.

Pod-level resources incompatibility. The VPA computes per-container recommendations and currently does not understand a Pod that declares a Pod-level resources stanza — its container recommendations can exceed the Pod-level cap, producing rejected or mis-sized Pods. Work on Pod-level support is tracked in AEP-7571; until it lands, avoid Pod-level resources on VPA-managed workloads.

Memory recommendation lag for spiky workloads. The Recommender’s memory estimate is peak-oriented and decays slowly. A workload with rare, large memory spikes may be sized for the spike (wasteful) or, if the spike is rarer than the decay window, sized too small and OOM-killed. Memory right-sizing is intrinsically harder than CPU.

Missing metrics. Like the HPA, the VPA Recommender needs a metrics source (Metrics Server or Prometheus). If it is absent, the Recommender produces no recommendation and the VPA is silently inert.

Admission Controller down. If the mutating webhook is unavailable, recreated Pods are not re-sized — they come back with their old requests. Depending on the webhook’s failurePolicy, Pod creation may also be blocked. The VPA Admission Controller is a deployment that itself needs to be healthy.

Alternatives and When to Choose Them

  • Horizontal Pod Autoscaler. The orthogonal axis. Choose HPA for stateless workloads that scale out well — adding identical replicas is cheap and increases throughput linearly. Choose VPA when the workload is a singleton, is stateful, or otherwise does not benefit from more replicas, and instead needs each Pod correctly sized. They can coexist on different metrics.
  • In-place Pod resize used directly. Kubernetes’ /resize subresource (GA in v1.35) lets an operator (or a custom controller) resize a Pod by hand or by their own logic, without the VPA. Choose this if you have a bespoke right-sizing policy and want the resize mechanism without the VPA’s recommendation engine.
  • Commercial right-sizing tools. ScaleOps, Cast AI, Densify, StormForge wrap the VPA’s mechanism (or replace it) with more sophisticated recommendation algorithms, dashboards, and HPA-coexistence handling. Choose when the open-source VPA’s recommendation quality or operability is insufficient.
  • Manual sizing with LimitRange defaults. For small or stable clusters, a LimitRange supplying default requests plus periodic manual review can be enough. The VPA earns its keep when workload count and variability make hand-tuning infeasible.

Production Notes

  • The canonical adoption path is Off → Initial → Recreate/InPlaceOrRecreate. Run in Off first and read the recommendations to learn whether the VPA’s numbers are sane for your workloads; then Initial to apply them safely (new Pods only, no eviction); only graduate to an enforcing mode once trust is established. Jumping straight to Recreate on production workloads is how teams get surprised by restart storms.
  • In-place resize is the headline change of the VPA’s recent history. The classic VPA’s restart-to-resize cost made many teams use it only in Off mode as an advisor. With in-place Pod resize GA in Kubernetes v1.35 and the VPA’s InPlaceOrRecreate mode GA in VPA v1.6.0, zero-downtime right-sizing is now a supported production path rather than an experiment — provided your cluster is on Kubernetes 1.33+ (the floor InPlaceOrRecreate was built against). The newer eviction-free InPlace mode (alpha in VPA v1.7.0) is the part still maturing.
  • Never co-deploy HPA and VPA on the same resource. This bears repeating because it is the single most common VPA misconfiguration. If you need both axes, split the signals.
  • The VPA’s three components are three separate Deployments — the standard install (the vertical-pod-autoscaler Helm chart or the hack/vpa-up.sh script from kubernetes/autoscaler) deploys all three plus the CRDs. All three must be healthy for the enforcing modes to work.
  • Managed-Kubernetes note: GKE and AKS ship a VPA integration (GKE, AKS), often a managed build of the upstream project. Their updateMode and in-place support track but can lag the upstream release.

See Also

Graph View

Backlinks