Autoscaling LLM Inference Workloads

Autoscaling a large language model (LLM) inference deployment is the same control problem as autoscaling any web service — measure a signal, compare it to a target, adjust replica count — with two properties that break every default. First, the conventional signals are blind: central processing unit (CPU) utilisation measures a component that is not the bottleneck, and graphics processing unit (GPU) utilisation as reported by the driver is pinned near 100% during token generation “regardless of whether the model is handling one request or a saturated batch” (llm-d autoscaling design). Second, the actuator is minutes slow: a new replica must schedule onto a GPU node, pull a multi-gigabyte image, transfer tens of gigabytes of weights, and run an engine start-up that takes over 20 seconds even for a 3-billion-parameter model with warm caches (Breaking the Ice, arXiv:2606.07362v3). The signals that do work are exactly the two that the Gateway API Inference Extension’s Model Server Protocol already mandates — queue depth and KV cache utilisation — which is why the autoscaler and the load balancer in a well-built stack read the same telemetry (see Load Balancing Strategies for Inference Servers). This note is the LLM-specific layer; the general control theory lives in Autoscaling in Practice and the Kubernetes mechanism in Horizontal Pod Autoscaler.

Mental Model: Two Clocks That Do Not Agree

Every autoscaler is a feedback loop, and every feedback loop has a delay. What makes LLM serving unusual is that the delay is not a tuning parameter — it is a physical property of the workload measured in minutes, while the thing being controlled degrades in seconds.

The KV cache is the resource that runs out. When a vLLM replica cannot allocate blocks for the next scheduling step, it does not queue politely; it preempts a running request and discards its computed prefill (num_computed_tokens = 0), forcing a full recomputation later. Preemption and Recomputation in LLM Serving and Observability for LLM Serving trace that code path in detail. The operational consequence is that the transition from “healthy at 95% KV utilisation” to “recompute storm, goodput collapsing” happens over one or two scheduler steps — tens of milliseconds. A reactive autoscaler that notices the problem and starts a pod is, by construction, five minutes late.

flowchart LR
    subgraph FAST["Degradation clock — milliseconds to seconds"]
      A1["KV utilisation<br/>0.85 → 0.95"] --> A2["allocate_slots<br/>returns None"] --> A3["preemption"] --> A4["recompute storm<br/>goodput collapses"]
    end
    subgraph SLOW["Actuation clock — minutes"]
      B1["metric scraped<br/>15-30 s"] --> B2["HPA/KEDA decides<br/>15-60 s"] --> B3["node provisioned<br/>3-4 min if none free"] --> B4["image pull<br/>+ weight load"] --> B5["engine start<br/>20 s to minutes"] --> B6["replica READY"]
    end
    A4 -.->|"the signal that<br/>triggers B1"| B1
    B6 -.->|"relief arrives<br/>long after A4"| A4

The two clocks. What it shows: the failure develops on the top track in under a second, while the remedy travels the bottom track for several minutes. The insight to take: you cannot close this loop by tuning it faster. Every practical design either (a) moves the trigger point earlier so the slow track starts before the fast track fires, (b) pre-pays the slow track with warm capacity, or (c) accepts the gap and absorbs it with queueing and admission control. Anything else is wishful thinking dressed as a HorizontalPodAutoscaler.

Why the Default Signals Are Wrong

CPU utilisation is the Kubernetes default and it is simply measuring the wrong machine. Google’s GKE guidance is direct: “For inference workloads running on GPUs, we don’t recommend CPU and memory utilization as the only indicators of the amount of resources a job consumes because inferencing workloads primarily rely on GPU resources” (GKE autoscaling best practices). A vLLM replica saturating an H100 may sit at 15% of one CPU core; a replica idling with zero requests may sit at 12%, because the API server, the metrics exporter, and the tokenizer threads never stop. There is no threshold that separates the two states.

GPU utilisation — the DCGM_FI_DEV_GPU_UTIL family — is the trap that catches people who correctly reject CPU. The metric reports the fraction of sampled intervals in which at least one kernel was resident, not how much work those kernels did. GKE states the limitation plainly: it “does not measure how much work is being done while the GPU is active. This makes it difficult to map inference based performance metrics, such as latency and throughput, to a GPU Utilization threshold.” Because decode runs a forward pass every iteration whether the batch holds 1 request or 200, a decode-bound replica reports ~100% at both extremes. Scaling on it produces a step function with no useful gradient.

GPU memory utilisation fails for a different and more embarrassing reason: vLLM allocates its KV cache arena up front, sized by gpu_memory_utilization (0.92 by default in v0.26.0). Device memory therefore reads as ~92% used one second after the process starts and never moves again. GKE notes that for “workloads that preallocate GPU memory or never deallocate memory (such as workloads running on TGI and vLLM), this metric only works for scaling up, and won’t scale down when traffic decreases” — and in practice it does not even work for scaling up, because it is already above any sane threshold at idle.

Requests per second (RPS) is the subtlest failure, and it is the one that will pass a load test and then fall over in production. RPS assumes requests are interchangeable units of work. In LLM serving they differ by three orders of magnitude: a 20-token prompt with a 10-token answer and a 100,000-token document summarisation with a 4,000-token answer are both “one request”. Worse, output length is unknown at admission — the server learns how expensive a request was only when the model emits the end-of-sequence token. A fleet tuned to 8 RPS per replica against a chat traffic mix will melt when an agentic workload arrives at 8 RPS with 50× the token volume. AIBrix makes the same point from production experience: “Request complexity and I/O size vary widely, often overwhelming systems before autoscalers can react” (Bytedance et al., arXiv:2504.03648).

Concurrency — Knative’s default metric, targeting 100 in-flight requests per pod by default (Knative autoscaling targets) — is better than RPS because it at least accounts for how long a request occupies the server. It is still blind to token volume: 30 concurrent short chats and 30 concurrent long-context summarisations impose wildly different KV pressure at identical concurrency.

Signal	What it actually measures	Why it fails for LLM serving	Verdict
CPU utilisation	Host cores	Not the bottleneck; idle and saturated look alike	Never
GPU utilisation (DCGM)	Fraction of intervals with a resident kernel	~100% during any decode, batch size invisible	Never
GPU memory used	Bytes allocated on device	Pre-allocated arena; constant after start-up	Never
Requests per second	Arrival rate	Request cost varies 1000×; output length unknown at admission	Only with a rigidly homogeneous traffic mix
Concurrency / in-flight	Little’s-Law occupancy	Blind to token volume and context length	Acceptable floor, poor ceiling
Queue depth	Requests the batch could not admit	Directly counts unmet demand	Yes — primary
KV cache utilisation	Fraction of block budget in use	Leading indicator of preemption	Yes — leading

flowchart TD
    Q{"Choosing an<br/>autoscaling signal"}
    Q --> C["CPU / memory"] --> CX["✗ measures the wrong device"]
    Q --> G["GPU utilisation"] --> GX["✗ saturated at any batch size"]
    Q --> R["requests per second"] --> RX["✗ a request is not a unit of work"]
    Q --> QD["queue depth<br/>vllm:num_requests_waiting"] --> QY["✓ cost-optimal;<br/>lags by one queueing delay"]
    Q --> KV["KV utilisation<br/>vllm:kv_cache_usage_perc"] --> KY["✓ leading;<br/>fires before latency moves"]
    Q --> BS["batch size<br/>vllm:num_requests_running"] --> BY["✓ latency-sensitive;<br/>reacts sooner than queue"]

Signal selection as a decision tree. What it shows: three dead ends and three live options. The insight: the live options are not interchangeable — GKE’s guidance is that queue size is the choice “when optimizing throughput and cost”, while batch size is for “latency-sensitive workloads where queue-based scaling isn’t fast enough”. Queue depth only rises once requests are already waiting; batch size and KV utilisation move before that.

The Shared-Signal Idea: One Telemetry Surface, Two Consumers

The connective fact across this note and Load Balancing Strategies for Inference Servers is that the Kubernetes ecosystem already standardised the right metrics, and it did so for the router, not the autoscaler. Proposal 003 of the Gateway API Inference Extension requires every conforming model server to expose three gauges — TotalQueuedRequests, TotalRunningRequests, and KVCacheUtilization — with the spec noting that names may differ but “the metric types and semantics MUST follow this doc”. Gateway API Inference Extension enumerates the full mapping across vLLM, SGLang, TensorRT-LLM and Triton.

The autoscaler wants the same three numbers. A load balancer asks “which replica is least loaded right now”; an autoscaler asks “is the whole pool loaded”. Those are a per-endpoint read and a pool-wide aggregate of the identical gauges.

flowchart TB
    subgraph POD["Each model-server replica"]
      M1["vllm:num_requests_waiting"]
      M2["vllm:num_requests_running"]
      M3["vllm:kv_cache_usage_perc"]
    end
    POD --> EPP["Endpoint Picker<br/>per-endpoint read<br/>→ pick a replica"]
    POD --> PROM["Prometheus<br/>pool-wide aggregate"]
    PROM --> AS["HPA / KEDA / WVA<br/>→ pick a replica COUNT"]
    EPP --> ROUTE(["routing decision<br/>milliseconds"])
    AS --> SCALE(["scaling decision<br/>minutes"])
    ROUTE -.->|"shapes the load<br/>the autoscaler sees"| POD
    SCALE -.->|"changes the pool<br/>the router sees"| POD

One telemetry surface feeding two control loops at different timescales. What it shows: the same three gauges are read per-endpoint by the router and pool-wide by the autoscaler. The insight to take: these loops interact. A prefix-affinity router that concentrates traffic on cache-warm replicas (Prefix-Cache-Aware Request Routing) drives the pool average down while a single replica queues — so an autoscaler averaging KV utilisation across the pool will refuse to scale during a hot-spot incident. Route on maxima and alert on maxima; scale on averages only when the router is genuinely balancing.

Two source-level details matter when wiring this up. At vLLM tag v0.26.0, vllm:num_requests_running and vllm:num_requests_waiting are declared with multiprocess_mode="mostrecent" (vllm/v1/metrics/loggers.py), meaning the exported value is the latest sample from one engine process rather than a sum — a subtlety that matters for data-parallel deployments. And the same file adds vllm:num_requests_waiting_by_reason, splitting the queue into capacity (“waiting for scheduling capacity”) and deferred (“deferred by transient constraints (LoRA budget, KV transfer, blocked status)”). Only the capacity share is a genuine scale-up signal. Scaling on total waiting count means adding GPUs because a LoRA adapter budget was momentarily exhausted.

The Cold-Start Constraint

Cold start is not a detail of LLM autoscaling; it is the constraint the whole design bends around. It is worth decomposing honestly, because each stage has a different fix and the mechanics of weight distribution belong to Model Weight Distribution and Cold Start.

flowchart LR
    S(("scale-up<br/>decision")) --> A["metric scrape<br/>+ HPA sync<br/>~15-30 s"]
    A --> B["Cluster Autoscaler<br/>reacts<br/>&lt;30 s"]
    B --> C["GPU node provisioned<br/>and joins cluster<br/>3-4 min on GCE"]
    C --> D["container image pull<br/>tens of GB<br/>~minutes"]
    D --> E["model weights staged<br/>to the node<br/>~minutes"]
    E --> F["vLLM engine start<br/>20.32 s measured<br/>for Llama3.2-3B"]
    F --> G["torch.compile<br/>3-6 s cached<br/>11-21 s cold"]
    G --> R(("replica<br/>READY"))
    style S fill:#2d6a4f,color:#fff
    style R fill:#2d6a4f,color:#fff
    style C fill:#9d0208,color:#fff
    style D fill:#9d0208,color:#fff
    style E fill:#9d0208,color:#fff

A worst-case cold-start budget. What it shows: the control plane contributes well under a minute; everything else is infrastructure and engine start-up. The two red ~minutes boxes are deliberately unquantified — see the resolved callout below, which establishes that no vendor publishes an absolute duration for either stage, only speedup ratios. The insight to take: tuning HPA windows optimises the smallest term. The Cluster Autoscaler FAQ puts node provisioning at “3 to 4 minutes from CA request to when pods can be scheduled” on GCE with total HPA+CA time “usually about 5 minutes” — and that is before any LLM-specific work begins.

Resolved 2026-08-15 — the absence is the answer: nobody publishes an absolute duration for these two stages, only ratios

I went looking for a citable number and found that the entire ecosystem publishes speedup multiples against unstated baselines rather than durations. That is not an oversight on my part — it is what the sources contain, and it is the reason the diagram leaves those two boxes unquantified. What I checked on 2026-08-15:

Source	What it publishes	What it does not publish
GKE Hyperdisk ML docs (updated 2026-08-11)	“accelerate the loading of model weights by up to 11.9X relative to loading directly from a model registry”	any duration, and no statement of what the registry baseline was
Cloud Storage FUSE performance docs	parallel downloads “resulting in nine times faster model load time”	the model, the size, or the seconds
Google Cloud × NVIDIA Run:ai Model Streamer post	a chart for a 141 GB Llama 3.3 70B fetched from Cloud Storage, plus the prose “for large models, this loading phase can take many minutes”	tabulated values; the axis numbers live only in the rendered figure
NVIDIA Dynamo README v1.3.1	“7x faster model startup — ModelExpress weight streaming (DeepSeek-V3 on H200)”	the absolute before/after
“Breaking the Ice” (arXiv:2606.07362v3), already cited below	20.32 s of engine start-up, decomposed into six steps	staging — its weight-loading figures are measured with a warm Linux buffer cache, which by construction excludes the download

So the “~minutes” labels are corroborated as an order of magnitude — Google and NVIDIA independently use the phrase “many minutes” for weight loading of large models — but no primary source supports converting them into a number, because the number is a function of image size, registry locality, model size, precision and storage backend, and every vendor writing about it is selling a way to make it smaller.

The practical resolution is a measurement recipe rather than a citation, and Kubernetes hands you half of it for free: the kubelet already reports image-pull duration in its Pulled event. The GKE Image streaming documentation shows the exact output — Successfully pulled image "…/gb-frontend:v5" in 23.929723476s — and notes that the same pull took 1.5 s with image streaming enabled. That is a small sample image, not a multi-gigabyte vLLM image, so do not transfer the figure; transfer the method. Run kubectl get events and read your own Pulled duration, then time from container start to the first HTTP 200 on /health for the staging-plus-engine term. Those two numbers, on your cluster with your model, are the only trustworthy version of this row.

The engine’s own contribution is now well characterised. “Breaking the Ice” decomposes vLLM start-up into six steps and measures 20.32 seconds total for Llama3.2-3B on an H100, finding the process “predominantly CPU-bound” — only KV cache profiling and CUDA graph capture are GPU-bound. Weight loading scales linearly with parameter count and precision (Pearson correlation coefficient, a −1 to +1 measure of linear association, equal to 1.00 in their fit): roughly 0.5–1 s for a 1.8–3 B model and “nearly five seconds” for DeepSeek-V2-Lite-16B, with a warm Linux buffer cache. Reading from PCIe 5.0 SSDs instead slowed the loading step by about half, but moved total start-up by only 1.04× “because the loading step constitutes only about 7–10% of the total startup duration”.

The most actionable finding concerns torch.compile. vLLM caches compiled computation graphs after the first run; with VLLM_DISABLE_COMPILE_CACHE=1 the graph-storing step takes 11–21 seconds versus 3–6 seconds when cached. A truly cold pod — new node, empty compile cache directory — therefore pays a start-up penalty that a pod on a recycled node does not. Mounting a shared, warm compile cache is one of the cheapest cold-start wins available.

Three architectural responses follow, and they are the standard vocabulary of LLM capacity planning (GPU Capacity Planning for Inference):

Over-provision deliberately. Because the loop cannot close in time, the honest design runs the steady-state fleet below its knee — the point past which latency rises steeply, treated generally in Utilization Targets and the Latency Knee — so a traffic step can be absorbed by existing replicas while new ones boot. The headroom is not waste; it is the premium on an insurance policy whose claim takes five minutes to pay out.

Keep warm capacity. Nodes with the image pre-pulled and weights staged on local disk (or a warm pool of stopped instances, as in Autoscaling Groups and Managed Instance Groups) collapse the two largest bars. The GPU still costs money while idle, which makes this an explicitly financial decision: compare the hourly cost of n warm nodes against the revenue and SLO cost of a five-minute brownout, an analysis that connects directly to Cost per Million Tokens.

Scale on the leading indicator, and scale early. Because the actuator is slow, the trigger must be moved left. This is why KV cache utilisation is the signal of choice: it rises before latency degrades, because it measures how close the scheduler is to the preemption cliff rather than how bad things already are. llm-d’s Workload Variant Autoscaler ships a default kvCacheThreshold: 0.80 — well below the ~0.95 at which a healthy, well-tuned server operates — precisely to buy back the actuation delay.

Mechanics: HPA on Custom Metrics, and KEDA

Kubernetes has no native path from a Prometheus gauge to a replica count. The two production routes are Prometheus Adapter feeding the external.metrics.k8s.io API to a plain Horizontal Pod Autoscaler (see Metrics Pipeline for Autoscaling), or KEDA, which runs its own metrics server and generates the HPA for you.

KServe’s generative-inference guide takes the KEDA route. Its worked example, annotated line by line:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-qwen
  annotations:
    serving.kserve.io/deploymentMode: "Standard"   # KEDA autoscaling is Standard-mode only
    serving.kserve.io/autoscalerClass: "keda"      # use KEDA instead of the built-in HPA path
    prometheus.io/scrape: "true"                   # the gauges must actually be scraped
    prometheus.io/port: "8080"
spec:
  predictor:
    model:
      modelFormat: {name: huggingface}
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      resources:
        limits: {nvidia.com/gpu: "1"}              # one GPU per replica: the scaling unit
    minReplicas: 1                                 # NOT zero — see the scale-to-zero section
    maxReplicas: 5                                 # a hard cost ceiling, and a quota guard
    autoScaling:
      metrics:
        - type: External
          external:
            metric:
              backend: "prometheus"
              query: vllm:num_requests_running     # concurrency, not queue depth
            target:
              type: Value
              value: "2"                           # 2 in-flight requests per replica

The documentation explains the arithmetic: “if our LLM receives 6 concurrent requests with our target of 2 requests per pod, the system will scale to 3 replicas” (KServe generative autoscaling). Note what a target of 2 implies — a batch size of 2 is a tiny batch for a modern engine, so this configuration optimises latency at a severe throughput cost. Choosing the number is the whole game, and GKE’s queue-size guidance is the most concrete published starting point: “start with a value between 3-5 and gradually increase it until requests reach the preferred latency”, with a warning that “for thresholds under 10, fine-tune HPA scale-up settings to handle traffic spikes”.

KEDA earns its place over raw HPA on three features that map onto LLM-specific pathologies, per KServe’s own comparison table:

• initialCooldownPeriod — “particularly useful for LLM deployments where the model takes time to load before it can serve traffic, preventing premature scale-up decisions during startup”. Without it, a freshly created replica that is still loading weights reports an empty queue, drags the pool average down, and can trigger a scale-down of the very replicas that are carrying the load.
• fallback — a fixed replica count when the metric source fails failureThreshold times. A plain HPA whose external metric disappears enters Unknown and freezes at its current count; if the outage coincides with a traffic ramp, the fleet is frozen at the wrong size.
• idleReplicaCount — scaling below minReplicas when no trigger is active, the honest middle ground between “always warm” and “scale to zero”.

KEDA also separates activation from scaling, which is a distinction the HPA does not have: the activationThreshold governs the 0↔1 transition and the ordinary threshold governs 1↔n. The documentation is emphatic that activation wins on disagreement: “threshold: 10 and activationThreshold: 50, in case of 40 messages the scaler is not active and it’ll be scaled to zero even the HPA requires 4 instances” (KEDA scaling concepts). Misreading this is a good way to have a fleet that refuses to leave zero under real traffic.

On the HPA behaviour block itself, the defaults are backwards for this workload in one direction and right in the other. Kubernetes skips scaling entirely when the metric ratio is within a tolerance of 0.1 by default, and syncs every 15 seconds. The conventional asymmetry — 0 s stabilisation on scale-up, 300 s on scale-down — is correct here and should be made more extreme, not less: scale down slowly, because a replica you kill costs five minutes to replace. KServe’s LLMInferenceService example encodes exactly this shape:

scaling:
  minReplicas: 2
  maxReplicas: 10
  wva:
    hpa:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0        # react immediately; the actuator is already slow
          policies: [{type: Percent, value: 100, periodSeconds: 60}]   # at most double per minute
        scaleDown:
          stabilizationWindowSeconds: 300      # 5 minutes of sustained quiet before shrinking
          policies: [{type: Pods, value: 1, periodSeconds: 120}]       # one replica per 2 minutes

Model-Based Autoscaling: Solving for Replicas Instead of Reacting

The most interesting development in this space (as of 2026-08) is the replacement of threshold-following with an explicit performance model. llm-d’s Workload Variant Autoscaler (WVA), which KServe v0.20 wires into LLMInferenceService.spec.scaling, inverts the usual arrangement: WVA computes the desired replica count and publishes it as a Prometheus gauge, wva_desired_replicas; the HPA or KEDA object is configured with averageValue: "1" so that it acts as a pass-through actuator. “WVA is the sole source of scaling decisions — the actuator (HPA or KEDA) acts as a pass-through that directly applies the replica count computed by WVA.”

Its QueueingModelAnalyzer is worth walking through because it makes the physics of LLM serving explicit (WVA queueing-model guide). Three hardware parameters connect abstract work to wall-clock time: α (alpha), the fixed per-iteration overhead in milliseconds — kernel launches and synchronisation barriers; β (beta), compute time per token, governed by GPU floating-point throughput; and γ (gamma), KV-cache memory-access time per token, governed by memory bandwidth. The split of β from γ is the Prefill and Decode as Two Different Workloads asymmetry expressed as two coefficients.

A request with i_l input tokens and o_l output tokens runs one prefill plus o_l decode steps. Averaging its marginal work across those o_l + 1 iterations gives

δ = β × (i_l + o_l) / (o_l + 1)  +  γ × (i_l + o_l / 2)

With n concurrent requests the iteration time is T_iter(n) = α + n × δ — the empirically observed linear relationship between inter-token latency and batch size, which WVA’s benchmarking across L40S, L4, H100, A100, MI300X and Gaudi3 confirms. Applying Little’s Law (in a stable queue, average occupancy equals arrival rate times average time in system: n = λ × (o_l + 1) × T_iter) and solving yields the closed form:

T_iter = α / (1 − ρ)     where   ρ = λ × ( β(i_l + o_l) + γ(o_l + 1)(i_l + o_l/2) )

Read symbol by symbol: λ is the request arrival rate; ρ (rho) is server utilisation, a dimensionless fraction; the system is stable only while ρ < 1, and as ρ → 1 the iteration time diverges. This is the same hyperbola behind every latency knee, now with named coefficients. Setting a target T_iter = k × α corresponds to holding utilisation at ρ = 1 − 1/k, and the shipped sloMultiplier of k = 3.0 therefore targets ρ = 0.67 — with the config file itself documenting k=2.0 → ρ=0.50 as conservative and k=5.0 → ρ=0.80 as “aggressive; maximise throughput, higher tail latency”.

The parameters are learned online by an Extended Kalman Filter (EKF) — a recursive estimator that treats (α, β, γ) as hidden state and observed time-to-first-token (TTFT) and inter-token latency (ITL) as noisy measurements. Each cycle it predicts, updates against the new observation, and validates using the Normalized Innovation Squared statistic: updates with NIS ≥ 7.378 (the 95th percentile of the chi-squared distribution with two degrees of freedom) are rejected as outliers. Convergence takes 3–10 reconcile cycles; until then the analyzer falls back to observed latency × 1.5, capped at 10,000 ms TTFT and 500 ms ITL — deliberately conservative so it “scales out rather than under-provisioning while the model is still learning”. Capacity sizing then binary-searches for the maximum arrival rate λ* meeting both SLOs, and required_replicas = ceil(total_arrival_rate / λ*).

The simpler default path, the saturation analyzer, is the one most deployments will run, and its thresholds are the punchline of this whole note. There are now two of them, and which one you get depends entirely on your WVA version — so both shipped wva-saturation-scaling-config ConfigMaps are worth reading side by side.

The V1 (percentage-based) analyzer, which is what WVA v0.7.0 ships — and therefore what a stock KServe v0.20.0 install gets, since kserve-deps.env pins WVA_VERSION=v0.7.0:

default: |
  kvCacheThreshold: 0.80      # replica is "saturated" at 80% KV cache
  queueLengthThreshold: 5     # ...or at 5 queued requests
  kvSpareTrigger: 0.1         # scale up if avg spare KV capacity < 0.1
  queueSpareTrigger: 3        # ...or avg spare queue capacity < 3
  enableLimiter: false        # don't cap scale-up by cluster GPU availability

Note what is absent: there is no analyzers: key at all. The V2 (token/capacity) analyzer arrives in WVA v0.9.0 (released 2026-08-13), and the presence of that key is precisely what selects it:

default: |
  analyzers:
    - name: saturation        # this section is what selects V2; delete it to fall back to V1
      score: 1.0
  scaleUpThreshold: 0.85      # V2-only
  scaleDownBoundary: 0.70     # V2-only
  kvCacheThreshold: 0.80      # shared by V1 and V2
  queueLengthThreshold: 5     # shared by V1 and V2
  kvSpareTrigger: 0.1         # V1-only — silently IGNORED under V2
  queueSpareTrigger: 3        # V1-only — silently IGNORED under V2

The upstream comment states the switch outright: the analyzers section “selects the V2 (token/capacity-based) saturation analyzer — the default since v0.9.0. Remove the analyzers section (and the V2-only thresholds) to opt out to the legacy V1 (percentage-based) analyzer.” The trap is the two *SpareTrigger keys: they remain valid YAML under V2 and are simply ignored, so a config tuned for V1 will apply cleanly and behave differently.

A replica is saturated when KV cache utilisation ≥ 0.80 or queue length ≥ 5 — the two Model Server Protocol gauges, with thresholds set below the true cliff to pay for actuation delay. AIBrix reports the payoff from the same family of techniques: bypassing the custom-metrics path and keeping sliding-window aggregation inside the autoscaler, combined with KPA/APA-style algorithms, “reduce[d] latency by 11.5%, increase[d] token throughput by 11.4%, and minimize[d] scaling oscillations by 33% compared to native HPA”.

Resolved 2026-08-15 — no. KServe v0.20.0 pins WVA v0.7.0, which predates the V2 analyzer entirely

Answered by reading the pin rather than the cluster. kserve-deps.env at KServe v0.20.0 contains, in its auto-generated “LLMISvc dependencies” block, WVA_VERSION=v0.7.0 — two minor versions behind. (The same block pins GIE_VERSION=v1.5.0, LLMD_ROUTER_VERSION=v0.9.0, LWS_VERSION=v0.8.0 and GATEWAY_API_VERSION=v1.5.1, and Makefile line 237 confirms the pin is load-bearing: it kustomize-builds the VariantAutoscaling CRD from github.com/llm-d/llm-d-workload-variant-autoscaler.git/config/crd?ref=$(WVA_VERSION).)

The timing makes it unambiguous: WVA v0.9.0 was published 2026-08-13, a week after KServe v0.20.0 shipped on 2026-08-06. It could not have been pinned. So a stock KServe v0.20.0 install runs the legacy V1 percentage-based analyzer — its wva-saturation-scaling-config at v0.7.0 has no analyzers: key and no scaleUpThreshold/scaleDownBoundary, only kvCacheThreshold, queueLengthThreshold, kvSpareTrigger, queueSpareTrigger and enableLimiter. Both YAML blocks above are now shown separately for exactly this reason.

Two things carry over unchanged regardless of version, which is why the paragraph below still stands: kvCacheThreshold: 0.80 and queueLengthThreshold: 5 are documented as “shared by V1 and V2”, so the saturation definition is stable across the switch.

The generalizable point: a control plane’s pinned dependency versions, not the dependency’s own latest release, decide what your cluster actually runs. kserve-deps.env is the file to read, and the gap here was two minor versions and one behavioural default.

Scale to Zero, Honestly

Scale to zero is the feature everyone asks for and most deployments should not enable. The mechanism is real: in Knative, only the Knative Pod Autoscaler (KPA) supports it — the HPA class explicitly “does not support scale to zero functionality” — and the last replica is removed only “after there has not been any traffic to the Revision for the entire duration of the stable window”, 60 s by default. A scale-to-zero-grace-period (30 s) bounds network reprogramming, and scale-to-zero-pod-retention-period (0 s) sets a floor on how long the last pod survives. KEDA reaches zero through minReplicaCount: 0 plus activationThreshold; WVA lists minReplicas: 0 as an alpha feature.

The economics are what decide it. Scale to zero is a good trade when cost while idle × idle duration exceeds cold-start latency cost × request rate on wake. That is true for a genuinely bursty internal tool, an evaluation environment, a per-tenant fine-tuned variant used a few times a day, or the long tail of a multi-model fleet where hundreds of low-traffic models share a GPU pool. It is a trap when the first user after an idle period is a human waiting on a chat response: they will wait minutes, and — worse — the wake-up is correlated, so the whole recovering fleet is cold at once and the request that triggered the wake is joined by every other request arriving during the boot.

stateDiagram-v2
    [*] --> Zero
    Zero --> Activating: request arrives<br/>activationThreshold exceeded
    Activating --> Warming: pod scheduled,<br/>image pulled, weights loading
    Warming --> Serving: /health returns 200
    Serving --> Draining: no traffic for<br/>the full stable window
    Draining --> Zero: cooldownPeriod elapsed
    Serving --> Serving: normal 1↔n HPA scaling

    note right of Activating
      The requesting client is BLOCKED
      from here until Serving.
      Minutes, not milliseconds.
    end note
    note right of Warming
      Replica exists but reports
      empty queue and low KV use.
      Naive averaging sees this as
      SPARE CAPACITY and may
      scale DOWN a busy peer.
    end note

The scale-to-zero lifecycle. What it shows: two distinct hazards, at the entry and in the middle. The insight to take: the Warming state is the one that bites operators who did enable scale-to-zero successfully — a not-yet-ready replica dilutes every pool average. This is exactly what KEDA’s initialCooldownPeriod and a correctly configured readiness probe exist to suppress, and it is why pool-average signals need Ready-gated endpoint selection.

The middle path most large deployments actually take is scale to a floor, not to zero: minReplicas: 1 (or n for availability), plus KEDA’s idleReplicaCount below it, plus aggressive scale-down of everything above the floor. You keep the weights resident, pay for one GPU, and never make a user absorb a cold start.

Interaction with Cluster Autoscaling for GPU Nodes

Pod-level autoscaling only helps if there is a node to schedule onto. For GPU fleets this interaction is more brittle than for CPU workloads, for reasons that are worth naming (Cluster Autoscaler covers the general mechanism):

An LLM replica is typically an all-or-nothing unit — one pod requesting 1, 2, 4 or 8 GPUs, sometimes a LeaderWorkerSet spanning multiple nodes for tensor-parallel serving (Tensor Parallelism for Inference). There is no partial scheduling: either the whole shape fits or the pod stays Pending. Bin-packing therefore fails discretely rather than gracefully, and a cluster with plenty of aggregate free GPU memory can still be unable to place one 8-GPU pod.

GPU capacity is scarce and quota-bound. The Cluster Autoscaler “expects requested nodes to appear within 15 minutes (configured by --max-node-provision-time)”; on a GPU-constrained region the request may simply never be satisfied, and the autoscaler will then try a different node group or give up. An autoscaling policy that assumes maxReplicas is reachable is, in practice, an untested failure path.

The mitigations are the usual ones with an LLM twist: reserve capacity ahead of demand (commitments, capacity reservations, or the Cluster Autoscaler’s ProvisioningRequest, which “reserves this capacity for the ProvisioningRequest for 10 minutes”); use low-priority placeholder pods to hold GPU nodes warm; and cap maxReplicas at your actual quota so that scale-up failures surface as a clear ceiling rather than a permanently Pending pod. WVA formalises the last idea with a GPU limiter that constrains scale-up by cluster inventory or by an operator-declared quota (limiters: [{type: quota, quotas: {H100: 32}}]), and with priority-weighted rescale that reclaims GPUs from lower-priority models under contention — a fleet-level admission control policy, related in spirit to Multi-Tenancy and Fairness in LLM Serving.

Failure Modes

Scaling on the wrong half of the queue. Total vllm:num_requests_waiting includes requests deferred for LoRA budget or KV-transfer reasons, not just capacity. Symptom: replica count climbs while KV utilisation stays modest and latency does not improve. Diagnose with vllm:num_requests_waiting_by_reason{reason="capacity"} and scale on that.

Thrash from an untuned scale-down window. Because a killed replica costs minutes to replace, oscillation is far more expensive here than for a stateless service. Symptom: sawtooth replica count, periodic TTFT spikes tracking each scale-down. Fix: a long scaleDown.stabilizationWindowSeconds (300 s or more) and a Pods-type policy of 1 per period. Knative’s KPA encodes the same asymmetry structurally, with max-scale-up-rate defaulting to 1000.0 and max-scale-down-rate to 2.0.

Cold replicas dragging the average down. A booting pod reports zero queue and near-zero KV usage. If it is included in the pool average before it is Ready, the autoscaler concludes the pool is over-provisioned mid-ramp. Symptom: scale-up followed immediately by scale-down. Fix: readiness probes that fail until the engine serves, plus initialCooldownPeriod.

Panic-mode churn under bursty traffic. Knative’s KPA enters panic mode when demand exceeds what current replicas can handle by panic-threshold-percentage (default 200.0), shrinking its evaluation window to panic-window-percentage (default 10.0) of the 60 s stable window — a 6-second window. For a workload whose replicas take minutes to arrive, a 6-second window can command a large scale-up in response to a burst that has already drained.

Pool-average blindness under prefix affinity. Covered above and in Prefix-Cache-Aware Request Routing: an affinity router concentrates load, so the maximum matters more than the mean. Symptom: one replica’s queue climbing while the HPA reports the pool comfortably below target.

Autoscaling a disaggregated deployment as one unit. In a prefill/decode split (Prefill-Decode Disaggregation), the two pools saturate on different signals — prefill on compute and arrival rate, decode on KV capacity and batch size. Scaling them together over- or under-provisions one of them permanently. KServe’s spec.prefill.scaling exists exactly to give the prefill workload “independent bounds”, with the constraint that both pools must use the same actuator backend.

Alternatives and When to Choose Them

Approach	Best when	Cost
Fixed replica count	Predictable, contract-bound capacity; scarce GPU quota	Pays for peak continuously
HPA on queue depth (via Prometheus Adapter)	You already run the adapter; throughput/cost optimisation	Lags by one queueing delay; no scale-to-zero
HPA on batch size (num_requests_running)	Latency-sensitive; queue-based scaling reacts too late	Scales earlier, so runs at lower utilisation
KEDA on a PromQL query	Need fallback, idle replicas, initial cooldown, or 0↔1	Another operator to run
Knative KPA	Genuinely bursty, scale-to-zero-tolerant workloads	Concurrency/RPS metrics only; token-blind
WVA / model-based	Multi-variant fleets, explicit TTFT/ITL SLOs, cost optimisation across GPU types	Needs profiling or an online tuner; more moving parts
Predictive / scheduled	Strong diurnal or known-event traffic	Wrong when the forecast is wrong; needs a reactive backstop

Scheduled scaling deserves a specific defence in this domain. For a workload with a strong daily cycle, a cron-driven floor that raises minReplicas fifteen minutes before the morning ramp completely sidesteps the cold-start problem: the capacity is warm when demand arrives, and the reactive autoscaler only handles deviation from the forecast. Given a five-minute actuation delay, “scale on a clock and react to the residual” is often a better engineering answer than any amount of controller tuning. AIBrix names the same direction as future work: “token-based proactive scaling and SLO-driven autoscaling”.

Production Notes

Three habits separate deployments that survive from those that page.

Alert on preemption rate, not on KV utilisation level. As Observability for LLM Serving argues, a server at 95% KV utilisation with zero preemptions is perfectly tuned; the same server with a steady preemption rate is in a recompute storm. Your autoscaler should trigger at 0.80 to buy actuation time; your alert should fire on rate(vllm:num_preemptions[5m]) > 0 sustained. Confusing the two produces either a fleet that never scales or a pager that never sleeps.

Load-test with your real token distribution. A benchmark with fixed 512-in/128-out prompts will validate an RPS-based autoscaler that fails immediately on production traffic. Replay real prompt and output length distributions (Benchmarking LLM Inference), including the long tail — the tail is where autoscaling assumptions die.

Treat maxReplicas as a quota assertion and test the ceiling. Set it to what you can actually obtain, and run a game day where you exceed it, so that the behaviour under exhausted capacity — queueing, shedding, or timing out — is a designed response rather than a discovery. That behaviour is Backpressure and Flow Control, and it is the thing that actually protects you during the five minutes the autoscaler cannot.

See Also

• LLM Inference Serving MOC — the parent map; this note is §2, Deployment and Orchestration
• Load Balancing Strategies for Inference Servers — the same three gauges, read per-endpoint instead of pool-wide
• Gateway API Inference Extension — where the Model Server Protocol’s three mandatory metrics are specified
• Prefix-Cache-Aware Request Routing — why pool averages mislead when the router concentrates traffic
• Observability for LLM Serving — the metric surface, preemption detection, and alert design
• The Inference Request Scheduler — what the queue depth gauge is actually counting
• Preemption and Recomputation in LLM Serving — the cliff that KV utilisation predicts
• Model Weight Distribution and Cold Start — the mechanics behind the cold-start constraint
• GPU Capacity Planning for Inference — sizing the fleet the autoscaler moves within
• KServe and the InferenceService · llm-d and Distributed Inference Orchestration · Serving LLMs on Kubernetes
• Autoscaling in Practice — the general control loop and its two speeds
• Horizontal Pod Autoscaler · Cluster Autoscaler · Metrics Pipeline for Autoscaling · Vertical Pod Autoscaler
• Utilization Targets and the Latency Knee — why a target of 0.80 rather than 0.95
• Backpressure and Flow Control — what protects you while the autoscaler is still booting
• Autoscaling Groups and Managed Instance Groups — warm pools in the cloud-VM idiom