vault

Autoscaling in Practice

16 min read

Autoscaling in Practice

Autoscaling is the operational practice of letting a control loop, rather than a human, adjust the amount of capacity a service runs with, so that capacity tracks demand automatically — adding resources when load rises and removing them when load falls. The promise is twofold: you neither pay for a permanently over-provisioned fleet sized for peak, nor page an on-call engineer to add machines at 3 a.m. when a traffic spike arrives. But autoscaling is deceptively hard to get right, and most of the difficulty is not in the mechanism — the HPA, the AWS Auto Scaling group, the Google Cloud managed instance group all implement roughly the same feedback loop — but in the decisions around it: which signal to scale on, how fast to react in each direction, what bounds to enforce, and the sobering recognition that an autoscaler can only rearrange capacity you already have or can acquire, and cannot conjure headroom that does not exist. This note is the general practice and the decision framework; the concrete Kubernetes instances live in Horizontal Pod Autoscaler, Vertical Pod Autoscaler, Cluster Autoscaler, and Metrics Pipeline for Autoscaling.

Mental Model — A Feedback Controller with Two Speeds

Every autoscaler is a closed-loop controller, and the cleanest way to think about it is as a thermostat with a memory. It has a target (a metric value it wants to hold — 60% CPU, 100 requests per second per instance, 30 messages of queue backlog), it measures the current value at intervals, and it computes how much capacity would bring the measurement back to target. The Kubernetes HPA writes this out as a bare proportion (Kubernetes HPA docs):

desiredReplicas = ceil( currentReplicas × currentMetricValue / targetMetricValue )

If the metric sits at twice the target, double the capacity; if it sits at half, halve it. AWS target-tracking policies and Google Cloud MIG autoscaling use the same proportional idea, describing it explicitly as a thermostat that keeps a chosen metric near a set point (AWS target tracking).

The subtlety that separates a working autoscaler from a flapping one is that the loop runs at two speeds, one for growing and one for shrinking, and those speeds are deliberately asymmetric.

flowchart TD
    M["Measure signal<br/>(CPU / RPS / queue depth)"] --> C["Compute desired capacity<br/>= curr × current/target"]
    C --> T{"Within tolerance<br/>of target?"}
    T -->|yes| N["No-op<br/>(dead band)"]
    T -->|"above target"| UP["Scale UP — fast<br/>short/zero stabilization<br/>large step allowed"]
    T -->|"below target"| DOWN["Scale DOWN — slow<br/>long stabilization window<br/>small step, cooldown"]
    UP --> B["Enforce min/max bounds"]
    DOWN --> B
    B --> A["Apply: launch/terminate<br/>capacity"]
    A -.->|"warm-up / cold start<br/>delay before new capacity helps"| M

What it shows and the insight to take: the loop is not symmetric. The right-hand branch (scale up) fires quickly and can take large steps because being under-provisioned hurts users now; the left-hand branch (scale down) is throttled by a long stabilization window and a small step size because being briefly over-provisioned only costs a little money, while shrinking too eagerly risks a painful re-expansion. The dashed feedback arrow carries the delay that makes autoscaling genuinely hard — new capacity does not become useful the instant it is requested, so the loop is always steering a vehicle that responds seconds or minutes after you turn the wheel.

Reactive versus Predictive and Scheduled Scaling

The dominant style is reactive (also called dynamic) scaling: watch a metric, and respond after it moves. This is what the HPA does, what an AWS target-tracking or step-scaling policy does, and what a Google Cloud CPU-utilization autoscaler does. Reactive scaling is simple, needs no forecasting, and copes with any load shape — but it is structurally late. It cannot begin to add capacity until load has already risen, and the capacity it adds is not useful until it has finished initializing. For a service whose instances boot in seconds this lag is tolerable; for one that takes minutes to warm a cache, load a model, or JIT-compile, the reactive loop spends the entire ramp under-provisioned, and users feel it.

Predictive scaling attacks that lag by scaling ahead of demand. AWS predictive scaling analyzes historical load to detect daily and weekly patterns and forecasts future capacity needs, launching instances in advance so they are already in service when the anticipated load arrives (AWS predictive scaling). Google Cloud predictive autoscaling does the same for managed instance groups, forecasting from historical data so new instances are ready to serve when load arrives (GCP autoscaler). AWS is explicit about when this pays off: cyclical traffic (busy business hours, quiet nights and weekends), recurring on-and-off batch patterns, and — the decisive case — applications that take a long time to initialize, where reactive scaling would impose a noticeable latency hit during every scale-out. Predictive scaling is not a replacement for reactive scaling but a complement: it handles the predictable baseline shape, while a reactive policy still catches the surprises the forecast did not see.

Scheduled scaling is the crudest and most reliable predictive tool: you simply tell the autoscaler to hold a minimum capacity during a known window. Google Cloud allows up to 128 scaling schedules per group, each specifying a minimum instance count with a start time, duration, and recurrence (GCP autoscaler). This is the right hammer for events you know about but a forecaster cannot infer — a product launch, a televised advertisement, a Black Friday sale, a nightly reporting job. When you know the spike is coming and roughly how big it is, do not make a statistical model guess; schedule the floor.

The decision among the three is a question of what you know about your load. If it is unpredictable, react. If it is cyclical and your instances are slow to warm, add prediction. If you have specific dated foreknowledge, schedule. Real production services usually run all three at once: a reactive policy as the safety net, prediction for the diurnal curve, and schedules for known events.

Choosing the Scaling Signal — Why CPU Is Often the Wrong Metric

The single most consequential autoscaling decision is which signal to scale on, and the default that every platform offers — average CPU utilization — is frequently the wrong one. CPU is the default because it is universally available and needs no application instrumentation, not because it is a good proxy for user-perceived load. It fails as a scaling signal whenever the work that hurts users is not CPU-bound.

Consider the failure cases. An I/O-bound web tier spends most of its time waiting on downstream calls; its CPU can sit at 20% while every request thread is blocked and latency has collapsed — CPU says “idle,” users say “down.” A service with a garbage-collected runtime can show CPU dominated by GC rather than useful work, so CPU rises without more requests being served. A queue-consumer’s CPU tells you nothing about how far behind it is falling. And CPU is easily confounded by co-tenancy and noisy neighbors on shared hosts, and by throttling, so the number you read is not even a faithful measure of the work being done.

The better signals are the ones that track the thing you actually care about — usually latency or backlog:

  • Requests per second (RPS) or concurrency per instance. For a request-serving service, scaling to hold a target of, say, 100 in-flight requests per instance ties capacity directly to offered load. AWS target tracking offers “request count per target” as a predefined metric precisely for this reason (AWS target tracking), and Google Cloud can scale on HTTP load-balancing serving capacity.
  • Queue depth or consumer lag. For asynchronous, event-driven work, the number of unprocessed messages is the truest measure of “am I keeping up.” This is the entire premise of KEDA (Kubernetes Event-Driven Autoscaling), which drives the HPA from the depth of a Kafka topic, an SQS queue, or a Redis list, and — critically — can scale a consumer to zero when no messages are pending and reactivate it when the first arrives (KEDA scaling). CPU cannot express “10,000 messages are waiting”; queue depth can.
  • p99 latency against an SLO. Scaling to defend a latency objective is the most direct expression of the SLO, though it must be paired with a load signal, because latency alone can be driven by causes (a slow dependency) that adding capacity will not fix.

The rule of thumb: scale on a signal that is causally, and ideally linearly, related to the capacity you control. AWS states this as a requirement — choose a metric that changes inversely proportional to capacity, so that doubling capacity roughly halves the metric (AWS target tracking). CPU sometimes satisfies this; RPS and queue depth usually satisfy it better. Getting real application metrics to the autoscaler is itself work — see Metrics Pipeline for Autoscaling for how Kubernetes plumbs custom and external metrics into the HPA.

The Scale-Up-Fast, Scale-Down-Slow Asymmetry and Flapping

The most important operational property of a well-tuned autoscaler is that it grows eagerly and shrinks reluctantly. The reason is an asymmetry of costs: being under-provisioned degrades or drops user requests immediately, while being over-provisioned for a few extra minutes costs only money. So the correct default is to scale up fast and scale down slow.

Kubernetes bakes this asymmetry into the HPA’s default behavior. Scale-up may add capacity as fast as the metric demands — by default it will double the pod count or add four pods per minute, whichever is larger. Scale-down is governed by a stabilization window that defaults to 300 seconds (5 minutes): before shrinking, the controller looks back over that window and uses the highest recent desired-replica value, so a momentary dip in load does not immediately tear down capacity that will be needed again seconds later (Kubernetes HPA docs). Google Cloud does the same with a default stabilization period of 10 minutes during which it holds capacity for the peak load seen in the interval, plus explicit “scale-in controls” that cap how much a group may shrink within a trailing window (GCP autoscaler). AWS target-tracking similarly treats scale-in more conservatively than scale-out.

The failure this machinery prevents is flapping (also called thrashing): a control loop that oscillates, repeatedly adding and removing capacity as the metric hovers near the target. Flapping is corrosive — every scale event has a cost (launching instances, draining connections, cold caches on new instances) so an autoscaler that flaps pays those costs continuously while delivering worse service than a static fleet. Three mechanisms suppress it, and understanding them is the core of tuning an autoscaler:

  1. A tolerance / dead band. The HPA skips scaling entirely when the ratio of current to target is within a tolerance of 0.1 (10%) of 1.0 (Kubernetes HPA docs). Without a dead band, ordinary metric noise around the set point would trigger endless tiny corrections.
  2. Stabilization windows (above), which smooth the down direction by remembering recent peaks.
  3. Cooldown periods, the AWS term for a rest interval after a scaling activity during which further like-scaling is suppressed, giving the previous change time to take effect before another is computed. The general lesson is that the loop must not act faster than its own actions take to register — otherwise it steers on stale error and overshoots in both directions.

There is a genuine tension here. Making scale-down slower and the dead band wider buys stability at the cost of efficiency — you carry idle capacity longer. Making them tighter saves money but risks flapping. There is no universal setting; the right point depends on how bursty your traffic is, how expensive a scale event is, and how much idle capacity costs you relative to a dropped request.

Cold Start, Warm-Up, and Warm Pools

The delay in the feedback loop — the interval between “the autoscaler decides to add capacity” and “that capacity is actually serving” — is the property that most often makes reactive autoscaling insufficient on its own. If an instance takes four minutes to boot, install its runtime, warm its cache, and pass health checks, then a reactive autoscaler responding to a spike delivers help four minutes too late, and the service is overloaded for the whole interval. Autoscalers account for this delay explicitly: the HPA has an initial-readiness delay (default 30 s) and a CPU-initialization period (default 5 min) during which a new pod’s metrics are treated cautiously so a not-yet-warm pod is not mistaken for evidence that scaling worked (Kubernetes HPA docs); Google Cloud’s initialization period (default 60 s) ignores a new VM’s usage for scale-out decisions until it has settled (GCP autoscaler).

But accounting for the delay does not remove it. The two structural answers are prediction (discussed above — start scaling before the load, so the warm-up finishes in time) and pre-warmed capacity. AWS EC2 warm pools are the canonical example: a pool of pre-initialized instances that sits alongside the Auto Scaling group, so that on a scale-out event the group draws already-booted instances from the pool (a warm start) instead of launching cold (AWS warm pools). The pool instances can be held Stopped (cheapest — you pay only for storage and any attached addresses), Hibernated (RAM contents saved to disk and restored on resume, so caches survive), or Running (fastest but discouraged, since you pay full instance price). AWS is blunt about the trade-off: creating a warm pool when your first-boot time does not actually cause noticeable latency is just wasted money, and if the pool is depleted during a spike, instances launch cold anyway. The general principle transfers beyond EC2 — Kubernetes practitioners keep spare over-provisioned “pause” pods so the Cluster Autoscaler can schedule real work instantly rather than waiting for a new node, and serverless platforms sell “provisioned concurrency” for exactly the same reason: pre-paying for warmth to hide the cold-start tail.

Min/Max Bounds — The Guardrails

Every autoscaler is configured with a minimum and a maximum capacity, and these are not incidental — they are the safety rails that keep the loop from doing something catastrophic. The minimum guarantees a floor of capacity regardless of what the metric says: it defends against a metrics-pipeline outage scaling you to zero, absorbs the first instant of a spike before the loop reacts, and preserves enough instances for redundancy across failure domains. The maximum is the more important guardrail: it caps blast radius when something goes wrong. A metrics bug, a retry storm, or a runaway load generator can make the desired-capacity formula demand absurd numbers of instances; without a ceiling, the autoscaler will faithfully try to launch them — draining your budget, exhausting your cloud quota, and potentially amplifying an incident by hammering an already-struggling downstream with more clients. The maximum turns “scale until the metric is satisfied” into “scale until the metric is satisfied or we hit the ceiling, then shed load and page.” Choosing the maximum is a capacity-planning exercise (see Capacity Planning and Demand Forecasting): it should be high enough to cover a realistic peak with headroom, low enough to bound cost and to stay within downstream dependency limits.

The Caveat That Matters Most — Autoscaling Cannot Fix a Shortage

The most important thing to understand about autoscaling is what it cannot do. Autoscaling redistributes and acquires capacity; it does not create capacity that does not exist, and it does not remove a bottleneck it does not control. Several concrete versions of this caveat bite in production:

  • You cannot scale past a hard dependency limit. If your service scales out but every instance opens connections to a single database with a fixed connection cap, adding instances past that point does not add throughput — it adds contention, and can tip the database over. Autoscaling the stateless tier in front of a non-scalable stateful backend just moves the queue and often makes things worse.
  • You cannot scale faster than you can acquire resources. A cloud region can run out of a given instance type; a Kubernetes cluster can run out of schedulable nodes until the Cluster Autoscaler adds one, which itself takes minutes. The autoscaler’s desire to grow is not the same as ability to grow.
  • Autoscaling can amplify a cascading failure. The Google SRE Book’s chapter on cascading failures warns that under overload “something needs to give,” and that mechanisms which react to load can make things worse (Google SRE — Addressing Cascading Failures). Concretely: if a service is failing because a downstream is slow, its own latency rises, an autoscaler keyed on latency or CPU adds instances, and those new instances each pile more concurrent load onto the already-overwhelmed downstream — the autoscaler pours fuel on the fire. New instances also start with cold caches, and a service not provisioned to serve under a cold cache is at greater risk of an outage precisely when the autoscaler is spinning up fresh, cold capacity in response to load.
  • Autoscaling is not a substitute for load shedding. When you hit the ceiling — or when the bottleneck is not the thing you can scale — the correct response is to shed or degrade load gracefully, not to keep scaling. Autoscaling and Load Shedding and Graceful Degradation are complementary defenses: one adds capacity when it helps, the other protects the service when adding capacity does not.

The synthesis: autoscaling is a powerful tool for the common case where load varies and capacity is elastic and the bottleneck is the thing you scale. It is not a reliability strategy on its own. It must be bounded (min/max), fed a good signal (usually not raw CPU), tuned against flapping (dead band, stabilization, cooldown), given time to warm (prediction and warm pools), and backstopped by load shedding for the cases it cannot solve. An autoscaler is an optimizer operating inside a capacity envelope that capacity planning must still provision.

See Also

Graph View

Backlinks