Capacity Planning and Demand Forecasting

Capacity planning is the SRE discipline of making sure a service has enough serving capacity — plus a deliberate margin of spare capacity — to meet future demand at its target reliability, and of provisioning that capacity before the demand arrives rather than after it has already caused an outage. Google states the requirement plainly: capacity planning must ensure “sufficient capacity and redundancy to serve projected future demand with the required availability” (Google SRE Book, Introduction). A service that runs out of capacity fails as surely as one with a bug — it just fails on a Tuesday during a launch instead of at deploy time. The discipline has three moving parts that this note develops in turn: forecasting demand (both the steady kind and the lumpy kind), sizing headroom and redundancy (so losing a unit at peak is survivable, not catastrophic), and provisioning ahead of the lead time it takes to actually acquire the capacity. Capacity planning sits high in Dickerson’s service reliability hierarchy — above testing, below development — because you cannot meaningfully plan capacity until you can measure the system (monitoring), survive its failures (incident response), and test how raw resources translate into serving capacity (Google SRE Book, Part III — Practices).

This is the general operational discipline. Its concrete Kubernetes instance — control-plane scale ceilings, worker headroom, etcd limits — lives in Capacity Planning on Kubernetes and is cross-linked, not duplicated here.

Mental Model

Provisioned capacity is not a single number; it is a stack of margins on top of the raw demand you actually expect to serve. Each layer answers a different “what if.”

flowchart TB
    subgraph STACK["Total capacity to provision"]
        ONEOFF["One-off / inorganic buffer<br/>launches · sales · migrations"]
        REDUN["Redundancy headroom<br/>N+1 / N+2 — survive losing a unit at peak"]
        GROWTH["Growth buffer<br/>organic demand over the lead time"]
        PEAK["Forecast peak organic demand<br/>(the load you expect to actually serve)"]
    end

    PEAK --> GROWTH --> REDUN --> ONEOFF

    LEAD["Provisioning lead time<br/>(weeks–months to acquire capacity)"] -. "forces you to<br/>plan this far ahead" .-> GROWTH
    UTIL["Utilization target<br/>(~60–70%, not 95%)"] -. "sets how much<br/>of it you dare use" .-> STACK

What it shows and the insight to take: the load you serve (bottom) is only the foundation. On top of it you stack (1) a growth buffer for the organic demand that will materialize before your next capacity delivery, sized to the provisioning lead time; (2) redundancy headroom so that when a machine, rack, zone, or region fails at peak, the survivors can absorb the load; and (3) a buffer for one-off inorganic events — a launch, a marketing push, a data migration. The utilization target then caps how much of the whole stack you allow yourself to run at, because queueing theory guarantees latency explodes as you approach 100%. The central lesson: the goal is not to run efficiently at 100%; it is to have somewhere to fall when reality deviates from the forecast — and reality always does.

Mechanical Walk-Through: The Capacity Planning Cycle

The traditional capacity planning process is a cycle, and Google describes it as exactly that: gather demand forecasts, devise a plan for how much to build and where to allocate it, get the plan reviewed and signed off, and deploy the resources (Google SRE Book ch.18, Software Engineering in SRE). Each turn of the cycle rests on three requirements that Google enumerates for any adequate plan (Introduction):

1. An accurate organic demand forecast that extends beyond the provisioning lead time. You must predict demand far enough into the future that you can act on the prediction — if it takes eight weeks to land new servers, a four-week forecast is useless.
2. Accurate incorporation of inorganic demand sources into that forecast. Steady growth is not the whole story; launches and campaigns must be folded in.
3. Regular load testing that correlates raw capacity to service capacity. You must know how many queries-per-second one server actually buys you, because “ten more machines” means nothing until you know what a machine is worth in service terms.

The third requirement is why capacity planning sits above testing in the reliability hierarchy: load testing is the measuring instrument that turns “servers, disks, CPUs” into “requests per second at p99 under 300 ms.” Without that calibration, the forecast (in demand units) and the plan (in hardware units) speak different languages.

Organic versus inorganic growth

The single most useful distinction in demand forecasting is between two kinds of growth, because they are predicted with completely different methods.

Organic growth is the steady, natural increase that “stems from natural product adoption and usage by customers” (Introduction). It is the slope you get for free as more users find the product and existing users use it more. Because it is smooth and driven by underlying trends, organic growth is forecastable with time-series methods — extrapolate the trend, fit the seasonality (daily, weekly, holiday cycles), project forward. It is the “easy” part precisely because history is a good guide.

Inorganic growth results from “events like feature launches, marketing campaigns, or other business-driven changes” (Introduction). It is lumpy, discontinuous, and not visible in the historical curve — a Super Bowl ad, a new feature that goes viral, a partner integration, a migration that shifts a neighboring service’s traffic onto yours. You cannot extrapolate inorganic growth from past traffic because by definition it has no past. It must be sourced from the business: the product roadmap, the marketing calendar, the launch schedule. This is why capacity planning is inherently cross-functional — the SRE cannot forecast a launch spike the product team has not told them about. The classic capacity failure is a launch that succeeds too well: the forecast captured the organic trend perfectly and missed the 50× inorganic spike entirely (Google’s own Pokémon GO case study is the canonical example — real traffic arrived at roughly 50× the initial estimate).

Headroom, Redundancy, and the N+ Notation

Serving the expected peak is not enough, because at the moment you most need capacity — peak load — is also when a failure hurts most. Headroom is the deliberate spare capacity you carry so the service survives the loss of a unit while at peak. The standard vocabulary for expressing how much redundancy you carry is N+ notation, where N is the number of units required to serve peak demand:

• N+0 — no redundancy. You have exactly enough to serve peak. Lose any unit and you are under-provisioned and shedding load or falling over. Essentially never acceptable for a user-facing service.
• N+1 — one spare unit. You can lose a single unit (a machine, a rack, a zone, depending on what “unit” means) and still serve peak. This is the minimum for most production services and directly analogous to redundancy in power supplies or RAID.
• N+2 — two spare units. You can lose one unit and still tolerate a second failure — critically, this lets you take a unit down for planned maintenance or a rollout and still survive an unplanned failure of another. N+2 is the common target for important services precisely because “one down for maintenance, one lost to a fault” is a routine Tuesday, not a black-swan.

Google’s intent-based capacity system treats these redundancy levels as first-class inputs — a service declares “I want N+2 redundancy” as an abstract intent, and the planner works out the concrete allocation that satisfies it (ch.18). The workbook makes the operational version of the same point: user-facing services must reserve enough spare capacity for both overload protection and redundancy, so that when an entire region goes offline the remaining regions can absorb its traffic without cascading (Managing Load).

The subtle, expensive trap is that redundancy must be sized against your failure domain. If your unit of failure is a region and you run in three regions active-active, then surviving the loss of one region means each region can run at most ~⅔ utilization at peak — losing one leaves two to carry 100% of the load. Provision three regions at 90% each and a single regional failure instantly overloads the survivors and triggers a cascading failure. The redundancy math and the failure-domain design are the same decision viewed from two angles.

Provisioning Lead Time: Why You Plan Ahead

Capacity is not summoned instantly. Provisioning lead time is the interval between deciding you need capacity and that capacity actually serving traffic — and it can be long: hardware must be ordered, racked, cabled, imaged, and integrated; even in the cloud, quota increases, reserved-instance commitments, and region build-outs take real time. This is why Google’s first forecasting requirement insists the organic forecast extend beyond the lead time: you must decide to acquire capacity early enough that it lands before the demand does.

Provisioning is also described in the reliability hierarchy as the step that combines capacity planning with change management, and Google stresses that it should be done quickly and only when necessary, because adding capacity is one of the riskier operations you can perform — new, unverified capacity that has never taken real traffic is a change, and changes cause outages (Part III — Practices). The discipline is therefore a balance: provision far enough ahead to beat the lead time, but validate the new capacity (load test it, canary traffic onto it) before you depend on it. “We added the servers but never confirmed they actually serve” is a real and common way to have a capacity plan on paper and an outage in production.

Intent-Based Capacity Planning

At scale, the manual cycle breaks down. Google documents why (ch.18): the traditional spreadsheet-driven approach is brittle — a small change (an efficiency regression, an adoption spike, a slipped delivery) forces you to re-check or re-derive the entire allocation plan, because one cluster’s slippage cascades into every service’s requirements — and it is labor-intensive, because the underlying allocation problem is bin-packing, which is NP-hard, and humans do it slowly and with errors. Teams cope by making simplifying assumptions that quietly leave the plan wrong.

The proposed fix is intent-based capacity planning: instead of specifying concrete demands (“50 cores in clusters X, Y, Z”), operators express why they need resources at increasing levels of abstraction — from explicit requests, to location-flexible requests (“any 3 clusters in region YYY”), to redundancy intent (“N+2”), to pure SLO intent (“five nines of availability”). Google reports that services get the best wins as they cross into declaring redundancy-level intent, because that is the abstraction that is both flexible enough to optimize and concrete enough that its consequences are understandable. The planner then needs service dependencies (who calls whom, with what latency budget) and performance metrics — the “glue” that converts a high-level demand into low-level resource needs by answering “for every N queries of service Foo, how many resources does dependency Bar consume?”

Google implemented this in a tool called Auxon, built by SREs over roughly two years, which ingests intent configs, demand forecasts, resource supply, and pricing, encodes the whole thing as a mixed-integer / linear program, and solves for an allocation plan — which resources go to which service in which cluster. The engineering lesson the chapter draws is as valuable as the tool: they shipped a crude heuristic “Stupid Solver” first and iterated, rather than waiting for a perfect optimizer — launch and iterate applied to internal tooling.

Uncertain

Verify: whether “Auxon” is still the name and design of Google’s production capacity-planning system today, and whether the four-level intent abstraction is described the same way in current internal practice. Reason: the primary source is the 2016 SRE Book ch.18, which is nearly a decade old; internal tooling evolves and rebrands. To resolve: check for a newer Google/USENIX write-up on intent-based capacity planning. The concepts (intent-based planning, N+ redundancy, organic/inorganic growth, the three forecasting requirements) are stable and independently corroborated; the specific tool name and its current status are the point-in-time claims. #uncertain

Utilization Targets: How Much of the Stack You Dare Use

Capacity planning and utilization targeting are two halves of one decision. Having provisioned the margin stack, you must decide what steady-state utilization to run at — and the answer is not “as close to 100% as possible,” because queueing theory (Little’s Law, the M/M/1 latency curve) says wait time rises hyperbolically as utilization approaches 1: at 90% utilization a small demand bump pushes you past the latency knee and tail latency explodes. Utilization targets in the ~60–70% range are common precisely so that a forecast miss, a traffic spike, or a lost redundancy unit still leaves you on the flat part of the curve. This is developed fully in Utilization Targets and the Latency Knee; the tie-in for capacity planning is that the utilization target is what converts “how much load do we serve” into “how much capacity do we buy” — buy load ÷ target, not load ÷ 1.

Failure Modes and Anti-Patterns

• Forecasting only organic growth. The classic outage: the time-series model nails the steady curve and completely misses the launch spike nobody told the SRE about. Inorganic demand must be sourced from the business calendar, not the traffic history.
• Provisioning to N+0. Sizing to exactly serve peak leaves no room for a single failure — and failures happen most at peak. Any user-facing service should be at least N+1, usually N+2.
• Redundancy sized against the wrong failure domain. Carrying “one spare machine” is meaningless if your real failure unit is a zone or region. Redundancy must match the blast radius you actually need to survive.
• Ignoring the lead time. A perfect forecast is worthless if you decide to act on it after the capacity can no longer arrive in time. The forecast horizon must exceed the provisioning lead time.
• Trusting new capacity blindly. Freshly provisioned capacity that has never served real traffic is an untested change. Load-test and canary it before you count on it.
• Optimizing utilization to the point of fragility. Running at 95% utilization to “save money” trades a small, visible cost (idle capacity) for a large, catastrophic, invisible-until-it-hits cost (latency collapse under any deviation). The idle capacity is the reliability.
• Static plans in a dynamic world. A capacity plan reviewed once a quarter goes stale between reviews as efficiency, adoption, and dependencies drift — the brittleness Google cites as the motivation for intent-based, continuously-solved planning.

Alternatives and When to Choose Them

• Manual / spreadsheet capacity planning. Fine for a handful of services with predictable demand and generous lead time. Breaks down at scale (brittleness, bin-packing complexity) — the failure mode that motivated intent-based systems.
• Intent-based / automated capacity planning (Auxon-style). Choose when you have many interdependent services, tight resource constraints, and frequent change, so the allocation problem is too large and too volatile for humans to keep correct.
• Autoscaling. Reactive capacity that follows demand within a provisioned envelope, on the timescale of minutes. It is complementary to, not a replacement for, capacity planning: autoscaling can only allocate capacity that planning has already procured, and it cannot conjure a new region during a launch. Autoscaling handles the sub-lead-time wiggle; capacity planning handles the envelope and the lead-time-scale growth. Note also the scale-up/scale-down asymmetry — autoscalers add capacity eagerly and remove it cautiously (Managing Load) — which is itself a form of headroom.
• Load shedding. The last-resort complement: when demand exceeds even your headroom, shed the least-important load to protect the core rather than collapsing entirely. Capacity planning tries to make shedding rare; a good service can still do it when the forecast is beaten.

Production Notes

The hardest part of capacity planning in practice is rarely the math — it is the organizational plumbing: getting the product and marketing teams to disclose their launch calendar early enough to fold inorganic demand into the forecast, and getting finance to fund headroom that, when the discipline works, looks like waste because the failure it was reserved for never happened. The utilization-target conversation is where reliability and cost meet head-on, and the SRE’s job is to make the trade-off explicit and numeric rather than letting it be decided implicitly by whoever last argued about the cloud bill. The interview-ready synthesis: capacity planning is demand forecasting (organic + inorganic) plus headroom (N+1/N+2 sized to your failure domain) plus provisioning ahead of the lead time, all governed by a utilization target that keeps you off the latency knee — and the spare capacity that looks like waste on the balance sheet is exactly the capacity that keeps you up when the forecast is wrong.

See Also

• Capacity Planning on Kubernetes — the concrete Kubernetes instance of this discipline (control-plane ceilings, etcd limits, worker headroom, pod over-provisioning)
• Utilization Targets and the Latency Knee — the queueing-theory reason you target ~60–70%, not 95%
• Load Testing Types and Practice — how you correlate raw capacity to service capacity (the third forecasting requirement)
• Autoscaling in Practice — reactive capacity within the provisioned envelope; complement, not substitute
• Load Shedding and Graceful Degradation — what to do when demand beats even your headroom
• Failure Domains and Blast Radius — the failure-domain design that sets how much redundancy you need
• Redundancy and Failover Strategies — active-active vs active-passive and the N+ math in depth
• The Service Reliability Hierarchy — where capacity planning sits (above testing, below development)
• Site Reliability Engineering MOC — parent map (§7 Capacity Planning and Load Management)