vault

Capacity Planning on Kubernetes

16 min read

Capacity Planning on Kubernetes

Capacity planning on Kubernetes is the discipline of answering two questions before the cluster forces the answer on you: “how big can this control plane grow before it falls over?” and “how much worker capacity do I need — and how much spare worker capacity do I need so the scheduler has room to maneuver?” Kubernetes itself ships explicit scale ceilings: the upstream “considerations for large clusters” guidance states the platform is designed to support clusters meeting all of — no more than 5,000 nodes, 150,000 total Pods, 300,000 total containers, and 110 Pods per node (Considerations for large clusters). But raw ceilings are the least useful number in practice. What actually breaks first is rarely the node count — it is etcd (disk IOPS and the database size limit) and watch fan-out on the API server. And on the worker side, the trap is not running out of capacity but running too close to capacity: a 100%-packed cluster cannot tolerate a single node failure or absorb a rollout surge, because the scheduler has nowhere to put the displaced Pods. Capacity planning is therefore as much about deliberately reserving headroom as it is about sizing.

Mental Model

flowchart TB
    subgraph CP["Control-plane capacity"]
        ETCD["etcd<br/>DB size, disk IOPS, fsync latency<br/>← scales with OBJECT COUNT"]
        APIS["kube-apiserver<br/>CPU/memory<br/>← scales with WATCH volume<br/>+ LIST request rate"]
        ETCD --> APIS
    end
    subgraph WORKER["Worker capacity"]
        CAP["Node capacity<br/>(what the kernel sees)"]
        RES["− kube-reserved<br/>− system-reserved<br/>− eviction threshold"]
        ALLOC["= Allocatable<br/>(what the scheduler may pack)"]
        REQ["− Σ Pod requests"]
        HEAD["= Headroom<br/>(spare for failover + surge)"]
        CAP --> RES --> ALLOC --> REQ --> HEAD
    end
    OBJ[Object count<br/>Pods, Secrets, CRs, Events] --> ETCD
    CTRL[Controllers + kubectl + operators<br/>each opens watches] --> APIS

What this diagram shows. Two independent capacity budgets. The control-plane budget (top) is driven by the total number of API objects and the number of watchers — not by how much CPU your workloads burn. The worker budget (bottom) is a subtraction chain: the kernel’s raw capacity minus what the kubelet reserves for itself and the OS, yielding allocatable; then minus the sum of every scheduled Pod’s requests (see Resource Requests and Limits), yielding headroom. The insight to extract: these two budgets fail for completely different reasons and must be planned separately. A cluster can have abundant worker CPU and still fall over because etcd’s disk can’t keep up with object churn — and vice versa.

Mechanical Walk-through

Control-plane sizing — what scales with what

The control plane does not care how busy your workloads are. It cares about the size and churn of the API object set and how many clients are watching it.

etcd is the binding constraint at scale. etcd is a Raft-replicated key-value store (etcd, Raft) and every write must be fsync’d to disk on a quorum of members before it is acknowledged. Three things stress it:

  1. Database size. etcd has a default storage quota of 2 GiB (raisable, but the project recommends staying under ~8 GiB). Every Pod, Secret, ConfigMap, custom resource, and — notoriously — every Event consumes space. The upstream large-cluster guidance explicitly recommends running a separate etcd instance dedicated to Event objects, because Events are high-churn and would otherwise dominate the main store.
  2. Disk IOPS and fsync latency. etcd is exquisitely sensitive to disk latency. The single most important capacity decision for a self-managed control plane is fast, dedicated SSD/NVMe storage for etcd — a slow or shared disk raises fsync latency, which raises Raft commit latency, which raises every API write’s latency cluster-wide.
  3. Object count → memory. etcd holds its data in a memory-mapped B+ tree; more objects means more memory.

kube-apiserver scales with a different axis: watch fan-out. Every controller, every operator, every kubectl get -w, every kubelet, and every node-agent opens long-lived watch streams (Watch and Informers). When an object changes, the API server must fan that change out to every interested watcher. A cluster with thousands of nodes and dozens of operators can have tens of thousands of open watches; the apiserver’s CPU and memory scale with that fan-out, not with workload traffic. The other apiserver stressor is un-paginated LIST requests — a client that does LIST pods across the whole cluster without limit/continue pagination forces the apiserver to load the entire result set into memory at once (see Failure Modes, and Common Kubernetes Failures Catalog).

Sizing approach. The upstream guidance is deliberately qualitative: scale the control plane vertically first (bigger apiserver/etcd machines), and only add control-plane replicas (horizontal) once vertical scaling shows diminishing returns — with at least one control-plane instance per failure zone for HA. The 5,000-node ceiling is what the project tests; most organizations should treat ~1,000–2,000 nodes as the point where control-plane tuning becomes a dedicated project, and consider splitting into multiple clusters (Multi-Cluster Kubernetes) rather than chasing the ceiling.

Worker capacity — the requests-vs-actual gap and the allocatable subtraction

Worker sizing has a chain of subtractions. Start with node capacity — what the Linux kernel reports (e.g. 8 vCPU, 32 GiB). The kubelet then carves out reservations (Reserve Compute Resources):

  • --kube-reserved — CPU/memory/disk set aside for the kubelet, the container runtime (containerd), and other K8s node daemons.
  • --system-reserved — set aside for OS daemons (systemd, sshd, the kernel itself).
  • Hard eviction threshold — a final sliver (e.g. memory.available<100Mi) the kubelet keeps free so it can act before the kernel OOM-killer does.

What remains is allocatable — the only number the kube-scheduler is allowed to pack against. This is the source of the perennial “my node has 32 GiB but only ~29 GiB is schedulable — why?” surprise. If you do not set these reservations, the kubelet packs Pods right up against raw capacity and the node itself starves — a documented k8s.af failure pattern (kubelet OOM-killed because system reservation was unset).

The scheduler then subtracts the sum of Pod requests — never limits, never live usage (see Resource Requests and Limits). This creates the request-vs-actual-usage gap: industry studies repeatedly find requested CPU running 5–8× actual usage (Sedai capacity-planning guide). A cluster can therefore report “100% requested / no room for new Pods” while its CPU dashboards sit at 15%. Capacity planning must reason in requested terms (that is what the scheduler sees) while cost planning must reason in actual terms (see Kubernetes Cost Management) — the gap between the two is the rightsizing opportunity.

Headroom — the part everyone forgets

A cluster packed to 100% of allocatable is not a well-utilized cluster — it is a fragile one. Headroom is needed for three distinct events:

  1. Node failure. If one node dies, every Pod on it must reschedule. With zero headroom there is nowhere for them to land — they sit Pending until the Cluster Autoscaler or Karpenter provisions a replacement, which takes minutes.
  2. Rollout surge. A Deployment’s Rolling Update Strategy with maxSurge: 25% temporarily creates extra Pods before terminating old ones. With no headroom the surge Pods go Pending and the rollout stalls.
  3. Scheduler maneuvering room. Topology Spread Constraints, anti-affinity, and bin-packing all need slack to satisfy — a perfectly full cluster gives the scheduler no freedom.

The naïve way to reserve headroom is to size the cluster larger and hope it stays empty — but the autoscaler, seeing idle nodes, will remove them. The disciplined way is overprovisioning Pods (also called pause-capacity Pods or balloon Pods): deploy a Deployment of Pods that run a do-nothing pause container, request real CPU/memory, and carry a negative-priority PriorityClass (AWS — Eliminate node scaling lag). These Pods occupy the headroom — keeping the autoscaler from reclaiming it — but the instant a real Pod needs the space, the scheduler preempts the low-priority pause Pods (they have negative priority, so they lose every contest), the real Pod schedules immediately, and the now-Pending pause Pods trigger the autoscaler to grow the cluster in the background. The result is a warm buffer that absorbs failures and surges at zero scheduling latency.

Right-sizing — closing the request-vs-usage gap

The single highest-leverage capacity-planning activity is right-sizing requests: bringing the requests figures (which the scheduler packs against) closer to the actual observed steady-state usage. A cluster where requests are 5× actual usage is paying for 5× the worker capacity it needs — those idle nodes still bill the cloud provider, still draw operational toil, and still count against the Cluster Autoscaler’s utilisation threshold.

The mechanical methodology is well-established. Collect baseline data: at least one week of CPU and memory series at fine resolution — long enough to span weekly cycles, daily traffic patterns, and at least one batch peak. Pick a percentile per resource: P95 or P99 for CPU (which can absorb brief spikes via the proportional cpu.weight split), and P99 or even max for memory (which cannot — over-limit means OOMKilled). Apply a safety margin of ~10–25% on top of the percentile. Set the request equal to that figure (or remove the limit entirely for CPU and keep only the request, the widely-used pattern that avoids CFS throttling — see cgroups Integration). This algorithm is exactly what the Vertical Pod Autoscaler’s recommender does autonomously — it observes resource usage histograms over a moving window and emits a recommendation that the updater can apply.

Until recently, VPA’s Auto mode worked by evicting a Pod and letting the controller recreate it with the new request — a disruptive operation that made VPA hard to use on stateful or latency-sensitive workloads. In-Place Pod Resize went GA in Kubernetes v1.35 (December 2025 blog): spec.containers[*].resources is now a mutable field, changes flow through a new /resize subresource, and the kubelet rewrites the underlying cgroup files (cpu.max, memory.max) without restarting the container. Pod-level in-place resize — adjusting the aggregate Pod cap rather than per container — graduated to beta in v1.36 (April 2026). The capacity-planning consequence is that the right-sizing feedback loop can finally close non-disruptively: VPA observes, recommends, and the kubelet applies the new request in place. Some applications still require a restart to react to a smaller memory limit (the JVM heap, for example, is sized at startup), and the per-container resizePolicy field declares that requirement so VPA / in-place resize know to fall back to a restart.

The complementary autoscaler is the Horizontal Pod Autoscaler (HPA docs), which adjusts replica count rather than per-replica size. The classical algorithm is desiredReplicas = ceil(currentReplicas × currentMetric / desiredMetric) against a target utilisation of the request; the ratio must differ from 1.0 by more than 10% (the default tolerance) to actually trigger scaling, and a 3-minute stabilisation window prevents flapping. HPA and VPA together cover both axes — VPA right-sizes each replica, HPA varies the replica count — but combining them on the same cpu/memory metric is unsafe (each would chase the other); the standard practice is VPA for CPU/memory and HPA for a separate signal such as request rate or queue depth.

Pod density limits

Even a huge node cannot run unlimited Pods. Two ceilings bind:

  • The 110-Pods-per-node default. The kubelet’s --max-pods defaults to 110. This is a kubelet limit, raisable, but raising it stresses the kubelet and the node’s cgroup hierarchy.
  • CNI IP-per-node limits. This is often the real binding constraint. The AWS VPC CNI assigns each Pod a VPC IP from ENIs attached to the node, and instance types cap how many ENIs/IPs a node may hold — a small instance might top out at ~8–29 Pods regardless of CPU. Overlay CNIs (Calico, Cilium) carve a per-node /24 (254 addresses) and so rarely hit this. Choosing instance types must account for the IP budget, not just CPU/memory.

Configuration / API Surface

An overprovisioning (pause-capacity) Deployment plus its negative PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -10                          # NEGATIVE → loses every preemption contest
globalDefault: false
description: "Placeholder Pods that reserve headroom; preempted by any real Pod."
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
  namespace: kube-system
spec:
  replicas: 10                      # 10 placeholders → 10 Pods' worth of warm headroom
  selector: { matchLabels: { app: overprovisioning } }
  template:
    metadata:
      labels: { app: overprovisioning }
    spec:
      priorityClassName: overprovisioning   # the negative-priority class above
      terminationGracePeriodSeconds: 0      # evict instantly — nothing to clean up
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10 # the do-nothing infra image
          resources:
            requests:                        # THIS is what reserves the headroom
              cpu: "1"
              memory: 4Gi
            limits:
              cpu: "1"
              memory: 4Gi

Line-by-line. value: -10 is the crux: a negative priority means any normal Pod (default priority 0 or higher) will preempt these placeholders the instant capacity is tight. replicas: 10 × requests of 1 CPU / 4Gi reserves exactly ten Pods’ worth of warm capacity. terminationGracePeriodSeconds: 0 lets the scheduler evict a placeholder immediately — there is no workload to drain. The pause image is the same near-empty binary K8s uses for the Pause Container; it burns no CPU. When a real Pod needs room, a placeholder is killed, the real Pod schedules with zero delay, and the displaced placeholder goes Pending — which the Cluster Autoscaler / Karpenter sees and reacts to by adding a node, restoring the buffer.

Worker reservation flags on the kubelet (the allocatable gap):

# kubelet flags / KubeletConfiguration fields
--kube-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi
--system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi
--eviction-hard=memory.available<500Mi,nodefs.available<10%
# Result on an 8-vCPU / 32Gi node:
#   capacity.cpu      = 8        allocatable.cpu    ≈ 7
#   capacity.memory   = 32Gi     allocatable.memory ≈ 29.5Gi

Failure Modes

etcd database full. The 2 GiB default quota is exceeded — typically by Event churn or an operator that creates thousands of custom resources. etcd goes read-only (mvcc: database space exceeded), and because the apiserver cannot write, the entire cluster freezes for writes. Prevention: dedicated etcd disk, a separate etcd for Events, etcd --auto-compaction plus periodic etcdctl defrag, and alerting on etcd_mvcc_db_total_size_in_bytes.

apiserver OOM from un-paginated LIST. A runaway client (a buggy controller, an kubectl get pods -A against a 100k-Pod cluster) issues a LIST with no limit. The apiserver materialises the whole result set in memory and OOMs. API Priority and Fairness mitigates this; the real fix is paginating clients and capacity-sizing the apiserver for the worst legitimate LIST.

The 100%-packed cluster cannot self-heal. A cluster sized with zero headroom loses a node; the displaced Pods sit Pending for the full autoscaler provisioning latency (3–8 minutes on a Cluster Autoscaler ASG round-trip). For that window, availability is degraded. The fix is the overprovisioning pattern above.

CNI IP exhaustion mistaken for CPU exhaustion. Pods stick in ContainerCreating with failed to assign an IP address. Operators size by CPU/memory and forget the AWS VPC CNI ENI/IP cap. The node has spare CPU but no addresses. Prevention: count the IP budget per instance type during capacity planning, or use a prefix-delegation CNI mode.

Scheduler can’t place despite “free” capacity. Aggregate allocatable looks fine, but every node individually fails a Topology Spread Constraints or anti-affinity predicate — the cluster is fragmented. A bigger total is no help; the Pods need slack per topology domain.

Alternatives and When to Choose Them

  • Reactive autoscaling only (Cluster Autoscaler / Karpenter with no headroom). Cheapest steady-state cost; pays a latency penalty on every spike and failure. Choose for cost-sensitive, disruption-tolerant batch workloads.
  • Static capacity (fixed node count, no autoscaler). Predictable, audit-friendly, the only option on fixed-footprint on-prem or regulated environments. Wastes money in troughs; risks Pending Pods in peaks. Choose when the workload is steady and the footprint is mandated.
  • Overprovisioning Pods + autoscaler (the recommended production shape). A warm buffer absorbs spikes instantly while the autoscaler refills it in the background. Slightly higher steady-state cost (you pay for the buffer). Choose for latency-sensitive production services.
  • Karpenter just-in-time provisioning. Karpenter’s ~60-second node provisioning shrinks the headroom you need — fast enough that a smaller buffer suffices. Choose on supported clouds for the best cost/latency balance.
  • Split into multiple clusters. Beyond ~2,000 nodes, or when one etcd is the bottleneck, Multi-Cluster Kubernetes sidesteps the single-control-plane ceiling entirely. Choose when control-plane scaling becomes its own engineering project.

Production Notes

  • The AWS EKS Best Practices guide recommends pairing overprovisioning Pods with the autoscaler and warns that without headroom, every scale-up is on the critical path of a latency spike.
  • Shopify’s BFCM (Black Friday / Cyber Monday) scaling work pre-warms node pools ahead of predicted peaks rather than relying on reactive autoscaling — a scheduled cron bumps the node-group desired count before the event. This is capacity planning as a time-series problem: the headroom you need is a function of when.
  • The k8s.af catalog features multiple capacity incidents: kubelet OOM-killing critical Pods because system reservation was unset; AZ-outage reschedule storms hitting clusters with no spare capacity; etcd freezes from Event-object churn (see Common Kubernetes Failures Catalog).
  • A recurring lesson: plan control-plane capacity by object count, not by workload size. A cluster running a few large workloads can have a tiny apiserver; a cluster running thousands of small CRs and operators needs a large one — even if total CPU is identical.
  • The upstream large-cluster page added guidance in February 2026 to run cluster-essential components under system-cluster-critical or system-node-critical PriorityClasses so CoreDNS, metrics-server, the CNI agent, and similar workloads cannot be preempted by user workloads when capacity tightens. Treat this as a capacity-planning concern, not just a configuration nicety: the moment overprovisioning Pods cannot soak a spike, the next thing the scheduler reaches for is whatever workload it can evict, and an evicted CoreDNS Pod can take down DNS resolution cluster-wide.

The 5,000-node / 150,000-Pod / 300,000-container / 110-Pod-per-node ceilings are the upstream-tested limits as documented on the Considerations for large clusters page, verified against the current docs (last modified February 8, 2026). etcd’s default storage quota is 2 GiB with a suggested maximum of 8 GiB above which etcd warns at startup — both are documented in the etcd dev guide and are configurable via --quota-backend-bytes. The one remaining order-of-magnitude figure is the request-vs-actual-usage gap (~5–8× for CPU): it is a workload-dependent observation reported repeatedly by FinOps studies, not an upstream-tested constant — your cluster’s gap may be far smaller (well-tuned VPA pipelines) or far larger (default-sized off-the-shelf charts). Measure with OpenCost (CNCF incubating, originally created by Kubecost) or a similar tool to ground capacity work in your own numbers, not the industry average.

See Also

Graph View

Backlinks