Pod Overhead

Pod Overhead is the Kubernetes mechanism that makes the scheduler and the kubelet account for the resources consumed by a Pod’s infrastructure — the per-Pod cost of the chosen container runtime itself — rather than only the resources requested by the Pod’s application containers. It surfaces as the spec.overhead field, populated automatically at admission time from the Pod’s RuntimeClass (Kubernetes — Pod Overhead). The motivating case is VM-isolated runtimes such as Kata Containers or gVisor: a Kata Pod boots a lightweight virtual machine, and that VM has a non-trivial fixed memory and CPU cost (the guest kernel, the VMM process, the agent) that exists before any application container runs. Without Pod Overhead the scheduler under-counts that cost, over-packs nodes, and the node later hits memory pressure and evicts Pods. Pod Overhead reached stable (GA) in Kubernetes v1.24 — the PodOverhead feature gate was locked to true and then removed (Feature Gates — removed). It tracks back to KEP-688: alpha in v1.16, beta in v1.18, GA in v1.24.

Mental Model

The default runtime (runc) has near-zero per-Pod overhead: a Pod is a set of Linux namespaces and cgroups plus the tiny Pause Container. The resources the scheduler must reserve are essentially sum(container requests). But a sandboxed runtime like Kata wraps the whole Pod in a micro-VM. That VM consumes, say, 120 MiB of memory and 250 millicores of CPU just to exist. If the scheduler reserves only the container requests, it believes a 4 GiB node can hold thirty-three 120 MiB Pods — when in reality each Pod also consumes 120 MiB of VM overhead, so the node holds far fewer.

Pod Overhead closes the gap by letting a RuntimeClass declare its fixed per-Pod cost once, and having an admission controller stamp that cost onto every Pod that selects the RuntimeClass. From then on, two consumers read spec.overhead: the scheduler adds it to the Pod’s resource footprint when checking node fit, and the kubelet adds it when sizing the Pod-level cgroup.

flowchart TD
    RC["RuntimeClass kata-fc<br/>overhead.podFixed:<br/>memory 120Mi, cpu 250m"]
    POD["Pod created with<br/>spec.runtimeClassName: kata-fc"]
    ADM["RuntimeClass admission controller<br/>(mutating)"]
    POD2["Pod now carries<br/>spec.overhead.podFixed:<br/>memory 120Mi, cpu 250m"]
    SCHED["kube-scheduler — NodeResourcesFit<br/>node needed = sum(container requests) + overhead"]
    KUBELET["kubelet — Pod cgroup sizing<br/>cgroup limit = sum(container limits) + overhead<br/>cpu.shares = sum(container requests) + overhead"]
    QUOTA["ResourceQuota accounting<br/>charges overhead too"]
    RC --> ADM
    POD --> ADM
    ADM --> POD2
    POD2 --> SCHED
    POD2 --> KUBELET
    POD2 --> QUOTA

What this diagram shows. A single fact — “this runtime costs 120Mi/250m per Pod” — declared once on the RuntimeClass, is injected into the Pod by the admission controller and then fans out to three independent consumers: the scheduler (placement math), the kubelet (cgroup math), and the quota controller (tenancy accounting). The insight to extract is that spec.overhead is not something a user sets — it is derived, immutable-after-admission bookkeeping, and the RuntimeClass author is the single source of truth for the number.

Mechanical Walk-through

A Pod author writes spec.runtimeClassName: kata-fc and the usual resources.requests / resources.limits on each container. They do not write spec.overhead — if they try, the RuntimeClass admission controller rejects the Pod (the field is owned by the system).

At admission, the built-in RuntimeClass admission controller (a mutating admission plugin compiled into the API server) looks up the named RuntimeClass, reads its overhead.podFixed map, and writes it verbatim into pod.spec.overhead. If the RuntimeClass has no overhead stanza, spec.overhead is left empty (zero cost — the runc case). The mutation happens once; spec.overhead is immutable for the life of the Pod.

When the scheduler runs its NodeResourcesFit Filter plugin (see kube-scheduler), it computes the Pod’s total demand as sum over containers of requests plus spec.overhead.podFixed. A node is feasible only if its allocatable capacity minus already-committed Pods can absorb that combined number. The same combined number feeds the NodeResourcesFit Score plugin, so bin-packing and spread heuristics also see the true footprint. Init containers are handled with the usual max-of-init-vs-sum-of-app rule, and the overhead is added on top of whichever wins.

On the node, the kubelet uses spec.overhead in two places when it constructs the Pod-level cgroup (the cgroup that is the parent of all the Pod’s container cgroups):

Cgroup limits. The Pod cgroup’s memory limit is sum(container memory limits) + overhead.memory, and its CPU quota (cpu.cfs_quota_us) is derived from sum(container CPU limits) + overhead.cpu. This means the VM’s memory genuinely lives inside the Pod’s cgroup accounting — if the runtime overruns, the Pod cgroup’s OOM killer fires, not a neighbour’s.
CPU shares. For Guaranteed and Burstable Pods, the Pod cgroup’s cpu.shares is computed from sum(container CPU requests) + overhead.cpu, so the Pod gets a proportionally larger slice of contended CPU to cover the runtime’s baseline draw.

ResourceQuota accounting also includes spec.overhead, so a namespace’s requests.memory / limits.cpu quotas reflect the true consumption of sandboxed Pods rather than under-charging tenants who use expensive runtimes.

You can confirm the injected value:

kubectl get pod kata-pod -o jsonpath='{.spec.overhead}'
# map[cpu:250m memory:120Mi]

Configuration / API Surface

The overhead is declared on the RuntimeClass and consumed implicitly by the Pod:

apiVersion: node.k8s.io/v1          # RuntimeClass is in the node.k8s.io group
kind: RuntimeClass
metadata:
  name: kata-fc                     # the name Pods reference via spec.runtimeClassName
handler: kata-fc                    # the CRI handler the node's runtime config maps to
overhead:                           # OPTIONAL — omit for zero-overhead runtimes (runc)
  podFixed:                         # a fixed (not per-container) resource map
    memory: "120Mi"                 # memory the VM/sandbox costs regardless of workload
    cpu: "250m"                     # CPU the VMM + guest kernel + agent cost
scheduling:                         # OPTIONAL — restricts Pods to nodes that have this runtime
  nodeSelector:
    katacontainers.io/kata-runtime: "true"
---
apiVersion: v1
kind: Pod
metadata:
  name: kata-pod
spec:
  runtimeClassName: kata-fc         # selecting this triggers the overhead injection
  # NOTE: spec.overhead is intentionally NOT set here — the admission
  # controller fills it. Setting it manually makes the API server reject the Pod.
  containers:
    - name: app
      image: nginx
      resources:
        requests:                   # the scheduler reserves (these + overhead)
          cpu: "500m"
          memory: "256Mi"
        limits:                     # the kubelet sizes the cgroup at (these + overhead)
          cpu: "1"
          memory: "512Mi"

Line-by-line. overhead.podFixed is a flat resource map — note “podFixed” emphasizes the cost is per-Pod and constant, not scaled by container count or workload size; this is the modeling simplification Pod Overhead deliberately accepts (a Kata VM’s cost is dominated by the fixed guest-kernel footprint, so a constant is a defensible approximation). The scheduling.nodeSelector is a separate RuntimeClass feature — it pins the Pod to nodes where the runtime is actually installed; it is unrelated to overhead but commonly appears alongside it. In the Pod, the absence of spec.overhead is the point: it is derived state. The effective scheduler reservation for kata-pod is cpu: 750m, memory: 376Mi, and the kubelet’s Pod cgroup limit is cpu: 1250m, memory: 632Mi.

Failure Modes

Manually setting spec.overhead. The Pod is rejected at admission with an error about the field being immutable / set-by-system. Users who copy a kubectl get pod -o yaml dump and re-apply it hit this. Strip spec.overhead (and spec.nodeName, status, etc.) before re-applying.

RuntimeClass changed after Pods exist. spec.overhead is snapshotted at admission. Editing the RuntimeClass’s overhead value does not retroactively update running Pods — only Pods admitted afterward see the new number. A fleet can therefore carry a mix of old and new overhead values; the scheduler and kubelet each use whatever was stamped on the individual Pod.

Overhead set too low. If the declared overhead under-estimates the real runtime cost, the scheduler over-packs the node; the node’s actual memory use exceeds what the scheduler believes, MemoryPressure is reached, and the kubelet starts node-pressure Pod Eviction. Symptom: nodes evicting Pods despite the scheduler showing free capacity. Diagnostic: compare kube_pod_overhead_memory_bytes (from kube-state-metrics) against measured VMM RSS.

Overhead set too high. The opposite — the scheduler reserves more than the runtime actually costs, nodes are under-packed, and cluster cost rises. Less dangerous than under-estimating but real money at scale.

RuntimeClass admission controller disabled. If a cluster operator disabled the RuntimeClass admission plugin, spec.overhead is never injected even though the RuntimeClass declares it — the scheduler silently under-counts. Verify the plugin is in the API server’s --enable-admission-plugins set.

Alternatives and When to Choose Them

Padding container requests by hand. Before Pod Overhead, operators inflated every container’s memory request by the VM’s cost. This works but is fragile: it must be re-applied on every workload, it pollutes the application’s own resource numbers, and it double-counts when a Pod has many containers (you only want the VM cost charged once). Pod Overhead is strictly better for sandboxed runtimes.
A dedicated node pool for sandboxed workloads. Put all Kata Pods on their own nodes and size those nodes with explicit headroom. This sidesteps the accounting problem but loses bin-packing density and is operationally heavier. Reasonable when sandboxed workloads are a small, isolated slice.
Ignoring it (runc). For the default runc runtime the per-Pod overhead is genuinely negligible — the Pause Container is a few hundred KiB — so leaving overhead unset on the RuntimeClass (or having no RuntimeClass at all) is correct. Pod Overhead is a feature you reach for only when running a heavyweight isolation runtime.

Production Notes

Pod Overhead is effectively a Kata Containers / gVisor (runsc) / Firecracker-microVM feature. Clusters running only runc or crun never set it. Confluent, Alibaba, and other multi-tenant SaaS operators who run untrusted tenant code in Kata cite Pod Overhead as a prerequisite for honest bin-packing of sandboxed Pods.
The correct overhead number is measured, not guessed: boot the runtime with a trivial Pod, measure the VMM process RSS and the guest’s baseline, and set podFixed slightly above the observed steady-state. The Kata project publishes representative numbers per VMM (QEMU is heavier; Firecracker / Cloud Hypervisor are lighter — hence handler names like kata-fc).
Monitoring: kube-state-metrics exposes kube_pod_overhead_cpu_cores and kube_pod_overhead_memory_bytes. Summed per node, these tell you how much of a node’s capacity is being consumed by runtime overhead versus application workload — a useful density and cost signal.
Because overhead is snapshotted at admission, a cluster-wide overhead correction requires a rolling restart of the affected workloads to re-admit their Pods. Plan RuntimeClass overhead edits like any other rollout.

vault

Explorer

Pod Overhead

Pod Overhead

Mental Model

Mechanical Walk-through

Configuration / API Surface

Failure Modes

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents