vault

Unbounded Resource Requests Anti-Pattern

11 min read

Unbounded Resource Requests Anti-Pattern

The unbounded-resources anti-pattern is running Pods with no resources.requests/limits — or with wildly wrong ones. It comes in two opposite shapes and both are damaging. With no requests, a Pod is BestEffort QoS — the first class the kubelet evicts under node pressure, the first the kernel OOM-kills, and a Pod the scheduler cannot reason about at all (it has zero declared footprint, so the scheduler packs nodes blind and creates the contention it then has to evict its way out of). With no memory limit, one leaking Pod can consume a node’s entire memory and OOM unrelated neighbors. And with requests set far above real usage, the cluster reports itself “full” while nodes sit at 20% actual utilization — pure cost waste. The anti-pattern is not setting requests/limits thoughtfully; the fix is the discipline (and the namespace-level enforcement) to always set them from observed reality.

Mental Model

flowchart TB
    subgraph NONE["No requests/limits"]
        BE["BestEffort QoS"]
        BE --> E1["evicted FIRST under node pressure"]
        BE --> E2["oom_score_adj 1000 → killed FIRST"]
        BE --> E3["scheduler sees 0 footprint → packs blind"]
    end
    subgraph NOLIM["No memory limit"]
        NL["leaking Pod grows unbounded"] --> NODEOOM["node OOM → kills neighbors too"]
    end
    subgraph TOOBIG["Requests ≫ real usage"]
        TB["scheduler reserves the request"] --> WASTE["cluster 'full' at 20% real CPU<br/>→ paying for idle nodes"]
    end
    NONE -.-> CHAOS["eviction storms · noisy neighbors"]
    NOLIM -.-> CHAOS
    TOOBIG -.-> COST["cost blowout"]

What this diagram shows. Three failure shapes from one root cause — not declaring resources accurately. No requests → BestEffort QoS, the bottom of every eviction and OOM ordering, plus a scheduler that cannot bin-pack because it sees a zero-sized Pod. No memory limit → a leak grows until the node runs out of memory and the kernel OOM-kills whatever it picks, including healthy neighbors. Requests far above real usage → the scheduler reserves the inflated request, so the cluster fills up (reports unschedulable) long before the hardware is full — you pay for nodes running at 20%. The insight: requests and limits are the only signals the scheduler and the kernel have about a workload — omit them and both subsystems make bad decisions; inflate them and you pay for capacity nobody uses.

Mechanical Walk-through — Anatomy of the Anti-Pattern

Why it’s tempting. Setting requests and limits correctly requires knowing your workload’s real CPU and memory profile — and on day one you do not. Measuring it takes a load test or a week of production metrics. The manifest works without them — a Pod with no resources block schedules and runs perfectly well on an empty cluster. So the path of least resistance is to ship without resources and “tune later” — and “later” never arrives until an incident forces it. The anti-pattern is not malice or ignorance; it is the gap between “works” and “works under contention,” and contention is exactly what you do not see in dev.

What BestEffort actually costs you. Kubernetes derives a Pod’s QoS class purely from its requests/limits (Pod QoS docs):

  • Guaranteed — every container has requests == limits for both CPU and memory.
  • Burstable — at least one container has a request or limit set, but not the Guaranteed shape.
  • BestEffort — no container has any request or limit. This is what “no resources block” gets you.

BestEffort is the bottom of two orderings. Eviction order: when a node hits MemoryPressure, the kubelet’s node-pressure eviction evicts BestEffort Pods first, then Burstable Pods over their requests, and protects Guaranteed Pods last. OOM order: the kubelet sets each container’s oom_score_adj from its QoS — BestEffort gets the maximum (1000), so when the kernel’s OOM killer fires, a BestEffort container is the kernel’s first pick. A BestEffort workload is, by design, the cluster’s sacrificial victim.

What “no requests” does to the scheduler. The kube-scheduler’s NodeResourcesFit filter sums every Pod’s requests to decide what fits on a node. A Pod with no requests contributes zero to that sum — so the scheduler believes it is free, and will pack it onto a node alongside everything else. It then actually uses CPU and memory the scheduler never accounted for. The scheduler cannot bin-pack honestly when some Pods lie about being weightless: it creates real node contention precisely because it could not see the load. The eviction storm that follows is the scheduler cleaning up a mess its own blindness created.

What “no memory limit” does. Memory is incompressible — the kernel cannot reclaim it without killing a process (see Resource Requests and Limits). A container with no limits.memory has no cgroup memory.max cap; a memory leak grows until the node is out of memory. The kernel’s OOM killer then fires at the node level and kills some process — not necessarily the leaker, possibly a healthy neighbor with the unlucky oom_score. One un-capped leaking Pod can take down unrelated workloads sharing its node.

What over-sized requests do. The opposite mistake. An engineer, burned by an OOMKill once, sets requests.memory: 4Gi on a service that uses 600 MiB “to be safe.” The scheduler reserves 4 GiB per replica. With 20 replicas the cluster has reserved 80 GiB it will never use — nodes report full, the Cluster Autoscaler adds more nodes to fit “unschedulable” Pods, and you pay for hardware running at 15–20% real utilization. Over-provisioning is a cost failure as real as the stability failures above.

Worked failure scenario

A team ships a dozen services with no resources blocks — all BestEffort. The cluster runs fine for months on lightly-loaded nodes. Then a traffic spike pushes one node’s memory toward the limit. The kubelet hits MemoryPressure and evicts BestEffort Pods first — which is all twelve services. The evicted Pods reschedule onto other nodes (the scheduler, still seeing them as zero-footprint, packs them onto already-loaded nodes), pushing those into MemoryPressure, which evicts their BestEffort Pods. The eviction wave cascades across the cluster. Because nothing had requests, nothing was protected and the scheduler had no information to place the churn sensibly. A localized spike became a cluster-wide eviction storm — the exact category-3 failure in the Common Kubernetes Failures Catalog.

Configuration / API Surface — The Wrong Way and the Right Way

The wrong way — no resources block at all:

# WRONG — BestEffort QoS. First evicted, first OOM-killed, invisible to the scheduler.
apiVersion: v1
kind: Pod
metadata: { name: api }
spec:
  containers:
    - name: api
      image: registry/api:v4
      # no resources: → BestEffort → the cluster's sacrificial victim

The right way — requests sized from observed usage; memory request == limit for a critical workload to reach Guaranteed:

# BETTER — requests from real metrics; Guaranteed QoS for a critical service.
apiVersion: v1
kind: Pod
metadata: { name: api }
spec:
  containers:
    - name: api
      image: registry/api:v4
      resources:
        requests:
          cpu: "500m"            # observed steady-state, e.g. from VPA recommender
          memory: "512Mi"        # observed steady-state with headroom
        limits:
          cpu: "500m"            # == request  ─┐ both CPU and memory request==limit
          memory: "512Mi"        # == request  ─┘ ⇒ Guaranteed QoS (last evicted)

Enforce it at the namespace level so no Pod can omit resources in the first place:

# LimitRange — supplies DEFAULTS so a Pod with no resources still gets sane values.
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: team-a }
spec:
  limits:
    - type: Container
      default:        { cpu: "500m", memory: "512Mi" }   # applied as 'limits' if omitted
      defaultRequest: { cpu: "200m", memory: "256Mi" }   # applied as 'requests' if omitted
---
# ResourceQuota — forces the discipline: a namespace with a quota REJECTS any Pod
# that has no requests/limits, because the quota cannot account for an un-sized Pod.
apiVersion: v1
kind: ResourceQuota
metadata: { name: budget, namespace: team-a }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi

Line-by-line. The LimitRange gives every container in the namespace a default request and limit when the manifest omits them — so “forgot to set resources” produces a Burstable Pod with sane numbers rather than a BestEffort one. The ResourceQuota is the harder forcing function: once a namespace has a CPU/memory quota, the API server rejects any Pod that does not declare requests and limits, because it cannot debit an un-sized Pod against the budget. Quota turns the discipline from optional to mandatory.

Failure Modes

  • Eviction storms. BestEffort Pods are evicted first under node pressure; the rescheduling churn cascades — a localized spike becomes a cluster-wide storm.
  • Noisy neighbors. With no limits, a Pod can consume an arbitrary share of node CPU and memory, starving co-located workloads.
  • Node OOM kills neighbors. No memory limit → a leak grows until the node OOMs → the kernel kills some process, possibly a healthy unrelated Pod.
  • Scheduler blindness. No requests → the scheduler bin-packs as if the Pod were weightless → real contention it then evicts its way out of.
  • The QoS cascade. BestEffort is the bottom of both the eviction ordering and the oom_score_adj ordering — see QoS Classes. Omitting resources opts the workload into being sacrificed first, every time.
  • Cost blowout. Requests set far above real usage reserve capacity nobody uses; the cluster reports “full” at 20% real utilization and the autoscaler buys nodes to host the phantom load.
  • Diagnostic. kubectl get pod -o jsonpath='{.status.qosClass}' → BestEffort is the red flag. kubectl describe node showing high requested but low actual utilization is the over-provisioning red flag.

The Correct Alternative

  1. Always set requests; size them from observed usage. Never ship a workload BestEffort. Derive requests from real metrics — a week of production data or the Vertical Pod Autoscaler recommender, which watches actual consumption and recommends right-sized values. Guess only as a temporary placeholder, then correct from data.
  2. Always set memory limits; consider request == limit for critical workloads. A memory limit caps a leak’s blast radius at one Pod instead of one node. Setting memory request == limit (and CPU likewise) puts the Pod in Guaranteed QoS — last to be evicted, most-protected oom_score. Do this for anything whose death is an incident. See QoS Classes.
  3. CPU limits are optional and workload-dependent. Unlike memory, omitting a CPU limit is a legitimate, often-recommended choice for latency-sensitive services (it avoids CFS-throttling p99 spikes — see Resource Requests and Limits). Keep the CPU request always (for fair-share scheduling); treat the CPU limit as a deliberate decision.
  4. Enforce defaults with LimitRange. A namespace LimitRange ensures a Pod that omits resources still gets sane defaults — it eliminates accidental BestEffort.
  5. Force the discipline with ResourceQuota and admission policy. A namespace quota rejects un-sized Pods outright. A Kyverno / OPA Gatekeeper policy can require requests and limits at admission. Make “no resources block” un-shippable.
  6. Right-size continuously. Workloads change; revisit requests with VPA recommendations and cost tooling (Kubernetes Cost Management, Kubecost/OpenCost) so you neither under-provision into instability nor over-provision into waste.

Production Notes

  • The k8s.af catalog repeatedly features both failure poles: OOMKills from limits set by a single load test rather than steady-state behavior, and eviction storms from BestEffort workloads with no requests at all. The Common Kubernetes Failures Catalog records “no resource requests → BestEffort eviction storms” as a standing category-3 failure.
  • The widely-shared production lesson: the first thing a platform team does when they inherit an unstable cluster is audit for BestEffort Pods and missing memory limits — it is consistently among the highest-leverage, lowest-effort stability fixes.
  • The cost angle is equally well-documented: cost-optimization write-ups (Kubecost, cloud FinOps teams) find that over-set requests are the single largest source of Kubernetes waste — clusters routinely run at 20–35% real utilization because requests were padded “to be safe.” Right-sizing requests downward (with VPA) typically recovers a large fraction of the bill.
  • The honest framing: there is no universally correct number — requests and limits are workload-specific and must come from measurement. The anti-pattern is not “wrong numbers”; it is no numbers and un-measured numbers. The discipline is: measure, set, enforce at the namespace level, and revisit.

See Also

Graph View

Backlinks