vault

QoS Classes

10 min read

QoS Classes

A QoS (Quality of Service) class is a label Kubernetes derives — you never set it — from a Pod’s requests and limits, and it answers one question: when a node runs out of resources, in what order are Pods sacrificed? There are exactly three classes — Guaranteed, Burstable, BestEffort — and a Pod’s class is computed at creation, stamped onto status.qosClass, and (with the narrow exception of in-place resize) immutable thereafter. QoS does not affect where a Pod is scheduled and does not affect preemption order — it governs node-pressure eviction order (which Pod the kubelet evicts when the node is starved) and the oom_score_adj value the kubelet hands the kernel (which process the OOM killer picks first). The canonical advice “set requests == limits for critical workloads” is, mechanically, advice to put those workloads in the Guaranteed class so they are evicted last and OOM-killed last (Pod Quality of Service Classes).

Mental Model

QoS is a consequence, not a control. You write requests and limits; Kubernetes reads them and sorts your Pod into one of three buckets that decide its survival priority under pressure.

flowchart TB
    SPEC["Every container's requests & limits"] --> Q{"Classify"}
    Q -->|"EVERY container:<br/>request == limit for<br/>BOTH cpu AND memory"| G["Guaranteed<br/>oom_score_adj = -997<br/>evicted LAST"]
    Q -->|"NO container has<br/>ANY request or limit"| B["BestEffort<br/>oom_score_adj = 1000<br/>evicted FIRST"]
    Q -->|"anything else<br/>(at least one request/limit,<br/>but not Guaranteed)"| U["Burstable<br/>oom_score_adj 2..999<br/>evicted in the MIDDLE"]
    G --> EVICT["Node-pressure eviction order:<br/>BestEffort → Burstable → Guaranteed"]
    B --> EVICT
    U --> EVICT

What this diagram shows. The classifier is a simple decision tree over all the Pod’s containers. Guaranteed requires the strictest condition — every container pins request == limit for both CPU and memory. BestEffort is the absence of any resource declaration anywhere. Burstable is the residual “everything else.” The class then maps to two enforcement consequences: the kubelet’s node-pressure eviction order (BestEffort first, Guaranteed last) and the kernel’s oom_score_adj bias. The insight to extract: QoS is entirely a function of requests and limits — there is no qosClass field to set — so the only way to make a Pod Guaranteed is to write request == limit for both resources on every container.

Mechanical Walk-through

Guaranteed — the protected class

A Pod is Guaranteed only if every container (including init and sidecar containers) satisfies all of:

  • a CPU request and a CPU limit, and they are equal;
  • a memory request and a memory limit, and they are equal.

If even one container misses one of these conditions, the Pod is not Guaranteed. Guaranteed Pods are the last to be node-pressure-evicted and are eligible for exclusive CPU pinning under the kubelet’s static CPU manager policy. Their oom_score_adj is -997 — far below most processes, so the kernel’s OOM killer avoids them.

Burstable — the common middle

A Pod is Burstable if it is not Guaranteed but at least one container has some CPU or memory request or limit. This is where most real-world Pods land: a typical service sets requests.cpu: 250m, limits.cpu: 1, requests.memory: 256Mi, limits.memory: 512Mi — requests present, but limit ≠ request, so it is Burstable. Burstable Pods may burst above their requests up to their limits when the node has spare capacity, and are evicted after all BestEffort Pods but before any Guaranteed Pod. Their oom_score_adj is computed per Pod, scaled by how large the memory request is relative to node capacity — a Burstable Pod requesting more memory gets a lower (more protective) score. The range is roughly 2 to 999.

BestEffort — the first to die

A Pod is BestEffort only if no container declares any CPU or memory request or limit. It is allowed to consume whatever node resources are not reserved by other classes — and is the first Pod evicted under node pressure and the first OOM-killed, with oom_score_adj fixed at 1000 (the most kill-prone value). A Pod that drifts into BestEffort by accident — a missing resources block — is a latent production hazard; the Unbounded Resource Requests Anti-Pattern note covers the failure.

How QoS drives node-pressure eviction

When the kubelet detects a node-pressure condition — MemoryPressure, DiskPressure — it must reclaim resources by evicting Pods (see Pod Eviction, Node Pressure Conditions). The selection algorithm sorts eviction candidates first by whether the Pod exceeds its requests (Pods over their request are evicted before Pods under it), then by Pod priority (Pod Priority and Preemption), and then by how far over its memory request the Pod is using. QoS class shapes this ordering: in practice the kubelet evicts BestEffort Pods first, then Burstable Pods that exceed their requests, and Guaranteed Pods only as a last resort (a Guaranteed Pod by definition cannot exceed its requests for the resource under pressure). QoS is thus the coarse eviction tier; priority and over-request usage are the tiebreakers within a tier.

How QoS drives oom_score_adj

Node-pressure eviction is the kubelet’s proactive defense; the kernel OOM killer is the reactive backstop when memory is exhausted faster than the kubelet can react. Every Linux process has an oom_score; the kernel kills the process with the highest score. The kubelet biases this by writing each container’s /proc/<pid>/oom_score_adj:

QoS classoom_score_adjEffect
Guaranteed-997Strongly protected — kernel avoids killing it
Burstable2 – 999 (computed; larger memory request ⇒ lower score)Intermediate
BestEffort1000Maximally kill-prone — killed first

So even within a single OOM event on a node, a BestEffort Pod’s process is killed before a Burstable one’s, and a Guaranteed Pod’s process is killed only if nothing else can be.

Immutability

status.qosClass is set once at admission. The one nuance: in-place Pod resize (GA v1.35) lets you change requests/limits on a running Pod, but a resize that would change the QoS class is rejected — you cannot promote a Burstable Pod to Guaranteed in place.

Configuration / API Surface

The three classes side by side — note none of them sets qosClass; it is derived:

# ---- Guaranteed: request == limit for BOTH cpu and memory, EVERY container ----
apiVersion: v1
kind: Pod
metadata: { name: critical }
spec:
  containers:
  - name: app
    image: app:v1
    resources:
      requests: { cpu: "500m", memory: "512Mi" }
      limits:   { cpu: "500m", memory: "512Mi" }   # equal ⇒ Guaranteed
---
# ---- Burstable: requests present, limits differ (or some field missing) ----
apiVersion: v1
kind: Pod
metadata: { name: service }
spec:
  containers:
  - name: app
    image: app:v1
    resources:
      requests: { cpu: "250m", memory: "256Mi" }
      limits:   { cpu: "1",    memory: "512Mi" }    # limit ≠ request ⇒ Burstable
---
# ---- BestEffort: no resources block at all ----
apiVersion: v1
kind: Pod
metadata: { name: scratch }
spec:
  containers:
  - name: app
    image: app:v1
    # no `resources:` ⇒ BestEffort ⇒ evicted first, oom_score_adj 1000

Commentary. The critical Pod is Guaranteed because requests equals limits for both CPU and memory — if it had three containers, all three would need that property. The service Pod is Burstable: it has requests (so it is not BestEffort) but limits.cpu of 1 exceeds requests.cpu of 250m, so it fails the Guaranteed test — it can burst to a full core on an idle node but is evicted before critical. The scratch Pod omits resources entirely and is BestEffort — it runs only on slack capacity, is evicted first under pressure, and carries oom_score_adj 1000. Verify any Pod’s class with kubectl get pod <name> -o jsonpath='{.status.qosClass}'.

Failure Modes

Accidental BestEffort. A container ships without a resources block; the Pod is silently BestEffort and is the first thing evicted in the next node-pressure event — often the least expected Pod to die. Diagnostic: kubectl get pod -o jsonpath='{.status.qosClass}'. Fix: a LimitRange enforcing namespace defaults so no Pod can be accidentally BestEffort.

“Guaranteed” Pod that isn’t. An operator sets request == limit on the main container but forgets the sidecar (mesh proxy, log shipper) — one non-conforming container demotes the whole Pod to Burstable. Every container, init and sidecar included, must conform.

Burstable Pod evicted despite “having limits.” Having a limit does not protect a Burstable Pod that is using more than its request. Under memory pressure, a Burstable Pod consuming above requests.memory is an eviction candidate. Set requests to true steady-state usage, not a wishful low number.

OOMKill order surprises. Inside one node OOM event, a Burstable Pod survives while a “more important” BestEffort Pod is killed — because the kernel obeys oom_score_adj, not human intent. If a workload must survive, it cannot be BestEffort.

Expecting QoS to influence scheduling or preemption. It does neither. The scheduler uses requests and priority; QoS only governs eviction and OOM order. Confusing these is a common interview and postmortem error.

Alternatives and When to Choose Them

QoS is not chosen directly — you choose requests and limits and the class follows. The decision is therefore about which class to target:

  • Target Guaranteed for latency-critical, stateful, or revenue-path workloads — set request == limit for both resources. They survive node pressure longest. The cost: no bursting, so you must size for peak.
  • Target Burstable for the typical stateless service — requests at steady-state usage, limits higher to absorb spikes. The default and usually correct choice; accepts mid-tier eviction risk for better node utilization.
  • Target BestEffort only for genuinely sacrificial work — opportunistic batch, scratch jobs that can be killed and retried freely. Never for anything user-facing.
  • Pod Priority and Preemption is the complementary knob: priority is a tiebreaker within the eviction sort and governs preemption, a separate mechanism. A high-priority Burstable Pod is still evicted before a low-priority Guaranteed one for the resource under pressure — QoS is the coarse tier, priority refines within it.
  • Pod Disruption Budget governs voluntary disruption (drains); it is orthogonal to QoS-driven node-pressure eviction.

Production Notes

  • “requests == limits for critical workloads” is the single most repeated piece of Kubernetes resource advice — and QoS is why: it is the only way to reach the Guaranteed class and its protective oom_score_adj of -997 and last-to-evict status.
  • Sidecars break Guaranteed silently. Service-mesh proxies, log shippers, and secrets agents injected by mutating webhooks frequently lack request==limit settings, demoting otherwise-Guaranteed Pods. Audit injected containers, or configure the injector to set conforming resources.
  • k8s.af failure stories include outages where a node-pressure event evicted a critical-but-BestEffort Pod the team never realized had no resources block — the fix in every case was a LimitRange plus an admission policy forbidding BestEffort in production namespaces.
  • cpuManagerPolicy: static grants exclusive whole cores only to Guaranteed Pods with integer CPU requests — another concrete reward for the Guaranteed class, relevant to latency-sensitive and NUMA-sensitive workloads.
  • Monitor kube_pod_status_qos_class (from kube-state-metrics) to catch BestEffort drift across the fleet before the next node-pressure event finds it for you.

See Also

Graph View

Backlinks