vault

ResourceQuota

10 min read

ResourceQuota

A ResourceQuota is a namespaced Kubernetes object (apiVersion: v1, kind: ResourceQuota) that places aggregate caps on what a single Namespace is allowed to consume — total CPU/memory requested, total persistent storage claimed, total number of Pods or Services or ConfigMaps that may exist. It is the primary tool for keeping multiple teams sharing one cluster from starving each other (kubernetes.io — Resource Quotas). Crucially, a ResourceQuota is not just a passive ceiling: it is enforced synchronously by the ResourceQuota admission controller at object-create time, and — for compute quotas — its mere presence changes the cluster’s behaviour, forcing every Pod in the namespace to declare resource requests and limits or be rejected outright. This makes ResourceQuota the load-bearing half of a two-object pair: it sets the caps, and LimitRange supplies the defaults that keep those caps from being painful.

Mental Model

flowchart TB
    USER[kubectl apply -f pod.yaml] --> API[kube-apiserver]
    API --> AUTHN[authn + authz]
    AUTHN --> MUT["Mutating admission<br/>(incl. LimitRange:<br/>inject default requests/limits)"]
    MUT --> RQ["ResourceQuota<br/>admission controller"]
    RQ -->|"sum(existing usage) + new<br/>≤ quota.hard ?"| DECISION{within<br/>quota?}
    DECISION -->|yes| STORE["persist to etcd<br/>+ atomically bump<br/>quota.status.used"]
    DECISION -->|"no — or Pod has<br/>no requests/limits<br/>under a compute quota"| REJECT["403 Forbidden<br/>'exceeded quota' /<br/>'must specify limits.cpu'"]
    STORE --> RECONCILE["quota controller<br/>(in kube-controller-manager)<br/>periodically recomputes<br/>status.used from reality"]

What this diagram shows. A ResourceQuota is enforced at two moments. The fast path is the admission controller: when any object that consumes quota is created, the controller reads the namespace’s current status.used, adds the new object’s contribution, and rejects with HTTP 403 if the sum would exceed spec.hard. This admission check is atomic with the write to etcd — there is no race window where two Pods both squeak under the same remaining headroom. The slow path is the quota controller in kube-controller-manager, which periodically rescans the namespace and recomputes status.used from the actual set of live objects, correcting any drift (e.g., if a Pod was force-deleted out from under the accounting). The insight to extract: a ResourceQuota is itself an object with spec and status, reconciled like everything else in Kubernetes — spec.hard is the desired cap, status.used is the observed consumption, and status.hard mirrors spec.hard once the controller has seen it.

Mechanical Walk-through

Three families of quota

A ResourceQuota’s spec.hard map mixes three categories of key, each behaving slightly differently.

1. Compute resource quota. Caps the sum of container resource requests and limits across all non-terminal Pods in the namespace. The keys are requests.cpu, requests.memory, limits.cpu, limits.memory (with cpu and memory accepted as aliases for requests.cpu/requests.memory), plus hugepages-<size>. Extended and device resources — GPUs, FPGAs — are quota-able only on the request side, prefixed requests.: requests.nvidia.com/gpu. Extended resources cannot be overcommitted, so request and limit are always equal and there is no limits.nvidia.com/gpu key.

2. Storage resource quota. Caps persistent storage. requests.storage bounds the sum of all PersistentVolumeClaim capacity requests; persistentvolumeclaims caps the count of PVCs. Both can be scoped to a single StorageClass with the prefix form <storage-class>.storageclass.storage.k8s.io/requests.storage and <storage-class>.storageclass.storage.k8s.io/persistentvolumeclaims — this is how an operator says “this namespace may claim 500 Gi of cheap HDD but only 50 Gi of premium SSD.” Local ephemeral storage is quota-able via requests.ephemeral-storage and limits.ephemeral-storage.

3. Object count quota. Caps the number of objects of a given kind. A handful of kinds have bare-name keys for historical reasons (pods, services, configmaps, secrets, persistentvolumeclaims, replicationcontrollers, resourcequotas, services.loadbalancers, services.nodeports). Everything else uses the generic syntax count/<resource>.<group>: count/deployments.apps, count/cronjobs.batch, count/jobs.batch, even count/<custom-resource>.<crd-group> for CRDs. Object-count quota is the defence against a runaway controller flooding etcd with objects.

The compute-quota behaviour change

The single most surprising fact about ResourceQuota: once a quota constrains cpu or memory (request or limit), every Pod created in that namespace must declare the corresponding request/limit, or admission rejects it. The reason is accounting integrity — the quota controller cannot decide whether a Pod fits if the Pod does not say how much it wants. A Pod that omits resources.requests.cpu in a namespace with a requests.cpu quota fails with Error from server (Forbidden): error when creating "pod.yaml": pods "x" is forbidden: failed quota: ...: must specify requests.cpu.

This turns a quota into a de-facto policy mandate. The fix is not to make every manifest author hand-write requests and limits — it is to install a LimitRange in the same namespace with default and defaultRequest set. The LimitRange admission controller runs in the mutating phase, before the ResourceQuota controller, and injects the missing values; by the time the quota controller sees the Pod, it has the numbers it needs. ResourceQuota + LimitRange are designed as a pair; deploying the first without the second is a known operational footgun.

Scopes and scopeSelector

A plain ResourceQuota counts every matching object in the namespace. Scopes narrow that to a subset, so an operator can write different caps for different classes of workload. The classic scopes:

  • Terminating / NotTerminating — splits Pods by whether they set spec.activeDeadlineSeconds (a self-imposed deadline). Lets an operator give batch/Job Pods a separate, larger compute budget from long-running services.
  • BestEffort / NotBestEffort — splits Pods by QoS class. BestEffort Pods (no requests, no limits) can be counted under a quota that only caps their count — you cannot meaningfully cap their CPU because they declared none.
  • PriorityClass — restricts the quota to Pods at one or more named priority classes. This is how a cluster reserves a slice of capacity for high-priority workloads: a quota scoped to PriorityClass in (high) caps how much the namespace may consume at that priority.
  • CrossNamespacePodAffinity — counts only Pods that use cross-namespace pod (anti-)affinity terms; lets an admin forbid or bound this expensive scheduling feature.

The legacy scopes: field takes a list of scope names ANDed together. The newer scopeSelector: field takes matchExpressions with operators (In, NotIn, Exists, DoesNotExist) and is required for PriorityClass-valued scopes. A namespace typically holds several ResourceQuota objects with disjoint scopes rather than one global quota.

Configuration / API Surface

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-compute            # multiple quotas per namespace is normal
  namespace: team-alpha
spec:
  hard:
    # ---- compute ----
    requests.cpu:    "20"             # sum of all Pods' CPU requests ≤ 20 cores
    requests.memory: 40Gi             # sum of all Pods' memory requests ≤ 40 Gi
    limits.cpu:      "40"             # sum of all Pods' CPU limits   ≤ 40 cores
    limits.memory:   80Gi             # sum of all Pods' memory limits ≤ 80 Gi
    requests.nvidia.com/gpu: "8"      # extended resource — request side only
    # ---- storage ----
    requests.storage:           500Gi # total PVC capacity requested
    persistentvolumeclaims:     "20"  # number of PVCs
    fast-ssd.storageclass.storage.k8s.io/requests.storage: 50Gi   # per-StorageClass cap
    # ---- object counts ----
    pods:                       "100"
    services:                   "10"
    services.loadbalancers:     "2"   # cloud LB objects cost real money
    count/deployments.apps:     "30"
    count/cronjobs.batch:       "10"
  scopeSelector:                      # this quota applies ONLY to non-BestEffort Pods
    matchExpressions:
    - scopeName: NotBestEffort
      operator: Exists
status:
  hard:                               # mirrors spec.hard once controller has observed it
    requests.cpu: "20"
  used:                               # live consumption — recomputed by the quota controller
    requests.cpu: "13500m"
    pods: "47"

Inspect with kubectl describe resourcequota -n team-alpha — the human-readable output prints a Used / Hard column per resource, which is the fastest way to answer “why was my Pod rejected.” kubectl get resourcequota -n team-alpha -o yaml shows the raw status.used.

Failure Modes

must specify requests.cpu on every Pod. A compute quota exists but no LimitRange supplies defaults, so every Pod authored without explicit requests/limits is rejected at admission. Symptom: Deployments stuck with 0/N replicas and ReplicaSet events showing FailedCreate ... failed quota. Fix: add a LimitRange with defaultRequest and default to the namespace.

exceeded quota rejections under churn. A rolling update temporarily needs maxSurge extra Pods, but the namespace’s pods count or requests.cpu quota has no headroom for the surge. The old ReplicaSet still holds quota while the new one tries to scale up — deadlock. Symptom: rollout stuck, events show exceeded quota: pods. Fix: size quotas with rollout headroom, or set maxSurge: 0 on tight-quota namespaces (trading availability for fit).

status.used drift. A Pod is force-deleted (--grace-period=0 --force) while its node is unreachable; the admission-time decrement may not fire correctly. The quota controller’s periodic full recompute eventually corrects status.used, but for a window the namespace appears fuller (or emptier) than reality. Diagnostic: compare kubectl describe resourcequota against kubectl get pods -o ... summed by hand.

Quota on a kind with no namespace meaning. Attempting count/nodes or count/persistentvolumes (cluster-scoped kinds) in a namespaced ResourceQuota silently does nothing — quota only counts namespaced objects. Mistaking this for enforcement is a real-world misconfiguration.

Scope omission. An operator wants to cap only batch Pods but writes a plain (unscoped) quota; long-running services then also consume the batch budget. Fix: scope with Terminating/NotTerminating or PriorityClass.

Alternatives and When to Choose Them

  • LimitRange — not an alternative; the mandatory companion. ResourceQuota caps the namespace total; LimitRange bounds and defaults the individual Pod/Container. Use both.
  • Pod Priority and Preemption — a complement: quota bounds how much a namespace may consume, priority decides who wins when nodes are full. A PriorityClass-scoped quota links the two.
  • Hierarchical Namespace Controller (HNC) / accounting CRDs — for nested team structures where one parent quota must subsume children. Stock ResourceQuota is flat per-namespace; HNC propagates quotas down a namespace tree.
  • Admission policy engines (OPA Gatekeeper, Kyverno) — for qualitative governance ResourceQuota cannot express (“every Pod must set a team label”, “no :latest image tags”). Quota is purely quantitative; policy engines handle the rest.
  • Cluster-per-tenant — when even quota-bounded namespaces share too much (control plane, kernel), the answer is a separate cluster. See Kubernetes Multi-tenancy Models.

Production Notes

  • Always pair with a LimitRange. Every production write-up of multi-tenant clusters (and the Kubernetes docs themselves) makes the same point: a compute ResourceQuota without a defaulting LimitRange turns into a wall of must specify requests rejections that manifest authors experience as a broken cluster.
  • Quota the loadbalancer count. services.loadbalancers maps to real cloud spend — each LoadBalancer Service provisions a billed cloud load balancer. Capping it at 1–2 per namespace is a standard cost-control move.
  • Object-count quota protects etcd. A buggy controller in a tenant namespace can create ConfigMaps or Events without bound; count/configmaps and friends are the etcd-stability backstop, not just a tidiness measure.
  • Size for rollout surge. The most common quota complaint in practice is rollouts wedging because pods or requests.cpu left no room for maxSurge. Add ~25% headroom over steady-state, or coordinate maxSurge/maxUnavailable with the quota.
  • Quota changes are not retroactive. Lowering spec.hard below current status.used does not evict anything — existing objects keep running; only new creations are blocked until usage falls back under the cap.

See Also

Graph View

Backlinks