vault

CPU Bandwidth Control cpu.max and Throttling

15 min read

CPU Bandwidth Control cpu.max and Throttling

CFS bandwidth control is the mechanism by which the cgroup cpu controller imposes a hard, absolute cap on a task group’s CPU consumption — independent of how much CPU is idle. It is configured through the cgroup v2 file cpu.max = "$quota $period" (default max 100000, i.e. unlimited over a 100 ms period): within each period of wall-clock time the group may run for at most quota microseconds of CPU time, summed across all CPUs; once the group’s quota for the period is exhausted, every run queue belonging to the group is throttled — its tasks are dequeued and cannot run again until the next period replenishes the quota (sched-bwc.rst, v6.12). The implementation centers on a per-group struct cfs_bandwidth holding a global runtime pool refilled every period, from which per-CPU run queues draw fixed slices; the function throttle_cfs_rq() parks an exhausted group and do_sched_cfs_period_timer() refills and unthrottles. This note covers that machinery and the notorious container throttling-latency problem it causes. It is the bandwidth/cpu.max counterpart to The cgroup v2 CPU Controller, which covers the proportional cpu.weight side.

This note is pinned to Linux 6.12 LTS (released 2024-11-17, kernel.org/releases) and holds equally on 6.18 LTS (2025-11-30). Despite the name “CFS bandwidth control,” the mechanism is unchanged under EEVDF (Earliest Eligible Virtual Deadline First, the default fair scheduler since 6.6, which replaced the Completely Fair Scheduler / CFS): bandwidth throttling operates on the group run queue (cfs_rq) abstraction, which EEVDF inherited from CFS, so the code, file names, and CONFIG_CFS_BANDWIDTH config option all retain the “CFS” lineage.

Mental Model

Picture each capped cgroup as owning a central bucket of CPU-time tokens (the cfs_bandwidth runtime pool) that is refilled to quota microseconds at the start of every period. The group runs on many CPUs at once, but you cannot hand every CPU direct access to the central bucket on every tick without crippling lock contention on large machines. So instead, each CPU’s group run queue draws a slice (default 5 ms) from the central bucket when it needs to run, holds that slice locally, and bills its own execution against it. When a CPU’s local slice runs dry it goes back for another. When the central bucket is empty, the next CPU that asks gets nothing — and every run queue in the group is throttled: dequeued from the scheduler, invisible, runnable-but-not-running, until the period timer fires and refills the bucket.

flowchart TB
  TIMER["period hrtimer<br/>fires every cpu.max period<br/>(default 100ms)"] -->|"__refill_cfs_bandwidth_runtime()"| POOL
  POOL[("cfs_b->runtime<br/>global token pool<br/>= quota + burst")]
  POOL -->|"slice = 5ms<br/>__assign_cfs_rq_runtime()"| RQ0["cfs_rq @ CPU0<br/>runtime_remaining"]
  POOL -->|"slice = 5ms"| RQ1["cfs_rq @ CPU1<br/>runtime_remaining"]
  POOL -->|"slice = 5ms"| RQN["cfs_rq @ CPUn<br/>runtime_remaining"]
  RQ0 -->|"runtime_remaining <= 0<br/>and pool empty"| THR["throttle_cfs_rq()<br/>dequeue group, park tasks"]
  RQ1 --> THR
  RQN --> THR
  TIMER -->|"do_sched_cfs_period_timer()<br/>distribute_cfs_runtime()"| UNTHR["unthrottle_cfs_rq()<br/>re-enqueue, tasks run again"]
  THR -.->|"throttled until next period"| UNTHR

The CFS bandwidth refill/draw/throttle loop. What it shows: a per-group hrtimer refills the central cfs_b->runtime pool to quota each period; per-CPU run queues draw 5 ms slices from it on demand; when both a run queue’s local slice and the global pool are exhausted, throttle_cfs_rq() removes the whole group from the scheduler; the next period timer refills the pool and distribute_cfs_runtime()/unthrottle_cfs_rq() brings the group back. The insight to take: throttling is binary and group-wide — a group either has tokens and runs, or has none and is fully parked until the period boundary; there is no graceful slowdown. This abruptness, combined with the per-CPU slice mechanic, is the root of the container latency problem.

Mechanical Walk-through

The cpu.max interface and its bounds

cpu.max is a read-write two-value file on every non-root cgroup, default max 100000 — quota max (unlimited) over a 100,000 µs (100 ms) period (cgroup-v2.rst, v6.12). Writing "$MAX $PERIOD" sets both; writing a single number updates only the quota. Parsing is done by cpu_period_quota_parse(), which multiplies both values by NSEC_PER_USEC (microseconds in, nanoseconds stored) and maps the literal "max" to the sentinel RUNTIME_INF (core.c, v6.12). The write then funnels into the central setter tg_set_cfs_bandwidth(), which enforces the legal envelope:

/* from tg_set_cfs_bandwidth(), kernel/sched/core.c, v6.12 */
if (quota < min_cfs_quota_period || period < min_cfs_quota_period)
        return -EINVAL;                 /* min 1ms for both */
if (period > max_cfs_quota_period)
        return -EINVAL;                 /* period <= 1s */
if (quota != RUNTIME_INF && quota > max_cfs_runtime)
        return -EINVAL;                 /* quota <= ~203 days */
if (quota != RUNTIME_INF && (burst > quota ||
                             burst + quota > max_cfs_runtime))
        return -EINVAL;                 /* burst <= quota */

with min_cfs_quota_period = 1 * NSEC_PER_MSEC (1 ms) and max_cfs_quota_period = 1 * NSEC_PER_SEC (1 s) (core.c, v6.12). So the minimum quota and minimum period are both 1 ms, and the maximum period is 1 s. After the checks pass, __cfs_schedulable() runs a feasibility check across the hierarchy (an individual child’s bandwidth must be attainable within its parent’s), then the function sets cfs_b->period, cfs_b->quota, cfs_b->burst, calls __refill_cfs_bandwidth_runtime(), (re)starts the period timer, and clears every per-CPU cfs_rq->runtime_remaining, unthrottling any currently-throttled run queue so the new limit takes effect immediately.

Refilling the pool each period

The pool is cfs_b->runtime, a signed token count in nanoseconds. Each period it is topped up:

void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b)
{
        s64 runtime;
        if (unlikely(cfs_b->quota == RUNTIME_INF))
                return;                                  /* unlimited: nothing to do */
        cfs_b->runtime += cfs_b->quota;                  /* add this period's quota */
        runtime = cfs_b->runtime_snap - cfs_b->runtime;
        if (runtime > 0) {                               /* track burst accounting */
                cfs_b->burst_time += runtime;
                cfs_b->nr_burst++;
        }
        cfs_b->runtime = min(cfs_b->runtime,
                             cfs_b->quota + cfs_b->burst); /* cap at quota + burst */
        cfs_b->runtime_snap = cfs_b->runtime;
}

(fair.c, v6.12). The crucial detail: it adds quota to whatever is left, then clamps to quota + burst. With the default burst = 0, the pool is effectively reset to quota each period — unused quota does not accumulate across periods. The burst feature (cpu.max.burst, default 0) raises that ceiling, letting a group bank up to burst microseconds of underused quota to spend in a later busier period — bounded over-subscription that “borrows time now against our future underrun, at the cost of increased interference against the other system users. All nicely bounded” (sched-bwc.rst, v6.12; the statistical-distribution argument traces to an Alibaba LKML thread).

Drawing slices to per-CPU run queues

A group’s tasks run on potentially every CPU. Rather than have each CPU consult the global pool on every accounting update, runtime is transferred in slices. __assign_cfs_rq_runtime() is called when a run queue’s local budget runs low:

static int __assign_cfs_rq_runtime(struct cfs_bandwidth *cfs_b,
                                   struct cfs_rq *cfs_rq, u64 target_runtime)
{
        u64 min_amount, amount = 0;
        min_amount = target_runtime - cfs_rq->runtime_remaining; /* how much to top up */
        if (cfs_b->quota == RUNTIME_INF)
                amount = min_amount;                  /* unlimited: always grant */
        else {
                start_cfs_bandwidth(cfs_b);
                if (cfs_b->runtime > 0) {
                        amount = min(cfs_b->runtime, min_amount);
                        cfs_b->runtime -= amount;     /* draw from pool */
                        cfs_b->idle = 0;
                }
        }
        cfs_rq->runtime_remaining += amount;
        return cfs_rq->runtime_remaining > 0;         /* 0 => pool empty => throttle */
}

(fair.c, v6.12). The target_runtime is normally one slice: sched_cfs_bandwidth_slice() returns sysctl_sched_cfs_bandwidth_slice * NSEC_PER_USEC, and sysctl_sched_cfs_bandwidth_slice defaults to 5000 µs = 5 ms (tunable via /proc/sys/kernel/sched_cfs_bandwidth_slice_us) (fair.c, v6.12). Larger slices reduce global-lock pressure; smaller slices allow finer-grained consumption and reduce the worst-case “stranded” quota on idle CPUs (sched-bwc.rst, v6.12). As tasks run, __account_cfs_rq_runtime() subtracts the elapsed delta_exec from cfs_rq->runtime_remaining; when it goes non-positive the run queue tries to grab another slice, and if assign_cfs_rq_runtime() returns 0 (pool empty) it calls resched_curr() so the throttle path runs at the next pick.

Throttling: parking the group

The throttle decision is made in check_cfs_rq_runtime() (called from put_prev_entity/pick), which calls throttle_cfs_rq() when a run queue is out of runtime and the pool cannot top it up even by 1 ns:

static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
{
        ...
        raw_spin_lock(&cfs_b->lock);
        if (__assign_cfs_rq_runtime(cfs_b, cfs_rq, 1)) {  /* race: bandwidth appeared */
                dequeue = 0;                              /* don't throttle after all */
        } else {
                list_add_tail_rcu(&cfs_rq->throttled_list,
                                  &cfs_b->throttled_cfs_rq); /* enqueue for unthrottle */
        }
        raw_spin_unlock(&cfs_b->lock);
        if (!dequeue)
                return false;
        ...
        /* walk up the hierarchy, dequeueing the group's sched_entity at each level */
        for_each_sched_entity(se) { ... dequeue_entity(qcfs_rq, se, flags); ... }
        ...
        cfs_rq->throttled = 1;
        cfs_rq->throttled_clock = rq_clock(rq);       /* start the throttled-time stopwatch */
        return true;
}

(fair.c, v6.12). The group’s sched_entity is removed from its parent run queue at every level of the hierarchy (so the group becomes invisible to pick_next_task), the throttled run queue is added to cfs_b->throttled_cfs_rq, cfs_rq->throttled is set, and throttled_clock records the start so throttled wall-time can be accounted into cpu.stat. From this instant the group’s tasks do not run at all until unthrottled — even if the host has many idle CPUs. That is the hard-cap semantics in action and the source of the latency problem below.

Unthrottling at the period boundary

do_sched_cfs_period_timer() runs when the period hrtimer fires. It refills the pool, bumps nr_periods, and if any run queue is on the throttled list, accounts nr_throttled and loops calling distribute_cfs_runtime(), which walks cfs_b->throttled_cfs_rq, hands each throttled run queue enough runtime to become positive (runtime = -cfs_rq->runtime_remaining + 1, bounded by the pool), and calls unthrottle_cfs_rq() (or the async variant unthrottle_cfs_rq_async() for run queues on other CPUs) to re-enqueue the group’s entities and let its tasks run again (fair.c, v6.12). If the pool is exhausted before all throttled run queues are served, the leftover ones stay throttled into the next period.

cpu.stat — measuring throttling

The five controller-enabled fields in cpu.stat are the primary diagnostic (cgroup-v2.rst, v6.12):

  • nr_periods — number of enforcement periods elapsed.
  • nr_throttled — number of periods in which the group was throttled.
  • throttled_usec — cumulative wall-time the group spent throttled (microseconds; the v1 analogue throttled_time was nanoseconds).
  • nr_bursts / burst_usec — periods in which burst was used and the cumulative above-quota time.

The single most useful derived signal is the throttle ratio nr_throttled / nr_periods: a healthy capped workload that genuinely needs its full quota will show a ratio near 1, but a latency-sensitive service showing a high ratio while usage_usec stays well below its quota is the classic symptom of pathological throttling.

The container throttling-latency problem (the Kubernetes “CFS throttling” pain)

This is the most consequential real-world behavior of cpu.max, and the reason experienced operators are wary of CPU limits. Two distinct effects compound:

1. The hard-cap latency tax. Because throttling is binary and lasts to the period boundary, a multi-threaded service that exhausts its quota early in a 100 ms period is frozen for the remainder — up to ~100 ms of added tail latency on an otherwise-idle machine. A request that should take 5 ms can stall for tens of milliseconds waiting for the next refill. The cap does its job (average CPU is bounded) but at the cost of bursty, period-quantized latency spikes that show up directly in p99/p999.

2. The historical slice-expiry bug (fixed in 5.4). Worse, for years the per-CPU slices expired at the period boundary even if unused. A highly-threaded, non-CPU-bound app on a many-core machine would scatter 5 ms slices across dozens of CPUs, use a little of each, and have the remainder evaporate — so it hit its quota’s throttle threshold while having used far less CPU than its quota allowed. Indeed Engineering’s “Unthrottled” write-up documented services being throttled at ~25% of their apparent limit (Indeed, 2019); Kubernetes issue #67577 tracked it upstream. The fix was kernel commit de53fd7aedb1 (“sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices”), merged in Linux 5.4 (2019), which made cpu-local slices not expire — the behavior the v6.12 min_cfs_rq_runtime caveat now documents: “Once a slice is assigned to a cpu it does not expire. However all but 1ms of the slice may be returned to the global pool if all threads on that cpu become unrunnable” (sched-bwc.rst, v6.12; LWN on the commit). The reported improvement was nearly 30x for an artificial 10ms/100ms-on-80-CPU testcase, while still enforcing the average cap (commit message). The fix later needed a follow-up correction for an over-accounting regression it introduced (Indeed regression write-up, 2019).

On any current LTS (6.12 / 6.18) the expiry bug is long gone, so effect 2 no longer applies — but effect 1, the fundamental hard-cap latency tax, is inherent to bandwidth control and remains. The standard mitigations are below.

Uncertain

Verify: the exact upstream merge release for commit de53fd7aedb1 (stated here as 5.4) and whether it was backported to specific stable trees. Reason: the commit and its ~30x figure are confirmed from the commit message and LWN, but I did not independently confirm the first tag containing it; stable backport requests were debated on lore.kernel.org. To resolve: git tag --contains de53fd7aedb100f03e5d2231cfce0e4993282425 against a torvalds/linux clone. (Does not affect 6.12/6.18 behavior, where the fix is present.) uncertain

Mitigations operators actually use

  • Prefer cpu.weight over cpu.max where a soft priority suffices — proportional sharing has no throttling and wastes no idle CPU (see The cgroup v2 CPU Controller). In Kubernetes terms: set CPU requests, avoid CPU limits unless you truly need a hard ceiling. (The K8s request/limit translation lives in cgroups Integration / Kubernetes MOC.)
  • Shorten the period (e.g. 10–20 ms) to cap the worst-case throttle stall, accepting more frequent refills and slightly higher overhead.
  • Right-size threads to quota. A common anti-pattern is a thread pool sized to all host CPUs while the quota allows only a couple — the pool burns the quota across many CPUs in milliseconds, then the whole group throttles. Cap concurrency (e.g. GOMAXPROCS, JVM ActiveProcessorCount) to roughly the quota-implied CPU count.
  • Use cpu.max.burst to absorb periodic spikes within bounded over-subscription.

Configuration / Code — worked examples

# 0.5 CPU: 50ms of runtime every 100ms period
echo "50000 100000" > /sys/fs/cgroup/svc/cpu.max
 
# 2 CPUs worth: 200ms quota over a 100ms period (quota may exceed period -> >1 CPU)
echo "200000 100000" > /sys/fs/cgroup/svc/cpu.max
 
# Tighten the period to 20ms to bound worst-case throttle latency, same 0.5 CPU
echo "10000 20000"  > /sys/fs/cgroup/svc/cpu.max
 
# Remove the cap entirely (back to work-conserving)
echo "max 100000"   > /sys/fs/cgroup/svc/cpu.max
 
# Allow banking up to 10ms of unused quota for a later burst
echo 10000 > /sys/fs/cgroup/svc/cpu.max.burst
 
# Diagnose throttling
cat /sys/fs/cgroup/svc/cpu.stat
# nr_periods 5234
# nr_throttled 4810        <- 92% of periods throttled...
# throttled_usec 1820345
# usage_usec  ...          <- ...if usage << quota*nr_periods, you are over-throttling

The quota/period arithmetic: quota / period is the number of CPU-equivalents. 50000/100000 = 0.5 CPU; 200000/100000 = 2 CPUs. A larger period gives more burst headroom but coarser, spikier latency; a smaller period gives smoother latency but less burst capacity (sched-bwc.rst examples, v6.12).

Failure Modes and Common Misunderstandings

“My container is throttled, so it must be using too much CPU.” Often false. Check usage_usec against quota × nr_periods. If usage is well under that while nr_throttled is high, you have either the thread-count-vs-quota mismatch or simply burst-y arrival hitting the hard cap — not genuine over-use.

Hierarchical throttling. A child can be throttled because its parent’s quota is exhausted, even if the child has runtime left (“case b” in sched-bwc.rst). Aggregate child quotas may exceed the parent’s (over-subscription is allowed for work-conserving semantics), but the parent cap still binds. Diagnose by checking cpu.stat at every ancestor, not just the leaf.

cpu.max is not cpu.weight. A hard cap throttles even on an idle host; a weight never wastes idle CPU. Reaching for cpu.max when cpu.weight would do is the most common self-inflicted throttling wound.

RT tasks are not capped by cpu.max. cgroup v2 does not yet support bandwidth control of real-time tasks via the cpu controller; RT bandwidth is the separate system-wide mechanism in Real-Time Throttling and the RT Bandwidth Limit.

Alternatives and When to Choose Them

  • cpu.weight (proportional) — The cgroup v2 CPU Controller. Prefer this whenever a relative priority under contention is acceptable; no throttling, no idle waste.
  • cpuset confinement — Cpusets and CPU Partitioning. Limits which CPUs, not how much time; combine with cpu.max for both.
  • SCHED_DEADLINE / CBS — The Constant Bandwidth Server and Admission Control. For genuine reservation semantics (a guaranteed runtime within a period with admission control), the deadline class’s CBS is the principled tool; CFS bandwidth control is a cruder cap layered on the fair class.

Choose cpu.max when you need a hard, enforceable ceiling on a group’s average CPU (billing, multi-tenant isolation, preventing a noisy neighbor) and can tolerate period-quantized latency — and reach for the alternatives first when you cannot.

Production Notes

The throttling story is one of the best-documented production scheduler issues in the Linux ecosystem: Indeed’s “Unthrottled” series (part 1, the regression follow-up) and Kubernetes issue #67577 drove the upstream de53fd7aedb1 fix and reshaped how the industry sets CPU limits. The durable takeaways for current (6.12/6.18) kernels: the slice-expiry pathology is fixed, so do not blame slice expiry on a modern kernel; the remaining hard-cap latency tax is real and inherent; measure with the cpu.stat throttle ratio before tuning; and default to CPU requests (cpu.weight) over limits (cpu.max) for latency-sensitive services. The mechanism is healthy; most production pain comes from applying a hard cap where a soft weight was the right tool.

See Also

Graph View

Backlinks