vault

The cgroup io Controller

17 min read

The cgroup io Controller

The io controller is the cgroups-v2 facility that regulates how a process tree consumes block-device I/O bandwidth and I/O operations. Unlike the cpu or memory controllers, I/O has no trivially observable cost metric — a 4 KiB random read and a 1 MiB sequential read cost wildly different amounts of device time on the same disk — so the controller exposes several complementary policies: io.max (hard per-device bandwidth/IOPS caps), io.weight (proportional, work-conserving sharing backed by the cost model or BFQ), io.latency (latency-target QoS protection), and io.stat (accounting). Its single most distinctive feature versus cgroups v1 is writeback integration: because v2 unifies the hierarchy, dirty page-cache writeback I/O can be attributed to the cgroup that dirtied the page rather than dumped on the kernel flusher’s root cgroup — a thing v1’s split hierarchies structurally could not do (per the v6.12 cgroup-v2.rst “IO” and “Writeback” sections). This note owns the cgroup-interface angle; the block-layer plumbing it sits on (the request queue, BFQ’s scheduling, rq-qos) belongs to the Linux Block Layer and Storage MOC and is cross-linked.

This note is pinned to Linux 6.12 LTS (released 2024-11-17), with file definitions read directly from the block/ source in that tree. Interface-file behaviour is stable across 6.12 and 6.18 LTS, but constants and defaults below are stated as of 6.12.

Mental Model — Three Policies, One Resource

The hardest thing to internalise about block I/O control is that there is no single “amount” of I/O. CPU time is wall-clock seconds; memory is bytes; both are directly countable. Block I/O is neither: the cost of an operation depends on size, on read-vs-write, on sequential-vs-random, on the device’s queue depth and its internal garbage collection. The kernel’s own blk-iocost.c opens by stating exactly this — that I/O is “distinguished from CPU and memory where wallclock time and the number of bytes can serve as accurate enough approximations” (blk-iocost.c header). Because of that, the io controller is not one mechanism but a family, and you choose the member that matches what you actually want to guarantee.

flowchart TB
  BIO["A bio (block I/O request)<br/>tagged with its originating cgroup"]
  BIO --> STAT["io.stat<br/>(always-on accounting:<br/>rbytes/wbytes/rios/wios/dbytes/dios)"]
  BIO --> MAX["io.max<br/>HARD CAP per device<br/>(rbps/wbps/riops/wiops)<br/>policy: blk-throttle"]
  BIO --> WEIGHT["io.weight<br/>PROPORTIONAL share<br/>(work-conserving)<br/>policy: iocost OR BFQ"]
  BIO --> LAT["io.latency<br/>LATENCY-target protection<br/>(throttle lower-priority peers)<br/>policy: blk-iolatency"]
  WEIGHT -. "needs a cost model" .-> COST["io.cost.qos / io.cost.model<br/>(root-only tunables)"]
  subgraph DOMAINS["Two cooperating domains (v2 only)"]
    MEMDOM["memory domain<br/>(who dirtied the page)"]
    IODOM["io domain<br/>(who writes it back)"]
  end
  MEMDOM -. "writeback attribution" .-> IODOM
  IODOM --> BIO

The io controller as a family of policies over one un-measurable resource. What it shows: every bio is accounted by io.stat; on top of that you layer at most a hard cap (io.max), a proportional weight (io.weight, which needs a cost model to convert bytes into device-time), or a latency guarantee (io.latency) — and, uniquely to v2, the memory and io domains cooperate so writeback I/O is charged to the cgroup that dirtied the page. The insight to take: you pick the policy by the guarantee you need — an absolute ceiling, a fair slice, or a latency floor — not by “limiting I/O” generically, because there is no single number to limit.

How the Interface Files Are Wired

A subtle but important fact: the files that all begin with io. are not all implemented by one piece of code. The cgroup-v2 controller named io is the kernel’s io_cgrp_subsys, defined in block/blk-cgroup.c, and several independent block-cgroup policies register their own interface files under that one subsystem’s directory. When a policy registers a cftype named "weight", cgroup core prefixes it with the subsystem name, producing the file io.weight. This is why one io. controller can expose files owned by four different source files:

  • io.stat is registered by the core in blk-cgroup.c (the blkcg_files[] array, name = "stat") — it is the always-present accounting file (blk-cgroup.c).
  • io.max is registered by the throttling policy blkcg_policy_throtl in blk-throttle.c (throtl_files[], name = "max") (blk-throttle.c).
  • io.weight, io.cost.qos, io.cost.model are registered by the cost-model policy blkcg_policy_iocost in blk-iocost.c (ioc_files[]) (blk-iocost.c).
  • io.latency is registered by blk-iolatency.c.
  • io.bfq.weight is registered by the BFQ scheduler’s cgroup glue in bfq-cgroup.c (bfq_blkg_files[], name = "bfq.weight") — note this is a separate file from io.weight (bfq-cgroup.c).
  • io.prio.class is registered by the I/O-priority policy in blk-ioprio.c.
  • io.pressure is the PSI file, populated by the generic PSI machinery, not a block policy.

The practical consequence: which io.* files appear in a given cgroup directory depends on which kernel config options are built and which policies are active on the device. A file existing does not mean it is doing anything — io.weight only has an effect once a cost model or BFQ is actually running on that disk.

io.max — The Hard Cap (Throttling)

io.max is the simplest and most predictable policy: an absolute ceiling on bytes-per-second and operations-per-second, per block device. It is a read-write nested-keyed file which exists on non-root cgroups (cgroup-v2.rst, “io.max”). Each line is keyed by a device’s $MAJ:$MIN number — the major/minor pair identifying the block device (e.g. 8:16 is the second SCSI/SATA disk, sdb). The four nested keys are:

KeyMeaning
rbpsMax read bytes per second
wbpsMax write bytes per second
riopsMax read I/O operations per second
wiopsMax write I/O operations per second

To set a 2 MiB/s read cap and a 120-write-IOPS cap on device 8:16:

echo "8:16 rbps=2097152 wiops=120" > io.max

Here 2097152 is 2 * 1024 * 1024, i.e. exactly 2 MiB/s. Any nested key you omit defaults to max (unlimited), so reading the file back yields 8:16 rbps=2097152 wbps=max riops=max wiops=120 (verbatim from the doc). To remove a specific limit, write the literal max as its value: echo "8:16 wiops=max" > io.max. The doc warns that if you specify the same key twice in one write “the outcome is undefined.”

Mechanically, the cap is not a literal token bucket but a time-sliced budget computed from jiffies (the kernel’s timer ticks). In blk-throttle.c, calculate_bytes_allowed(bps_limit, jiffy_elapsed) returns bps_limit * jiffy_elapsed / HZ — the bytes a group is permitted to dispatch in the elapsed fraction of a second (HZ is the timer frequency, so jiffy_elapsed / HZ is “seconds elapsed”) (blk-throttle.c, line 567). The analogous calculate_io_allowed does the same for IOPS. The controller tracks how much each group has already dispatched in the current slice (bytes_disp/io_disp); when a bio would exceed the slice’s allowance it is delayed (not dropped) until enough time has passed. The doc states this directly: “BPS and IOPS are measured in each IO direction and IOs are delayed if limit is reached. Temporary bursts are allowed.” The burst allowance comes from carryover_bytes/carryover_ios accumulated across trimmed slices, which is why a freshly-idle group can briefly exceed its steady-state rate.

io.max is the right tool when you need a guaranteed not-to-exceed ceiling — e.g. capping a noisy batch job so it cannot starve a latency-sensitive database, or enforcing a contractual I/O quota in a multi-tenant host. Its weakness is that it is not work-conserving: the cap applies even when the device is otherwise idle, so you can leave throughput on the table.

io.weight — Proportional, Work-Conserving Sharing

Where io.max is an absolute ceiling, io.weight is a relative share: it divides device capacity among siblings in proportion to their weights, but only when there is contention. It is work-conserving — an idle disk imposes no limit; weights only matter when multiple cgroups compete. io.weight is “a read-write flat-keyed file which exists on non-root cgroups. The default is default 100,” with weights in the range [1, 10000] specifying “the relative amount IO time the cgroup can use in relation to its siblings” (cgroup-v2.rst, “io.weight”). The first line is a default applied to all devices; subsequent $MAJ:$MIN $WEIGHT lines override per device:

default 100
8:16 200
8:0 50

A cgroup with io.weight 200 against a sibling at 100 gets twice the I/O time when both are busy on that device.

The catch is the one the controller’s whole design wrestles with: to share I/O time proportionally, you must convert each bio into an estimate of device time. That is the job of the cost model, implemented by blk-iocost.c (built under CONFIG_BLK_CGROUP_IOCOST). The cost model classifies each I/O as sequential or random, assigns a base cost accordingly, and adds a size-proportional term — “Each IO is classified as sequential or random and given a base cost accordingly. On top of that, a size cost proportional to the length of the IO is added” (blk-iocost.c header). The controller then runs a virtual-time (vtime) scheme: each cgroup’s vtime advances at a rate inversely proportional to its hierarchical weight (hweight), so a group with 12.5% of the weight has its clock run 8× slower; a group may issue a new I/O only if doing so would not “outrun the current device vtime,” otherwise the I/O is suspended until vtime catches up. Because real devices misbehave, the controller also runs vrate adjustment: it watches request-queue wait and completion latency, and dynamically scales the rate at which device vtime advances so the model self-corrects when the device is over- or under-loaded.

The two root-only tunables steer this:

  • io.cost.model sets the linear model’s coefficients ([r|w]bps, [r|w]seqiops, [r|w]randiops). The kernel ships defaults per device class; tools/cgroup/iocost_coef_gen.py can generate device-specific ones (cgroup-v2.rst, “io.cost.model”).
  • io.cost.qos sets the saturation criteria — latency percentiles and thresholds plus a min/max scaling range. The doc’s example, 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0, means: consider sdb saturated when the 95th-percentile read latency exceeds 75 ms (or write 150 ms), and adjust the overall issue rate between 50% and 150%.

Uncertain

Verify: the io section header’s claim in cgroup-v2.rst that “weight based distribution is available only if cfq-iosched is in use and neither scheme is available for blk-mq devices.” Reason: this sentence is stale documentation. The cfq-iosched scheduler was removed years ago with the legacy single-queue block layer, yet the same v6.12 tree ships blk-iocost.c (the cost model that does provide io.weight proportional control) and bfq-cgroup.c (which provides io.bfq.weight) — both work on blk-mq devices, directly contradicting the header. The doc never mentions BFQ or iocost in its IO header. To resolve: treat the file definitions in source (ioc_files[] registering io.weight; bfq_blkg_files[] registering io.bfq.weight) as authoritative over the header prose, and verify on a running 6.12 system which weight file is active for a given scheduler. uncertain

io.latency — Latency-Target Protection

io.latency is a different philosophy again: instead of a cap or a share, you give a cgroup a latency target in microseconds, and the controller throttles lower-priority peers to keep that target met. It is “work conserving; so as long as everybody is meeting their latency target the controller doesn’t do anything” (cgroup-v2.rst, “How IO Latency Throttling Works”). The interface is one line per device:

8:16 target=50

meaning “keep average latency on 8:16 for this group near 50 µs.” Crucially the doc stresses that protection is only enforced at the peer level in the hierarchy — siblings under the same parent influence each other, but a group on a different branch does not. When a protected group starts missing its target, blk-iolatency.c throttles peers with higher (looser) targets in two ways: queue-depth throttling (clamping outstanding I/Os from “no limit” down to as low as 1 at a time) and artificial delay induction for I/O that cannot be queue-throttled without harming higher-priority work — swapping and metadata I/O. Those un-throttleable I/Os are still issued, but “charged” to the originating group, which then accrues a per-process delay (visible in io.stat) capped at 1 second per event (blk-iolatency.c header). When the protected group recovers, peers are unthrottled.

io.latency is the policy you reach for to protect a single latency-sensitive workload (a database, a user-facing service) from being collateral damage when a co-tenant goes wild — it gives a floor on responsiveness rather than a ceiling on consumption.

io.stat — Always-On Accounting

io.stat is read-only and always present (it is registered by the core io_cgrp_subsys, independent of any policy). It is nested-keyed by $MAJ:$MIN, with rbytes/wbytes/rios/wios/dbytes/dios — bytes and operation counts for reads, writes, and discards (the discard/TRIM path that tells SSDs which blocks are free) (cgroup-v2.rst, “io.stat”). A sample line: 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0. When io.latency or io.cost is active, extra fields appear in io.stat (depth, avg_lat, win for latency; use_delay, delay for induced delays) — accounting that is genuinely useful for tuning: the doc recommends setting an io.latency target ~10–15% above the avg_lat you observe under normal load.

The Distinctive v2 Feature — Writeback Integration

This is the capability that justifies the whole unified-hierarchy redesign, and the part a v1 setup cannot replicate. Most disk writes a workload issues are not synchronous — a process does a buffered write(), the kernel marks the page dirty in the page cache, and returns immediately. The actual disk I/O happens later, asynchronously, performed by the kernel’s writeback machinery (the flusher threads). The problem: by the time those writes hit the disk, the original process may be long gone, so naively the I/O cannot be attributed to anyone — historically it was all charged to the kernel/root cgroup, meaning a cgroup could dirty gigabytes, escape its io.max cap entirely, and have its writeback storm degrade everyone.

cgroups v2 fixes this because memory and I/O live in the same hierarchy, so a page’s memory cgroup and its I/O cgroup are the same cgroup. The doc explains the cooperation precisely: “The memory controller defines the memory domain that dirty memory ratio is calculated and maintained for and the io controller defines the io domain which writes out dirty pages for the memory domain. Both system-wide and per-cgroup dirty memory states are examined and the more restrictive of the two is enforced” (cgroup-v2.rst, “Writeback”). Concretely, the vm.dirty_ratio/vm.dirty_background_ratio sysctls — which bound how much of memory may be dirty before writeback is forced — are applied per cgroup, with “available memory” capped by that cgroup’s memory limit. So a container with memory.max=1G cannot hoard dirty pages against the host’s total RAM; its dirty budget is computed against its own 1 GiB.

The attribution itself has a granularity mismatch worth understanding: memory is tracked per page, but writeback is tracked per inode. An inode (a single file) is assigned to one cgroup, and all its dirty-page writeback I/O is charged there. Pages dirtied by a different cgroup than the inode’s owner are called foreign pages; the writeback code “constantly keeps track of foreign pages and, if a particular foreign cgroup becomes the majority over a certain period of time, switches the ownership of the inode to that cgroup.” This works well when an inode is mostly written by one cgroup at a time (even if that cgroup changes), but breaks down when multiple cgroups write the same file simultaneously — the doc explicitly says “a significant portion of IOs are likely to be attributed incorrectly” and recommends avoiding such patterns.

There is a hard constraint: cgroup writeback requires explicit filesystem support. As of 6.12 it is implemented on ext2, ext4, btrfs, f2fs, and xfs; “on other filesystems, all writeback IOs are attributed to the root cgroup” — i.e. on an unsupported filesystem the writeback-attribution benefit silently vanishes and you are back to v1-like behaviour for buffered writes.

Failure Modes and Common Misunderstandings

The most common surprise is io.weight doing nothing. Setting a weight has no effect unless a weight-capable mechanism is actually running on the device — either the iocost cost model is enabled and configured, or BFQ is the scheduler (in which case the live file is io.bfq.weight, not io.weight). On a device using the default mq-deadline or none scheduler with no iocost configured, weights are inert. This is the practical reality behind the stale doc header flagged above.

A second trap is buffered writes escaping io.max on an unsupported filesystem. Because io.max throttles the cgroup that issues the bio, and on a filesystem without cgroup-writeback support the writeback bios are issued by the root cgroup, a workload’s asynchronous writes can bypass its own io.max cap. Synchronous and direct I/O (O_DIRECT) are charged correctly; large buffered-write workloads on, say, an overlay-on-tmpfs or an unsupported filesystem are where caps “leak.”

A third is conflating io.latency peer scope. Because protection is enforced only between siblings, configuring io.latency on a deeply nested leaf while leaving its ancestors unconfigured means it protects nothing against groups on other branches — the doc’s worked tree (groups A/B/C influence each other; D/F influence each other; G influences nobody) is the canonical illustration.

Finally, io.max is not work-conserving, which periodically surprises operators who set a cap “just in case”: the cap throttles even on an idle device, so a conservative io.max can cost real throughput that nobody is contending for. If the goal is fairness-under-contention rather than a hard ceiling, io.weight is the correct tool.

Alternatives and When to Choose Them

Within the controller, the choice is policy-by-guarantee: io.max for an absolute, predictable ceiling (multi-tenant quotas, protecting against runaway batch jobs); io.weight for work-conserving fairness that wastes no capacity (general co-tenancy where you want proportional shares but full utilisation when uncontended); io.latency for protecting one latency-critical workload from noisy neighbours. They compose: you can cap a batch job with io.max and protect a database with io.latency simultaneously.

Outside the io controller, PSI via io.pressure is the observability counterpart — it tells you how stalled a cgroup is on I/O, which is the input that drives policy decisions (and feeds userspace daemons like systemd-oomd). And the memory controller is its inseparable partner for writeback: tuning dirty-page behaviour is a joint memory+io exercise, which is exactly why v2 put them in one hierarchy.

Production Notes

The cost-model controller (iocost) was contributed by Tejun Heo and Andy Newell at Facebook in 2019 (blk-iocost.c copyright header) specifically to make proportional I/O control viable on modern blk-mq SSDs, where the old on-device-time approximation had collapsed — the header’s long design essay is itself the primary write-up, and the in-tree tools/cgroup/iocost_coef_gen.py it references is the supporting tooling. Among the policies, the throttling (io.max) path is the most predictable hard guarantee, while io.weight (iocost or BFQ) is the work-conserving option for co-tenancy where you want full device utilisation when uncontended, and io.latency is the protection mechanism for a single latency-critical workload.

Uncertain

Verify: that container runtimes / the Kubernetes kubelet write these io.* cgroup files when a workload requests block-I/O limits, and the claim that iocost weights see production fleet use (e.g. at Meta). Reason: these are memory-based deployment claims not pinned to a source fetched this session. To resolve: check the kubelet/CRI cgroup-writing code and a primary Meta engineering write-up on iocost deployment. The controller’s kernel-side behaviour and its Facebook-2019 authorship are pinned to the fetched source. uncertain

See Also

Graph View

Backlinks