The Memory Cgroup memcg

The memory cgroup — universally abbreviated memcg — is the Linux kernel subsystem that accounts for, and bounds, the memory consumed by a group of processes rather than the system as a whole. Where the global allocator tracks one number (free pages on the machine), memcg tracks a number per cgroup: every page a group of tasks touches — anonymous heap, page-cache file data, kernel slab objects, network socket buffers — is charged to that group’s struct mem_cgroup, and a small set of interface files (memory.current, memory.min, memory.low, memory.high, memory.max, …) let an administrator read that usage and set protection floors and enforcement ceilings on it. memcg is the kernel mechanism a container’s memory limit ultimately rests on: when Kubernetes sets a pod’s memory limit, the kubelet writes memory.max on a cgroup, and this controller does the rest. This note is the orientation map — why the cgroup-v2 interface has the shape it does, what backs it in the kernel, what counts toward a charge, how the four-knob pressure ladder behaves, and how a page’s charge is born, moves and dies. The enforcing code path (the charge fast path, throttle arithmetic, reclaim loop) lives in memcg Charging and Limits, and the cgroup OOM specifics in cgroup OOM and memory.oom.group. Everything here is pinned to the v6.12 LTS source (released 2024-11-17; still a maintained long-term-support series, 6.12.107 shipped as of 2026-08), cross-checked against the cgroup-v2 documentation.

This note describes cgroup v2 memcg, the modern interface — and treats cgroup v1 strictly as legacy. That is not editorial taste: as of v6.12 the entire v1 memory controller is gated behind CONFIG_MEMCG_V1, which is default n, and lives in its own file mm/memcontrol-v1.c. If you are reading a document that presents memory.limit_in_bytes and memory.memsw.limit_in_bytes as the memory-cgroup interface, that document is describing a controller the upstream kernel is actively dismantling (see cgroup v1 vs v2).


Mental Model — A Page Counter Per Group, Wired Into a Tree

The whole controller reduces to one idea: for each cgroup, maintain a running sum of charged pages, and clamp it against a few thresholds. That sum lives in a struct page_counter embedded in the cgroup’s struct mem_cgroup. Because cgroups form a tree (a parent cgroup contains child cgroups), the counters are hierarchical: charging a page to a leaf cgroup also charges it to every ancestor up to the root, so a parent’s memory.current is the sum of its own usage plus all descendants’. Limits propagate the same way — a child can never effectively be allowed more than its parent’s limit.

flowchart TB
  ROOT["root memcg<br/>(no limits, CSS_NO_REF)"]
  A["cgroup A<br/>memory.max = 4G<br/>page_counter.usage = 3.1G"]
  B["cgroup B<br/>memory.low = 1G (protected)"]
  A1["pod-1 (leaf)<br/>memory.max = 2G"]
  A2["pod-2 (leaf)<br/>memory.high = 1.5G"]
  ROOT --> A
  ROOT --> B
  A --> A1
  A --> A2
  PAGE["page faulted in by pod-1<br/>(anon / file / slab / sock)"]
  PAGE -->|"try_charge_memcg()"| A1
  A1 -.->|"charge propagates up"| A
  A -.->|"and up"| ROOT

memcg is a tree of page counters. What it shows: when a task in pod-1 faults in a page, the charge is applied to pod-1’s counter and every ancestor (A, root) — so A’s memory.current already includes pod-1’s and pod-2’s usage. Limits set on A (memory.max = 4G) bound the whole subtree; limits on a leaf bound just that workload. The insight to take: there is no separate “container accounting” code — it is the same page that the buddy allocator handed out and the page cache indexed, with one extra step at allocation time that bumps a per-group counter. memcg is accounting layered onto the existing page lifecycle, not a parallel allocator.

Two consequences follow immediately from “it is a tree of counters”, and both trip people up:

  • A parent’s number is never smaller than the sum of its children’s. memory.current on an inner node is not “what processes directly in this cgroup use” — inner nodes in cgroup v2 cannot even hold processes when they have enabled controllers and children (see cgroup Delegation and the No Internal Process Rule). It is the subtree total.
  • The charge is applied to the counter, not to the process. Nothing in a task_struct says “this task owns 300 MiB”. The number lives on the cgroup, and the page carries a back-pointer to the cgroup — which is why moving the task between cgroups does not move the memory (see The Life of a Charge).

Why cgroup v2 Redesigned the Memory Controller

The v2 interface is not a rename of the v1 interface. It is a redesign that came out of a specific, documented set of failures, and the kernel’s own cgroup-v2.rst contains the argument in a section titled “Controller Issues and Remedies → Memory” (cgroup-v2.rst). Reading it is the fastest way to understand why there are four memory knobs instead of one. There were three complaints.

The soft limit did not work, and could not be made to work. v1’s lower boundary was memory.soft_limit_in_bytes, unset by default. Because it was opt-in, global reclaim had to search for cgroups that had set it: “the set of cgroups that global reclaim prefers is opt-in, rather than opt-out. The costs for optimizing these mostly negative lookups are so high that the implementation, despite its enormous size, does not even provide the basic desirable behavior.” Worse, it had no hierarchical meaning at all — “All configured groups are organized in a global rbtree and treated like equal peers, regardless where they are located in the hierarchy. This makes subtree delegation impossible.” And when it did fire, it fired too hard: “the soft limit reclaim pass is so aggressive that it not just introduces high allocation latencies into the system, but also impacts system performance due to overreclaim, to the point where the feature becomes self-defeating.” The v2 replacement, memory.low, inverts the model: it is “a top-down allocated reserve,” so a subtree can be delegated and a child’s protection is bounded by its parent’s.

A hard limit alone forces users to over-provision. v1 offered exactly one ceiling, memory.limit_in_bytes, “defined as a strict limit that can not budge, even if the OOM killer has to be called. But this generally goes against the goal of making the most out of the available memory.” Since workload footprints move around at runtime, using a strict cap safely requires either accurately predicting the working-set size or padding the limit — and “Since working set size estimation is hard and error prone, and getting it wrong results in OOM kills, most users tend to err on the side of a looser limit and end up wasting precious resources.” memory.high exists to break that trade-off: it “throttles allocations by forcing them into direct reclaim to work off the excess, but it never invokes the OOM killer,” so a too-tight high boundary produces “gradual performance degradation” that an operator can observe and correct rather than a dead process.

Combined memory+swap accounting is the wrong abstraction. v1’s memory.memsw.limit_in_bytes capped the sum of resident memory and swap. The stated justification was that a parent should always be able to swap out a child’s anonymous memory regardless of the child’s own configuration — but the docs reject the argument: “untrusted groups can sabotage swapping by other means - such as referencing its anonymous memory in a tight loop - and an admin can not assume full swappability when overcommitting untrusted jobs.” And for trusted jobs “a combined counter is not an intuitive userspace interface, and it flies in the face of the idea that cgroup controllers should account and limit specific physical resources.”

There is even a concrete race that v2 fixed: writing a lower value to v1’s memory.limit_in_bytes “was subject to a race condition, where concurrent charges could cause the limit setting to fail,” whereas memory.max “will first set the limit to prevent new charges, and then reclaim and OOM kill until the new limit is met - or the task writing to memory.max is killed.”

flowchart LR
  subgraph V1["cgroup v1 — two knobs, both broken"]
    SL["memory.soft_limit_in_bytes<br/>opt-in, global rbtree,<br/>no hierarchy, over-reclaims"]
    HL["memory.limit_in_bytes<br/>strict cap, OOM on breach"]
    MSW["memory.memsw.limit_in_bytes<br/>caps RAM + swap as one number"]
  end
  subgraph V2["cgroup v2 — a graduated ladder"]
    MIN["memory.min<br/>hard protection floor"]
    LOW["memory.low<br/>best-effort floor"]
    HIGH["memory.high<br/>throttle, never kills"]
    MAX["memory.max<br/>hard cap, cgroup OOM"]
    SWAP["memory.swap.max<br/>swap is its own resource"]
  end
  SL -->|"replaced by a top-down<br/>allocated reserve"| LOW
  SL -.->|"and a strictly<br/>guaranteed variant"| MIN
  HL -->|"split: throttle first…"| HIGH
  HL -->|"…kill only as backstop"| MAX
  MSW -->|"unbundled into its<br/>own page_counter"| SWAP

How each v1 knob was replaced. What it shows: the v2 interface is not additive decoration — every v2 file exists because a specific v1 file failed in a specific way. The single v1 hard limit was split into a throttle (high) and a backstop (max); the single broken soft limit was split into best-effort (low) and guaranteed (min) protection; the combined memsw counter was unbundled. The insight to take: if you find yourself reaching for “the memory limit,” you are still thinking in v1. In v2 the intended day-to-day control knob is memory.high, and memory.max is a safety net you hope never fires — the docs say so in as many words: “memory.high is the main mechanism to control memory usage. Over-committing on high limit (sum of high limits > available memory) and letting global memory pressure to distribute memory according to usage is a viable strategy.”

The deprecation of v1 has been a slow, deliberate campaign rather than a flag day. At the 2024 Linux Storage, Filesystem, Memory-Management and BPF Summit, a session led by Shakeel Butt and Roman Gushchin laid out the plan: move the v1 code into a separate file with its own config option, default that option off (Michal Hocko’s position was that “if it remains enabled, the community will never manage to get rid of it”), add a deprecation warning backported to stable kernels, then “wait a while”, defined as “two or three long-term-support cycles” before hollowing out the implementation (LWN: The twilight of the version-1 memory controller). The individual v1 features were triaged in that session: move_charge_at_immigrate, TCP memory accounting (“nobody is using TCP memory accounting, its performance is terrible, and the version-2 implementation is far better” — Butt, with no disagreement recorded), and soft limits were all agreed to be on the chopping block; combined memory+swap accounting survived because Google still depends on it and no good replacement exists.


struct mem_cgroup — What Backs a Cgroup’s Memory Accounting

Every memory cgroup is a struct mem_cgroup, defined in include/linux/memcontrol.h. It is large; the fields that matter for understanding the controller are:

struct mem_cgroup {
    struct cgroup_subsys_state css;   /* ties this into the cgroup tree */
    struct mem_cgroup_id id;          /* short numeric id, recycled */
 
    struct page_counter memory;       /* main usage counter (v1 and v2) */
    union {
        struct page_counter swap;     /* v2: swap usage, bounded by memory.swap.max */
        struct page_counter memsw;    /* v1: combined memory+swap counter */
    };
 
    struct work_struct high_work;     /* deferred memory.high reclaim */
    struct vmpressure vmpressure;     /* legacy pressure notification */
    bool oom_group;                   /* kill the whole group on OOM? */
    int swappiness;                   /* per-cgroup anon-vs-file reclaim bias */
 
    struct memcg_vmstats *vmstats;                       /* the memory.stat numbers */
    struct memcg_vmstats_percpu __percpu *vmstats_percpu;/* per-CPU stat batching */
    atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS]; /* memory.events counters */
    atomic_long_t memory_events_local[MEMCG_NR_MEMORY_EVENTS]; /* memory.events.local */
 
    struct obj_cgroup __rcu *objcg;   /* for kernel/slab object accounting */
    struct list_head objcg_list;      /* reparented objcgs inherited from dead children */
    int kmemcg_id;
 
    struct mem_cgroup_per_node *nodeinfo[]; /* per-NUMA-node LRU/stat state */
};

Walking the load-bearing pieces:

  • css (cgroup_subsys_state) is the generic handle that splices this memcg into the cgroup hierarchy and carries its reference count. The kernel pins a memcg with css_get() while a charge is outstanding and releases it with css_put() when the page is uncharged — so a cgroup directory can be rmdir’d but its mem_cgroup lives on until the last charged page is freed. The root memcg is special: it is marked CSS_NO_REF (reference counting disabled), because every page on the system is ultimately charged to it and refcounting that would be pure overhead.
  • id is not the css id. memcg maintains a private 16-bit ID space, and the reason is stated in a comment in mm/memcontrol.c: “Swap-out records and page cache shadow entries need to store memcg references in constrained space, so we maintain an ID space that is limited to 16 bit (MEM_CGROUP_ID_MAX), limiting the total number of memory-controlled cgroups to 64k.” A swap entry or a page-cache shadow entry has only a few spare bits, not room for a 64-bit pointer, so it stores this short id instead. The same comment explains why the id is recycled aggressively at offline time rather than at free time: “there usually are many references to the offline CSS after the cgroup has been destroyed, such as page cache or reclaimable slab objects, that don’t need to hang on to the ID. We want to keep those dead CSS from occupying IDs, or we might quickly exhaust the relatively small ID space and prevent the creation of new cgroups even when there are much fewer than 64k cgroups - possibly none.”
  • memory is the central struct page_counter. Its usage field is memory.current; its max, high, min, low fields back the four corresponding interface files. The mechanics of charging against it are in memcg Charging and Limits.
  • swap / memsw is a union because v1 and v2 model swap differently — see cgroup v1 vs v2. A given kernel build and runtime uses one or the other, never both.
  • vmstats / vmstats_percpu hold the detailed breakdown that memory.stat exposes. Updates are batched per-CPU rather than hitting a shared counter on every page operation — the machinery and its accuracy consequences are the subject of memory.current vs memory.stat.
  • objcg (obj_cgroup) is the indirection used for kernel-memory (slab) accounting — see Kernel Memory and the objcg Indirection.
  • nodeinfo[] is the per-NUMA-node state: each node has its own per-cgroup LRU lists, because per-cgroup reclaim walks this cgroup’s LRU on the node under pressure, not the global one. It is a trailing flexible array, sized at allocation to the machine’s node count.

page_counter — the actual arithmetic object

struct page_counter (include/linux/page_counter.h) is worth reading in full because every memory interface file is a view onto one of its fields. Its layout is deliberately cache-line-conscious:

struct page_counter {
	atomic_long_t usage;              /* <- this is memory.current */
	CACHELINE_PADDING(_pad1_);        /* keep usage alone in its line */
 
	unsigned long emin;               /* effective memory.min, after hierarchy */
	atomic_long_t min_usage;
	atomic_long_t children_min_usage;
 
	unsigned long elow;               /* effective memory.low, after hierarchy */
	atomic_long_t low_usage;
	atomic_long_t children_low_usage;
 
	unsigned long watermark;          /* all-time peak */
	unsigned long local_watermark;    /* peak since last memory.peak reset */
	unsigned long failcnt;
 
	CACHELINE_PADDING(_pad2_);        /* read-mostly fields below */
	bool protection_support;
	unsigned long min, low, high, max;/* <- the four configured knobs */
	struct page_counter *parent;      /* <- the hierarchy link */
} ____cacheline_internodealigned_in_smp;

The padding is not cosmetic — the source comment says so directly: “Make sure usage does not share cacheline with any other field. The memcg->memory.usage is a hot member of struct mem_cgroup.” Every page fault in the cgroup touches usage; if the configured limits shared its cache line, reading memory.max from a neighbouring CPU would bounce the line that the fault path is hammering.

Three details fall out of this struct that answer common questions:

  1. min/low vs emin/elow. The file you write is min; the number reclaim actually consults is emin, the effective protection after the hierarchy has been applied (page_counter_calculate_protection() computes it). A child cannot protect more than its parent protects, and when siblings collectively ask for more than the parent grants, each gets a share proportional to its usage below the floor.
  2. “unlimited” is a real number. PAGE_COUNTER_MAX is LONG_MAX / PAGE_SIZE on 64-bit, and the default max at page_counter_init() time is exactly that. Writing the string max to memory.max writes this value. There is no separate “no limit” flag.
  3. failcnt is v1-only in practice. cgroup v2 does not expose a failcnt file. The 2024 LSFMM+BPF session noted this explicitly and concluded it “would be easy to add … to version 2, but there should be a use case defined first” (LWN 974575). v2’s equivalent signal is the max key in memory.events.
flowchart LR
  subgraph MC["struct mem_cgroup"]
    CSS["css : cgroup_subsys_state<br/>id : 16-bit, recycled at offline<br/>oom_group : bool<br/>swappiness : int<br/>high_work : deferred high reclaim"]
  end
  subgraph PCM["page_counter memory"]
    PCMF["usage → memory.current<br/>min / low / high / max → the four knobs<br/>emin / elow → effective, after hierarchy<br/>watermark, local_watermark → memory.peak<br/>parent → the hierarchy link"]
  end
  subgraph PCS["page_counter swap (union with memsw)"]
    PCSF["usage → memory.swap.current<br/>high → memory.swap.high<br/>max → memory.swap.max"]
  end
  subgraph VS["memcg_vmstats + per-CPU deltas"]
    VSF["state[] and events[] → memory.stat<br/>folded in by cgroup_rstat, every 2s"]
  end
  subgraph NI["mem_cgroup_per_node[ ]"]
    NIF["lruvec : this cgroup's LRU on this node<br/>per-node stats → memory.numa_stat"]
  end
  subgraph OC["obj_cgroup"]
    OCF["refcnt : percpu_ref<br/>memcg : rewritten on reparent<br/>nr_charged_bytes : fractional page"]
  end
  MC --- PCM
  MC --- PCS
  MC --- VS
  MC --- NI
  MC --- OC
  PCM -->|"parent"| PARENT["parent cgroup's<br/>page_counter memory"]

The objects behind the interface files. What it shows: each user-visible file maps onto exactly one field of one of these structures — memory.current is page_counter.usage, memory.peak is local_watermark, memory.numa_stat comes out of the per-node array. The parent self-link on the memory counter is the entire hierarchy mechanism. The insight to take: memcg has two accounting machines, not one — the exact atomic page_counter that gates allocation, and the approximate, batched memcg_vmstats that produces memory.stat. Confusing them is the source of most “the numbers don’t add up” reports; see the section below.


What Counts Toward a Charge

The cgroup-v2 documentation states the controller tracks, “While not completely water-tight, all major memory usages by a given cgroup … Userland memory - page cache and anonymous memory. Kernel data structures such as dentries and inodes. TCP socket buffers.” (cgroup-v2.rst). The phrase “not completely water-tight” is load-bearing and honest: memcg is a very good approximation of a group’s footprint, not a proof.

Concretely, four families of allocation land on a cgroup’s memory counter:

  • Anonymous memory — pages with no file backing: heap (brk/sbrk), mmap(MAP_ANONYMOUS), stack. Reported as anon in memory.stat, with anon_thp breaking out the part backed by transparent huge pages.
  • Page cache (file) memory — cached file contents, “including tmpfs and shared memory”. A read() that pulls a file page into the page cache charges that page to the reading cgroup. memory.stat splits this further into file_mapped (mapped with mmap()), file_dirty, file_writeback, shmem (swap-backed cache: tmpfs, shm segments, shared anonymous mappings) and file_thp.
  • Kernel memory — slab objects (slab, split into slab_reclaimable for dentries and inodes and slab_unreclaimable), kernel stacks (kernel_stack), page tables (pagetables), secondary page tables (sec_pagetables — “KVM mmu allocations on x86 and arm64 and IOMMU page tables”, a line item that matters enormously on virtualization hosts), per-CPU allocations (percpu), and vmalloc. The umbrella kernel key sums these. Kernel memory accounting is on by default for all cgroups; an individual allocation opts in by carrying __GFP_ACCOUNT, usually via GFP_KERNEL_ACCOUNT — defined as GFP_KERNEL | __GFP_ACCOUNT and documented in-tree as “the same as GFP_KERNEL, except the allocation is accounted to kmemcg” (include/linux/gfp_types.h) — or, for a whole slab cache, by setting SLAB_ACCOUNT (see Kmem Caches and Object Slabs and GFP Flags and Allocation Contexts).
  • Socket memory — network transmit and receive buffers, reported as sock. In cgroup v2 this is not a side counter: mem_cgroup_charge_skmem() calls the ordinary try_charge() against memcg->memory and then merely labels the pages by bumping the MEMCG_SOCK statistic. Socket memory therefore competes with anon and file memory for the same memory.max. (In v1 it went through a separate tcpmem page counter with its own limit files — the arrangement Shakeel Butt described as having “terrible” performance.) The per-socket side of this is Socket Buffers and Memory Accounting.

Why bother charging kernel memory at all? Because unlike user pages it cannot be reclaimed by swapping. The kernel documentation puts the threat model plainly: “Kernel memory is fundamentally different than user memory, since it can’t be swapped out, which makes it possible to DoS the system by consuming too much of this precious resource” (cgroup-v1/memory.rst §2.7). Without kmem accounting, a container could stay comfortably under memory.max on its user pages while exhausting the host through dentries, page tables, or socket buffers.

All three accounting families can be switched off at boot. Documentation/admin-guide/kernel-parameters.txt documents exactly three tokens for the cgroup.memory= parameter in v6.12:

Boot tokenEffect
cgroup.memory=nosocketDisable socket memory accounting
cgroup.memory=nokmemDisable kernel memory accounting
cgroup.memory=nobpfDisable BPF memory accounting

Turning any of them on is a decision to make a class of memory invisible to every container limit on the machine; they exist for debugging and for workloads where the accounting overhead measurably hurts, not as routine tuning.

flowchart TB
  subgraph SRC["Allocation sites that charge"]
    ANON["page fault on anon VMA<br/>heap · stack · MAP_ANONYMOUS"]
    FILE["page cache fill<br/>read() · mmap() · readahead · tmpfs"]
    SLABK["slab / kernel object<br/>SLAB_ACCOUNT or __GFP_ACCOUNT"]
    SOCK["socket buffer<br/>mem_cgroup_charge_skmem()"]
  end
  CHG["try_charge()<br/>against memcg-&gt;memory"]
  OBJ["obj_cgroup byte charging<br/>(sub-page granularity)"]
  CNT["page_counter.usage<br/>= memory.current"]
  STAT["memcg_vmstats<br/>anon / file / slab / sock / …<br/>= memory.stat"]
  ANON --> CHG
  FILE --> CHG
  SOCK --> CHG
  SLABK --> OBJ --> CHG
  CHG --> CNT
  ANON -.->|"labels the charge"| STAT
  FILE -.-> STAT
  SLABK -.-> STAT
  SOCK -.-> STAT

The four charge sources and where each number ends up. What it shows: every family funnels into the same try_charge() and the same page_counter.usage; the split you see in memory.stat is a parallel labelling, not a set of separate budgets. Slab is the one path with an extra hop — obj_cgroup — because slab objects are smaller than a page. The insight to take: there is no per-category limit in cgroup v2. You cannot cap “kernel memory to 200 MiB” the way v1’s memory.kmem.limit_in_bytes pretended to; page cache, anon, slab and sockets all draw on one budget, which is precisely why memory.current is usually much larger than the RSS you expected.


The Pressure Ladder: min → low → high → max

The four numeric knobs are not four ways of saying “limit”. They are four rungs of a ladder, and each rung has different failure semantics. Getting this wrong is the single most common memcg misconfiguration, so it is worth stating the distinction precisely before looking at the interface as a whole.

Read from the bottom up, the ladder is: memory.min and memory.low say “do not take memory away from me”; memory.high and memory.max say “do not let me take more”. Protection is about reclaim; limits are about allocation.

The two protection knobs: min vs low

Both create a protection floor — a number of bytes below which this cgroup’s memory is shielded from reclaim — but with opposite behaviour when the machine runs out of options.

  • memory.low is best-effort. Per the documentation: “If the memory usage of a cgroup is within its effective low boundary, the cgroup’s memory won’t be reclaimed unless there is no reclaimable memory available in unprotected cgroups.” Under severe global pressure the protection is sacrificed rather than letting the system fail.
  • memory.min is hard. “If the memory usage of a cgroup is within its effective min boundary, the cgroup’s memory won’t be reclaimed under any conditions. If there is no unprotected reclaimable memory available, OOM killer is invoked.”

memory.min was added by Roman Gushchin (then at Facebook) precisely because memory.low “works only until there is a supply of reclaimable memory,” which makes it “pretty useless against any sort of slow memory leaks” — and on a swapless machine the failure is immediate rather than merely postponed. His conclusion, quoted in the LWN coverage of the patch, is the whole rationale for the knob: “the only effective way to guarantee the memory protection in this case is to invoke the OOM killer” (LWN: mm: introduce memory.min).

Both are hierarchical and both have a proportional region above the floor rather than a cliff: “Above the effective min boundary (or effective low boundary if it is higher), pages are reclaimed proportionally to the overage, reducing reclaim pressure for smaller overages.” A cgroup 5% over its floor is scanned much more gently than one 300% over. The documentation also warns against the obvious over-use: “Putting more memory than generally available under this protection is discouraged and may lead to constant OOMs,” and notes a rule that surprises people — “If a memory cgroup is not populated with processes, its memory.min is ignored,” so an empty cgroup cannot hoard a reserve.

The two limit knobs: high vs max

  • memory.high is “Memory usage throttle limit.” Exceeding it does not kill anything: “the processes of the cgroup are throttled and put under heavy reclaim pressure… Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached.”
  • memory.max is the “Memory usage hard limit. This is the main mechanism to limit memory usage of a cgroup. If a cgroup’s memory usage reaches this limit and can’t be reduced, the OOM killer is invoked in the cgroup.”

The documentation adds two qualifications to memory.max that matter operationally. First, breaching it is not always fatal: “In default configuration regular 0-order allocations always succeed unless OOM killer chooses current task as a victim” — a single-page allocation will force-charge rather than fail. Second, not every allocation that cannot be satisfied triggers a kill: “Some kinds of allocations don’t invoke the OOM killer. Caller could retry them differently, return into userspace as -ENOMEM or silently ignore in cases like disk readahead.”

flowchart TD
  U["allocation charges the cgroup<br/>usage rises"]
  U --> Q1{"usage &lt; effective min?"}
  Q1 -->|yes| P1["reclaim SKIPS this cgroup entirely.<br/>If nothing else is reclaimable →<br/>OOM killer, rather than touch it.<br/>events: (none)"]
  Q1 -->|no| Q2{"usage &lt; effective low?"}
  Q2 -->|yes| P2["reclaim skips it while ANY<br/>unprotected cgroup has reclaimable pages.<br/>Sacrificed under extreme pressure.<br/>events: low++ when breached"]
  Q2 -->|no| Q3{"usage &gt; memory.high?"}
  Q3 -->|yes| P3["allocator forced into direct reclaim,<br/>then a penalty sleep proportional<br/>to the overage. NEVER OOMs.<br/>Limit may be breached.<br/>events: high++"]
  Q3 -->|no| P0["normal: reclaim treats it like<br/>any other cgroup"]
  P3 --> Q4{"usage would exceed memory.max?"}
  Q4 -->|"reclaim frees enough"| P0
  Q4 -->|"reclaim fails"| P4["cgroup-scoped OOM kill<br/>(whole group if oom.group=1)<br/>events: max++, oom++, oom_kill++"]

What actually happens at each rung. What it shows: the two protection rungs change what reclaim is allowed to take; the two limit rungs change what the allocator is allowed to get. Only the bottom-right path ends in a dead process. The insight to take: high and max are not “soft limit” and “hard limit” in the usual sense — high is a latency mechanism (it makes the workload slower, which a monitoring agent can see and react to), while max is a correctness mechanism (it makes the workload stop). If your platform sets only memory.max, you have given yourself no early-warning band at all: the first symptom is a kill.

RungDefaultQuestion it answersEnforcementCan it kill?
memory.min0How much must never be reclaimed?reclaim skips the cgroup unconditionallyyes — indirectly, by leaving nothing else to reclaim
memory.low0How much should preferably not be reclaimed?reclaim skips it unless nothing unprotected remainsno
memory.highmaxAbove what point should this workload be slowed down?direct reclaim + penalty sleep in the allocatorno, never
memory.maxmaxAbove what point must this workload stop?reclaim, then cgroup-scoped OOMyes

The full enforcement code paths — the per-CPU charge stock, the memory.high penalty-sleep formula, the reclaim-then-OOM loop behind memory.max — are the subject of memcg Charging and Limits and cgroup OOM and memory.oom.group; the reclaim side (how emin/elow are consulted while scanning an LRU) is in Per-cgroup Reclaim and Memory Pressure.


The cgroup-v2 Interface Files

memcg exposes its state and knobs as files in each cgroup directory under /sys/fs/cgroup/. The list below is the complete v2 set as registered in memory_files[], swap_files[] and zswap_files[] in mm/memcontrol.c, plus memory.pressure, which the cgroup core installs generically. Every one of them “exists on non-root cgroups” only — the root cgroup has no memory.current, because everything on the machine is charged to it and a limit there would be meaningless.

FileR/WMeaning
memory.currentRO“The total amount of memory currently being used by the cgroup and its descendants.” Exactly page_counter.usage.
memory.peakRWHigh-water mark “since either the creation of the cgroup or the most recent reset for that FD”. Writing any non-empty string resets it — per file descriptor, see below.
memory.minRWHard reclaim protection. Default 0.
memory.lowRWBest-effort reclaim protection. Default 0.
memory.highRWThrottle limit — the intended main control knob. Default max.
memory.maxRWHard limit — the last line of defence. Default max.
memory.statROFlat-keyed breakdown: anon, file, kernel, slab, sock, zswap, LRU sizes, workingset_* refault counters, pg* event counters.
memory.numa_statROThe same keys, split per NUMA node: type N0=<bytes> N1=<bytes> …. Keys tagged npn in memory.stat (“non-per-node”) do not appear here.
memory.eventsROHierarchical counters: low, high, max, oom, oom_kill, oom_group_kill.
memory.events.localROThe same counters, non-hierarchical — this cgroup only.
memory.reclaimWOTrigger proactive reclaim: echo "1G" > memory.reclaim. Accepts a nested swappiness= key. Returns -EAGAIN if it reclaimed less than asked.
memory.oom.groupRWTreat the cgroup as an indivisible OOM unit — see cgroup OOM and memory.oom.group.
memory.swap.currentROSwap bytes used by the subtree.
memory.swap.highRWSwap throttle. Default max.
memory.swap.maxRWSwap hard limit. Default max.
memory.swap.peakRWSwap high-water mark, same FD-reset semantics as memory.peak.
memory.swap.eventsROhigh, max, fail counters.
memory.zswap.currentROBytes held in the zswap compressed pool for this cgroup.
memory.zswap.maxRWCap on the zswap pool for this cgroup. Default max.
memory.zswap.writebackRW1 by default; 0 forbids writing this cgroup’s pages out to a swap device. Hierarchical.
memory.pressureROPSI for memory — some and full, avg10/avg60/avg300 plus total.

Three interface subtleties deserve a callout because they are easy to misuse:

memory.peak resets are per-file-descriptor, not global. In v6.12 the implementation (peak_show() / peak_write() in mm/memcontrol.c) keeps a per-open-file struct cgroup_of_peak alongside the counter’s local_watermark, and a read returns max(this FD's peak, the counter's local watermark). Two monitoring agents that both open the file and both write to it will therefore not clobber each other’s baselines — but an agent that opens the file fresh on every scrape gets a value relative to the last global reset, not to its own previous scrape. If you want interval peaks, hold the descriptor open.

memory.events is hierarchical and memory.events.local is not, and the difference is the whole diagnostic value. A high count on a parent tells you something in this subtree is being throttled; the same count on memory.events.local of a leaf tells you it is that workload. The documentation is explicit that a parent’s file can generate an inotify modification event “due to an event down the hierarchy,” which is exactly the trap for a naive watcher.

memory.reclaim is deliberately not a pressure signal. The docs warn: “the proactive reclaim (triggered by this interface) is not meant to indicate memory pressure on the memory cgroup. Therefore socket memory balancing triggered by the memory reclaim normally is not exercised in this case. This means that the networking layer will not adapt based on reclaim induced by memory.reclaim.” Proactive reclaim is a tool for a userspace agent that has decided a workload’s cold memory should be squeezed out; it does not tell TCP to back off the way genuine pressure does.


memory.current vs memory.stat — Two Accounting Machines {#memory-current-vs-memory-stat}

A recurring bug report on container platforms is “memory.current does not equal the sum of the memory.stat categories.” It is not a bug. The two files are produced by two different mechanisms with deliberately different accuracy guarantees, and knowing which is which tells you which number to alert on.

memory.current is exact and synchronous. It is a direct read of atomic_long_read(&counter->usage). It has to be exact, because it is the number the allocator compares against memory.max before deciding whether a charge may proceed. There is a small caveat — the charge fast path pre-charges in batches of MEMCG_CHARGE_BATCH (64 pages, defined in include/linux/memcontrol.h) into a per-CPU stock, so a few hundred kilobytes per CPU may be charged-but-unused at any instant — but the counter itself never lags.

memory.stat is approximate and eventually consistent. Every statistic update goes through memcg_rstat_updated(), which accumulates into a per-CPU memcg_vmstats_percpu and marks the cgroup dirty in the generic cgroup_rstat tree. Flushing those per-CPU deltas into the readable aggregate is expensive, so mm/memcontrol.c states the policy in a comment:

 * 1) Periodically and asynchronously flush the stats every 2 seconds to not let
 *    the rstat update tree grow unbounded.
 *
 * 2) Flush the stats synchronously on reader side only when there are more than
 *    (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization
 *    will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but
 *    only for 2 seconds due to (1).

Walking that bound symbol by symbol: MEMCG_CHARGE_BATCH is 64 pages; nr_cpus is num_online_cpus(). On a 64-core machine with 4 KiB pages the worst-case skew is therefore 64 × 64 × 4 KiB = 16 MiB, and it is corrected within FLUSH_TIME, defined as 2UL*HZ — two seconds. On a 192-core host the same arithmetic gives roughly 48 MiB. That is the size of the discrepancy you should expect to see and not investigate.

sequenceDiagram
    autonumber
    participant T as task (page fault)
    participant S as per-CPU charge stock
    participant PC as page_counter.usage
    participant PS as per-CPU vmstats
    participant RS as cgroup_rstat tree
    participant R as reader of memory.stat
    T->>S: charge 1 page
    alt stock has credit
        S-->>T: satisfied locally (no atomic)
    else stock empty
        S->>PC: try_charge 64 pages (one atomic)
        PC-->>S: ok, 63 pages banked
    end
    T->>PS: memcg_rstat_updated(+1 page, anon)
    PS->>RS: cgroup_rstat_updated(cgroup, cpu)
    Note over RS: flushed asynchronously<br/>every FLUSH_TIME = 2*HZ
    R->>RS: read memory.stat
    alt pending updates > MEMCG_CHARGE_BATCH * nr_cpus
        R->>RS: force a synchronous flush
    end
    RS-->>R: aggregated per-key values
    R->>PC: read memory.current
    PC-->>R: exact usage, no flush needed

Why the two files disagree. What it shows: memory.current is read straight off the atomic counter on the right-hand side of the charge path, while memory.stat is assembled from per-CPU deltas that are only folded in every two seconds (or sooner, if enough updates have piled up). The insight to take: alert on memory.current and memory.events, because they are exact and immediate; use memory.stat for composition (“is this anon or page cache?”), not for threshold comparisons. A monitoring system that computes memory.current − memory.stat[file] − memory.stat[anon] and alarms on the remainder will page you for a scheduling artefact.

There is a second, deeper reason the numbers will not reconcile even after a flush: memory.stat’s keys are not a partition. The documentation says so where it describes the LRU keys — “As these represent internal list state (eg. shmem pages are on anon memory management lists), inactive_foo + active_foo may not be equal to the value for the foo counter, since the foo counter is type-based, not list-based.” Some keys are subsets of others (slab_reclaimable and slab_unreclaimable sum to slab, which is itself part of kernel); some are counters of events rather than bytes (pgfault, pgscan, workingset_refault_file); and some overlap deliberately (swapcached is “accounted against both memory and swap usage”).

For the categories that are byte-valued and disjoint, a useful working decomposition of memory.current is:

memory.current  ≈  anon          (heap, stack, anonymous mmap)
                +  file          (page cache, incl. tmpfs via shmem)
                +  kernel        (slab + kernel_stack + pagetables
                                  + sec_pagetables + percpu + vmalloc)
                +  sock          (network buffers)
                +  zswap         (compressed pool held for this cgroup)

with the ”≈” carrying the rstat skew above, plus whatever memcg does not track at all — the “not completely water-tight” disclaimer. Notably absent from a container’s charge by default: HugeTLB pages (see the mount-option section below), DMA buffers pinned by a driver on the workload’s behalf, and memory held by kernel threads doing work that was never attributed to the requesting cgroup.

Three mount options that change memcg semantics system-wide

Some memcg behaviour is not per-cgroup at all — it is fixed at cgroup2 mount time and applies to the whole machine. Three of the documented mount -t cgroup2 -o … options touch the memory controller directly, and each exists because the correct behaviour would have broken somebody’s existing setup:

Mount optionEffectWhy it is not the default
memory_localevents“Only populate memory.events with data for the current cgroup, and not any subtrees.”This is the legacy v1-ish behaviour; hierarchical counting is the default. Provided for tools written against the old semantics. Can only be set at mount or remount from the init namespace.
memory_recursiveprot“Recursively apply memory.min and memory.low protection to entire subtrees, without requiring explicit downward propagation into leaf cgroups.”The docs say outright it “should have been the default behavior,” but making it so would regress setups that rely on “specifying bogusly high ‘bypass’ protection values at higher tree levels.”
memory_hugetlb_accounting“Count HugeTLB memory usage towards the cgroup’s overall memory usage”New behaviour “that could regress existing setups, so it must be explicitly opted in.”

memory_recursiveprot is the one most worth knowing about: without it, protecting a subtree means writing memory.low into every leaf, and a leaf you forget is unprotected. With it, protection set on a parent covers the whole subtree while children still compete freely inside it. systemd sets this option on modern distributions, which is why hand-computed protection arithmetic sometimes does not match what reclaim actually does.

The HugeTLB option carries four caveats the documentation states explicitly, and two of them bite hard. Charging happens at use, not at pool allocation — “when a new HugeTLB folio is allocated to the pool, it is not accounted for from the perspective of the memory controller. It is only charged to a cgroup when it is actually used (for e.g at page fault time)” — so host overcommit planning cannot see the pool. And a failed charge is not an OOM kill but a SIGBUS: “Failure to charge a HugeTLB folio to the memory controller results in SIGBUS. This could happen even if the HugeTLB pool still has pages available (but the cgroup limit is hit and reclaim attempt fails).” A database or VM host that enables this option and sets a tight memory.max will see SIGBUS where it expected an OOM kill. Pages used while the option was off stay untracked forever, “even if cgroup v2 is remounted later on.”


Kernel Memory and the objcg Indirection {#objcg}

Charging a whole page is easy: the page has one owner, so it gets one back-pointer. Slab objects break that model. A dentry is on the order of 200 bytes; a 4 KiB slab page holds twenty of them, and there is no reason those twenty came from the same container. Charging the whole page to whoever allocated the first object would be wildly unfair, and refusing to share pages between cgroups is exactly what the old implementation did — and it was a disaster.

The pre-2020 design gave every memory cgroup its own copy of every accounted slab cache. A machine running a few hundred containers would carry a few hundred private dentry caches, each with its own partially-filled pages. Roman Gushchin’s rework, merged in 5.9, replaced it with a single global set of caches plus per-object accounting. The measured result, from his cover letter as carried by LWN’s coverage of the new slab controller, was “memory savings in order of 1 GB per host” with slab utilization improvements “up to 45%”, though the numbers “varied heavily depending on the actual workload, size of RAM, number of CPUs, memory pressure, etc.” Facebook ran it in production “for several months with different workloads and haven’t seen any noticeable regressions” before it was proposed upstream.

The object that makes this possible is struct obj_cgroup, deliberately tiny:

struct obj_cgroup {
	struct percpu_ref refcnt;      /* cheap per-CPU reference counting */
	struct mem_cgroup *memcg;      /* who to charge — REWRITTEN on reparent */
	atomic_t nr_charged_bytes;     /* fractional page left over */
	union {
		struct list_head list; /* protected by objcg_lock */
		struct rcu_head rcu;
	};
};

The charge path (obj_cgroup_charge() in mm/memcontrol.c) works in bytes, not pages:

  1. Try to satisfy the request from a per-CPU byte stock (consume_obj_stock()). Most allocations end here with no atomic operation at all.
  2. If the stock is short, round the request up to whole pages, charge those pages through the ordinary obj_cgroup_charge_pages()try_charge() path, and refund the fractional remainder into the stock: refill_obj_stock(objcg, PAGE_SIZE - nr_bytes, false). The comment in the source notes the bookkeeping consequence — “The maximum reachable value of the pre-charged bytes is (sizeof(object) + PAGE_SIZE - 2) if there is no data race.”
  3. Each accounted object also pays for its own metadata. obj_full_size() adds the space used “to store obj_cgroup membership” to the charge, so the per-object back-pointer that makes shared slab pages possible is itself charged to the cgroup that benefits from it.
erDiagram
    MEM_CGROUP ||--|| OBJ_CGROUP_ACTIVE : "objcg (RCU pointer)"
    MEM_CGROUP ||--o{ OBJ_CGROUP_INHERITED : "objcg_list (from dead children)"
    OBJ_CGROUP_ACTIVE ||--o{ SLAB_OBJECT : "charged in bytes"
    SLAB_PAGE ||--|{ SLAB_OBJECT : "holds many, from many cgroups"
    SLAB_PAGE }o--|| GLOBAL_KMEM_CACHE : "one shared cache per type"
    MEM_CGROUP {
        page_counter memory "the bytes ultimately land here"
        int kmemcg_id
    }
    OBJ_CGROUP_ACTIVE {
        percpu_ref refcnt
        ptr memcg "rewritten on reparent"
        atomic nr_charged_bytes "fractional page"
    }
    SLAB_OBJECT {
        ptr slabobj_ext "back-pointer to its objcg"
        bytes obj_full_size "object + its own metadata"
    }

How a shared slab page is accounted to many cgroups at once. What it shows: the global kmem_cache owns the page; each individual object carries a pointer to an obj_cgroup; the obj_cgroup carries a pointer to the mem_cgroup that pays. Two levels of indirection instead of one. The insight to take: the extra hop exists for two reasons at once — it lets one physical page carry objects belonging to different containers, and it makes cgroup teardown O(1). Redirecting a dead cgroup’s kernel memory to its parent is a single WRITE_ONCE(iter->memcg, parent) per objcg, not a walk over millions of objects.

Reparenting: what happens to kernel memory when a cgroup dies

When a cgroup directory is removed, its css goes offline but its mem_cgroup cannot be freed while charges remain. For kernel memory, memcg does not wait — memcg_offline_kmem() calls memcg_reparent_objcgs(memcg, parent), which:

  1. Detaches the cgroup’s active objcg (rcu_replace_pointer(memcg->objcg, NULL, true)).
  2. Walks the cgroup’s objcg_list — which contains its own objcg plus any objcgs it previously inherited from its own dead children — and rewrites each one’s memcg pointer to the parent.
  3. Splices that whole list onto the parent’s objcg_list, so the parent inherits the obligation to reparent them again when it dies.
  4. Kills the refcount on the now-detached objcg, so it is freed once the last object charged through it is released.

The list splice in step 3 is what makes this work recursively: charges from a grandchild that died three cgroup generations ago keep bubbling up one level at a time rather than being lost or re-walked. The same treatment is applied to list_lru state (memcg_reparent_list_lrus()) and to shrinker deferred counts (reparent_shrinker_deferred()), which is why the memcg-aware dentry/inode shrinkers described in Shrinkers and Slab Reclaim do not leak work items when containers churn.

Zombie memcgs. Page-cache and anonymous charges are not reparented. A mem_cgroup whose directory is long gone stays allocated until the last page charged to it is freed, which — for page cache that nothing is pressuring — can be indefinitely. These are the “zombie” or “dying” memcgs visible as a growing count in /sys/fs/cgroup/cgroup.stat’s nr_dying_descendants. The mitigation the kernel does apply is the ID recycling described earlier: an offlined memcg gives up its 16-bit id immediately so the 64 k id space is not exhausted by corpses. The memory itself is only released by reclaim, which is why a host that creates and destroys many short-lived containers benefits from occasional proactive reclaim on the parent cgroup.


The Life of a Charge {#the-life-of-a-charge}

The single rule that governs charge ownership is stated in one sentence in the documentation’s “Memory Ownership” section: “A memory area is charged to the cgroup which instantiated it and stays charged to the cgroup until the area is released” (cgroup-v2.rst). This is first-touch ownership, and everything surprising about memcg accounting follows from it.

The immediate corollary is that process migration and memory migration are decoupled: “Migrating a process to a different cgroup doesn’t move the memory usages that it instantiated while in the previous cgroup to the new cgroup.” Move a 4 GiB Java process from cgroup A to cgroup B and A’s memory.current does not drop by 4 GiB — B starts at zero and A keeps paying until those pages are freed. cgroup v1 had an opt-in knob, memory.move_charge_at_immigrate, that attempted to recharge on move; it was one of the features the 2024 LSFMM+BPF session agreed “makes maintenance of the memory controller harder and should be removed”, and a deprecation warning was added in 6.3 and backported (LWN 974575). cgroup v2 never had it.

The second corollary concerns shared pages, and the documentation is unusually candid about it: “A memory area may be used by processes belonging to different cgroups. To which cgroup the area will be charged is in-deterministic; however, over time, the memory area is likely to end up in a cgroup which has enough memory allowance to avoid high reclaim pressure.” The self-correcting mechanism is worth understanding: a shared page charged to a memory-starved cgroup gets reclaimed early, and whichever cgroup faults it back in next becomes the new owner. Over enough cycles ownership drifts toward whoever can afford it. The documented escape hatch when you cannot wait for that drift is POSIX_FADV_DONTNEED — “If a cgroup sweeps a considerable amount of memory which is expected to be accessed repeatedly by other cgroups, it may make sense to use POSIX_FADV_DONTNEED to relinquish the ownership of memory areas belonging to the affected files to ensure correct memory ownership.”

stateDiagram-v2
    [*] --> Uncharged
    Uncharged --> ChargedToA : first touch by a task in cgroup A<br/>try_charge() + folio_memcg set
    ChargedToA --> ChargedToA : task moves A → B<br/>(charge does NOT follow)
    ChargedToA --> ChargedToA : another cgroup reads the page<br/>(no recharge; A still pays)
    ChargedToA --> Reclaimed : reclaim evicts it<br/>(anon → swap, file → dropped)
    Reclaimed --> ChargedToB : next faulter is in B<br/>B becomes the new owner
    Reclaimed --> ChargedToA : A faults it back first
    ChargedToA --> Uncharged : page freed<br/>uncharge + css_put
    state ZombieNote as "if A's directory was rmdir'd,<br/>A's mem_cgroup stays allocated<br/>(dying/zombie memcg) until this happens"
    ChargedToA --> ZombieNote : rmdir A
    ZombieNote --> Uncharged : last page freed → memcg destroyed
    note right of ChargedToA
        Kernel (slab) charges are the exception:
        on rmdir they are REPARENTED to A's parent
        immediately, by rewriting objcg->memcg.
    end note

The lifecycle of one page’s charge. What it shows: there are exactly two events that change who pays — the page being instantiated, and the page being freed and re-instantiated. Task migration is not one of them, and neither is a second cgroup reading the page. The insight to take: the only supported way to hand off ownership of shared page cache is to drop the pages and let the intended owner fault them back. This is why “the sidecar that warmed the cache owns the workload’s page cache” is a real, recurring production problem rather than a hypothetical — and why the kernel treats slab differently, reparenting it eagerly instead of waiting for the pages to die.


Swap and zswap Accounting

cgroup v2’s swap model follows directly from the design argument above: swap is a distinct physical resource, so it gets its own page_counter and its own files rather than being folded into the memory number. A cgroup can therefore be capped at 4 GiB of RAM while being permitted 8 GiB of swap, or — increasingly common on Kubernetes nodes — permitted RAM but denied swap entirely.

The two swap ceilings mirror the memory ceilings, but the documentation is emphatic that they are not used the same way:

FileSemantics (verbatim, condensed)
memory.swap.max“Swap usage hard limit. If a cgroup’s swap usage reaches this limit, anonymous memory of the cgroup will not be swapped out.”
memory.swap.high“Swap usage throttle limit. If a cgroup’s swap usage exceeds this limit, all its further allocations will be throttled to allow userspace to implement custom out-of-memory procedures.”

The docs then draw the distinction explicitly: memory.swap.high “marks a point of no return for the cgroup. It is NOT designed to manage the amount of swapping a workload does during regular operation. Compare to memory.swap.max, which prohibits swapping past a set amount, but lets the cgroup continue unimpeded as long as other memory can be reclaimed. Healthy workloads are not expected to reach this limit.” So memory.swap.max is the routine sizing knob and memory.swap.high is a tripwire for a userspace killer — the reverse of the memory.high/memory.max relationship, and a genuine trap for anyone who pattern-matches on the names.

Lowering a swap limit below current usage does not force immediate eviction: “When reduced under the current usage, the existing swap entries are reclaimed gradually and the swap usage may stay higher than the limit for an extended period of time. This reduces the impact on the workload and memory management.”

memory.swap.events exposes three counters, and the third is the one worth alerting on: high and max count threshold crossings, while fail counts “the number of times swap allocation failed either because of running out of swap system-wide or max limit” — a signal that conflates a cgroup-local misconfiguration with a host-wide swap exhaustion, so it should always be read alongside the host’s swap free space.

zswap is accounted separately again

zswap compresses pages into a RAM-backed pool instead of writing them to a swap device. That pool is real memory, so memcg accounts it — and gives it its own ceiling. Three files:

  • memory.zswap.current — “The total amount of memory consumed by the zswap compression backend” for this subtree. This is compressed size; the corresponding uncompressed figure appears in memory.stat as zswapped (“Amount of application memory swapped out to zswap”), while memory.stat’s zswap key matches the compressed pool figure. The ratio between them is the compression ratio the workload is actually achieving.
  • memory.zswap.max — a hard cap on the pool: “If a cgroup’s zswap pool reaches this limit, it will refuse to take any more stores before existing entries fault back in or are written out to disk.”
  • memory.zswap.writeback — default 1, and hierarchical (“the writeback would be implicitly disabled for child cgroups if the upper hierarchy does so”). Setting it to 0 means “all swapping attempts to swapping devices are disabled,” including both zswap writeback and swapping caused by zswap store failures. The docs flag the sharp edge: “If the zswap store failures are recurring (for e.g if the pages are incompressible), users can observe reclaim inefficiency after disabling writeback (because the same pages might be rejected again and again).”

The last file is subtler than it looks and the documentation says why: it “is subtly different from setting memory.swap.max to 0, as it still allows for pages to be written to the zswap pool.” In other words, memory.zswap.writeback=0 gives you a compressed tier with no disk backing — memory pressure is absorbed by compression only — whereas memory.swap.max=0 disables the whole anonymous-reclaim path. The zswap-side events show up in memory.stat as zswpin, zswpout and zswpwb (“Number of pages written from zswap to swap”), and a rising zswpwb with memory.zswap.writeback=1 is the signature of a zswap pool that is too small for the workload.

flowchart TB
  A["reclaim picks an anonymous page<br/>in this cgroup"]
  A --> Z0{"zswap enabled<br/>and memory.zswap.current<br/>&lt; memory.zswap.max?"}
  Z0 -->|yes| Z1["compress into the zswap pool<br/>memory.zswap.current += compressed size<br/>memory.stat: zswpout++, zswapped += page"]
  Z0 -->|"no, or pool full,<br/>or incompressible"| S0{"memory.swap.current<br/>&lt; memory.swap.max?"}
  Z1 --> Z2{"pool under pressure?<br/>memory.zswap.writeback == 1?"}
  Z2 -->|yes| S0
  Z2 -->|"writeback = 0"| Z3["page stays compressed in RAM.<br/>Store failures recur if incompressible<br/>→ reclaim inefficiency"]
  S0 -->|yes| S1["write to swap device<br/>memory.swap.current += page<br/>memory.stat: zswpwb++ if from zswap"]
  S0 -->|"no (incl. memory.swap.max = 0)"| S2["anonymous memory of this cgroup<br/>will NOT be swapped out.<br/>memory.swap.events: max++ / fail++"]
  S2 --> S3["pressure falls entirely on file pages;<br/>if none are reclaimable → cgroup OOM"]

Where a reclaimed anonymous page actually goes, and which cgroup file gates each hop. What it shows: three independent per-cgroup ceilings sit on this path — memory.zswap.max gates the compressed tier, memory.zswap.writeback gates the hop from compressed to disk, and memory.swap.max gates the disk itself. The insight to take: memory.swap.max = 0 does not merely “turn off swap for this container”; it removes anonymous memory from the reclaimable set entirely, so all reclaim pressure is redirected onto page cache, and a cgroup with no reclaimable cache left goes straight to OOM. That is a defensible choice for a latency-sensitive service and a bad default for a batch job.

Two adjacent memory.stat keys complete the picture and are easy to miss: swpin_zero and swpout_zero count pages whose swap I/O was elided entirely “due to the content being detected as zero” — a workload showing large numbers here is swapping freshly-allocated, never-written memory, which usually means the limit is too tight rather than the working set too large.


Configuration — Worked Examples

All of the following assume a unified (v2-only) hierarchy mounted at /sys/fs/cgroup, which is the default on every current mainstream distribution. Check with stat -fc %T /sys/fs/cgroupcgroup2fs means v2, tmpfs means you are on a legacy or hybrid layout (see cgroups v2 Unified Hierarchy).

Bounding a workload by hand

# 1. The memory controller must be enabled in the PARENT before a child
#    can have memory.* files at all. cgroup v2 delegates top-down.
cat /sys/fs/cgroup/cgroup.controllers          # what is available here
echo "+memory" > /sys/fs/cgroup/cgroup.subtree_control
 
# 2. Create the cgroup. It is just a directory; the kernel populates it.
mkdir -p /sys/fs/cgroup/analytics
ls /sys/fs/cgroup/analytics/memory.*           # memory.current, memory.max, ...
 
# 3. Hard ceiling: 2 GiB. Crossing this after reclaim fails -> cgroup OOM.
echo 2G > /sys/fs/cgroup/analytics/memory.max
 
# 4. Throttle band at 1.75 GiB. Between 1.75G and 2G the workload is
#    slowed by direct reclaim + penalty sleeps but is NEVER killed.
#    This is the early-warning window that memory.max alone does not give you.
echo 1750M > /sys/fs/cgroup/analytics/memory.high
 
# 5. Best-effort protection: keep 512 MiB safe from GLOBAL reclaim, but
#    let the kernel take it if the host is truly out of options.
#    Use memory.low, not memory.min, unless you accept host-wide OOM risk.
echo 512M > /sys/fs/cgroup/analytics/memory.low
 
# 6. Deny swap entirely for this workload (latency-sensitive service).
echo 0 > /sys/fs/cgroup/analytics/memory.swap.max
 
# 7. Move a running process tree in. Note: this moves the TASKS, not
#    their existing charges — see "The Life of a Charge".
echo $$ > /sys/fs/cgroup/analytics/cgroup.procs

Two ordering rules are worth internalising. Writing +memory to a parent’s cgroup.subtree_control is what creates the memory.* files in its children — a fresh mkdir under a cgroup whose parent has not enabled the controller yields a directory with no memory knobs at all, which reads like a kernel bug and is not one. And step 4 must satisfy memory.high < memory.max; the kernel will accept high > max, but the configuration is meaningless because max will always fire first.

Reading the state back

cd /sys/fs/cgroup/analytics
 
cat memory.current            # exact bytes, safe to alert on
cat memory.peak               # high-water mark since creation/reset
echo reset > memory.peak      # resets the watermark FOR THIS FD only
 
# Composition: is this a real working set, or reclaimable cache?
grep -E '^(anon|file|kernel|slab|sock|shmem|zswap) ' memory.stat
 
# Has the kernel actually intervened? These are the numbers that matter.
cat memory.events
#   low 0            <- protection was breached under pressure
#   high 1284        <- throttled this many times (expected if high is set)
#   max 3            <- came within a hair of the hard limit
#   oom 1            <- an allocation was about to fail
#   oom_kill 1       <- a process died
#   oom_group_kill 0 <- the whole group was killed as a unit
 
# Is the workload actually hurting, or merely busy?
cat memory.pressure
#   some avg10=... avg60=... avg300=... total=...
#   full avg10=... avg60=... avg300=... total=...

The memory.events block above is the diagnostic core of memcg, and the mapping from counter to conclusion is mechanical:

Counter risingWhat it meansUsual action
lowGlobal reclaim took memory you asked to protect — the protection is over-committed across the hostReduce memory.low somewhere, or add RAM
highThe cgroup is being throttled. Expected and healthy if you set memory.high deliberatelyNothing, unless latency is unacceptable
maxReclaim only just saved it. You are one bad allocation from a killRaise memory.max or shrink the workload
oomAn allocation was about to failInvestigate now
oom_killA process was killedPost-mortem; check dmesg for the memcg OOM report
oom_group_killThe whole cgroup was killed together (memory.oom.group=1)See cgroup OOM and memory.oom.group

Crucially, high rising is not an incident and max rising is. A monitoring rule that alerts on “memory.events changed” will drown you in throttle events; alert on max, oom and oom_kill, and graph high alongside memory.pressure to size the limit.

Proactive reclaim

# Ask the kernel to reclaim 512 MiB from this subtree, right now.
echo "512M" > memory.reclaim
 
# Same, but bias hard toward evicting anonymous pages to swap
# instead of dropping page cache (see [[Swappiness and Reclaim Balance]]).
echo "512M swappiness=180" > memory.reclaim

memory.reclaim returns -EAGAIN (visible as a write error) if the kernel freed less than you asked for, and the docs caution that “the kernel can over or under reclaim from the target cgroup.” This is the mechanism a memory-tiering or overcommit agent uses to squeeze cold memory out of an idle container ahead of demand rather than during a crisis; the reclaim side of it is covered in Per-cgroup Reclaim and Memory Pressure and Memory Reclaim Overview.

The systemd equivalent

On a systemd host you almost never write these files directly — systemd owns the hierarchy and will overwrite manual edits on the next daemon reload. The unit directives map one-to-one:

[Service]
MemoryMin=512M      # -> memory.min
MemoryLow=1G        # -> memory.low
MemoryHigh=3G       # -> memory.high
MemoryMax=4G        # -> memory.max
MemorySwapMax=0     # -> memory.swap.max
MemoryZSwapMax=1G   # -> memory.zswap.max
MemoryZSwapWriteback=no  # -> memory.zswap.writeback

See systemd and cgroup Integration for how systemd builds the slice hierarchy these land in, and cgroup Delegation and the No Internal Process Rule for why an unprivileged manager can only touch a delegated subtree.


cgroup v1 vs v2 {#cgroup-v1-vs-v2}

The cgroup-v1 memory controller is deprecated, and its own documentation says so in a banner at the top of the file: “This document is hopelessly outdated and it asks for a complete rewrite” (cgroup-v1/memory.rst).

As of v6.12 the entire v1 controller lives in its own translation unit, mm/memcontrol-v1.c, gated behind CONFIG_MEMCG_V1. That option is default n, with help text that states the intent plainly: “Legacy cgroup v1 memory controller which has been deprecated by cgroup v2 implementation. The v1 is there for legacy applications which haven’t migrated… If you do not have any such application then you are completely fine leaving this option disabled” (init/Kconfig).

The option’s introduction can be dated precisely by checking init/Kconfig at successive tags: it is absent in v6.9 and v6.10, and present from v6.11 onward (still present in v6.18). So CONFIG_MEMCG_V1 first shipped in Linux 6.11 (released 2024-09), one release before the 6.12 LTS this note is pinned to.

The substantive interface differences:

Concerncgroup v1cgroup v2
Hard limit filememory.limit_in_bytesmemory.max
Usage filememory.usage_in_bytesmemory.current
Setting a limit below usageRacy — concurrent charges could make the write failSets the limit first to block new charges, then reclaims/OOMs until met
Swap accountingCombined memory.memsw.limit_in_bytes (memory + swap as one number)Separate memory.swap.max with its own page_counter
Soft protectionmemory.soft_limit_in_bytes — global rbtree, no hierarchy, over-reclaimsmemory.low / memory.min — top-down reserves, proportional pressure above the floor
Throttle-before-killnone — hard limit and OOM onlymemory.high (graduated throttle, never OOMs)
Kernel memorymemory.kmem.* knobs; kernel hard limit unsupported since 5.16accounted by default via obj_cgroup, no separate limit file
Socket memoryseparate tcpmem counter and memory.kmem.tcp.* limitscharged through the ordinary try_charge() against memory.max, labelled as the sock stat
Recharge on task movememory.move_charge_at_immigrate (deprecation warning added in 6.3)none — first-touch ownership, charge never moves
OOM controlmemory.oom_control (notification + OOM-disable)memory.oom.group + memory.events; no way to disable the OOM killer
Pressure notificationmemory.pressure_level + cgroup.event_control (vmpressure)memory.pressure (PSI)
Limit-failure counter*.failcntnot exposed; use memory.events max
Hierarchyopt-in memory.use_hierarchyalways hierarchical

The single most consequential difference for understanding charging remains swap accounting: v1 folds memory and swap into one memsw counter and limits their sum, while v2 gives swap its own counter and its own limit. This is why struct mem_cgroup declares swap and memsw as a union — a given kernel build and runtime uses one or the other, never both.

It is also the one v1 feature with no v2 replacement. At the 2024 LSFMM+BPF session, a participant argued that “without that… applications simply cannot know how much swap space they need”; Google is still using it, “and there does not seem to be a way to create a good replacement.” A -google mount option for cgroupfs was floated and rejected by Michal Hocko on the grounds that it “would cause it to never go away” (LWN 974575). If you are on v1 specifically for memsw, you are the one class of user with a defensible reason to still be there.

The general cgroup v1→v2 redesign — multiple hierarchies, thread granularity, the no-internal-process rule — is a separate topic covered in cgroups v1 vs v2; this table is only the memory controller’s slice of it.


Failure Modes and Common Misunderstandings

memory.current says 3 GiB but the process only has 400 MiB resident.” Both numbers are right. memory.current includes page cache the workload caused to be read, and page cache is reclaimable — a container that has streamed a 3 GiB file will sit near its limit indefinitely without being in any danger. This is why container platforms report a working set metric (roughly memory.current minus inactive file pages) rather than memory.current itself. Diagnose with grep -E '^(anon|file|inactive_file) ' memory.stat: if file dominates and anon is small, the cgroup is holding cache, not leaking.

A sidecar owns the main container’s page cache. First-touch ownership means whichever cgroup reads a shared file first pays for the cache. An init container that warms a dataset, a logging agent that tails the same files, or a backup job that reads a database’s data directory can all end up owning gigabytes that “morally” belong elsewhere — and then get reclaimed or OOM-killed for it. There is no recharge mechanism; the documented remedies are to have the intended owner touch the data first, or to call POSIX_FADV_DONTNEED from the accidental owner so the pages are dropped and re-instantiated by the next reader.

memory.high can livelock a fast allocator. This is the most instructive real-world memcg failure, and it is documented in Kubernetes’ own Memory QoS enhancement proposal after the feature’s Beta promotion was cancelled in v1.28. The finding: “for a container process trying to allocate large chunks of memory, once the memory.high level is reached, it doesn’t progress further and stays stuck indefinitely… the process gets stuck because its memory consumption rate is faster than what the memory reclaim can recover. This creates a livelock situation where the process rapidly consumes the memory reclaimed by the kernel causing the memory usage to reach memory.high level again, leading to another round of memory reclamation” (KEP-2570). Because memory.high never OOMs, the workload never reaches the memory.max intervention point either — it just stops making progress. Kernels from 5.9 onward carry a fix, and the kubelet now emits a warning on older kernels; the diagnostic signature is the high counter in memory.events “incrementing rapidly” while the process makes no forward progress.

memory.min equal to memory.max guarantees an OOM kill. If a cgroup’s protection floor equals its ceiling, the kernel cannot reclaim anything inside the cgroup — memory.min means “won’t be reclaimed under any conditions” — so the first allocation that would exceed memory.max has no reclaim path and goes straight to the OOM killer. This is not hypothetical: it is a documented hazard of enabling Kubernetes’ memoryReservationPolicy: TieredReservation for Guaranteed pods, where requests equal limits by definition and therefore memory.min equals memory.max. The KEP spells out the consequence — “the kernel cannot reclaim page cache within the cgroup to make room for new allocations, causing OOM kills when the cgroup reaches memory.max” — and notes that Burstable pods escape it only because memory.low is soft. If you set memory.min, leave headroom below memory.max for the cgroup’s own page cache.

Alerting on memory.events without distinguishing the keys. high rising means the throttle you configured is doing its job; on a busy Burstable workload it can increment thousands of times an hour. max, oom and oom_kill are the incident signals. Compounding this, memory.events is hierarchical: a count on a parent may come from any descendant. Read memory.events.local when you need to attribute an event to a specific cgroup.

Reading memory.stat as if it were a partition of memory.current. Discussed above: the keys overlap, some are event counters rather than byte counts, some are subsets of others, and the whole file lags the exact counter by up to MEMCG_CHARGE_BATCH × nr_cpus pages for up to two seconds. Reconciliation arithmetic on these files produces false alarms.

memory.swap.high is not the swap analogue of memory.high. memory.high is the routine control knob; memory.swap.high is a tripwire — “a point of no return… Healthy workloads are not expected to reach this limit.” The routine swap sizing knob is memory.swap.max. Setting memory.swap.high the way you would set memory.high will throttle all of a cgroup’s allocations, not just swapping.

Zombie cgroups and the 64 k limit. Deleting a cgroup does not free its mem_cgroup while page-cache or anonymous charges remain; those show up as nr_dying_descendants in the parent’s cgroup.stat, described in the docs as remaining “in dying state for some time undefined time (which can depend on system load) before being completely destroyed.” A host that creates and destroys containers rapidly can accumulate thousands. The 16-bit memcg ID space (64 k) is protected because IDs are recycled at offline time rather than at free time, but the struct mem_cgroup allocations and their per-node arrays are not. Proactive reclaim on the parent (memory.reclaim) is the practical way to drain them.

Setting memory.max and nothing else. This is the v1 mental model transplanted onto v2, and it is the configuration the v2 design was written to argue against: it gives you no throttle band, no early-warning latency signal, and no protection floor. The kernel’s own guidance is the opposite — “memory.high is the main mechanism to control memory usage” and memory.max “serves mainly to provide the final safety net.”

Uncertain

Verify: that the memory.high livelock is fully resolved by the 5.9-era fix rather than merely mitigated. Reason: the claim comes from KEP-2570’s prose (“With the kernel 5.9+ fix preventing livelock…”) and the kubelet’s version check, not from an identified kernel commit; the LKML thread it cites (lkml.org/lkml/2023/6/1/1300) was not retrievable during this research, and lore.kernel.org is currently behind a proof-of-work bot wall that blocks curl. To resolve: locate the commit that introduced the memory.high reclaim-progress bail-out in mm/memcontrol.c and read its changelog, then re-check whether later reports of memory.high stalls exist against 6.x. uncertain


Production Notes — How Kubernetes Actually Sets These Files

The kernel side of memcg does not know or care who writes its files. In practice, on a Kubernetes node, the writer is the kubelet, and the mapping from the familiar requests/limits fields to cgroup files is worth knowing concretely — both because it demystifies pod behaviour and because the mapping has changed substantially and recently.

A pod’s QoS class is derived from its requests and limits, and the class then selects which memcg knobs the kubelet writes. Guaranteed requires that “every Container in the Pod must have a memory limit and a memory request, both greater than zero” and that “the memory limit must equal the memory request” (and likewise for CPU). Burstable is anything with some request or limit that is not Guaranteed. BestEffort has none at all.

flowchart TB
  POD["Pod spec<br/>resources.requests.memory<br/>resources.limits.memory"]
  POD --> QOS{"QoS class"}
  QOS -->|"requests == limits<br/>on every container"| G["Guaranteed"]
  QOS -->|"some request or limit"| B["Burstable"]
  QOS -->|"none"| BE["BestEffort"]

  G --> GMAX["memory.max = limits.memory"]
  G --> GMIN["memory.min = requests.memory<br/>(only if memoryReservationPolicy:<br/>TieredReservation)"]
  G --> GHI["memory.high NOT set<br/>— nothing to throttle:<br/>requests == limits"]

  B --> BMAX["memory.max = limits.memory<br/>(unset → no ceiling)"]
  B --> BLOW["memory.low = requests.memory<br/>(only if TieredReservation)"]
  B --> BHI["memory.high = floor[(requests +<br/>f × (limits − requests)) / pageSize]<br/>× pageSize"]

  BE --> BEHI["memory.high computed against<br/>node allocatable instead of limits"]

From a pod manifest to cgroup files. What it shows: memory.max comes straight from limits.memory for every class; the protection and throttle knobs are what QoS actually selects between, and Guaranteed pods are deliberately exempted from memory.high throttling. The insight to take: “requests” and “limits” are not two ends of one knob — in cgroup-v2 terms, requests becomes a reclaim-protection floor (memory.min or memory.low) and limits becomes an allocation ceiling (memory.max). They act on entirely different kernel mechanisms, which is why a pod can be simultaneously well within its limit and starved of memory it “requested”.

The formulas come from KEP-2570, Memory QoS, which as of Kubernetes v1.37 has the MemoryQoS feature gate at Beta, enabled by default, with a documented kernel floor of 5.9+. Two KubeletConfiguration fields drive it:

  • memoryThrottlingFactor (call it f) — the fraction of the headroom between request and limit at which throttling begins. It defaults to nil in v1.37, meaning no memory.high is written at all unless an operator opts in with a value in (0, 1.0].
  • memoryReservationPolicyNone by default; setting it to TieredReservation is what makes the kubelet write memory.min for Guaranteed and memory.low for Burstable workloads.

The memory.high formula itself is worth walking symbol by symbol, because it was changed once already for good reason:

memory.high = floor[ (requests.memory + f × (L − requests.memory)) / pageSize ] × pageSize
       where L = limits.memory, or node allocatable memory if no limit is set
             f = memoryThrottlingFactor

Reading it: start at the request, take fraction f of the gap up to the limit, and round down to a page boundary. With requests=500Mi, limits=1000Mi, f=0.9, that is 500 + 0.9×500 = 950 Mi — throttling starts 50 Mi below the kill point. The original v1.22 formula was simply f × limits.memory, and the KEP documents exactly why it was abandoned: “if requests.memory = 85, limits.memory=100, and we have a throttling factor of 0.8, then as per the Alpha implementation memory.high = 80. In this case the level at which throttling is supposed to occur i.e. memory.high is less than requests.memory. Hence there won’t be any throttling.” A container whose request sat close to its limit — the KEP calls out “some known Java workloads that use 85% of the memory” — got either no throttling at all or throttling below its own guaranteed request.

The two formulas side by side on a 1000 Mi limit, from the KEP’s own worked table:

Request, factorv1.22: f × limitv1.27+: floor[(req + f×(limit−req))/pageSize] × pageSize
request 800 Mi, f = 0.6600 Mi → below request, no throttling920 Mi
request 850 Mi, f = 0.8800 Mi → below request, no throttling970 Mi
request 500 Mi, f = 0.4400 Mi → below request, no throttling700 Mi

Two further operational details from the KEP that surprise people. First, memory.high is set on the container cgroup, never the pod cgroup: “If a container in a pod sees a spike in memory usage, it could result in total pod-level memory usage to reach memory.high level set at pod-level cgroup. This will induce throttling in other containers as the pod-level memory.high was hit. Hence to avoid containers from affecting each other, we set memory.high for only container-level cgroup.” Second, disabling the feature does not immediately undo it — per-container memory.high is reset to max only when the container is recreated, restarted, or updated via in-place resize; “For already-running containers with no restart or resize, stale memory.high values persist.”

To see any of this on a real node, the paths follow systemd’s slice naming:

# A Burstable pod's cgroup, on a systemd-cgroup node:
cd /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<UID>.slice
cat memory.max memory.low memory.high memory.current
cat memory.events                      # high/max/oom_kill for the whole pod
cat cri-containerd-<CID>.scope/memory.high   # per-container throttle lives here

The kubelet passes these through to the runtime using runc’s Unified field, which has accepted raw cgroup-v2 parameters “since v1.0.0-rc93” — so an operator can also set memcg knobs the kubelet does not manage by writing unified annotations, at the cost of the kubelet later overwriting them during reconciliation.

Finally, the ecosystem above memcg is worth knowing about even though it is out of scope here. Meta runs a userspace OOM killer, oomd, driven by memory.pressure rather than by the kernel’s own OOM path, on the theory that by the time the kernel OOM killer runs the machine has already been unusable for some time; systemd-oomd is the distribution-integrated version of the same idea. Both are covered in cgroup Memory Pressure and oomd. And per the 2024 LSFMM+BPF discussion, memory.high is used in production as a de-facto OOM-disable mechanism — “Johannes Weiner said that Meta is using it that way, and it works; evidently Android also uses memory.high for this purpose” (LWN 974575).


See Also