Per-cgroup Reclaim and Memory Pressure

Per-cgroup (memcg) reclaim is the kernel reclaim machinery re-scoped to a single memory cgroup subtree instead of the whole machine. The exact same LRU-scanning code that runs under global memory pressure — shrink_node, shrink_lruvec, the LRU lists — is pointed at one cgroup by setting scan_control.target_mem_cgroup. It runs in four distinct situations: target reclaim when a cgroup breaches its memory.high (throttle) or memory.max (hard limit); proactive reclaim triggered from userspace by writing to memory.reclaim; the memory.min/memory.low protection that deflects global reclaim away from a cgroup’s pages; and the per-cgroup pressure signal (PSI, exposed at memory.pressure) that tells a management agent how much a cgroup is stalling on memory. This note traces those paths in Linux 6.12 LTS (verified against mm/memcontrol.c, mm/vmscan.c, and Documentation/admin-guide/cgroup-v2.rst at the v6.12 tag, with volatile interfaces spot-checked against v6.18). This is the kernel mechanism that a Kubernetes pod memory limit ultimately bottoms out in (orchestration policy lives in the Kubernetes MOC; this note owns the kernel side).

Context

This note sits in section 14 of the Linux Memory Management MOC. It is the reclaim-path counterpart to its siblings: The Memory Cgroup memcg covers what a memcg is and the controller’s structure; memcg Charging and Limits covers how pages get charged to a cgroup and how the page_counters for memory.current/memory.high/memory.max work; this note covers what happens on the reclaim path when those limits are hit and how global reclaim respects per-cgroup protection. Pressure Stall Information covers the PSI mechanism in general; here we cover only its per-cgroup memory.pressure facet. The generic reclaim flow it builds on is Memory Reclaim Overview, and the lists it scans are The LRU Lists (or Multi-Generational LRU when enabled).

The one idea: reclaim with a target_mem_cgroup

Global reclaim and per-cgroup reclaim are the same code. The pivot is one field in struct scan_control (v6.12, mm/vmscan.c):

struct scan_control {
	...
	struct mem_cgroup *target_mem_cgroup;  /* NULL => global; set => scoped */
	...
};
 
static bool cgroup_reclaim(struct scan_control *sc)   { return sc->target_mem_cgroup; }
static bool root_reclaim(struct scan_control *sc)     { return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); }

When target_mem_cgroup is NULL, reclaim walks every memcg’s lruvec on the node, freeing pages system-wide. When it is set to a cgroup, reclaim iterates only that cgroup and its descendants. The entry point for the scoped case is try_to_free_mem_cgroup_pages() (v6.12, mm/vmscan.c), which builds a scan_control with target_mem_cgroup = memcg and calls the ordinary do_try_to_free_pages → shrink_node → shrink_node_memcgs → shrink_lruvec chain. Everything below shrink_node is shared; only the scope differs.

flowchart TB
  CHARGE["page charge in try_charge()<br/>usage vs limits"]
  HIGH{"usage > memory.high?"}
  MAX{"usage > memory.max?"}
  USER["userspace writes<br/>memory.reclaim"]
  GLOBAL["global reclaim<br/>(kswapd / direct)"]

  CHARGE --> HIGH
  CHARGE --> MAX
  HIGH -->|"yes (batched)"| OVERHIGH["mem_cgroup_handle_over_high()<br/>reclaim + throttle penalty"]
  MAX -->|"yes"| TRYCHARGE["try_charge() reclaim loop<br/>then memcg OOM if stuck"]
  USER --> PROACTIVE["memory_reclaim handler<br/>MEMCG_RECLAIM_PROACTIVE"]

  OVERHIGH --> T2F["try_to_free_mem_cgroup_pages(memcg)"]
  TRYCHARGE --> T2F
  PROACTIVE --> T2F
  T2F --> SNM["shrink_node_memcgs(): iterate subtree"]
  GLOBAL --> SNM
  SNM --> PROT{"memory.min / memory.low<br/>protection check"}
  PROT -->|"below min"| SKIP1["skip (hard protect)"]
  PROT -->|"below low"| SKIP2["skip unless 2nd pass"]
  PROT -->|"above"| SL["shrink_lruvec(): scan LRU,<br/>scaled by overage"]

The four entry points into per-cgroup reclaim and where they converge. What it shows: memory.high (throttle, batched), memory.max (hard limit, can OOM), and memory.reclaim (proactive) all funnel into try_to_free_mem_cgroup_pages with a target memcg; global reclaim enters the same shrink_node_memcgs from the other side. The single shared chokepoint is the protection check, where memory.min/memory.low can deflect reclaim. The insight: the limits differ only in what they do when reclaim fails — high throttles, max OOM-kills, reclaim returns -EAGAIN — but the reclaim work itself is one code path scoped by target_mem_cgroup.

shrink_node_memcgs: the subtree iteration

The hinge between “reclaim a node” and “reclaim a cgroup” is shrink_node_memcgs() (v6.12, mm/vmscan.c). It walks the cgroup subtree rooted at target_mem_cgroup and reclaims each member’s lruvec in turn:

static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
{
	struct mem_cgroup *target_memcg = sc->target_mem_cgroup;
	struct mem_cgroup *memcg;
	...
	memcg = mem_cgroup_iter(target_memcg, NULL, partial);
	do {
		struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
		...
		mem_cgroup_calculate_protection(target_memcg, memcg);
 
		if (mem_cgroup_below_min(target_memcg, memcg)) {
			continue;                       /* HARD protect: never reclaim */
		} else if (mem_cgroup_below_low(target_memcg, memcg)) {
			if (!sc->memcg_low_reclaim) {
				sc->memcg_low_skipped = 1;  /* SOFT protect: skip 1st pass */
				continue;
			}
			memcg_memory_event(memcg, MEMCG_LOW);
		}
 
		shrink_lruvec(lruvec, sc);          /* the actual LRU scan */
		shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority);
 
		if (!sc->proactive)
			vmpressure(...);                /* feed legacy vmpressure */
 
		if (partial && sc->nr_reclaimed >= sc->nr_to_reclaim) {
			mem_cgroup_iter_break(target_memcg, memcg);
			break;
		}
	} while ((memcg = mem_cgroup_iter(target_memcg, memcg, partial)));
}

Symbol by symbol:

• mem_cgroup_iter(target_memcg, ...) walks target_memcg and all its descendants. For global reclaim target_memcg is the root, so it visits every cgroup; for scoped reclaim it visits only the named subtree. The same loop, two scopes.
• mem_cgroup_lruvec(memcg, pgdat) resolves the per-cgroup-per-node lruvec — the five LRU lists belonging to this cgroup on this node. That is the entire trick: because every memcg has its own lruvec, reclaim of one cgroup never touches another’s pages.
• mem_cgroup_calculate_protection then the below_min/below_low checks implement protection (next section).
• shrink_lruvec does the real scanning (active→inactive demotion and inactive eviction — see The LRU Lists), and shrink_slab reclaims the cgroup’s slab objects (dentries, inodes — see Shrinkers and Slab Reclaim), which are memcg-aware so a cgroup’s slab is reclaimed alongside its pages.
• Partial walks. For direct reclaimers the iterator keeps a cookie (mem_cgroup_reclaim_cookie) so successive invocations resume where they left off, balancing fairness against latency; for kswapd (and memcg_full_walk) it always does a full walk because “reliable forward progress is more important than a quick return to idle” (source comment).

memory.min and memory.low: protection during global reclaim

This is the most important and most misunderstood per-cgroup reclaim behavior. memory.min and memory.low do not trigger reclaim — they deflect it. They tell global reclaim “leave this cgroup’s pages alone.” Per Documentation/admin-guide/cgroup-v2.rst (v6.12):

• memory.min — hard protection. “If the memory usage of a cgroup is within its effective min boundary, the cgroup’s memory won’t be reclaimed under any conditions. If there is no unprotected reclaimable memory available, OOM killer is invoked.” This is the mem_cgroup_below_min → continue branch above: such a cgroup is skipped entirely, and if that starves reclaim of candidates, the OOM killer fires rather than violate min.
• memory.low — best-effort protection. “If the memory usage of a cgroup is within its effective low boundary, the cgroup’s memory won’t be reclaimed unless there is no reclaimable memory available in unprotected cgroups.” This is the mem_cgroup_below_low branch: on the first reclaim pass the cgroup is skipped and sc->memcg_low_skipped is set; if that pass cannot meet the goal, reclaim retries with sc->memcg_low_reclaim = 1 and overrides low protection, emitting a MEMCG_LOW event so the breach is observable.

Both boundaries are hierarchical and proportional, not binary. The cgroup-v2.rst design essay (v6.12) is explicit that this fixes the old soft_limit_in_bytes design, which had “no hierarchical meaning” and was “so aggressive that it… impacts system performance due to overreclaim, to the point where the feature becomes self-defeating.” memory.low is instead “a top-down allocated reserve… A cgroup enjoys reclaim protection when it’s within its effective low… It also enjoys having reclaim pressure proportional to its overage when above its effective low.” The “effective” qualifier matters: a child’s protection is capped by its ancestors’ (memory.min of all ancestors limits the effective min), and if siblings overcommit the parent’s protection, each child gets a share proportional to its usage below the boundary.

The proportional part is implemented in get_scan_count (v6.12, mm/vmscan.c), which scales the per-list scan target nr[] down by how much of the cgroup’s usage sits under protection:

mem_cgroup_protection(sc->target_mem_cgroup, memcg, &min, &low);
if (min || low) {
	unsigned long cgroup_size = mem_cgroup_size(memcg);
	unsigned long protection;
	if (!sc->memcg_low_reclaim && low > min) {
		protection = low;
		sc->memcg_low_skipped = 1;
	} else {
		protection = min;
	}
	cgroup_size = max(cgroup_size, protection);
	scan = lruvec_size - lruvec_size * protection / (cgroup_size + 1);
	scan = max(scan, SWAP_CLUSTER_MAX);
}

Walk the key line scan = lruvec_size - lruvec_size * protection / (cgroup_size + 1): if a cgroup uses exactly its protection amount, protection ≈ cgroup_size so scan ≈ 0 (fully protected); as usage grows above protection, the fraction protection/cgroup_size shrinks and scan rises toward lruvec_size (full pressure). The comment explains why this gradient exists: without it, “scanning aggression becomes extremely binary — from nothing as we approach the memory protection threshold, to totally nominal as we exceed it,” forcing users to set absurdly liberal thresholds. The max(scan, SWAP_CLUSTER_MAX) floor keeps reclaim inching forward so a protected cgroup can still be the OOM victim rather than livelocking reclaim.

Target reclaim: memory.high (throttle) vs memory.max (hard limit)

Both limits live in the charge path (try_charge, memcg Charging and Limits), but they invoke reclaim very differently.

memory.max is the hard limit, enforced synchronously inside try_charge() (v6.12, mm/memcontrol.c). When page_counter_try_charge(&memcg->memory, ...) fails because usage would exceed max, try_charge raises a MEMCG_MAX event and calls try_to_free_mem_cgroup_pages(mem_over_limit, ...) in the allocating task’s own context — wrapped in psi_memstall_enter()/psi_memstall_leave() so the stall is attributed to memory pressure (this is the PSI hook, below). It retries up to MAX_RECLAIM_RETRIES times; if reclaim cannot make room, it invokes the memcg OOM killer scoped to that cgroup. The docs: “If a cgroup’s memory usage reaches this limit and can’t be reduced, the OOM killer is invoked in the cgroup.” max is the backstop that cannot be exceeded for normal allocations (though some allocations bypass it and return -ENOMEM instead).

memory.high is a throttle, not a wall. Per the docs: “Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached.” Its enforcement is batched and deferred: when a charge pushes usage over high, try_charge records the overage in current->memcg_nr_pages_over_high and the work happens later in mem_cgroup_handle_over_high() (v6.12), called on the return-to-userspace path. That function calls reclaim_high() — which reclaims from the cgroup and every ancestor over its own high — and then computes a penalty sleep (calculate_high_delay) proportional to the overage, capped at MEMCG_MAX_HIGH_DELAY_JIFFIES (2 seconds, per the source comment “Clamp the maximum sleep time per allocation batch to 2 seconds”). The deliberate design (docs): “the processes of the cgroup are throttled and put under heavy reclaim pressure” but never killed, so “a management agent has ample opportunities to monitor and take appropriate actions such as granting more memory or terminating the workload.” The throttle slows the workload down gracefully instead of killing it — which is exactly why cgroup-v2.rst calls memory.high “the main mechanism to control memory usage” and recommends memory.max only as the buggy/malicious-app backstop.

Proactive reclaim: memory.reclaim

memory.reclaim (v6.12, Documentation/admin-guide/cgroup-v2.rst and mm/memcontrol.c) is a write-only file letting userspace trigger reclaim on a cgroup with no pressure at all: echo "1G" > memory.reclaim. Its handler memory_reclaim() parses a byte count (and an optional swappiness= nested key), then loops calling try_to_free_mem_cgroup_pages() with reclaim_options = MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE until the target is met or it gives up with -EAGAIN. Two details matter:

1. It converges in quarters. Each iteration requests (nr_to_reclaim - nr_reclaimed) / 4 pages, so it homes in on the target rather than overshooting in one giant pass.
2. MEMCG_RECLAIM_PROACTIVE suppresses pressure side-effects. The docs are explicit: “the proactive reclaim (triggered by this interface) is not meant to indicate memory pressure on the memory cgroup,” so socket-memory balancing and vmpressure are not exercised (note the if (!sc->proactive) guard around vmpressure in shrink_node_memcgs). Proactive reclaim must not look like real pressure or it would poison the very signals an agent uses to make decisions.

The swappiness= nested key lets the caller override the anon/file bias for this one reclaim. The two LTS releases differ here, and the difference is verified against source: in 6.12 the token is exactly "swappiness=%d", parsed with match_int and bounded by MIN_SWAPPINESS(0)–MAX_SWAPPINESS(200) (per v6.12 mm/memcontrol.c memory_reclaim and include/linux/swap.h). The swappiness=max form — documented in 6.18 as range [0-200, max] where max “exclusively reclaims anonymous memory” (v6.18 cgroup-v2.rst) — is a post-6.12 addition and is not accepted in 6.12 LTS. memory.reclaim is the canonical tool for userspace-driven memory tiering / offloading: a userspace agent writes to it to keep a workload at its minimal footprint, watching memory.pressure to know how hard it can push before the workload starts stalling.

Uncertain

I attribute proactive reclaim to “userspace agents” generically; a specific named tool (Meta’s senpai) is commonly cited for this pattern but I did not fetch a primary source naming it in this task. Verify the specific tool name/behavior before asserting it. The mechanism (memory.reclaim + memory.pressure feedback) is verified against the v6.12 source and cgroup-v2.rst. uncertain

Per-cgroup pressure: memory.pressure (PSI)

Every non-root cgroup exposes memory.pressure — a read-only file showing Pressure Stall Information scoped to that cgroup (the system-wide mechanism is Pressure Stall Information). It reports two lines:

some avg10=0.00 avg60=0.00 avg300=0.00 total=12345
full avg10=0.00 avg60=0.00 avg300=0.00 total=6789

Per Documentation/accounting/psi.rst (v6.12) and the original PSI proposal (LWN 759781, “Tracking pressure-stall information”): the some line is the share of wall-clock time in which at least one task in the cgroup was stalled waiting on memory (e.g. blocked in direct reclaim, swap-in, or page-cache refault); the full line is the share in which all non-idle tasks were stalled simultaneously — the CPU does no productive work. The avg10/60/300 are decaying averages over 10/60/300 seconds; total is the cumulative microseconds stalled. The same proposal added the per-cgroup cpu.pressure/memory.pressure/io.pressure files alongside the system-wide /proc/pressure/*.

The plumbing that makes per-cgroup PSI accurate is the psi_memstall_enter()/psi_memstall_leave() brackets you can see wrapping try_to_free_mem_cgroup_pages in both reclaim_high and try_charge (shown above). Whenever a task enters memcg reclaim, that interval is accounted as a memory stall and propagated up the cgroup PSI hierarchy. This is the early-warning signal that the The Memory Cgroup memcg design intends agents to act on: rather than waiting for an OOM kill, an orchestrator watches memory.pressure, and when some/full rise it grants more memory or sheds load. PSI also supports triggers — a process can poll() on memory.pressure for a threshold like “stalled > 150 ms in any 1 s window” and get woken to react; per Meta’s PSI documentation, this is the basis of oomd, “a userspace tool that uses PSI thresholds as triggers to carry out specified OOM actions when memory pressure increases.”

Uncertain

An older “Usage Guidelines” paragraph in cgroup-v2.rst still says “memory pressure monitoring mechanism isn’t implemented yet.” That sentence predates PSI and is stale — the implemented mechanism is precisely memory.pressure/PSI documented elsewhere in the same file and in psi.rst. Treat memory.pressure as the real, shipped per-cgroup pressure signal; the stale line should not be cited as current. uncertain

Pitfalls / Misconceptions

• “memory.min/memory.low cause reclaim.” They do the opposite — they protect a cgroup from global reclaim. The reclaim-triggering limits are high and max. Confusing protection with limiting is the single most common memcg mistake.
• “memory.high is a hard cap.” No. high throttles (penalty sleep + reclaim) and can be breached under extreme pressure; it never OOM-kills. Only memory.max is a hard cap that can invoke the (cgroup-scoped) OOM killer. Use high for graceful control, max as the backstop.
• “memory.reclaim shows up as memory pressure.” It is flagged MEMCG_RECLAIM_PROACTIVE specifically so it does not register as pressure (no vmpressure, no socket-memory balancing). Proactive reclaim is invisible to the pressure signals by design.
• “Per-cgroup reclaim is different code from global reclaim.” It is the same shrink_node/shrink_lruvec code, distinguished only by scan_control.target_mem_cgroup. Global reclaim is just shrink_node_memcgs with the root as the target.
• “PSI full includes the CPU controller.” For memory, full means every non-idle task stalled on memory; it is per-resource. A high memory full at the cgroup level is a direct measure of wasted work and the right trigger for granting memory or invoking oomd — long before an OOM kill.
• “Charging follows the process to its new cgroup.” No. Per cgroup-v2.rst “Memory Ownership,” a page stays charged to the cgroup that first instantiated it; migrating the task does not move already-charged memory (see memcg Charging and Limits).

Alternatives and when they apply

The legacy pressure signal is vmpressure (the cgroup-v1-era notification mechanism, still fed by shrink_node_memcgs), but PSI’s memory.pressure superseded it for v2 workloads because it measures time stalled (actionable) rather than reclaim efficiency (a derived, noisier proxy). The cgroup-v1 controller exposed memory.limit_in_bytes, memory.soft_limit_in_bytes, and memory.pressure_level; the v2 redesign replaced the global-rbtree soft limit with the hierarchical min/low reserves described above precisely because the soft limit was non-hierarchical and over-aggressive. For triggering reclaim from outside the kernel, the alternatives to memory.reclaim are crude (echo 1 > /proc/sys/vm/drop_caches is system-wide and unscoped; setting a low memory.high then raising it works but throttles the workload meanwhile) — memory.reclaim is the purpose-built, scoped, pressure-silent tool.

See Also

• The Memory Cgroup memcg — what a memcg is and the controller structure; this note is its reclaim path.
• memcg Charging and Limits — how pages are charged and the page_counters behind current/high/max.
• Memory Reclaim Overview — the generic reclaim flow that per-cgroup reclaim reuses.
• The LRU Lists — the per-lruvec lists shrink_lruvec scans; the per-memcg lruvec is what scopes reclaim.
• Pressure Stall Information — the PSI mechanism in full; memory.pressure is its per-cgroup facet.
• Swappiness and Reclaim Balance — the anon/file bias memory.reclaim’s swappiness= key overrides.
• cgroup OOM and memory.oom.group — what memory.max reclaim falls through to when it cannot make room.
• Shrinkers and Slab Reclaim — the memcg-aware slab reclaim shrink_node_memcgs runs alongside LRU reclaim.
• Kubernetes MOC — pod memory limits/requests/QoS map onto these knobs (orchestration policy, not kernel mechanism).
• Linux Memory Management MOC — parent map.

Sources

• Linux v6.12 mm/memcontrol.c — try_charge, reclaim_high, mem_cgroup_handle_over_high, memory_high_write, memory_max_write, memory_reclaim, psi_memstall_enter/leave brackets.
• Linux v6.12 mm/vmscan.c — try_to_free_mem_cgroup_pages, shrink_node_memcgs, get_scan_count protection scaling, cgroup_reclaim/root_reclaim.
• Linux v6.12 Documentation/admin-guide/cgroup-v2.rst — memory.min/low/high/max/reclaim/pressure semantics, hierarchical protection design essay, memory ownership.
• Linux v6.12 Documentation/accounting/psi.rst — some/full definitions.
• Linux v6.18 mm/memcontrol.c and cgroup-v2.rst — spot-check that memory_reclaim and the swappiness nested key persist (range documented as [0-200, max] in 6.18; swappiness=max is a post-6.12 addition).
• LWN 759781 (“Tracking pressure-stall information” — PSI some/full and per-cgroup *.pressure files); Meta PSI overview (facebookmicrosites.github.io/psi — oomd uses PSI triggers).