vault

cgroup Memory Pressure and oomd

18 min read

cgroup Memory Pressure and oomd

By the time the kernel’s in-cgroup out-of-memory (OOM) killer fires, the damage is often already done: a workload that has been thrashing for ten seconds — swapping pages out and faulting them straight back in, spending nearly all its CPU in reclaim — is functionally dead long before an allocation finally fails and the kernel kills a task. Userspace OOM management flips the timing. A daemon watches each control group’s Pressure Stall Information (PSI) memory.pressure file, and when a cgroup’s tasks spend too large a fraction of wall-clock time stalled on memory, it proactively SIGKILLs the offending cgroup — earlier, and with a smarter victim choice, than the kernel’s last-resort killer ever could. The two production implementations are Meta’s (Facebook’s) oomd (facebookincubator/oomd), the original PSI-driven killer that PSI itself was built to enable, and systemd-oomd (systemd-oomd.service(8)), the distribution-integrated descendant shipped with systemd since v247 (2020). This note is about those userspace policy engines — how they read pressure, how they pick a victim, and how their config languages differ. The kernel-side machinery they sit on top of — the PSI poll interface, the memory.high/memory.max charge path, the in-cgroup OOM killer — is owned by Pressure Stall Information and cgroup OOM and memory.oom.group; this note cross-links those and does not re-derive them.

Why a Userspace Killer at All

The kernel’s cgroup OOM killer is a backstop, not a policy. It fires only when an allocation is about to fail — when a charge against [[The Memory Cgroup memcg|memory.max]] cannot be satisfied even after per-cgroup reclaim has tried and failed. That is the latest possible moment to act. The kernel deliberately waits that long because killing a process is irreversible and the kernel cannot know which workload is “least valuable” — it has no notion of a batch job versus a latency-sensitive service, no SLO, no business priority. So it scores tasks by a crude memory-footprint heuristic ([[OOM Score and Victim Selection|oom_badness()]]) and kills the biggest.

The problem is that “an allocation is about to fail” is not the same moment as “the machine has become unusable.” A box can be perfectly responsive at 99% memory utilization if nothing is reclaiming, and it can be a thrashing brick at 80% utilization if the working set no longer fits and every access refaults from swap. Utilization does not distinguish the two; stall time does — which is exactly what PSI measures (see Pressure Stall Information for the full mechanism). The insight Meta operationalized is that a userspace agent, fed PSI, can detect the onset of thrash — the rising slope of memory pressure — seconds before the kernel allocator gives up, and can act on it with knowledge the kernel lacks: which cgroup corresponds to a kill-able batch job, which one is protected, which one to spare. The kernel killer remains essential as the floor (it is the only thing that holds when userspace itself is starved of CPU), but the first responder should be userspace.

Mental Model — A Control Loop Above the Kernel

flowchart TB
  subgraph KERNEL["Kernel (per-cgroup files)"]
    PSI["memory.pressure (PSI)<br/>some/full avg10 + total"]
    CUR["memory.current<br/>memory.swap.current"]
    HIGH["memory.high (throttle band)"]
    MAX["memory.max (hard wall)"]
    KOOM["in-cgroup OOM killer<br/>(last resort)"]
  end
  subgraph DAEMON["Userspace OOM daemon (oomd / systemd-oomd)"]
    POLL["poll() on memory.pressure<br/>+ periodic sampling"]
    DETECT["detector: is pressure<br/>above threshold for duration?"]
    RANK["victim ranking:<br/>most pressure / most swap /<br/>biggest growth"]
    KILL["SIGKILL the chosen cgroup<br/>(all tasks / cgroup.kill)"]
  end
  PSI --> POLL --> DETECT
  CUR --> RANK
  HIGH -. "keeps pressure high<br/>without killing" .-> PSI
  DETECT -->|threshold crossed| RANK --> KILL
  MAX --> KOOM
  KILL -. "acts BEFORE" .-> KOOM

The userspace OOM control loop sitting on top of the kernel’s per-cgroup files. What it shows: the daemon reads memory.pressure (the cost signal), memory.current/memory.swap.current (the usage signals), runs a detector that asks “has pressure stayed above threshold long enough?”, ranks candidate cgroups when it fires, and SIGKILLs the whole winning cgroup. The kernel’s own OOM killer (bottom right) is the backstop that only triggers at the memory.max wall. The insight to take: the userspace daemon’s whole reason to exist is to fire earlier and smarter than the kernel backstop — and memory.high (left) is the kernel feature that makes this possible, because it lets a cgroup generate sustained, observable pressure without ever being killed, giving the daemon a window to watch and decide.

The kernel features the daemon leans on

Two cgroup-v2 memory knobs make the whole pattern viable; both are detailed in cgroup OOM and memory.oom.group and recapped here only enough to explain the daemon’s dependence on them:

  • memory.high is a killless throttle band. When a cgroup’s usage crosses memory.high, its tasks are throttled into heavy direct reclaim but are never OOM-killed for it (“Going over the high limit never invokes the OOM killer,” per the v6.12 cgroup-v2 docs). This is the feature explicitly designed for an external monitor: set memory.high below memory.max, and a leaking cgroup will sit in the band generating sustained, measurable memory.pressure — visible to the daemon — instead of either running unchecked or being killed by the kernel. The daemon watches that pressure and decides what to do.
  • memory.max is the hard wall the kernel killer enforces. You keep it set even with a userspace daemon, as the floor.

The daemon also depends on the PSI poll-trigger interface — poll()/epoll on a cgroup’s memory.pressure file with a trigger line of the form "some <stall_us> <window_us>", which wakes the daemon with POLLPRI the instant cumulative stall crosses the threshold. That interface is documented symbol-by-symbol in Pressure Stall Information; the point for this note is only that it is how the daemon avoids busy-polling — it sleeps in poll() and is woken precisely when pressure spikes.

Meta’s oomd — A Plugin-Based Policy Engine

oomd is the original userspace OOM killer, written at Facebook; PSI was added to the kernel substantially to enable it (oomd README). Its defining design choice is that it is not a fixed algorithm but a configurable plugin engine: the operator declares rulesets in JSON, each ruleset pairing detectors (conditions that decide when to act) with actions (what to do, usually which cgroup to kill). This separation is what lets a fleet encode site-specific policy — “protect the system slice, kill the workload that is growing fastest” — rather than accepting one hard-coded heuristic.

The ruleset structure

An oomd config is a list of rulesets. Each ruleset has a name, a detectors array, and an actions array (oomd configuration docs). The detectors array is itself an array of detector groups; a detector group is a named chain of detector plugins. The firing rule is precise: “DETECTOR_GROUPs evaluate true if and only if all DETECTORs in the chain return CONTINUE.” When any detector group in a ruleset fires, the ruleset’s action chain begins executing in sequence; each action returns CONTINUE, STOP, or ASYNC_PAUSE to control whether the chain proceeds. A worked example from the docs:

{
    "name": "memory pressure protection",
    "detectors": [[
        "workload is under pressure and system is under moderate pressure",
        {
            "name": "pressure_rising_beyond",
            "args": {
              "cgroup": "workload.slice",
              "resource": "memory",
              "threshold": "5"
            }
        }
    ]],
    "actions": [{
        "name": "kill_by_memory_size_or_growth",
        "args": {"cgroup": "system.slice/*"}
    }]
}

Reading it: the ruleset is named "memory pressure protection". Its single detector group (the inner array, with a human-readable label string followed by one detector object) uses the pressure_rising_beyond detector — it returns CONTINUE only when the one-minute memory pressure of workload.slice exceeds 5% and is trending upward. Because that is the only detector in the group, the group fires whenever that condition holds; the ruleset then runs its action, kill_by_memory_size_or_growth scoped to children of system.slice. The cgroup glob (system.slice/*) restricts the action’s hunt for a victim to that subtree.

Detector plugins — deciding when

The core detectors (oomd core_plugins docs) each read a different signal:

  • pressure_above — fires when a cgroup’s 10-second memory (or I/O) pressure stays above threshold for duration. The blunt “is it stalling now?” detector.
  • pressure_rising_beyond — fires when the one-minute pressure exceeds threshold for duration and is rising (a fast_fall_ratio argument tunes how quickly a falling signal de-arms it). This catches the onset of thrash rather than its steady state — the early-warning detector.
  • memory_above — fires when a cgroup’s total (or threshold_anon, anonymous-only) memory usage surpasses a limit. Usage-based rather than pressure-based.
  • memory_reclaim — fires when reclaim activity occurred in the cgroup within duration. A proxy for “this cgroup is being squeezed.”
  • swap_free — fires when free swap drops below threshold_pct of total swap. The signal that swap is about to be exhausted.
  • nr_dying_descendants, exists, dump_cgroup_overview — structural/diagnostic helpers (count zombie child cgroups, assert a cgroup is present, log a system overview that always returns CONTINUE so it never blocks the chain).

Action plugins — deciding which cgroup dies

The kill plugins are where oomd’s victim policy lives. Each scans the children of its cgroup argument (optionally recursive) and ranks them by a different metric (core_plugins docs):

  • kill_by_memory_size_or_growth — “Kill the biggest (memory.current - memory.low) child cgroup if larger than size_threshold percent, or kill the fastest growing.” The default workhorse: it favours the cgroup using the most unprotected memory (subtracting memory.low, the protection floor, so a cgroup you have deliberately protected is ranked lower), and falls back to growth rate.
  • kill_by_swap_usage — “Kills the child with the largest swap usage.” Pairs naturally with the swap_free detector: when swap is nearly exhausted, evict whoever is hogging it.
  • kill_by_pressure — “Kills the child generating the most pressure.” Attributes the stall to its largest contributor.
  • kill_by_pg_scan — “Kills the child with the highest page-scan rate” (the cgroup whose pages the kernel is scanning hardest in reclaim — a direct thrash signal).
  • kill_by_io_cost — “Kills the child generating the most I/O cost” (for I/O-pressure rulesets rather than memory).

Critically, these kill the whole child cgroup — all its tasks — not a single process, which is why oomd composes naturally with multi-process workloads (it does not leave the zombie-survivor mess that a single-task kernel kill can; see cgroup OOM and memory.oom.group for that failure mode). A prekill_hook_timeout on a ruleset lets a workload run a graceful-shutdown hook before the SIGKILL.

systemd-oomd — The Distribution-Integrated Killer

systemd-oomd is the descendant that ships in systemd (added in v247, 2020; current systemd is 260.2, released 2026-05-27 per Phoronix). Its own man page states the mandate plainly: “systemd-oomd is a system service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take corrective action before an OOM occurs in the kernel space” (systemd-oomd.service(8)). Where oomd exposes a general plugin engine, systemd-oomd trades flexibility for integration: it is configured not by JSON rulesets but by systemd unit properties plus global defaults in oomd.conf, and it understands the systemd slice hierarchy natively.

What it monitors and the eligibility rules

systemd-oomd only watches units that explicitly opt in via the resource-control properties ManagedOOMSwap= or ManagedOOMMemoryPressure= set to kill (systemd.resource-control(5)). Both default to auto, which means “do not actively monitor this cgroup, but it may still be killed if a monitored ancestor selects it.” Setting one to kill makes the unit a monitoring root: systemd-oomd will watch its descendants and, on a violation, pick a descendant to kill.

The candidate-eligibility rules are strict and worth internalising, because they explain who can actually be killed (systemd-oomd.service(8)):

  • “Only descendant cgroups are eligible candidates for killing; the unit with its property set to kill is not a candidate” (unless one of its ancestors also set the property). You arm a slice; the daemon kills something inside it, never the slice’s own root.
  • “Only leaf cgroups and cgroups with memory.oom.group set to 1 are eligible candidates.” This is the elegant tie-in to the kernel feature: a non-leaf cgroup is only kill-able as a unit if it has declared itself an indivisible workload via [[cgroup OOM and memory.oom.group|memory.oom.group=1]]. Otherwise the daemon descends to leaves. This prevents the daemon from half-killing a multi-cgroup workload.

The two trigger conditions and their defaults

systemd-oomd acts on two distinct signals, each with a default that resolves the uncertainty flags carried by the sibling notes — these are now pinned to the oomd.conf(5) man page (added in the versions noted):

  1. Swap pressure — SwapUsedLimit= (default 90%, since v247). “If the fraction of memory used and the fraction of swap used on the system are both more than what is defined here, systemd-oomd will act on eligible descendant control groups with swap usage greater than 5% of total swap.” So the swap rule is global (it watches whole-system swap+memory fractions), but the victim it then kills is the descendant hogging the most swap.
  2. Memory pressure — DefaultMemoryPressureLimit= (default 60%) over DefaultMemoryPressureDurationSec= (default 30s). “Sets the limit for memory pressure on the unit’s control group… The memory pressure for this property represents the fraction of time in a 10 second window in which all tasks in the control group were delayed.” In other words: when a monitored cgroup’s PSI memory.pressure shows its tasks were fully stalled for more than 60% of a 10-second window, sustained continuously for at least 30 seconds, systemd-oomd kills a candidate in that cgroup. (DefaultMemoryPressureDurationSec= was added in v248 and “Must be set to 0, or at least 1 second”; 0 means “use the 30-second built-in default.”) A per-unit override, ManagedOOMMemoryPressureLimit=<percentage> (0–100%, default 0% meaning “inherit the oomd.conf default”), tunes the threshold per workload and is ignored unless ManagedOOMMemoryPressure=kill (systemd.resource-control(5)).

Resolves a sibling-note uncertainty flag

Both Pressure Stall Information and cgroup OOM and memory.oom.group carry [!warning] Uncertain flags noting their systemd-oomd numbers (“around 60% over 30s”, ManagedOOMMemoryPressureLimit) rested on memory because the man-page fetch had 403’d. The values above are now pinned to the systemd man-page XML sources and the freedesktop man pages, fetched 2026-06-14: SwapUsedLimit=90%, DefaultMemoryPressureLimit=60% over a 10-second PSI full window, DefaultMemoryPressureDurationSec=30s. Those sibling flags can be downgraded on their next edit.

The kill action and victim preference

The kill is unambiguous: systemd-oomd “will select a descendant cgroup and send SIGKILL to all processes under it” (systemd.resource-control(5)) — the whole cgroup dies, as with oomd. The man pages do not spell out the internal ranking metric the way oomd’s kill plugins do (for the memory-pressure rule it kills among the pressure-violating candidates; for the swap rule, among the swap-hogging candidates), but they do expose one explicit policy lever: ManagedOOMPreference=none|avoid|omit (default none, added in v248). avoid means “systemd-oomd will only select this cgroup if there are no other viable candidates”; omit means “systemd-oomd will ignore this cgroup entirely as a candidate.” This is how you protect a critical service — mark it avoid (deprioritise) or omit (untouchable). A caveat the man page stresses: the extended attribute that carries this preference “is not applied recursively to cgroups under this unit’s cgroup,” so you must set it where the kill candidate actually lives.

Uncertain

Verify: the precise victim-ranking metric systemd-oomd uses among multiple memory-pressure candidates (e.g. highest pressure, largest usage, or swap). Reason: the systemd man pages (fetched 2026-06-14) define eligibility and the ManagedOOMPreference ordering lever clearly, but do not publish the internal scoring function the way oomd’s kill_by_* plugins do. To resolve: read src/oom/oomd-util.c / oomd-manager.c in the systemd source tree at the v260 tag for the candidate-sort comparator. uncertain

Inspecting it with oomctl

The companion tool oomctl “can be used to list monitored cgroups and pressure information” (systemd-oomd.service(8)) — it is the operator’s window into what systemd-oomd currently sees: which slices are armed, their current PSI and swap readings, and where the thresholds sit.

Configuration Examples

Arming systemd-oomd on a user slice

# /etc/systemd/system/user@.service.d/oomd.conf  (drop-in)
[Service]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%

Line by line: ManagedOOMMemoryPressure=kill makes the user slice a monitoring root — systemd-oomd will now poll its descendants’ memory.pressure. ManagedOOMMemoryPressureLimit=50% overrides the global 60% default for this slice’s children, so a user session that stalls on memory more than 50% of a 10-second window (for the 30-second default duration) has its worst-offending leaf cgroup SIGKILLed. On Fedora this exact pattern is shipped by default for user@.service and -.slice so that a runaway desktop app is killed before the whole session freezes.

Protecting a critical service

# /etc/systemd/system/database.service.d/oomd.conf
[Service]
ManagedOOMMemoryPressure=auto
ManagedOOMPreference=omit

Here ManagedOOMMemoryPressure=auto means the database is not itself a monitoring root, and ManagedOOMPreference=omit makes it ineligible as a victim even if an ancestor slice is armed — systemd-oomd will pick something else to kill. Use avoid instead of omit if you want it killed only as a last resort.

The global defaults file

# /etc/systemd/oomd.conf
[OOM]
SwapUsedLimit=90%
DefaultMemoryPressureLimit=60%
DefaultMemoryPressureDurationSec=30s

These are the shipped defaults (oomd.conf(5)); the file exists so a site can tighten them fleet-wide (e.g. lower SwapUsedLimit to act sooner on swap exhaustion).

Failure Modes and Misunderstandings

  • No memory.high, no early window. If you set only memory.max and leave memory.high at max, a leaking cgroup jumps from “fine” to “kernel OOM” with no observable pressure band in between — the userspace daemon never gets its early-warning window. The pattern requires memory.high (or a tight memory.max that produces sustained reclaim) to generate the pressure the daemon watches.
  • PSI disabled. If the kernel was built CONFIG_PSI_DEFAULT_DISABLED=y and booted without psi=1, /proc/pressure and the per-cgroup memory.pressure files are absent, and both daemons are inert. oomctl showing no pressure data, or systemd-oomd logging that PSI is unavailable, is the symptom (see Pressure Stall Information for the boot parameter).
  • Nothing is armed. A common systemd-oomd surprise: the service is running but kills nothing, because no unit has ManagedOOMMemoryPressure=kill/ManagedOOMSwap=kill. Default is auto, which only makes a unit a candidate, not a monitoring root. oomctl listing zero monitored cgroups is the tell.
  • Killing the wrong cgroup. Without ManagedOOMPreference (systemd) or a tuned kill_by_* action and memory.low protection (oomd), the daemon may evict a cgroup you cared about. The fix is policy: omit/avoid the protected units, or set memory.low on them so kill_by_memory_size_or_growth ranks them lower.
  • Daemon starved too. Userspace OOM management is an augmentation of the kernel killer, never a replacement. If the daemon itself is CPU-starved during a severe thrash, it may not get scheduled to act — which is exactly why you keep memory.max set so the kernel backstop still holds.
  • Confusing v1 and v2 usage files. These daemons read cgroup-v2 files (memory.current, memory.swap.current, memory.pressure). On a legacy cgroup-v1 host the v2 unified hierarchy and PSI per-cgroup files are absent; both daemons require cgroup v2.

Alternatives and When to Choose Them

  • The kernel cgroup OOM killer alone (cgroup OOM and memory.oom.group) — zero extra moving parts, always present, but acts only at the memory.max wall with a crude victim heuristic. Adequate when you do not care about acting before thrash and a single-task kill is acceptable. Pair it with memory.oom.group=1 to at least make the kill clean.
  • earlyoom — a simpler, older userspace daemon that polls /proc/meminfo for free-memory and free-swap percentages rather than PSI, and kills by oom_score. It predates per-cgroup PSI and is not cgroup-aware; choose it only on systems without cgroup-v2 PSI, or for its minimalism. It cannot make the per-cgroup, pressure-based decisions oomd/systemd-oomd can.
  • oomd vs systemd-oomd. Choose oomd when you need programmable policy — custom detector/action chains, I/O- and pg-scan-based kills, prekill hooks, fleet-specific rulesets (its origin is Meta’s fleet). Choose systemd-oomd when you want zero-config integration with a systemd distro: it is already wired to the slice hierarchy, configured by unit properties, and shipped on by default on Fedora and other systemd distributions. Most servers and desktops use systemd-oomd for exactly that reason; oomd is for operators who want to hand-author the policy.

Production Notes

The pattern’s pedigree is industrial: PSI and oomd were built together at Facebook to run fleet-wide, and oomd is the documented foundation of Meta’s resource-control stack (oomd repository). systemd-oomd brought the idea to the mainstream — Fedora 34 (released spring 2021) enabled it by default for all variants and spins via a FESCo-approved System-Wide Change (Fedora 34 “Enable systemd-oomd by default” change), arming the user slice so a runaway desktop application is killed before the session livelocks, which is the single most visible benefit to an end user: the machine stays responsive through a memory blowup instead of freezing for tens of seconds while the kernel grinds toward an OOM. The operational playbook is: (1) ensure cgroup v2 and PSI are enabled; (2) set memory.high below memory.max on workloads so they produce an observable pressure band; (3) arm the right monitoring roots (ManagedOOMMemoryPressure=kill on the slice that should be policed, or an oomd ruleset scoped to it); (4) protect what must not die (ManagedOOMPreference=omit/avoid, or memory.low + kill_by_* tuning); (5) keep memory.max set as the kernel backstop; (6) watch oomctl (systemd) or the daemon’s logs to confirm it is actually monitoring and acting. In Kubernetes this lives below the orchestration layer — the kubelet wires pod limits into memory.max, and node-level userspace OOM management is a node-configuration concern; the kernel-vs-userspace boundary is the same one drawn here.

See Also

Graph View