Pressure Stall Information (PSI) is a Linux kernel facility that quantifies, in real time, how much productivity is being lost because tasks are stalled waiting on a contended resource — originally CPU, memory and I/O, and since Linux 6.1 also IRQ/softirq time. Rather than reporting raw utilization (which says nothing about whether load is healthy or crushing), PSI measures the wall-clock time during which work could not proceed because of contention, and exports it as both decaying running averages and an absolute stall-time counter through /proc/pressure/{cpu,memory,io,irq} and per-cgroup cpu.pressure/memory.pressure/io.pressure/irq.pressure files (kernel PSI documentation). It was written by Johannes Weiner at Facebook and merged in Linux 4.20 (December 2018, commit eb414681); the poll()-based threshold-trigger interface, contributed by Suren Baghdasaryan, followed in Linux 5.2 (mid-2019) — both dates re-verified here by existence-checking the source at successive release tags rather than taken from changelogs. PSI is the kernel signal that modern userspace out-of-memory daemons such as systemd-oomd and Facebook’s oomd use to act on memory pressure before the in-kernel OOM killer is forced to fire, and — in Meta’s senpai — the feedback term in a control loop that sizes containers automatically.
Two facts are worth carrying away before anything else, because both contradict the most widely-circulated descriptions of PSI and both are measured on live hardware below: memory pressure is not the same thing as reclaim (a memory.high throttle produces 57 % pressure with pgscan = 2), and the kernel’s own documented trigger example returns EINVAL for an unprivileged process (its 1-second window is not a multiple of the 2 seconds unprivileged callers are restricted to).
Feature History, Established by Existence-Checking
An earlier revision of this note flagged two version claims as unverified. Both are now resolved, and the method is worth stating because it is more reliable than release notes, which routinely describe intent rather than what shipped: fetch the same file path at successive release tags and record what comes back. Two signals are available — the HTTP status (404 vs 200 dates a file’s creation or deletion) and the presence of a grepped symbol in the body (dates a feature inside an existing file). A symbol absent at tag N and present at tag N+1 was introduced in N+1. Every row below was established by fetching raw.githubusercontent.com/torvalds/linux/<tag>/kernel/sched/psi.c on 2026-09-11.
The single jump from 22,386 bytes to 35,887 bytes between v5.1 and v5.2 is the entire trigger subsystem arriving at once. The file has been roughly stable at 47 KB since v6.6 (v6.12: 47,321 B; v7.1: 47,625 B).
Version pinning convention for this note: citations are to Linux v6.12, a longterm release (6.12.109 as of 2026-09-11 per kernel.org/releases.json). v6.12 is this vault’s pin, not the only longterm branch — 6.18.51, 6.6.156, 6.1.187 and 5.15.220 are also longterm, and mainline is 7.3-rc2. All live measurements in this note were taken on Linux 7.1.13-200.fc44.x86_64 (Fedora 44), a stable-series kernel; where 7.1 and 6.12 differ the note says so.
Uncertain
Verify: the exact upstream commit and merge window that removed WINDOW_MIN_US in 6.5, and whether Documentation/accounting/psi.rst is intentionally left describing the old 500 ms floor. Reason: the removal was established by existence-checking file contents across tags, which dates the change to a release but does not name the commit; lore.kernel.org returns HTTP 403 to this environment so the patch discussion could not be read. To resolve: git log -S WINDOW_MIN_US kernel/sched/psi.c on a local clone, or fetch the commit through a reachable mirror. The behavioural claim — that 6.12 and 7.1 enforce no lower bound beyond window_us != 0 — is verified directly from source and measured on a running 7.1 kernel (see What the Kernel Actually Accepts — a Measured Validation Matrix). uncertain
Why PSI Exists: Utilization Lies, Stall Time Doesn’t
The classic Linux load and utilization metrics are deeply ambiguous. A load average of 8 on an 8-core box might mean perfect saturation with zero queueing, or a thrashing mess where every task spends most of its life blocked. Memory utilization at 95 % might be a well-tuned cache or the brink of an OOM kill. iowait conflates a disk that is busy-but-keeping-up with one that has become a bottleneck. The kernel’s own top-of-file commentary in kernel/sched/psi.c frames the problem precisely: utilization tells you a resource is in use, but not whether that use is costing you forward progress (psi.c, v6.12).
PSI reframes the question around lost potential. The baseline for productivity is the time a task could be executing on a CPU. Pressure is the fraction of that potential that cannot be realized because of resource contention. This is a strictly more actionable signal: a memory-pressure reading of 30 % over the last 10 seconds means “tasks spent 30 % of recent wall-clock time stalled in the memory subsystem (reclaim, refault, swap-in, thrash)” — a number you can put a service-level objective on, alert on, and feed into a control loop that sheds load, migrates jobs, or kills a low-priority batch process before the box tips into an OOM event (Facebook PSI overview).
Mental Model: SOME vs FULL, Per Resource
flowchart TB
subgraph TASKS["Tasks on a runqueue"]
R["running / productive task"]
D1["task stalled on resource<br/>(reclaim, swap-in, IO wait, runqueue wait)"]
D2["another stalled task"]
end
subgraph STATE["Per-resource stall state"]
SOME["SOME state<br/>at least one task stalled<br/>(workload slowdown)"]
FULL["FULL state<br/>ALL non-idle tasks stalled<br/>CPU goes idle = thrashing"]
end
D1 --> SOME
D2 --> SOME
D1 --> FULL
D2 --> FULL
R -. "as long as something productive runs<br/>FULL is NOT reached" .-> FULL
SOME --> OUT["/proc/pressure/<res><br/>some avg10/avg60/avg300 total"]
FULL --> OUT2["/proc/pressure/<res><br/>full avg10/avg60/avg300 total"]
The two PSI stall states for one resource. What it shows: the SOME state holds whenever one or more tasks are stalled on the resource — this captures workload slowdown even when the CPU is still doing useful work for other tasks. The FULL state is the strictly worse subset: every non-idle task is stalled at once, so the CPU itself goes idle and cycles are wasted outright — the signature of thrashing. The insight to take:SOME answers “is my workload being slowed?”; FULL answers “is the machine grinding to a halt?”. They are tracked separately and exported as separate lines because they demand different responses — a little SOME is normal; sustained FULL is an emergency. Per the kernel docs, cpu has no system-level FULL (it is reported as zero for backward compatibility since it has been present since 5.13), but FULL is meaningful for cpu at the cgroup level.
The two states are defined in the source by two crisp predicates (psi.c model comment):
SOME = nr_delayed_tasks != 0
FULL = nr_delayed_tasks != 0 && nr_productive_tasks == 0
What counts as “productive” differs per resource: for I/O, productive means a running task; for memory, productive means a running task that is not itself a reclaimer (a task spinning in direct reclaim is not making application progress); for CPU, productive means an on-CPU task. This is why a box can show high memory FULL even while a core is technically executing — if that core is executing kswapd-style reclaim rather than the workload, no application progress is being made.
That distinction is not prose hand-waving; it is literally an extra counter in the kernel’s task-state enum. include/linux/psi_types.h at v6.12 tracks four per-CPU task counts, and the comment explains precisely why the fourth exists:
enum psi_task_count { NR_IOWAIT, NR_MEMSTALL, NR_RUNNING, /* * For IO and CPU stalls the presence of running/oncpu tasks * in the domain means a partial rather than a full stall. * For memory it's not so simple because of page reclaimers: * they are running/oncpu while representing a stall. To tell * whether a domain has productivity left or not, we need to * distinguish between regular running (i.e. productive) * threads and memstall ones. */ NR_MEMSTALL_RUNNING, NR_PSI_TASK_COUNTS = 4,};
Reading it symbol by symbol: NR_IOWAIT counts tasks blocked on I/O; NR_MEMSTALL counts tasks inside a memory-stall bracket; NR_RUNNING counts runnable/on-CPU tasks; and NR_MEMSTALL_RUNNING counts the overlap — tasks that are simultaneously running and memory-stalled, i.e. reclaimers burning CPU on the kernel’s behalf rather than the application’s. Memory FULL is computed by subtracting that overlap out, which is the mechanical realisation of “a reclaimer is not productive” (psi_types.h, v6.12).
The Four Resources and Which Lines They Export
The some/full pair is not uniform across resources. There are four resources in v6.12, not three, and two of them export only one of the two lines. This table is the single most useful reference in the note, and it is reconciled against three independent places: the state enum in psi_types.h, the printing logic in psi_show(), and a live read of /proc/pressure/* on a running 7.1 kernel.
Resource
/proc file
some line
full line
What a stall is
“Productive” means
Gate
io
/proc/pressure/io
✅ meaningful
✅ meaningful
task blocked waiting on block I/O (TSK_IOWAIT)
any running task
always
memory
/proc/pressure/memory
✅ meaningful
✅ meaningful
reclaim, refault of a workingset page, swap-in, memory.high throttle (TSK_MEMSTALL)
a running task that is not a reclaimer
always
cpu
/proc/pressure/cpu
✅ meaningful
⚠️ printed, always 0 at system level; meaningful per-cgroup
runnable but not on a CPU
an on-CPU task
always
irq
/proc/pressure/irq
❌ not printed at all
✅ meaningful
time consumed by hardirq/softirq handlers
n/a — there is no “some” notion
CONFIG_IRQ_TIME_ACCOUNTING
Three things in that table routinely surprise people:
IRQ pressure exists and most write-ups predate it.PSI_IRQ was added in 6.1, is conditional on CONFIG_IRQ_TIME_ACCOUNTING, and appears both as /proc/pressure/irq and as a per-cgroup irq.pressure file. It is an outlier in the model: the enum declares only PSI_IRQ_FULL, with no PSI_IRQ_SOME counterpart, and psi_show() handles this with a dedicated only_full flag that halves the print loop. Time stolen by interrupt handling is unproductive for everybody on that CPU simultaneously, so the “some tasks but not all” gradation has no meaning.
Critically, Documentation/accounting/psi.rst never mentions IRQ pressure — not at v6.12, and not at v7.1. The file still opens “Pressure information for each resource is exported through the respective file in /proc/pressure/ — cpu, memory, and io.” The only upstream documentation of irq.pressure is a four-line stanza buried in the cgroup v2 admin guide’s Core Interface Files list, which unhelpfully points the reader back at psi.rst for details that are not there (cgroup-v2.rst, v6.12). If you learned PSI from psi.rst, you do not know IRQ pressure exists.
The CPU full line is printed but hard-zeroed, and that is a deliberate backward-compatibility wart rather than a value. psi.rst states it outright: “CPU full is undefined at the system level, but has been reported since 5.13, so it is set to zero for backward compatibility.” The zeroing is one condition in psi_show() (psi.c, v6.12):
/* CPU FULL is undefined at the system level */if (!(group == &psi_system && res == PSI_CPU && full)) { for (w = 0; w < 3; w++) avg[w] = group->avg[res * 2 + full][w]; total = div_u64(group->total[PSI_AVGS][res * 2 + full], NSEC_PER_USEC);}
Read the guard carefully: the values are copied only if the triple (this is the system group, and the resource is CPU, and we are on the full iteration) is false. When all three hold, avg[] and total keep their zero-initialised values and a line of zeroes is printed. Note the three conjuncts — group == &psi_system is the one that matters. Swap in a cgroup’s psi_group and the guard passes, which is exactly why cgroup-level cpu.pressure full carries real information while the system-level one never can. The live measurement below shows both at the same instant: a throttled cgroup reading full avg10=82.92 while /proc/pressure/cpu reads full avg10=0.00 total=0.
Stall Classification as a State Machine
stateDiagram-v2
direction LR
[*] --> IDLE
IDLE: NONE<br/>nr_nonidle_tasks == 0<br/>CPU idle, nothing to do<br/>(no clock runs)
SOMEONLY: SOME (partial stall)<br/>nr_delayed > 0<br/>AND nr_productive > 0<br/>tSOME clock runs
FULLST: FULL (total stall)<br/>nr_delayed > 0<br/>AND nr_productive == 0<br/>tSOME AND tFULL both run
HEALTHY: PRODUCTIVE<br/>nr_delayed == 0<br/>work is getting done<br/>(no stall clock runs)
IDLE --> HEALTHY: task becomes runnable
HEALTHY --> SOMEONLY: a task enters a stall bracket<br/>(io_schedule / psi_memstall_enter /<br/>waits on a runqueue)
SOMEONLY --> FULLST: the last productive task<br/>also stalls
FULLST --> SOMEONLY: any task resumes<br/>productive work
SOMEONLY --> HEALTHY: last stalled task<br/>is unblocked
HEALTHY --> IDLE: all tasks sleep
FULLST --> IDLE: all stalled tasks exit or sleep
note right of FULLST
FULL is a strict SUBSET of SOME.
tFULL can never exceed tSOME,
so full% <= some% always.
end note
The per-CPU stall state machine, one instance per resource per runqueue. What it shows: each runqueue is, for each tracked resource, in exactly one of four states, and PSI runs a separate nanosecond clock for each of the two stall states. SOME and FULL are not siblings — FULL is the sub-state of SOME in which productivity has dropped to zero, so entering FULL runs both clocks. The insight to take: because FULL is a strict subset, full can never exceed some in any reading; if you ever see full > some you are misreading the file. And the two boundaries answer different questions — the HEALTHY → SOME edge is “my workload got slower”, the SOME → FULL edge is “the machine stopped doing work”. A sustained crossing of the second edge is the thrashing signature that PSI was built to catch.
A subtlety worth internalizing: FULL does not exist for CPU at the system level. If even one task is on a CPU, somebody is advancing, so by definition the whole machine is not CPU-FULL. But inside a cgroup, cpu.pressure full is meaningful: it means every non-idle task in that group is delayed on the CPU resource because it is being consumed by tasks outside the group or throttled by the group’s own cpu.max quota (psi.c comment).
The File Format
Reading /proc/pressure/memory on a 6.12/6.18 kernel built with CONFIG_PSI=y returns (psi.rst, v6.12):
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
Walking the fields symbol by symbol:
some / full — the two stall states described above, one line each. /proc/pressure/cpu has only the some line meaningfully populated at the system level; its full line is present but pinned to zero.
avg10 / avg60 / avg300 — the percentage of wall-clock time spent in that stall state, as an exponentially-decaying running average over a 10-second, 60-second, and 300-second (5-minute) window respectively. These are ratios in percent (0.00–100.00). avg10 reacts to short spikes; avg300 reveals sustained trends. They are the same family of decaying-average math the kernel uses for the classic load average.
total — the cumulative absolute stall time in microseconds since boot (or, for a cgroup, since the cgroup was created). This counter is the most precise signal: a brief but severe latency spike might never move avg10 appreciably yet adds a visible chunk to total. Tooling that wants to average pressure over a custom window samples total at two instants and divides the delta by the elapsed time.
The same three averages plus total appear for every resource — but, as the resource table above established, how many lines you get varies. As a field reference:
Field
Type
Units
Range
Resets
Notes
line label
some | full
—
—
—
full absent for irq; some absent for irq; cpu’s full present but pinned to 0 system-wide
avg10
decaying average
percent
0.00–100.00
never
2 dp fixed-point; reacts to spikes within seconds
avg60
decaying average
percent
0.00–100.00
never
the usual choice for alerting
avg300
decaying average
percent
0.00–100.00
never
trend detection, capacity planning
total
monotonic counter
microseconds
u64
never (system: since boot; cgroup: since cgroup creation)
the precise signal; diff it to get pressure over any window
Two traps live in that table. First, total is in microseconds while the kernel accumulates in nanoseconds — psi_show() divides by NSEC_PER_USEC on the way out, so you lose three digits of resolution at the interface but never inside it. Second, total never resets, not even when pressure returns to zero; the measurements below show a cgroup’s total frozen at its final value long after the workload exited while avg10 decays away. Any tool that treats total as “current pressure” rather than “cumulative stall since epoch” is broken.
The three averages are not independent measurements — they are three exponentially-weighted views of the same 2-second sample stream, differing only in decay constant. Those constants are compile-time literals in psi.c (lines 156–160, v6.12):
Constant
Value
Meaning
Decay per 2 s tick
Sample’s weight after
PSI_FREQ
(2*HZ + 1)
aggregation period ≈ 2 s
—
the tick itself
EXP_10s
1677
1/exp(2s/10s) ≈ 0.8187
×0.819
10 s → ~13.5 % remains
EXP_60s
1981
1/exp(2s/60s) ≈ 0.9672
×0.967
60 s → ~13.5 % remains
EXP_300s
2034
1/exp(2s/300s) ≈ 0.9934
×0.993
300 s → ~13.5 % remains
Walking the arithmetic symbol by symbol: each constant is e^(−2/W) for window W seconds, scaled into the kernel’s FSHIFT fixed-point format (1 << 11 = 2048, which is why every value sits just under 2048 — EXP_300s = 2034 is 0.9934 × 2048). On each 2-second tick the exported average is updated as avg = avg × EXP + sample × (2048 − EXP), all in fixed point. Because e^(−1) ≈ 0.368 and the windows are named for one time constant, a sample is not “forgotten” after avg10’s ten seconds — about 13.5 % of its weight (e^(−2)) survives. This is the same calc_load machinery that produces /proc/loadavg, reused at higher resolution; PSI needs a shorter period than loadavg’s 5 s, hence the odd 2*HZ + 1. If the aggregation worker was descheduled and several periods elapsed at once, calc_avgs() folds in missed_periods in a single calc_load_n() call rather than looping — so a briefly-stalled aggregator produces a correct average, not a spike.
Live Output on This Machine
Read on Linux 7.1.13-200.fc44.x86_64 (Fedora 44, CONFIG_PSI=y, CONFIG_PSI_DEFAULT_DISABLEDnot set, no psi= on the kernel command line) at rest:
Four things to read out of that idle capture, none of which are visible in the documentation:
irq emits a single full line. A parser that assumes two lines per file, or that keys on some, silently drops IRQ data or crashes.
CPU some total=180917952 — 181 seconds of cumulative runqueue-wait since boot — while CPU full total=0 exactly. Not “small”: zero, because it is never accumulated into.
All four files are mode 0666.proc_create("pressure/irq", 0666, ...) and its siblings deliberately grant world write access, because registering a trigger is a write(). Any user can arm a PSI monitor; the CAP_SYS_RESOURCE check gates only how expensive a monitor may be, never whether one may exist.
IRQ total (297 s) exceeds CPU some total (181 s) over the same 110,394 s of uptime. These are not comparable quantities, and the reason is structural rather than statistical: every other PSI state is derived from task-state transitions, but psi_account_irqtime() takes the raw delta from the kernel’s existing IRQ-time accounting and adds it straight into the bucket — groupc->times[PSI_IRQ_FULL] += delta; (psi.c, v6.12) — summed over all 32 CPUs on this box. It is a time measurement grafted onto a stall-state framework. Treat total counters as meaningful when differenced against themselves over time, never when compared across resources.
The Predicates, as Actual Code
The prose definitions above (SOME = nr_delayed_tasks != 0, and so on) are the model comment’s idealisation. The function that actually decides the state mask on every task-state change is eleven lines long, and reading it removes all remaining ambiguity (test_states(), psi.c lines 221–247, v6.12):
static u32 test_states(unsigned int *tasks, u32 state_mask){ const bool oncpu = state_mask & PSI_ONCPU; if (tasks[NR_IOWAIT]) { state_mask |= BIT(PSI_IO_SOME); if (!tasks[NR_RUNNING]) state_mask |= BIT(PSI_IO_FULL); } if (tasks[NR_MEMSTALL]) { state_mask |= BIT(PSI_MEM_SOME); if (tasks[NR_RUNNING] == tasks[NR_MEMSTALL_RUNNING]) state_mask |= BIT(PSI_MEM_FULL); } if (tasks[NR_RUNNING] > oncpu) state_mask |= BIT(PSI_CPU_SOME); if (tasks[NR_RUNNING] && !oncpu) state_mask |= BIT(PSI_CPU_FULL); if (tasks[NR_IOWAIT] || tasks[NR_MEMSTALL] || tasks[NR_RUNNING]) state_mask |= BIT(PSI_NONIDLE); return state_mask;}
Line by line, and note how each clause is a sharper statement than the prose version:
I/O: SOME iff any task is in NR_IOWAIT. FULL additionally requires !tasks[NR_RUNNING] — literally nothing runnable on this runqueue. Straightforward.
Memory: SOME iff any task is in NR_MEMSTALL. FULL requires tasks[NR_RUNNING] == tasks[NR_MEMSTALL_RUNNING] — not “nothing is running”, but “every running task is a reclaimer”. This is the one predicate the informal nr_productive_tasks == 0 phrasing hides, and it is why memory FULL can be 100 % on a CPU that is pegged at 100 % utilisation. Note it also fires when both counts are zero, which is fine because the outer if already established that something is memstalled.
CPU: oncpu is a single bit meaning “a task from this group is currently on this CPU”. SOME is tasks[NR_RUNNING] > oncpu — more runnable tasks than the (zero or one) currently executing, i.e. somebody is queued. FULL is tasks[NR_RUNNING] && !oncpu — runnable work exists but none of it is on the CPU. At the system level that is impossible by definition (if anything is runnable, the scheduler runs it), which is the honest reason system CPU full is meaningless; inside a cgroup it is entirely possible, because the CPU is being used by tasks outside the group or the group is cpu.max-throttled.
PSI_NONIDLE is set if the runqueue has any task in any of the three counts. It is not exported; it exists purely as the per-CPU weight in the aggregation step below, so that a CPU which was idle for most of a period does not dilute the pressure reported by CPUs that were busy.
How the Kernel Computes Stall Time
The honest mechanism — what psi.c actually does, not a gloss. Freezing the whole machine on every task state change to time the stall states exactly would not scale, so PSI uses a two-tier, per-CPU-then-aggregate scheme (psi.c implementation comment).
Per-CPU bookkeeping (hot path). On each runqueue the scheduler maintains live counts: nr_delayed_tasks (tasks stalled on the resource), nr_productive_tasks, and nr_nonidle_tasks. As tasks change state, hooks (psi_task_change, psi_task_switch, and the psi_memstall_enter/psi_memstall_leave brackets around reclaim/refault code) flip the per-CPU state and accumulate three local time clocks:
These are cheap counters bumped on the existing scheduler fast path; there is no extra global lock per task switch.
Periodic aggregation (cold path). A delayed work item runs every PSI_FREQ — defined as (2*HZ + 1), i.e. once every ~2 seconds (psi.c line 157). It folds the per-CPU clocks into a global figure, weighting each CPU’s stall time by how non-idle that CPU was, so idle CPUs don’t dilute the signal:
The multi-CPU correction is the clever part. Naively summing task states breaks on big machines. The source gives the canonical example: 257 CPU-bound tasks on 256 CPUs would, by naive accounting, show CPU SOME of 100 % (somebody is always waiting on a runqueue), yet the real contention is only 1 thread out of 256 possible execution threads ≈ 0.4 %. PSI fixes this by scaling against the number of potential execution threads:
threads = min(nr_nonidle_tasks, nr_cpus)
SOME = min(nr_delayed_tasks / threads, 1)
FULL = (threads - min(nr_productive_tasks, threads)) / threads
For the 257-on-256 case this yields SOME = min(1/256, 1) = 0.4% and FULL = 0% — matching intuition (psi.c worked example). Substituting nr_cpus = 1 recovers the single-CPU model, so the formula is a natural generalization rather than a special case.
The decaying averages.update_averages() converts the per-period %SOME/%FULL into the exported avg10/avg60/avg300 using the same fixed-point exponential-decay arithmetic (calc_load) as the system load average. The decay constants are precomputed for the 2-second sampling period (psi.c lines 158–160):
Each is 1/e^(2s / window) scaled into the kernel’s FSHIFT fixed-point format; on every 2-second tick the previous average is multiplied by this factor and the new sample blended in, giving the exponential weighting toward recent data. If aggregation work was delayed and several periods were “missed,” calc_avgs() applies calc_load_n() to fold in missed_periods at once rather than looping.
The Full Aggregation Pipeline
flowchart TB
subgraph HOT["HOT PATH — scheduler, every task state change, no global lock"]
H1["psi_task_change / psi_task_switch<br/>psi_memstall_enter / psi_memstall_leave<br/>psi_account_irqtime"]
H2["update per-CPU counts<br/>tasks[NR_IOWAIT, NR_MEMSTALL,<br/>NR_RUNNING, NR_MEMSTALL_RUNNING]"]
H3["test_states() -> state_mask<br/>record_times(): accumulate ns into<br/>groupc->times[PSI_*] for the OLD state"]
H1 --> H2 --> H3
end
H3 --> SEQ["struct psi_group_cpu, one per CPU per cgroup<br/>seqcount-protected, cacheline-aligned<br/>u32 times[NR_PSI_STATES]"]
subgraph COLD["COLD PATH — delayed_work, every PSI_FREQ = 2*HZ+1 (~2 s)"]
C1["psi_avgs_work()"]
C2["collect_percpu_times():<br/>seqcount snapshot of every CPU<br/>delta vs times_prev[PSI_AVGS]"]
C3["weight by per-CPU non-idle time<br/>tSOME = sum(tSOME[i] * tNONIDLE[i]) / tNONIDLE"]
C4["update_averages():<br/>fixed-point calc_load with<br/>EXP_10s / EXP_60s / EXP_300s"]
C1 --> C2 --> C3 --> C4
end
SEQ --> C1
C4 --> OUT["group->avg[state][3] -> avg10/avg60/avg300<br/>group->total[PSI_AVGS][state] -> total (ns, /1000 on read)"]
OUT --> FILE["/proc/pressure/<res><br/><controller>.pressure"]
subgraph RT["RT PATH — only while a privileged trigger exists"]
R1["psimon kthread (per psi_group)<br/>woken on entering a monitored stall state"]
R2["collect_percpu_times(PSI_POLL)<br/>10x per tracking window"]
R3["window growth >= threshold?<br/>-> POLLPRI, max 1 per window"]
R1 --> R2 --> R3
end
SEQ -.->|"independent aggregator:<br/>times_prev[PSI_POLL]"| R1
The two-and-a-half tier aggregation pipeline. What it shows: the scheduler fast path only ever bumps integer counters and accumulates nanoseconds into a per-CPU, seqcount-protected struct — no locks, no cross-CPU traffic. A 2-second delayed work item then snapshots every CPU, weights each CPU’s stall time by how non-idle that CPU was, and folds the result into the three exponential averages. A third, independent consumer — the psimon real-time kthread — reads the same per-CPU buckets through a separate times_prev[PSI_POLL] cursor whenever a privileged trigger is armed. The insight to take: the two aggregators (PSI_AVGS and PSI_POLL) are the reason NR_PSI_AGGREGATORS = 2 and why times_prev is a two-dimensional array — averages and triggers advance at different rates over one shared stream of samples, so arming a trigger never perturbs the numbers you read from the file. It is also why an unprivileged trigger (which rides PSI_AVGS) is forced to a 2-second-multiple window: it has no kthread of its own and can only be evaluated when the averages worker runs.
Where Memory Stall Time Actually Comes From
The scheduler discovers I/O and CPU stalls for free — a task blocking in io_schedule() or sitting on a runqueue is already a scheduler event. Memory stalls are different: nothing about the scheduler knows that a running task is inside reclaim rather than doing the workload’s work. So memory pressure is accounted by explicit source-code brackets: psi_memstall_enter(&flags) … psi_memstall_leave(&flags) around each region of kernel code that represents lost forward progress. The complete set of bracket sites in the memory-management and block layers at v6.12 is small enough to enumerate, and enumerating it is the only way to know what memory pressure does and does not include:
Site (v6.12)
Bracketed operation
What it represents
mm/vmscan.c:6849
balance_pgdat()
kswapd background reclaim — yes, kswapd’s own work counts as stall
mm/vmscan.c:7480
__node_reclaim()
node-local reclaim on the allocation path
mm/page_alloc.c:3948
__perform_reclaim()
direct reclaim from the page allocator
mm/page_alloc.c:3679
try_to_compact_pages()
direct compaction for a high-order allocation
mm/compaction.c:3185
kcompactd_do_work()
background compaction
mm/filemap.c:1244
folio_wait_bit_common()
waiting on a locked folio that has the workingset flag — a refault
mm/filemap.c:1397
migration_entry_wait_on_locked()
waiting on a workingset folio under migration
mm/filemap.c:2366
filemap_read_folio()
synchronous read of a workingset folio
mm/page_io.c:627
swap_read_folio() submission
swap-in of a workingset folio, including submission time
mm/memcontrol.c:1902
try_to_free_mem_cgroup_pages() from memory.high
reclaim triggered by the memory.high soft limit
mm/memcontrol.c:2210
try_to_free_mem_cgroup_pages() from try_charge
reclaim triggered by hitting memory.max
mm/memcontrol.c:2149
schedule_timeout_killable(penalty_jiffies)
the memory.high throttle sleep — deliberate allocator penalty, no reclaim at all
block/blk-cgroup.c:1924
blkcg_maybe_throttle_blkg() delay
I/O throttling delay charged as memory stall when use_memdelay is set
Three of these rows overturn a common mental model.
Reclaim is not the only source, and sometimes not the main one.mm/memcontrol.c:2149 brackets a sleep. When a cgroup exceeds memory.high, the kernel does not fail the allocation — it lets it through and then makes the allocating task sleep for a computed penalty. That sleep is charged as memory pressure even though no page was scanned. The measurement below captures precisely this case: a cgroup registering 57 % memory pressure and 1,159 memory.high events with pgscan of 2 and pgsteal of 0 — 19 seconds of stall with essentially no reclaim happening. If you reason “memory pressure means reclaim” you will look at memory.stat and conclude the pressure reading is broken.
Only workingset refaults count. Three of the mm/filemap.c and mm/page_io.c sites are guarded by folio_test_workingset(folio), not by “is this a major fault”. This is the deliberate design point Corbet highlighted in the 2018 LWN write-up: “A process that is just starting up will experience a lot of page faults as its working set is brought in, but those are not really indicative of system load. For that reason, refaulted pages — those which were evicted due to memory pressure and subsequently brought back in — are used to calculate these metrics.” A cold-start binary paging itself in generates major faults but no memory pressure, which is exactly right and is why PSI is usable as an alerting signal where pgmajfault is not. See Memory Reclaim Overview and Linux Swap Subsystem for how the workingset flag is maintained.
I/O throttling can appear as memory pressure. The block/blk-cgroup.c site charges a block-layer delay as a memory stall when the caller set use_memdelay — the case where the I/O being throttled is itself swap or writeback traffic. This is correct (the task is stalled because of memory management) but means memory.pressure and io.pressure are not cleanly separable resources at the margins.
Poll Triggers: Wake Userspace at a Threshold
The averages are fine for dashboards, but a userspace OOM daemon needs to react within milliseconds — polling /proc/pressure/memory in a busy loop would be wasteful and laggy. PSI therefore offers a poll()-able trigger interface (psi.rst “Monitoring for pressure thresholds”).
A trigger describes “X microseconds of cumulative stall within any Y-microsecond window.” To register one, open the resource’s pressure file and write a line in the format:
<some|full> <stall amount in us> <time window in us>
For example, writing some 150000 1000000 into /proc/pressure/memory sets a 150 ms partial-stall threshold measured over a 1-second window; writing full 50000 1000000 into /proc/pressure/io sets a 50 ms full-stall threshold over 1 second. The opened file descriptor is then handed to select()/poll()/epoll(); when the threshold is crossed, the fd becomes readable for POLLPRI.
The Write Format and Its Exact Bounds
The trigger string has three fields and each is validated independently in psi_trigger_create() (psi.c lines ~1279–1325, v6.12):
The documentation is wrong about the minimum window — verified two ways
Documentation/accounting/psi.rst states, at v6.12 and still at v7.1: “The kernel accepts window sizes ranging from 500ms to 10s, therefore min monitoring update interval is 50ms and max is 1s.”There is no 500 ms floor in the code. The constant WINDOW_MIN_US existed from 5.2 through 6.4 and was removed in 6.5; the surviving check is:
if (window_us == 0 || window_us > WINDOW_MAX_US) return ERR_PTR(-EINVAL);
Established by (a) existence-checking WINDOW_MIN_US across tags — present in v6.4, absent in v6.5/v6.12/v6.18/v7.1 — and (b) measuring the live kernel. Trust the code: a privileged caller on ≥ 6.5 may register a 100 ms or even 1 µs window. The documented consequence — that the monitoring interval is “50 ms to 1 s” — is likewise stale, since window / UPDATES_PER_WINDOW can now be far smaller than 50 ms.
The remaining constraints do hold, and each has a mechanical reason:
One trigger per fd. Multiple triggers (on one or several metrics) are allowed, but each needs its own open() because each is polled independently. Writing a second trigger to an fd that already has one fails with EBUSY — measured below.
Upper bound 10 s, inclusive: WINDOW_MAX_US = 10000000. The rationale in psi.rst is “a high enough number after which monitors are most likely not needed and psi averages can be used instead.”
Monitoring resolution: UPDATES_PER_WINDOW = 10. While the system is in the monitored stall state, trigger growth is checked ten times per window. Monitors only run when the system actually enters the stall state and deactivate on exit, so a quiet system incurs near-zero overhead — demonstrated below by a poll() on an idle cgroup returning nothing for five seconds. Once activated, a monitor stays active for at least one full window to avoid flapping, and notifications are rate-limited to one per tracking window.
The trigger de-registers automatically when its fd is closed (psi_trigger_destroy).
The Privileged / Unprivileged Split Is an Implementation Detail You Can Feel
The most consequential and least documented fact about triggers is that there are two entirely different trigger engines, chosen by a capability check, and which one you get changes the windows you are allowed to ask for. From psi_trigger_create():
/* * Checking the privilege here on file->f_cred implies that a privileged user * could open the file and delegate the write to an unprivileged one. */privileged = cap_raised(file->f_cred->cap_effective, CAP_SYS_RESOURCE);.../* * Unprivileged users can only use 2s windows so that averages aggregation * work is used, and no RT threads need to be spawned. */if (!privileged && window_us % 2000000) return ERR_PTR(-EINVAL);...t->aggregator = privileged ? PSI_POLL : PSI_AVGS;if (privileged) { if (!rcu_access_pointer(group->rtpoll_task)) { task = kthread_create(psi_rtpoll_worker, group, "psimon"); ... } list_add(&t->node, &group->rtpoll_triggers); ...} else { list_add(&t->node, &group->avg_triggers);}
Privileged (CAP_SYS_RESOURCE)
Unprivileged
Aggregator
PSI_POLL
PSI_AVGS
Evaluated by
dedicated psimon kthread, created lazily per psi_group
the existing 2-second psi_avgs_work delayed work
Legal window
any 1 … 10000000 µs
multiples of 2 000 000 µs only → 2 s, 4 s, 6 s, 8 s, 10 s
Effective latency
down to window/10
≥ 200 ms in principle, bounded by the 2 s aggregation tick in practice
Cost to the system
one extra kernel thread per monitored group
none
The comment on the capability check is worth dwelling on, because it describes a real delegation pattern: the check is on file->f_cred, the credentials captured when the file was opened, not the credentials of the writer. A privileged supervisor can therefore open() the pressure file, drop privileges or hand the descriptor to an unprivileged helper over a Unix socket, and that helper gets full real-time trigger semantics. This is how a container manager can give a sandboxed agent low-latency pressure notifications without granting it CAP_SYS_RESOURCE.
The practical trap: the example program printed in the kernel’s own documentation does not work for an unprivileged user.psi.rst uses "some 150000 1000000" — a 1-second window — which fails window_us % 2000000 and returns EINVAL. Measured below.
What the Kernel Actually Accepts — a Measured Validation Matrix
Every row below was executed on Linux 7.1.13-200.fc44.x86_64 as uid 1000 with no CAP_SYS_RESOURCE, opening the file O_RDWR | O_NONBLOCK and writing the NUL-terminated spec:
File
Written spec
Result
Why
/proc/pressure/memory
some 150000 1000000
EINVAL
the kernel doc’s own example — 1 s window is not a multiple of 2 s
/proc/pressure/memory
some 150000 2000000
accepted
2 s window
/proc/pressure/memory
some 100000 500000
EINVAL
500 ms — the doc’s stated minimum — rejected for an unprivileged caller
/proc/pressure/memory
some 100000 100000
EINVAL
100 ms, same reason
/proc/pressure/memory
some 100000 10000000
accepted
exactly WINDOW_MAX_US; the bound is inclusive
/proc/pressure/memory
some 100000 20000000
EINVAL
20 s > WINDOW_MAX_US
/proc/pressure/memory
some 3000000 2000000
EINVAL
threshold (3 s) exceeds window (2 s)
/proc/pressure/memory
some 0 2000000
EINVAL
zero threshold
/proc/pressure/memory
full 150000 2000000
accepted
full is legal on memory
/proc/pressure/cpu
some 150000 2000000
accepted
/proc/pressure/cpu
full 150000 2000000
accepted
⚠️ registration succeeds but the trigger can never fire — system CPU full is hard-zeroed
/proc/pressure/irq
full 150000 2000000
accepted
/proc/pressure/irq
some 150000 2000000
EINVAL
IRQ has no some state
/proc/pressure/memory
sometimes 1 2
EINVAL
sscanf matches neither "some %u %u" nor "full %u %u"
/proc/pressure/memory
second write on the same fd
EBUSY
one trigger per descriptor
Two rows deserve emphasis. The full trigger on /proc/pressure/cpu is silently useless: psi_trigger_create happily builds it because the state PSI_CPU_FULL is a valid enum value, but the zeroing in psi_show() and the fact that system-level CPU full time is never accumulated mean the threshold is never crossed. A monitoring daemon that arms this trigger will simply never wake, with no error to diagnose. And the EINVAL on the documentation’s own example is the single most likely thing to waste an afternoon.
Trigger Example (from the kernel docs)
#include <fcntl.h>#include <poll.h>#include <string.h>#include <unistd.h>int main() { /* 150ms of SOME memory stall within any 1s window */ const char trig[] = "some 150000 1000000"; struct pollfd fds; fds.fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK); /* 1 */ fds.events = POLLPRI; /* 2 */ if (write(fds.fd, trig, strlen(trig) + 1) < 0) /* 3 */ return 1; while (1) { int n = poll(&fds, 1, -1); /* 4 */ if (n < 0) return 1; if (fds.revents & POLLERR) return 0; /* source gone */ /* 5 */ if (fds.revents & POLLPRI) /* threshold hit */ /* 6 */ /* shed load / kill a victim / log */; }}
Line-by-line: (1) open the resource file O_RDWR | O_NONBLOCK — write access is required because registering a trigger is a write(). (2) PSI signals readiness via POLLPRI, not the usual POLLIN. (3) the single write() of the formatted trigger string registers it; a second write to this fd would return EBUSY. (4) block in poll() until the threshold is crossed. (5)POLLERR means the cgroup backing the file was removed — the daemon should drop the fd. (6)POLLPRI is the wakeup that “150 ms of memory stall accumulated within the last second” — the cue to act (psi.rst example).
The Notification Path, End to End
sequenceDiagram
autonumber
participant App as userspace daemon<br/>(oomd / systemd-oomd)
participant VFS as procfs / kernfs<br/>(psi_write)
participant Grp as struct psi_group<br/>(system or cgroup)
participant Mon as psimon kthread<br/>(privileged only)
participant Sched as scheduler hot path
App->>VFS: open("/proc/pressure/memory", O_RDWR|O_NONBLOCK)
Note over VFS: f_cred captured HERE -<br/>CAP_SYS_RESOURCE checked<br/>against the OPENER
App->>VFS: write("some 150000 1000000")
VFS->>Grp: psi_trigger_create()
Note over Grp: validate state / threshold / window<br/>-EINVAL on any breach<br/>-EBUSY if fd already has a trigger
Grp->>Mon: kthread_create("psimon") if first<br/>privileged trigger on this group
Grp-->>App: write() returns success
App->>VFS: poll(fd, POLLPRI, -1) -- blocks
Note over Sched,Mon: ... system is healthy: monitor is DORMANT,<br/>zero wakeups, zero cost ...
Sched->>Grp: psi_memstall_enter() -> state_mask gains PSI_MEM_SOME
Grp->>Mon: psi_schedule_rtpoll_work()<br/>(only if rtpoll_states has this bit)
loop UPDATES_PER_WINDOW = 10 times per window
Mon->>Grp: collect_percpu_times(PSI_POLL)
Mon->>Mon: growth = total[POLL][state] - win.start_value
end
alt growth >= threshold AND >= 1 window since last event
Mon-->>App: POLLPRI on the fd
App->>App: read pressure, pick a victim,<br/>write cgroup.kill / send SIGKILL
else rate-limited
Mon->>Mon: set pending_event, defer to next window
end
Note over Mon: monitor stays active for >= 1 full<br/>window after the stall clears,<br/>to avoid flapping
App->>VFS: close(fd)
VFS->>Grp: psi_trigger_destroy() - unlink, last one out stops psimon
The life of a PSI trigger. What it shows: registration is a write() on an ordinary file descriptor, the capability check happens at open() time against f_cred, and the notification arrives as POLLPRI — not POLLIN. Between registration and the first stall the monitor does nothing at all: the psimon thread is only woken from the scheduler hot path, and only when group->rtpoll_states has the monitored state’s bit set. The insight to take: the two rate limits are different things and both matter. UPDATES_PER_WINDOW = 10 bounds how finely growth is sampled; the one-event-per-window rule bounds how often userspace is woken. A daemon must therefore be written to treat each POLLPRI as “pressure is above threshold right now”, re-read the file, and act — never as “one discrete event occurred”.
Measured: a Trigger Firing
The whole path, run as uid 1000 on Linux 7.1.13, against a cgroup’s memory.pressure (not the system file), with a 2-second window because the process is unprivileged:
trigger registered on .../psi-demo/pollhog/memory.pressure : 'some 400000 2000000'(400ms stall / 2s window; 2s window forced because uid=1000 lacks CAP_SYS_RESOURCE)poll() with 5s timeout on an IDLE cgroup ... -> returned [] after 5.01s (no pressure, no wakeup)memory hog started inside the cgroup; waiting for POLLPRI ... [ 2.05s] WOKE UP: POLLPRI some avg10=26.05 avg60=51.93 avg300=19.97 total=70480525 [ 4.80s] WOKE UP: POLLPRI some avg10=36.32 avg60=52.47 avg300=20.52 total=71709802 [ 6.85s] WOKE UP: POLLPRI some avg10=34.81 avg60=51.67 avg300=20.57 total=72302058 [10.82s] WOKE UP: POLLPRI some avg10=49.29 avg60=53.46 avg300=21.38 total=75482380
Read the timings: the idle poll() blocked the full five seconds and returned an empty event list — an armed trigger on a healthy system costs nothing and wakes nobody. Once the hog started, wakeups arrived at +2.05 s, +4.80 s, +6.85 s and +10.82 s: gaps of 2.75 s, 2.05 s and 3.97 s, every one at least one 2-second tracking window, exactly as the rate-limit rule specifies. This is the empirical shape of “notifications are rate-limited to one per tracking window” — you do not get a wakeup per stall, you get at most one per window for as long as the condition persists.
Per-cgroup PSI
On a kernel with CONFIG_CGROUPS=y and the cgroup2 hierarchy mounted, PSI is tracked per cgroup as well as system-wide. Every cgroup directory exposes cpu.pressure, memory.pressure, and io.pressure with the identical some/fullavg10/avg60/avg300/total format, accounting only the stall time of tasks in that subtree (psi.rst “Cgroup2 interface”, cgroup-v2 admin guide). Per-cgroup triggers work exactly like the system-wide ones — open the cgroup’s memory.pressure file and write a trigger. This is what lets a container runtime watch a single workload’s pressure in isolation, and it is the natural complement to memcg limits: memory.maxbounds a group’s memory, while memory.pressuremeasures how much that bound is hurting, and per-cgroup reclaim is what generates the memory stall in the first place. Note that the system-level cpu.pressure full is undefined/zero, but per-cgroup cpu.pressure fullis meaningful (all the group’s tasks throttled by cpu.max or starved by outside tasks).
The Per-cgroup Pressure Files Are Core Files, Not Controller Files
A detail that catches people writing cgroup tooling: the *.pressure files do not depend on any controller being enabled. Creating a bare cgroup directory on a v2 hierarchy yields all four immediately, before a single token has been written to cgroup.subtree_control. Measured on 7.1.13 by making a fresh directory under a delegated subtree:
$ mkdir psi-demo && ls psi-demo | grep -E 'pressure'cgroup.pressurecpu.pressureio.pressureirq.pressurememory.pressure$ cat psi-demo/cgroup.subtree_control # empty - no controller enabled at all$ cat psi-demo/irq.pressurefull avg10=0.00 avg60=0.00 avg300=0.00 total=0
This matters because the usual v2 rule is that controller interface files (memory.max, cpu.weight, …) appear in a child only when the parent enables that controller (see cgroups v2 Unified Hierarchy). Pressure files are exempt: PSI is part of the cgroup core, so memory.pressure exists even where the memory controller is not enabled. A tool that probes for memory.pressure to decide “is the memory controller active here?” will get the wrong answer.
cgroup.pressure — the Per-cgroup Off Switch
Also present on every non-root cgroup since 6.2 is cgroup.pressure, a read-write 0/1 file defaulting to 1. Writing 0disables PSI accounting for that cgroup. The kernel doc states the motivation plainly (cgroup-v2.rst, v6.12):
The reason this control attribute exists is that PSI accounts stalls for each cgroup separately and aggregates it at each level of the hierarchy. This may cause non-negligible overhead for some workloads when under deep level of the hierarchy, in which case this control attribute can be used to disable PSI accounting in the non-leaf cgroups.
This is the honest cost statement for per-cgroup PSI, and it follows directly from the aggregation pipeline: psi_account_irqtime() and the task-change hooks walk group = group->parent in a do { } while loop, touching a psi_group_cpu at every ancestor level. A container stack that nests root → system.slice → kubepods.slice → burstable → pod → container pays that walk on every task state change. Crucially the attribute is not hierarchical — “disable or enable PSI accounting in a cgroup does not affect PSI accounting in descendants and doesn’t need pass enablement via ancestors from root” — so the intended pattern is to switch it off on intermediate slices you never query and leave it on at the leaves you actually monitor. On this Fedora 44 system every cgroup reads cgroup.pressure = 1, i.e. the distro leaves full accounting on.
Measured on a Live System
Everything below was run on Linux 7.1.13-200.fc44.x86_64, 32 CPUs, 125 GiB RAM, as uid 1000 — no root anywhere. That is possible because systemd delegates /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/ to the login user (drwxr-xr-x linman linman), with cgroup.controllers reading cpu io memory pids. Delegation is what makes PSI experimentation accessible without touching the machine’s real workload; see cgroup Delegation and the No Internal Process Rule.
Memory Pressure Without Any Reclaim
The setup is a cgroup with memory.high = 64 MiB (a throttling soft limit, not a killing hard limit), memory.swap.max = 0, and memory.max = max, containing one Python process that allocates 512 MiB of anonymous memory and then walks it forever. Anonymous pages with no swap are unreclaimable, so the kernel cannot get the cgroup back under memory.high and instead applies the escalating penalty sleep:
### hog cgroup: memory.high=67108864 memory.max=max memory.swap.max=0t=0 BEFORE cgroup memory.pressure: some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0t=0 BEFORE system /proc/pressure/memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=533501 full avg10=0.00 avg60=0.00 avg300=0.00 total=529031t=+10s cgroup memory.pressure: some avg10=51.31 avg60=12.25 avg300=2.69 total=8046052 full avg10=51.31 avg60=12.25 avg300=2.69 total=8046052 cgroup memory.current=72912896 system /proc/pressure/memory: some avg10=16.81 avg60=3.84 avg300=0.83 total=3738932 full avg10=14.80 avg60=3.38 avg300=0.73 total=3290689t=+30s cgroup memory.pressure: some avg10=57.13 avg60=24.81 avg300=6.24 total=19424367 full avg10=57.13 avg60=24.81 avg300=6.24 total=19424367 cgroup memory.current=74878976 system /proc/pressure/memory: some avg10=22.75 avg60=9.31 avg300=2.31 total=8082561 full avg10=19.17 avg60=7.91 avg300=1.97 total=7019040### memory.stat reclaim counters: pgscan 2 pgsteal 0 pgmajfault 0 workingset_refault_anon 0 workingset_refault_file 0### memory.events: low 0 high 1159 max 0 oom 0 oom_kill 0
Five readings, each teaching something distinct:
The cgroup accumulated 19.4 seconds of stall in 30 seconds of wall clock (total 0 → 19,424,367 µs), settling at avg10 = 57.13 %. The signal is unambiguous and needed no tuning to produce.
some and full are numerically identical throughout. This is not a bug: the cgroup contains exactly one task, so “at least one task stalled” and “every non-idle task stalled” are the same predicate. Whenever you see a cgroup where some == full exactly, you are looking at a single-threaded workload — a genuinely useful diagnostic tell.
pgscan = 2, pgsteal = 0, pgmajfault = 0, but memory.events high = 1159. Almost no reclaim occurred. The 19 seconds of “memory pressure” is entirely the schedule_timeout_killable(penalty_jiffies) at mm/memcontrol.c:2149. This is the concrete proof of the claim made earlier: memory pressure ≠ reclaim. Anyone correlating memory.pressure against pgscan to validate their monitoring would conclude, wrongly, that PSI is lying.
System-wide pressure is lower than the cgroup’s (22.75 % vs 57.13 %) even though the cgroup is the only source of stall on the box. This is the multi-CPU correction doing its job: one stalled task out of 32 potential execution threads is a small fraction of the machine’s lost potential, while inside a one-task cgroup it is most of it. Pressure is always relative to the scope you read it at.
System full (19.17) is strictly below system some (22.75), while the cgroup’s are equal. At system scope other tasks were running productively during part of the stall, so the machine was only partially unproductive. full being the strict subset is visible directly in the numbers.
CPU Pressure, and the full Line That Is and Isn’t There
The cleanest possible demonstration of the CPU full asymmetry: 64 spinning processes inside a cgroup with cpu.max = "100000 100000" — one CPU-second per second on a 32-CPU machine, so 64 runnable tasks share a single CPU’s worth of quota.
### cpu.max = 100000 100000 (nproc=32), 64 busy-loop processes in the cgroup=== BEFORE ===cgroup cpu.pressure: some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0system /proc/pressure/cpu: some avg10=0.00 avg60=0.11 avg300=0.05 total=182156780 full avg10=0.00 avg60=0.00 avg300=0.00 total=0=== DURING (t=+20s) ===cgroup cpu.pressure: some avg10=84.62 avg60=27.65 avg300=6.47 total=19694237 full avg10=82.92 avg60=27.11 avg300=6.35 total=19348759 <-- REAL, NON-ZEROsystem /proc/pressure/cpu: some avg10=82.74 avg60=26.21 avg300=6.12 total=201864519 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 <-- STILL EXACTLY ZEROcgroup cpu.stat: nr_periods 100 nr_throttled 100 throttled_usec 308533066
This single capture settles the question the whole section has been circling. At the same instant, on the same machine, under the same load: the cgroup reports cpu.pressure full avg10 = 82.92 while /proc/pressure/cpu reports full avg10 = 0.00, total = 0. Not “small”, not “noisy” — the system-level counter has never been incremented since boot, 30 hours earlier. And the mechanism is exactly what the source comment predicts: nr_throttled 100 / nr_periods 100 means the cgroup was throttled in every single 100 ms period, so its tasks were runnable-but-not-on-CPU — tasks[NR_RUNNING] && !oncpu, the PSI_CPU_FULL predicate — while the machine at large always had something to run.
Note also that system some (82.74) tracks the cgroup’s some (84.62) closely here. With 64 runnable tasks pinned behind a 1-CPU quota there is nearly always somebody queued, and min(nr_nonidle, nr_cpus) saturates. The two figures diverge only when the stall is small relative to the machine, as in the memory experiment above.
I/O Pressure
I/O is the resource most easily faked by accident, so the setup matters: 24 concurrent processes doing random 4 KiB O_DIRECT reads against a 3 GiB file on a real NVMe-backed filesystem. An earlier attempt against a file in /tmp produced zero I/O pressure and looked like a broken kernel — /tmp is tmpfs, so there was no block device and nothing ever blocked. O_DIRECT bypasses the page cache so every read must reach the device.
=== BEFORE ===cgroup io.pressure: some avg10=0.00 ... total=0 full avg10=0.00 ... total=0system /proc/pressure/io: some avg10=0.00 ... total=48341150=== DURING (t=+20s) ===cgroup io.pressure: some avg10=36.45 avg60=11.91 avg300=2.79 total=8542350 full avg10=31.43 avg60=10.27 avg300=2.40 total=7344689cgroup io.stat: 259:0 rbytes=20155174912 wbytes=0 rios=4920697 wios=0 dbytes=0 dios=0system /proc/pressure/io: some avg10=36.14 avg60=11.71 avg300=2.74 total=56835517 full avg10=29.48 avg60=9.56 avg300=2.23 total=50890255system /proc/pressure/cpu: some avg10=1.41 ... <-- CPU is nearly idle
Here some (36.45) exceeds full (31.43) by a clear margin — unlike the single-task memory case where they were equal — because with 24 tasks there is usually at least one running while others wait on the device. Meanwhile /proc/pressure/cpu some sits at 1.41 %: the machine is doing almost nothing, which is precisely the situation classic utilization metrics describe as “idle, healthy” and PSI correctly describes as “36 % of potential work is not happening because of the disk”. That contrast is the entire argument for PSI in one capture.
Reproducing These
The complete procedure, runnable as an unprivileged user on any systemd distro with cgroup delegation:
# 1. Your delegated subtree (systemd sets Delegate=yes on user@.service)D=/sys/fs/cgroup/user.slice/user-1000.slice/user@$(id -u).servicecat $D/cgroup.controllers # -> cpu io memory pids# 2. Memory pressure via memory.high throttling (NOT memory.max, which OOM-kills)mkdir -p $D/hogecho 67108864 > $D/hog/memory.high # 64 MiB soft capecho 0 > $D/hog/memory.swap.max # no swap => anon pages are unreclaimablebash -c 'echo $$ > '$D'/hog/cgroup.procs; exec python3 -c "import timeb=[bytearray(1<<20) for _ in range(512)] # 512 MiB anon, 8x the cape=time.time()+30i=0while time.time()<e: x=b[i%len(b)] for o in range(0,1<<20,4096): x[o]=(x[o]+1)&255 i+=1"' &watch -n1 "cat $D/hog/memory.pressure; grep -E 'high|max' $D/hog/memory.events"# 3. CPU pressure via cpu.max throttlingmkdir -p $D/cpu; echo "100000 100000" > $D/cpu/cpu.max # 1 CPU of quotafor i in $(seq 64); do bash -c 'echo $$ > '$D'/cpu/cgroup.procs; exec timeout 25 python3 -c "while True: pass"' & donecat $D/cpu/cpu.pressure ; cat /proc/pressure/cpu # compare the two 'full' lines# 4. Cleanup - rmdir only works once the cgroup is empty (else -EBUSY)rmdir $D/hog $D/cpu
Two setup mistakes are worth naming because both produce a convincing-looking null result. Using memory.max instead of memory.high gets the hog OOM-killed within a second or two — the first attempt at this experiment recorded oom_kill 1 and a mere 116 µs of stall, which reads as “PSI doesn’t work” rather than “the workload died”. And putting the I/O test file on tmpfs removes the block device entirely. Both are cases where the measurement apparatus, not the kernel, was wrong.
How systemd-oomd and oomd Use PSI
PSI’s flagship consumer is the class of userspace OOM daemons that act on pressure before the kernel’s last-resort OOM killer fires. The motivation is that by the time the in-kernel OOM killer runs, the machine has often already been thrashing for many seconds — swapping and refaulting so hard that it is effectively unresponsive (“livelock on the edge of OOM”). PSI lets userspace catch the thrash early.
Facebook’s oomd (oomd on GitHub) registers PSI triggers on cgroup memory.pressure files and, when sustained pressure crosses a configured threshold, proactively kills the least valuable cgroup. It was the original driving use case for PSI.
systemd-oomd (systemd-oomd(8), oomd.conf) is the distro-integrated descendant. It monitors memory.pressure of cgroups configured with ManagedOOMMemoryPressure=kill and, when a unit’s memory PSI exceeds ManagedOOMMemoryPressureLimit (a percentage, default around 60 % over a 30-second averaging window) for long enough, kills cgroups in that slice. It also watches swap usage. Because it acts on the trend of stall time rather than waiting for the allocator to fail, it can reclaim a desktop or server before the user perceives a freeze.
The decision flow is: PSI trigger fires → daemon evaluates candidate cgroups → SIGKILL the chosen victim’s cgroup (often via cgroup.kill or memory.oom.group semantics) → pressure subsides. This is strictly an augmentation of the kernel OOM killer, not a replacement; the kernel killer remains the backstop if userspace is itself starved.
Failure Modes and Misunderstandings
/proc/pressure is missing. PSI must be compiled in (CONFIG_PSI=y, present since 4.20 per LKDDB). Even when compiled in, many distributions ship it disabled by default behind the psi=1 kernel boot parameter (controlled by CONFIG_PSI_DEFAULT_DISABLED) to avoid the small scheduler-hot-path cost; if /proc/pressure/ does not exist, check the config and the cmdline. Verify the default-disabled state on your specific distro kernel — it is a build choice, not a fixed kernel fact.
Confusing SOME with utilization.some memory avg10=0 does not mean memory is free; it means no task was stalled on memory in the last 10 s. A box can be at 99 % memory utilization with zero memory pressure if nothing is reclaiming or refaulting.
Expecting CPU full at the system level. It is always zero there by design; only the cgroup-level CPU full carries information.
Polling on POLLIN. Trigger fds signal POLLPRI, not POLLIN; a poll loop watching the wrong event will never wake.
EBUSY on a second trigger. Each trigger needs its own fd; reusing one fails. Likewise POLLERR on a cgroup pressure fd means the cgroup vanished — re-open or drop it.
Sampling skew. The averages update only every ~2 s and the trigger resolution is at most 10 checks/window, so PSI is a near-real-time signal, not an instantaneous one. For microsecond-precise spike detection, diff the total counter rather than reading the averages.
Assuming memory pressure implies reclaim. The most expensive misunderstanding in the list. memory.high throttling, block-layer use_memdelay delays, and swap-in submission time are all charged as memory stall without necessarily scanning a single page. Measured above: 57 % pressure with pgscan = 2 and pgsteal = 0. Correlate memory.pressure with memory.events’ high and max counters, not only with memory.stat’s scan counters.
Expecting cold-start page faults to show as pressure. They do not, by design — only refaults of pages carrying the workingset flag are bracketed. A process paging in its binary produces pgmajfault with zero memory pressure. This is a feature (it is what makes PSI alertable), but it surprises people who expect PSI to track major faults.
Parsing /proc/pressure/irq with a two-line parser. It emits one line, labelled full, and only when CONFIG_IRQ_TIME_ACCOUNTING is set. psi.rst does not mention the file at all, so parsers written from the documentation break the first time they glob /proc/pressure/*.
Arming full on /proc/pressure/cpu. Accepted by the kernel, never fires. There is no error and no log line — the daemon simply sits in poll() forever. Use some at system level, or move the trigger to a cgroup’s cpu.pressure where full is real.
Writing the documentation’s example trigger as a normal user.some 150000 1000000 → EINVAL, because 1 s is not a multiple of 2 s and the process lacks CAP_SYS_RESOURCE. Either run privileged or round the window to 2 s.
Believing the documented 500 ms minimum window. Removed in 6.5. On ≥ 6.5 a privileged caller may use any window from 1 µs to 10 s; the docs still say otherwise at v7.1.
Measuring I/O pressure against tmpfs. No block device, no io_schedule(), no stall — a perfect null result that looks like a broken kernel. Verify with findmnt -no SOURCE,FSTYPE <path> before concluding anything.
Forgetting that total is per-cgroup-lifetime. A cgroup’s total starts at zero when the cgroup is created, not at boot. Comparing a freshly-created container’s total against the system’s is meaningless.
Deep hierarchies paying invisible aggregation cost. Every task state change walks group->parent to the root, touching one psi_group_cpu per level. On a Kubernetes-style six-level tree that is six cacheline updates per switch. cgroup.pressure = 0 on intermediate slices is the supported mitigation.
Alternatives and When to Choose Them
Load average (/proc/loadavg). Cheap and ubiquitous but resource-blind and ambiguous (cannot distinguish a healthy saturated box from a thrashing one, and lumps CPU + uninterruptible-sleep). Use it for a rough liveness glance; use PSI when you need to know whether load is hurting.
vmstat / /proc/meminfo / cgroup memory.stat. Report utilization and event counts (page faults, reclaim, swap-in). Complementary to PSI: counts tell you what is happening, PSI tells you how much it is costing in stalled time. Pair memory.pressure with memory.stat’s refault counters when diagnosing thrash.
iostat/iowait. I/O-only and known to misattribute time; io.pressure is the cleaner “is I/O actually blocking forward progress” signal.
eBPF latency tracing. Far more precise and per-event, but heavier and bespoke. PSI is the always-on, near-free aggregate; reach for eBPF when you need to attribute a stall to a specific syscall or device.
Production Notes
The Problem PSI Was Built To Solve
PSI is not a general-purpose observability feature that happened to find users; it was built for one specific production problem and its design only makes sense in that light. The problem is datacenter overcommit. psi.rst opens with the dilemma in two sentences: “Without an accurate measure of such contention, users are forced to either play it safe and under-utilize their hardware resources, or roll the dice and frequently suffer the disruptions resulting from excessive overcommit.” Every metric available before PSI sat on the wrong side of that trade. Utilization cannot distinguish a well-packed machine from a thrashing one. The load average conflates CPU queueing with uninterruptible sleep and must be interpreted against core count. iowait misattributes time. So fleet operators provisioned for the worst case and left capacity on the floor.
Weiner’s own framing, quoted in the 2018 LWN write-up, is about closing the loop after the fact as much as in real time: “We now log and graph pressure for the containers in our fleet and can trivially link latency spikes and throughput drops to shortages of specific resources after the fact, and fix the job config/scheduling.” That is the actual production workflow — pressure as a permanently recorded per-container time series that turns “the service was slow at 03:12” into “the service was slow at 03:12 because memory”.
Three Consumers, Three Different Uses of the Same Signal
The three flagship consumers use PSI in genuinely different ways, and the differences are instructive:
Tool
What it reads
How it uses PSI
Action
oomd (Meta)
per-cgroup memory.pressure, via poll() triggers
threshold alarm — sustained pressure above a configured level
SIGKILL the least valuable cgroup
systemd-oomd
memory.pressure of units with ManagedOOMMemoryPressure=kill, plus swap usage
threshold alarm on a percentage over an averaging window
kill cgroups in the slice
senpai (Meta)
memory.pressuretotal, differenced each interval
control-loop feedback term — a target amount of pressure, deliberately induced
adjust memory.high up or down
DAMON_RECLAIM (in-kernel)
system-wide some memory pressure
auto-tuning feedback (quota_mem_pressure_us)
modulate reclaim aggressiveness
The first two treat pressure as an emergency signal. senpai inverts it, and that inversion is the most interesting thing built on PSI.
senpai — Pressure as a Probe, Not an Alarm
The senpai README states the underlying problem crisply: “Allocated memory is not a good proxy for required memory.” Libraries and startup pages are loaded and never touched again; the page cache holds cold data until something else needs the memory. So a container’s memory.current tells you what it has been given, not what it needs — and sizing containers by observed usage systematically over-provisions the whole fleet.
Senpai’s insight is that PSI makes it safe to find out experimentally, on a live production workload. It repeatedly squeezes memory.high downward, watches the memory.pressuretotal counter, and backs off the moment stall time exceeds a small target. Its default configuration makes the control law explicit: pressure = 1000 (microseconds of stall tolerated per interval), max_probe = 0.01 (shrink the limit by at most 1 % per step), max_backoff = 0.1 (grow it by up to 10 % when pressure appears), interval = 5 seconds. The README’s worked example is a kernel build whose peak memory.current is 803,934,208 bytes — about 800 MiB — run unconstrained. Manual bisection shows it finishes in the same wall-clock time at a 600 MiB and even a 400 MiB memory.high, and only collapses (3m58s → 9m+, killed) at 300 MiB. Senpai, run against the same job with no prior knowledge, converges on 347,762,688 bytes ≈ 332 MiB — less than half the memory the unconstrained job appeared to need, with a 4m09s runtime against the 3m58s baseline.
The mechanism senpai exploits is precisely the one the memory experiment above isolated: memory.high reclaims and then throttles rather than killing, and that throttle is bracketed with psi_memstall_enter. Senpai’s entire feedback signal is mm/memcontrol.c:2149. This is also why the technique needs PSI specifically and could not have been built on pgscan or memory.current — it needs a measure of how much the squeeze is costing the workload, denominated in time, and nothing else in the kernel provides that. See The Memory Cgroup memcg for memory.high semantics and Per-cgroup Reclaim and Memory Pressure for the reclaim path itself.
Practical Guidance
The most valuable single number for “is this machine in trouble?” is memory.pressuresome avg10 per-cgroup (or full avg10 for “is it thrashing?”). For dashboards prefer avg60; for alerting prefer differencing total over your own window, which avoids the exponential average’s tail and lets you pick a window the decay constants do not offer. For capacity work, avg300 plus memory.events’ high counter tells you whether a limit is merely tight or actively destructive.
PSI’s design pedigree is industrial: it was built at Facebook to run fleet-wide and is the foundation of their oomd-based memory management and resource-control stack (Facebook PSI site). The introductory LWN coverage and the merge commit document the original rationale. In practice the most valuable single number for “is this machine in trouble?” is memory.pressuresome avg10 (or full avg10 for “is it thrashing?”), watched per-cgroup. Kubernetes and other orchestrators increasingly surface PSI as a node and pod health signal, and the kernel’s own DAMON proactive-reclaim module (DAMON_RECLAIM) can use system-wide some memory PSI as a feedback metric (quota_mem_pressure_us) to auto-tune how aggressively it reclaims — a direct in-kernel consumer of the same signal (DAMON reclaim docs).
See Also
Memory Reclaim Overview — the reclaim path is what generates memory stall time; psi_memstall_enter/leave brackets the reclaim and refault code.
The Memory Cgroup memcg — per-cgroup PSI complements memcg limits; memory.pressure measures the cost of memory.max.
cgroup Memory Pressure and oomd — the userspace OOM-daemon policy engines in depth (oomd rulesets, systemd-oomd configuration); this note covers the kernel signal they consume.
cgroups v2 Unified Hierarchy — why memory.pressure exists on every cgroup regardless of which controllers are enabled, and the subtree_control rules that govern everything else in the directory.