vault

Direct Reclaim

15 min read

Direct Reclaim

Direct reclaim is the kernel’s synchronous, last-ditch way to find free memory: when a task calls into the page allocator, the allocator cannot satisfy the request even against its lowest watermark, and the request is one that is allowed to block, the allocating task itself descends into the reclaim machinery — running try_to_free_pages() in its own context — to free pages before retrying the allocation. It is the opposite of background reclaim: instead of a dedicated kernel thread quietly freeing memory ahead of demand, the thread that wanted the memory is forced to stop and do the janitorial work first. Because the allocation blocks until reclaim makes progress, direct reclaim shows up directly as allocation latency — a stall in the application’s own timeline — which is why a high direct-reclaim rate (the allocstall / pgscan_direct counters) is one of the clearest signals that a machine is genuinely short of memory. This note covers the synchronous path in the allocator slow path; the asynchronous counterpart lives in kswapd and Background Reclaim and the shared page-scanning machinery in Memory Reclaim Overview and The LRU Lists.

Version

All code paths and function names here are pinned to Linux 6.12 LTS (mm/page_alloc.c, mm/vmscan.c) and were re-verified against 6.18 LTS (released 2025-11-30). try_to_free_pages(), throttle_direct_reclaim(), and __perform_reclaim() are byte-identical between the two; balance_pgdat() differs only cosmetically (6.18 uses a new for_each_managed_zone_pgdat() iterator macro). Treat every claim as “as of 6.12/6.18, June 2026.”

Mental Model — Reclaim in the Caller’s Own Hands

The page allocator has a fast path and a slow path. The fast path (Watermarks and the Allocation Fast Path) tries get_page_from_freelist() against the low watermark (ALLOC_WMARK_LOW) and, if a zone has enough free pages above that line, hands a page back with no further ceremony. When the fast path fails, control falls into __alloc_pages_slowpath(), and that function is where direct reclaim lives. The slow path does a specific, ordered set of escalations, and direct reclaim is deliberately not the first of them.

flowchart TD
  REQ["__alloc_pages()<br/>fast path: get_page_from_freelist<br/>(ALLOC_WMARK_LOW)"] -->|"fail"| SLOW["__alloc_pages_slowpath()"]
  SLOW --> WAKE["wake_all_kswapds()<br/>(if __GFP_KSWAPD_RECLAIM)"]
  WAKE --> RETRY1["retry get_page_from_freelist<br/>(ALLOC_WMARK_MIN)"]
  RETRY1 -->|"success"| DONE["return page"]
  RETRY1 -->|"fail"| CANBLOCK{"can_direct_reclaim?<br/>(__GFP_DIRECT_RECLAIM set,<br/>not PF_MEMALLOC)"}
  CANBLOCK -->|"no"| NOPAGE["fail / NULL<br/>(atomic alloc)"]
  CANBLOCK -->|"yes"| DR["__alloc_pages_direct_reclaim()<br/>= psi_memstall_enter +<br/>__perform_reclaim()"]
  DR --> TTFP["try_to_free_pages()<br/>do_try_to_free_pages → shrink_zones<br/>(ALLOCSTALL counted here)"]
  TTFP -->|"freed some"| RETRY2["get_page_from_freelist again"]
  RETRY2 -->|"success"| DONE
  RETRY2 -->|"still fail"| COMPACT["direct compaction,<br/>then retry / OOM"]

The allocator slow path in Linux 6.12. What it shows: kswapd is woken first (wake_all_kswapds), the allocator retries against the lower min watermark, and only if that still fails — and the request is allowed to block — does the task enter __alloc_pages_direct_reclaim() and run try_to_free_pages() synchronously. The insight to take: by the time a task is in direct reclaim, the per-node kswapd has already been asked to help and could not free pages fast enough. Direct reclaim is therefore by construction a sign that background reclaim has fallen behind demand — it is the allocator’s admission that it must block the caller to survive.

The single most important idea: direct reclaim is reclaim performed in the context of, and at the expense of, the task that wanted the memory. There is no separate thread; current — your malloc(), your page-fault handler, your read() populating the page cache — runs try_to_free_pages() on its own stack and does not return from the allocation until that function either freed enough pages or gave up. That is the whole reason it is a latency problem.

Mechanical Walk-through

How the slow path reaches direct reclaim

When get_page_from_freelist() fails on the fast path, __alloc_pages_slowpath() runs. Its first reclaim-related act, gated on the ALLOC_KSWAPD flag (set when the GFP mask includes __GFP_KSWAPD_RECLAIM), is:

if (alloc_flags & ALLOC_KSWAPD)
    wake_all_kswapds(order, gfp_mask, ac);

wake_all_kswapds() walks every zone in the allocation’s zonelist and calls wakeup_kswapd() on each distinct node’s pg_data_t — kicking the per-node background daemon (see kswapd and Background Reclaim). The slow path then retries get_page_from_freelist(), this time using the relaxed ALLOC_WMARK_MIN watermark (the min line is below low, so a little more of the reserve is now usable). If that succeeds, the task escapes without ever doing reclaim itself — kswapd, or simply the relaxed watermark, was enough.

Only when that retry also fails does the slow path consider direct reclaim. Two gates must be open:

/* Caller is not willing to reclaim, we can't balance anything */
if (!can_direct_reclaim)
    goto nopage;
 
/* Avoid recursion of direct reclaim */
if (current->flags & PF_MEMALLOC)
    goto nopage;

can_direct_reclaim is true only if the GFP mask has __GFP_DIRECT_RECLAIM set. An atomic allocation (GFP_ATOMIC, or any context that cannot sleep — interrupt handlers, code holding a spinlock) clears that bit; such an allocation simply fails rather than blocking. The second gate, PF_MEMALLOC, prevents recursion: a task already inside reclaim (or kswapd, which sets PF_MEMALLOC permanently — see kswapd and Background Reclaim) must not re-enter reclaim and risk an unbounded loop. See GFP Flags and Allocation Contexts for how the mask encodes “may I sleep / do I/O / trigger reclaim.”

__alloc_pages_direct_reclaim and the PSI stall window

With both gates open, the slow path calls:

page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, ac,
                                    &did_some_progress);

This wrapper does three things. It brackets the work in Pressure Stall Information (PSI) accounting — psi_memstall_enter(&pflags) … psi_memstall_leave(&pflags) — so that the time the task spends blocked here is attributed to memory stall and surfaces in /proc/pressure/memory (see Pressure Stall Information). Between those it calls __perform_reclaim(), and afterward it retries get_page_from_freelist(); if that still fails it drains the per-CPU page lists (drain_all_pages) and releases any high-order atomic reserves once before trying a final time. The PSI bracket is why direct reclaim is the canonical thing PSI was built to measure — it is, almost by definition, “an application stalled waiting on memory.”

try_to_free_pages — the actual reclaim

__perform_reclaim() is thin. It calls cond_resched(), marks the task as being in fs_reclaim context (a lockdep aid that catches deadlocks where a filesystem allocation recurses into the same filesystem during writeback), sets memalloc_noreclaim to prevent the just-described recursion, and calls:

progress = try_to_free_pages(ac->zonelist, order, gfp_mask, ac->nodemask);

try_to_free_pages() builds a struct scan_control — the kernel’s per-reclaim “request object” — with nr_to_reclaim = SWAP_CLUSTER_MAX (32 pages, the granularity of a reclaim batch), priority = DEF_PRIORITY (12), and may_writepage/may_unmap/may_swap all permitted. It then calls do_try_to_free_pages(), which is the shared engine that both direct reclaim and kswapd funnel into. That engine loops, decreasing sc.priority from 12 toward 0 (lower priority = scan a larger fraction of the LRU per pass), calling shrink_zones() → shrink_node() → shrink_lruvec() on each iteration until it has freed nr_to_reclaim pages or exhausted its passes. The page-scanning, LRU-aging, dirty-writeback, and swap logic inside shrink_node() is the same machinery the background daemon uses — it is documented in Memory Reclaim Overview, The LRU Lists, and Shrinkers and Slab Reclaim, and is deliberately not re-explained here. What is specific to direct reclaim is who runs it (the allocating task) and the accounting that results.

The allocstall / PGSCAN_DIRECT counters

The very first thing do_try_to_free_pages() does, for non-cgroup reclaim, is:

if (!cgroup_reclaim(sc))
    __count_zid_vm_events(ALLOCSTALL, sc->reclaim_idx, 1);

So every entry into direct reclaim bumps allocstall (exposed per-zone-class in /proc/vmstat as allocstall_dma, allocstall_normal, allocstall_movable, etc.). This is the headline metric: allocstall counting up means tasks are being forced into synchronous reclaim. Separately, the page scanner attributes the pages it scans and steals to either the kswapd or the direct bucket via reclaimer_offset():

item = PGSCAN_KSWAPD + reclaimer_offset();   /* PGSCAN_DIRECT when not kswapd/khugepaged */
...
item = PGSTEAL_KSWAPD + reclaimer_offset();

reclaimer_offset() returns 0 when current_is_kswapd(), otherwise the offset to the PGSCAN_DIRECT/PGSTEAL_DIRECT slot. Hence /proc/vmstat exposes pgscan_direct and pgsteal_direct (pages scanned and reclaimed by direct reclaimers) alongside pgscan_kswapd/pgsteal_kswapd. A healthy system does most of its scanning under pgscan_kswapd; a rising pgscan_direct and allocstall together say “kswapd is not keeping up and applications are paying for it.” (See kswapd and Background Reclaim for the kswapd-side counters.)

Backoff and throttling — reclaim_throttle, not congestion_wait

A common misconception, fossilized in older documentation and blog posts, is that direct reclaimers “wait on congestion” via congestion_wait() / wait_iff_congested(). Those functions were removed. In 6.12/6.18 the backoff primitive is reclaim_throttle(pgdat, reason) (mm/vmscan.c), which puts the reclaiming task to sleep on one of four per-node wait queues keyed by an explicit reason (enum vmscan_throttle_state), with reason-specific timeouts:

  • VMSCAN_THROTTLE_WRITEBACK (timeout HZ/10 ≈ 100 ms): too many pages under writeback at the tail of the inactive LRU — reclaim sleeps until enough dirty pages are cleaned (__acct_reclaim_writeback() wakes it once SWAP_CLUSTER_MAX * nr_throttled pages have been written back).
  • VMSCAN_THROTTLE_CONGESTED and VMSCAN_THROTTLE_NOPROGRESS (timeout 1 jiffy): the node is congested or a reclaim pass made no forward progress; back off briefly so the task does not spin.
  • VMSCAN_THROTTLE_ISOLATED (timeout HZ/50 ≈ 20 ms): too many pages are already isolated by other parallel reclaimers; a short, transient wait.

The code comment is refreshingly honest — “These figures are pulled out of thin air” — and the LWN coverage of the 2021 rework (LWN — reclaim throttling) explains the motivation: congestion_wait() keyed off block-device congestion, a signal that became meaningless for fast SSDs and stacked/virtual devices, so it was replaced with these explicit, reason-tagged throttle points. reclaim_throttle() deliberately exempts kthreads and user-workers other than kswapd (PF_KTHREAD/PF_USER_WORKER), because those threads may be the very ones doing the writeback that would let reclaim proceed — throttling them would deadlock.

The pfmemalloc throttle — a different thing, do not confuse it

try_to_free_pages() opens with:

if (throttle_direct_reclaim(sc.gfp_mask, zonelist, nodemask))
    return 1;

This is not the general “reclaim is slow” backoff. throttle_direct_reclaim() is specifically the pfmemalloc-reserve backpressure for the swap-over-network / swap-over-NBD case. It checks allow_direct_reclaim(pgdat), which is true unless the sum of free pages across ZONE_NORMAL-and-below has fallen below half the combined min watermark (the “pfmemalloc reserve”). If the reserve is dangerously depleted, the task waits on pgdat->pfmemalloc_wait until kswapd makes progress and wakes it (wake_up_all(&pgdat->pfmemalloc_wait) in balance_pgdat). Each such throttle bumps the PGSCAN_DIRECT_THROTTLE vmstat counter. Kernel threads and tasks with a fatal signal pending are exempted (they may be needed to clean pages, or are exiting and will free memory). The returned 1 (rather than 0) on a fatal-signal abort tells the allocator not to OOM-kill at this point. Keep this distinct from reclaim_throttle(): throttle_direct_reclaim() guards a specific reserve against a specific deadlock (allocating network buffers in order to swap), whereas reclaim_throttle() is the general per-reason backoff inside shrink_node().

Why Direct Reclaim Is a Sign of Pressure

Walk the slow path again with the timing in mind. The fast path failed (free < low watermark in every eligible zone). kswapd was woken. The min-watermark retry failed. Then the task reclaimed itself. For all of that to happen, the per-node kswapd — which is supposed to keep free memory between the low and high watermarks — was either not yet awake, or awake but unable to free pages as fast as allocations were consuming them. Direct reclaim is the system saying: background reclaim could not stay ahead of demand, so I am charging the cost to the application. That is qualitatively different from kswapd running — kswapd running is normal and asynchronous; direct reclaim running means a thread is blocked.

The cost is real and measurable. The blocked task accrues PSI memory-stall time; it may sleep in reclaim_throttle() for tens to hundreds of milliseconds; and if many threads hit the slow path at once they serialize on shared reclaim state and on the dirty-page writeback they are all waiting for. For latency-sensitive services this manifests as tail-latency spikes that correlate with allocstall/pgscan_direct climbing in /proc/vmstat.

Configuration / Diagnosis

There is no knob to “turn off” direct reclaim (short of using non-blocking GFP flags in your own kernel code). The lever is to make kswapd start earlier and reclaim more, so the fast path rarely fails. The two relevant sysctls (kernel vm docs):

# Widen the gap between the watermarks so kswapd wakes sooner and
# reclaims more before sleeping. Units: fraction of 10000 of node memory.
# Default 10 (= 0.1%). Raising it pushes the low/high lines further apart.
sysctl vm.watermark_scale_factor=200      # 2% of node memory
 
# Raise the absolute reserve floor (the min watermark). Bigger min ->
# bigger low/high -> kswapd has more headroom before tasks stall.
sysctl vm.min_free_kbytes=262144          # 256 MiB

The watermark arithmetic (from mm/page_alloc.c:__setup_per_zone_wmarks) makes the relationship concrete. With gap = max(min/4, managed_pages * watermark_scale_factor / 10000):

zone->_watermark[WMARK_LOW]  = min_wmark_pages(zone) + gap;
zone->_watermark[WMARK_HIGH] = low_wmark_pages(zone) + gap;

So low and high are equally spaced above min, and watermark_scale_factor controls that spacing. A larger spacing means kswapd is woken further from exhaustion (more slack before the fast path fails) and reclaims further past the danger line before sleeping — directly reducing the probability of direct reclaim. See Watermarks and the Allocation Fast Path for the watermark model and kswapd and Background Reclaim for how kswapd consumes these lines.

Diagnosis recipe:

# Direct-reclaim activity: these climbing = applications stalling on memory.
grep -E 'allocstall|pgscan_direct|pgsteal_direct|pgscan_kswapd|pgscan_direct_throttle' /proc/vmstat
 
# How much wall-clock time tasks lost stalled on memory (PSI):
cat /proc/pressure/memory      # 'some'/'full' avg10/avg60/avg300 + total (us)
 
# Trace each direct-reclaim episode (begin/end, nr_reclaimed):
echo 1 > /sys/kernel/tracing/events/vmscan/mm_vmscan_direct_reclaim_begin/enable
echo 1 > /sys/kernel/tracing/events/vmscan/mm_vmscan_direct_reclaim_end/enable
cat /sys/kernel/tracing/trace_pipe

The mm_vmscan_direct_reclaim_begin/_end tracepoints are emitted by try_to_free_pages() itself and give per-episode latency and pages freed — the most direct view of who is stalling and for how long.

Failure Modes and Common Misunderstandings

  • “Direct reclaim means the system is out of memory.” No — it means the system is short right now relative to the watermarks, and kswapd is behind. The OOM killer is much later: only after reclaim makes zero progress across the retry loop does the slow path consider OOM (see The OOM Killer). Direct reclaim is the warning, not the funeral.
  • Confusing the two throttles. reclaim_throttle() (4 reasons, general backoff inside shrink_node) is not throttle_direct_reclaim() (pfmemalloc reserve, swap-over-network). Older write-ups also still mention congestion_wait() — that primitive is gone in 6.12/6.18.
  • Assuming atomic allocations stall. GFP_ATOMIC and any non-blocking context clear __GFP_DIRECT_RECLAIM; they fail fast rather than reclaim. A driver doing GFP_ATOMIC in an interrupt never enters this path — it just gets NULL (and must handle it). See GFP Flags and Allocation Contexts.
  • Thinking direct reclaim only frees pages for you. try_to_free_pages() reclaims into the zone’s free lists; the pages you free may be grabbed by another concurrent allocator before your retry. The drain-and-retry-once logic in __alloc_pages_direct_reclaim() partly compensates, but under heavy concurrency a thread can reclaim and still not get a page.
  • High-order amplification. Reclaiming for an order-0 page is cheap; reclaiming (and then compacting) for a high-order/THP allocation can be very expensive, which is why costly high-order requests often carry __GFP_NORETRY and bail to compaction or failure rather than grind in reclaim. See Memory Compaction and Page Order and Fragmentation.

Alternatives and When Each Fires

  • kswapd and Background Reclaim — the asynchronous, per-node alternative. Ideally all reclaim is kswapd’s; direct reclaim is the fallback when kswapd cannot keep pace. Tuning the watermarks (above) shifts work from direct reclaim onto kswapd.
  • Direct compaction — for high-order allocations, the slow path also tries __alloc_pages_direct_compact(), which migrates pages to assemble a contiguous run rather than freeing pages outright. Reclaim frees; compaction defragments.
  • The OOM Killer — the terminal alternative: when reclaim makes no progress and the allocation cannot fail, kill a process.
  • Failing the allocation — for __GFP_NORETRY/atomic requests, the “alternative” is simply returning NULL and letting the caller cope.

Production Notes

On busy multi-tenant hosts, direct reclaim is the usual culprit behind unexplained latency cliffs: a memory-hungry neighbor pushes a node below its watermarks, kswapd falls behind, and unrelated latency-sensitive tasks get conscripted into try_to_free_pages() and lose tens of milliseconds each. The modern diagnostic chain is PSI first (/proc/pressure/memory “some” rising → tasks are stalling on memory), then /proc/vmstat allocstall/pgscan_direct to confirm it is direct reclaim, then the mm_vmscan_direct_reclaim_* tracepoints to attribute it. The standard mitigations are raising vm.watermark_scale_factor / vm.min_free_kbytes so kswapd starts earlier, and — in containerized environments — setting memory.high below memory.max so a cgroup does proactive, throttled reclaim before it ever hits the hard limit and triggers synchronous reclaim or OOM (see Per-cgroup Reclaim and Memory Pressure).

Uncertain

Verify: the exact reclaim_throttle() timeout values (HZ/10, HZ/50, 1 jiffy) and the four vmscan_throttle_state reasons are read directly from mm/vmscan.c at v6.12/v6.18 and should be stable, but the comment “pulled out of thin air” signals these are heuristic and could change between releases. Reason: heuristic constants in fast-moving reclaim code. To resolve: re-check reclaim_throttle() in the target kernel before quoting the millisecond figures in production tuning. uncertain

See Also

Graph View

Backlinks