vault

khugepaged and THP Collapse

12 min read

khugepaged and THP Collapse

khugepaged (“kernel huge-page daemon”) is the background kernel thread that makes Transparent Huge Pages (THP) work after the fact: it periodically scans the address spaces of processes that have huge-page-eligible mappings, finds runs of 512 contiguous 4 KiB base pages that together fill a naturally-aligned 2 MiB region, and collapses them into a single Page Middle Directory (PMD)-mapped huge page by allocating a fresh huge folio, copying the data in, and atomically swapping the page-table mapping. This is the “promotion” half of THP — the path that upgrades memory the page-fault handler originally satisfied with base pages (because no huge page was free, or because the region was filled in piecemeal). The daemon runs at the lowest scheduling priority and on a deliberate sleep/scan cadence so it never competes with real work; its behavior is tuned through a family of knobs under /sys/kernel/mm/transparent_hugepage/khugepaged/ (v6.18 admin guide). As of 6.12 LTS and 6.18 LTS, khugepaged collapses only to the PMD size — it makes no attempt to produce smaller multi-size THP (mTHP).

This is the companion to Transparent Huge Pages, which covers the fault-time path, the enabled/defrag policies, mTHP, and splitting. Read that first for the THP umbrella; this note is specifically the collapse daemon.

Mental Model

The page-fault handler is eager but cheap: when a large mapping is faulted, it tries for a huge page using GFP_TRANSHUGE_LIGHT, which fails instantly if no contiguous 2 MiB run is free, and falls back to base pages rather than stalling (see GFP Flags and Allocation Contexts). That leaves a lot of memory that could be huge but isn’t — either because the system was fragmented at fault time, or because the region was written one base page at a time. khugepaged is the patient, after-the-fact cleanup crew: it walks process page tables in the background, and where it finds a fully-eligible 2 MiB region it does the expensive work the fault path refused to do — including, if defrag is on, stalling in compaction to manufacture the contiguous huge page.

flowchart TD
  REG["mmap registers an mm in<br/>khugepaged's scan list<br/>(THP-eligible VMA seen)"] --> SLEEP["sleep scan_sleep_millisecs<br/>(default 10000 = 10 s)"]
  SLEEP --> SCAN["scan up to pages_to_scan<br/>PTEs this pass (default 4096)"]
  SCAN --> CHECK{"2 MiB region fully<br/>eligible?<br/>(none/swap/shared<br/>under thresholds)"}
  CHECK -->|"no"| SLEEP
  CHECK -->|"yes"| ALLOC["alloc_charge_folio:<br/>GFP_TRANSHUGE (may compact),<br/>charge to memcg"]
  ALLOC -->|"fail"| BACKOFF["sleep alloc_sleep_millisecs<br/>(default 60000 = 60 s)"]
  ALLOC -->|"ok"| COLLAPSE["collapse_huge_page:<br/>lock, copy 512 pages,<br/>install huge PMD"]
  BACKOFF --> SLEEP
  COLLAPSE --> SLEEP

The khugepaged scan/collapse loop. What it shows: the daemon is purely periodic — it sleeps, scans a bounded number of PTEs, and for each fully-eligible region does an allocate-then-collapse, backing off hard (60 s) on allocation failure. The insight: khugepaged decouples the decision to use a huge page from the moment of fault, paying the expensive allocation/compaction cost on a background thread at MAX_NICE priority instead of on a latency-sensitive user thread. The thresholds at the “eligible?” branch are exactly the max_ptes_* knobs.

The Daemon’s Main Loop

The thread is a standard kernel kthread whose entry point is khugepaged() (khugepaged.c, v6.12). It immediately drops to the lowest priority — set_user_nice(current, MAX_NICE) — so it yields to everything else, then loops: khugepaged_do_scan() followed by khugepaged_wait_work().

khugepaged_do_scan() reads pages_to_scan once into a local, then repeatedly calls khugepaged_scan_mm_slot() to scan registered address spaces (“mm slots”) until the accumulated progress reaches pages_to_scan for this pass. Each pass scans a bounded amount of work and then sleeps, which is what keeps the daemon’s CPU footprint low. The wait between passes is khugepaged_wait_work(): if there is work pending it sleeps for scan_sleep_millisecs (via wait_event_freezable_timeout), otherwise it sleeps until a THP-eligible mapping appears.

Crucially, an address space only enters the scan list when something registers it: a process gets added when it has a VMA eligible for THP (an anonymous or shmem mapping under the current policy, or one marked MADV_HUGEPAGE). The daemon is started automatically when PMD-sized THP is enabled (the per-size anon control or top-level enabled is always/madvise) and shut down when both are never (admin guide).

The Scan Parameters

All knobs live under /sys/kernel/mm/transparent_hugepage/khugepaged/. The defaults below are read directly from the v6.12 source initializer, expressed in terms of HPAGE_PMD_NR — the number of base pages in a PMD huge page (512 on x86-64 with a 4 KiB base and 2 MiB huge page) (khugepaged.c, v6.12):

khugepaged_pages_to_scan        = HPAGE_PMD_NR * 8;   /* = 4096 */
khugepaged_scan_sleep_millisecs = 10000;             /* 10 s   */
khugepaged_alloc_sleep_millisecs = 60000;            /* 60 s   */
khugepaged_max_ptes_none        = HPAGE_PMD_NR - 1;  /* = 511  */
khugepaged_max_ptes_swap        = HPAGE_PMD_NR / 8;  /* = 64   */
khugepaged_max_ptes_shared      = HPAGE_PMD_NR / 2;  /* = 256  */
  • pages_to_scan (default 4096 = 8 PMD-pages’ worth of PTEs) — how many PTEs the daemon examines per pass. Larger = faster promotion, more CPU.
  • scan_sleep_millisecs (default 10000, i.e. 10 s) — how long to sleep between passes. The doc notes: set it to 0 to run khugepaged at 100 % of one core — a useful trick to rapidly THP-ify a freshly-started workload, then turn back down.
  • alloc_sleep_millisecs (default 60000, i.e. 60 s) — how long to back off after a huge-page allocation failure, throttling the next attempt so a fragmented system isn’t hammered with failing order-9 requests. (The loop also has a fast-fail: on SCAN_ALLOC_HUGE_PAGE_FAIL it sleeps once, then aborts the whole pass on a second failure.)
  • defrag (echo 0/1) — whether khugepaged may invoke defrag (reclaim/compaction) on allocation. Independent of the fault-path transparent_hugepage/defrag knob; this one governs the daemon only.

The max_ptes_* eligibility thresholds

These three thresholds decide whether a candidate 2 MiB region is “collapsible enough.” During the scan (__collapse_huge_page_isolate and the scan-side check), the daemon walks all 512 PTEs and counts (khugepaged.c, v6.12):

  • max_ptes_none (default 511) — how many of the 512 slots may be empty (not mapped, or pointing at the zero page). A higher value lets the daemon collapse very sparse regions, allocating real memory for never-touched holes — the source of THP memory bloat. The default of 511 means even a region with a single populated PTE is collapsible; if the count of none/zero PTEs exceeds the threshold the scan returns SCAN_EXCEED_NONE_PTE. This same threshold defines an “underused” THP for the 6.18 underused-shrinker (see Transparent Huge Pages).
  • max_ptes_swap (default 64) — how many of the 512 pages may be swapped out. To collapse a region with swapped pages, the daemon must first swap them back in (__collapse_huge_page_swapin); a higher value risks excessive swap I/O, a lower value blocks collapse of partly-swapped regions (SCAN_EXCEED_SWAP_PTE).
  • max_ptes_shared (default 256) — how many pages may be shared across processes (mapped with refcount/mapcount > 1, e.g. after fork COW). Exceeding it blocks the collapse (SCAN_EXCEED_SHARED_PTE), because collapsing shared pages would force private copies.

The scan records its outcome as one of the enum scan_result values (SCAN_SUCCEED, SCAN_EXCEED_NONE_PTE, SCAN_PAGE_RO, SCAN_LACK_REFERENCED_PAGE, SCAN_ALLOC_HUGE_PAGE_FAIL, SCAN_CGROUP_CHARGE_FAIL, and ~30 others) — a fine-grained taxonomy of exactly why a collapse did or didn’t happen.

Allocation and the GFP Mask

When a region passes the scan, khugepaged allocates the huge folio via alloc_charge_folio(), which calls alloc_hugepage_khugepaged_gfpmask() (khugepaged.c, v6.12):

static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
{
    return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT;
}

This is the key difference from the fault path. If the daemon’s defrag knob is on, it uses GFP_TRANSHUGE — which includes __GFP_DIRECT_RECLAIM, so the allocation can stall in reclaim and compaction to manufacture a contiguous 2 MiB run. If defrag is off it uses the fail-fast GFP_TRANSHUGE_LIGHT. Because khugepaged runs in the background at MAX_NICE, paying compaction cost here is acceptable in a way it never is on a faulting user thread (see GFP Flags and Allocation Contexts and the GFP discussion in Transparent Huge Pages). The new folio is then charged to the originating task’s memory cgroup via mem_cgroup_charge(); if the cgroup is at its limit the collapse aborts with SCAN_CGROUP_CHARGE_FAIL, and thp_collapse_alloc/thp_collapse_alloc_failed in /proc/vmstat record success/failure.

The Collapse — Step by Step

The actual collapse, collapse_huge_page(), is a careful dance of three locks because it must atomically replace 512 PTEs with one huge PMD while racing faults, GUP-fast, and reclaim. The sequence (khugepaged.c, v6.12):

  1. Drop the mmap read lock, then allocate (alloc_charge_folio). The comment is explicit: the allocation “can take potentially a long time if it involves sync compaction, and we do not need to hold the mmap_lock during that.” Holding the address-space lock across compaction would stall every other thread of the process.
  2. Re-take mmap read lock and revalidate (hugepage_vma_revalidate) — the VMA may have been unmapped/remapped while the lock was dropped. Re-find the PMD.
  3. Swap in if needed — if any PTE was swapped out (and within max_ptes_swap), __collapse_huge_page_swapin() faults the pages back. On failure it returns with the lock released and the collapse aborts to avoid inconsistency.
  4. Take the mmap write lock, revalidate again, and check the PMD is still valid. The write lock is what serializes against concurrent page-table walkers — the design doc’s locking contract says any huge-PMD-aware walker holds mmap_lock precisely so collapse can’t run underneath it (design doc).
  5. Acquire anon_vma write lock and an MMU-notifier range (so KVM/IOMMU shadow mappings are invalidated).
  6. pmdp_collapse_flush() — clear the PMD and flush the TLB, so no CPU holds a stale base-page TLB entry for this range. The comment notes this avoids the hardware hazard of mixed huge/small TLB entries for the same address; GUP-fast is safe because it backs off when it sees the PMD changed.
  7. Isolate and copy — __collapse_huge_page_isolate() pulls the 512 base pages off the LRU and locks them; __collapse_huge_page_copy() copies their contents into the new huge folio. If the copy hits an uncorrectable memory error it returns SCAN_COPY_MC and restores the original page table.
  8. Install the huge PMD — mk_huge_pmd() builds the entry, folio_add_new_anon_rmap() / folio_add_lru_vma() wire the new folio into the reverse-mapping and LRU, pgtable_trans_huge_deposit() stashes the old page-table page (needed for a future split), and set_pmd_at() atomically publishes the huge mapping. The folio is also placed on the deferred-split list immediately. The old base pages, now isolated, are freed.

If any step fails after the PMD was cleared, the code restores the original page-table page via pmd_populate() so the region simply reverts to its 512 base PTEs — graceful fallback again.

When Collapse Fails

The common reasons, surfaced as scan_result codes and reflected in counters/tracepoints:

  • Too sparse / too much swap / too much sharing — exceeding max_ptes_none, max_ptes_swap, or max_ptes_shared (SCAN_EXCEED_*).
  • No referenced or accessed pages (SCAN_LACK_REFERENCED_PAGE) — the daemon won’t promote cold memory it has no evidence is in use.
  • Read-only or non-anon page where an anon page was expected (SCAN_PAGE_RO, SCAN_PAGE_ANON).
  • Allocation failure (SCAN_ALLOC_HUGE_PAGE_FAIL) — no contiguous 2 MiB run and (with defrag) compaction couldn’t make one; triggers the alloc_sleep_millisecs back-off. On a chronically fragmented system this is the dominant failure, and it is why Memory Compaction health is the single biggest determinant of whether khugepaged succeeds.
  • memcg over limit (SCAN_CGROUP_CHARGE_FAIL) — the cgroup can’t afford the huge folio.
  • Pinned pages — a GUP pin elevates the refcount past the expected value and the page can’t be isolated; collapse aborts (and conversely, a later split_huge_page of the result would also fail while pinned — see the refcount invariant in Transparent Huge Pages).

MADV_COLLAPSE — Synchronous, On-Demand Collapse

khugepaged is asynchronous and best-effort. For workloads that want collapse now, on a known region, madvise(addr, len, MADV_COLLAPSE) invokes the same collapse machinery synchronously in the caller’s context. Its implementation, madvise_collapse() in mm/khugepaged.c (v6.12), confirms the reuse: it allocates a collapse_control with cc->is_khugepaged = false, walks the requested range in PMD-aligned steps, and calls the very same hpage_collapse_scan_pmd() (or hpage_collapse_scan_file() for shmem) that the daemon’s scan uses — the only difference is the is_khugepaged flag, which (among other things) makes it ignore the TVA_ENFORCE_SYSFS check and thus the enabled/never sysfs settings. That is why, as the 6.18 doc warns, setting never everywhere does not truly disable THP, and why MADV_COLLAPSE is the deterministic counterpart to the background daemon.

Uncertain

Verify: the exact upstream release that introduced MADV_COLLAPSE. Reason: madvise_collapse() is present in the v6.12 source I traced, but the madvise(2) man-page copy I fetched (master branch) did not yet document the MADV_COLLAPSE constant, so I could not pin its introducing release from a primary source (it is attributed to Linux 6.1, late 2022, from memory). To resolve: check the madvise(2) “since Linux” annotation once the man page documents it, or git log mm/madvise.c/mm/khugepaged.c for the MADV_COLLAPSE introduction. uncertain

Production Notes — Tuning the Daemon

The two most common interventions: set scan_sleep_millisecs=0 briefly to aggressively THP-ify a process right after it allocates its working set (then restore the default to stop burning a core), and lower max_ptes_none well below 511 to stop khugepaged from collapsing sparse regions and inflating RSS — the cheap defense against THP memory bloat short of disabling THP entirely. Watch thp_collapse_alloc vs thp_collapse_alloc_failed in /proc/vmstat: a high failure ratio with low compact_success points at fragmentation, not at khugepaged itself. Because the daemon runs at MAX_NICE, it rarely shows up as a CPU problem unless scan_sleep_millisecs is set to 0 on a large, busy machine.

See Also

Graph View

Backlinks