MGLRU Page Table Walks

The Multi-Generational LRU (MGLRU) decides which pages are still hot by reading the hardware accessed bit (also called the young bit) that the CPU’s memory-management unit sets in a page-table entry (PTE) whenever software touches the page it maps. The classic reclaim scanner finds those PTEs the slow way — for each physical page on a list it runs a reverse-map (rmap) lookup to discover every virtual mapping, a path the kernel itself calls the highest CPU cost in reclaim for mapped-heavy workloads (kernel mm docs). MGLRU instead walks the process page tables forward (walk_page_range() over each mm_struct), sweeping whole runs of contiguous PTEs at once, test-and-clearing their accessed bits, and promoting the pages it finds young to the youngest generation. A Bloom filter records which page-table regions were densely populated with hot pages so the walker can skip the sparse ones on the next pass. This note explains that machinery — the per-mm walk, the rmap-driven lru_gen_look_around(), the accessed-bit clearing, and the Bloom-filter feedback loop — as it stands in Linux 6.12 LTS and 6.18 LTS.

This note assumes you know what MGLRU’s generations are for; the generational model itself lives in MGLRU Generations and Aging, and the head-to-head contrast with the old two-list scanner is in MGLRU vs the Classic LRU. Here we focus on the one specific question: how does the aging pass actually discover young pages?

What this note owns

This is the scanning note. It owns everything between “the aging decided to run” and “a folio’s gen counter was updated”: the mm_struct list, walk_page_range() and its four levels, the accessed-bit test-and-clear, the non-leaf PMD batching, lru_gen_look_around(), the Bloom filters, and the architecture predicates that decide whether any of it happens at all. It does not own what a generation is or when the aging is triggered — MGLRU Generations and Aging owns min_seq/max_seq, tiers, should_run_aging() and the refault controller. It does not own enablement or the enabled sysfs ABI as an operations surface — Multi-Generational LRU owns that; here the same bits appear only as the two code predicates should_walk_mmu() and should_clear_pmd_young(). It does not own the PTE flag layout — Page Table Entries and Hardware Flags does — nor the Bloom filter as a data structure, which is Bloom Filter.

Version pin

Every line number, constant and code excerpt below was read from the Linux v6.12 tree during this write-up. Checked by fetching both files at both tags: Documentation/mm/multigen_lru.rst is byte-identical from v6.12 through v7.2, and the # multi-gen LRU { block of mm/Kconfig (which contains CONFIG_LRU_GEN_WALKS_MMU) is byte-identical across v6.12, v6.16, v6.17, v6.18, v7.0, v7.1 and v7.2 — same 29 lines, same MD5. Nothing in the walk’s configuration surface has moved in that window. The live measurements at the end come from a 7.1.13-200.fc44 kernel and are labelled as such.

Mental Model — Two Ways to Reach a PTE

Every page-replacement algorithm ultimately needs the same datum: was this page accessed recently? On most architectures the only hardware signal is the accessed bit the MMU sets in the leaf PTE on access (see Page Table Entries and Hardware Flags for the bit layout). The whole design question is how you get from a page to its PTEs, and there are exactly two directions:

flowchart LR
  subgraph RMAP["Reverse map (rmap) direction — physical → virtual"]
    P["A physical page<br/>on an LRU list"]
    P -->|"rmap lookup:<br/>anon_vma / address_space"| PTE1["PTE in process A"]
    P -->|"...for every mapping"| PTE2["PTE in process B"]
  end
  subgraph WALK["Page-table walk direction — virtual &rarr; physical"]
    MM["An mm_struct<br/>(one process)"]
    MM -->|"walk_page_range()"| RANGE["Sweep a whole<br/>PTE table (512 entries)"]
    RANGE --> PA["...touch every mapped page<br/>in one cache-friendly pass"]
  end

The two directions for reaching a page-table entry. What it shows: the classic scanner starts from a physical page and runs an rmap lookup to fan out to every virtual mapping (left); MGLRU’s aging starts from a process address space and sweeps forward through the page tables, harvesting many PTEs in one contiguous pass (right). The insight: the rmap direction is random-access across virtual address space (cache-unfriendly, one page at a time); the page-table-walk direction has spatial locality (one cacheline of PTEs covers eight pages). MGLRU uses both, picking whichever is cheaper for the region in front of it — the Bloom filter is how it decides.

Why the Classic rmap Scan Is Expensive

To see what MGLRU is optimizing away, recall the classic scanner’s problem. The reverse map exists because a single physical page can be mapped at many virtual addresses in many processes (shared libraries, forked children sharing copy-on-write pages, files mapped by multiple tasks). To answer “is this page young?” the classic scanner takes a page off an LRU list and asks rmap for every mapping; for an anonymous page that means walking the anon_vma interval tree, for a file page the address_space interval tree. The kernel’s own MGLRU design document is blunt about the cost: “Searching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs (PA space) are not cache friendly to the rmap (VA space). For workloads mostly using mapped pages, searching the rmap can incur the highest CPU cost in the reclaim path.” (kernel mm docs). LWN’s original write-up makes the same point: “the scanning of anonymous pages is expensive, partly because it uses a complex reverse-mapping mechanism that does not perform well when a lot of scanning must be done” (Corbet 2021).

The deeper issue is locality mismatch. The LRU lists are ordered roughly by access recency, which has no relationship to virtual-address adjacency. So consecutive pages the scanner pulls off a list map to scattered, unrelated PTEs in different page tables. Each rmap lookup is a fresh tree walk and a cold cacheline. MGLRU’s response: “A rmap walk targets a single page and does not try to profit from discovering a young PTE. A page table walk can sweep all the young PTEs in an address space, but the address space can be too sparse to make a profit. The key is to optimize both methods and use them in combination.” (kernel mm docs).

The two directions, costed

The design document’s “use them in combination” is the whole thesis, so it is worth laying the two methods side by side with the cost model made explicit. This is a structural comparison of the two traversal directions, not a benchmark — no measured numbers are quoted anywhere in this note (see the closing source note for why):

	rmap walk (physical → virtual)	page-table walk (virtual → physical)
Starting point	one struct folio already on an LRU list	one struct mm_struct on lruvec_memcg()->mm_list
Data structure traversed	anon_vma interval tree, or address_space interval tree	the 4- or 5-level page table, top-down
Kernel entry point	rmap_walk() / folio_referenced(); under MGLRU, lru_gen_look_around() from shrink_folio_list()	walk_page_range() with .p4d_entry = walk_pud_range
Yield per unit of work	one PTE, unless look_around extends it	up to 512 PTEs per leaf table touched
Cacheline behaviour	one cold line per mapping; consecutive list entries are unrelated addresses	one 64-byte line covers 8 contiguous PTEs on x86-64
Cost driver	number of mappings per folio (shared libraries, forked children)	sparsity — a mostly-unmapped address space costs the same to walk as a dense one
Works when the folio is	mapped by anything	mapped by a process on this memcg’s mm_list, and this node
Handles unmapped page cache	not applicable — no PTEs to find	no: file pages touched only through read()/write() never appear in any page table
Required hardware support	none (it can read a software-maintained bit)	a hardware accessed bit, i.e. arch_has_hw_pte_young()
Skips idle processes	no	yes — the scheduler hook lets the walker skip tasks that have not run
Selected by	always available; the eviction path uses it	should_walk_mmu(), and then per-region by the Bloom filter
How the other one helps it	the walk’s promotions mean fewer folios reach the rmap at all	lru_gen_look_around() inserts dense PMDs into the filter, telling the walk where to go

The two accessed-bit harvesting directions, compared structurally. What it shows: neither direction dominates. The rmap wins when the target is a single folio whose mappings are few; the walk wins when a contiguous region is densely mapped and mostly hot. The insight to take: the last two rows are the actual design. MGLRU does not choose one direction — it runs both and lets each one reduce the other’s work. The Bloom filter is the channel the eviction’s rmap walks use to tell the aging’s page-table walks which PMDs are worth sweeping. Take the filter away and you have two independent scanners; put it in and you have a feedback loop. Note also the “handles unmapped page cache” row: the page-table walk is structurally blind to file pages that are only ever read(), which is why tiers (the folio->flags access counter, see MGLRU Generations and Aging) exist as an entirely separate mechanism for the file-descriptor access channel.

The Per-mm Page-Table Walk (the Aging Path)

The aging pass — the half of MGLRU that produces young generations by incrementing max_seq (see MGLRU Generations and Aging) — discovers young pages primarily by walking page tables. The mechanism, as the design doc states, is: “it iterates lruvec_memcg()->mm_list and calls walk_page_range() with each mm_struct on this list to scan PTEs, and after each iteration, it increments max_seq.” (kernel mm docs).

Each memory control group (memcg) keeps an mm_struct list — a FIFO of the address spaces belonging to its tasks — and an mm_struct follows its owning task to a new memcg if the task is migrated. The walker iterates this list; when several walkers run concurrently (parallel reclaim across nodes) each takes a unique mm_struct off the list, so they parallelize without stepping on each other. Crucially, “This infrastructure also tracks the usage of mm_struct between context switches so that page table walkers can skip processes that have been sleeping since the last iteration” — a scheduler hook records which tasks actually ran, so an idle process’s page tables are skipped entirely (kernel mm docs; Corbet 2021).

The driver state for one walk is struct lru_gen_mm_walk (defined in include/linux/mmzone.h, v6.12):

struct lru_gen_mm_walk {
        struct lruvec *lruvec;          /* the lruvec under reclaim */
        unsigned long seq;              /* max_seq from lru_gen_folio: can be out of date */
        unsigned long next_addr;        /* the next address within an mm to scan */
        int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];  /* to batch promoted pages */
        int mm_stats[NR_MM_STATS];      /* to batch the mm stats */
        int batched;                    /* total batched items */
        bool can_swap;
        bool force_scan;
};

Source: include/linux/mmzone.h, v6.12 (identical in v6.18). next_addr lets the walk resume mid-mm if it bails out; nr_pages[...] batches the count of pages promoted into each generation so the expensive LRU-lock update happens once per batch rather than once per page; can_swap tells the walker whether anonymous pages are reclaimable (so it can skip them if there is no swap).

Iterating the list: iterate_mm_list() and its three cases

The list walk is not a plain for_each. iterate_mm_list() (mm/vmscan.c, v6.12) advances a cursor stored in lru_gen_mm_state and its own comment enumerates the situations it must handle:

mm_state->seq is incremented after each iteration of mm_list. There are three interesting cases for this page table walker:

1. It tries to start a new iteration with a stale max_seq: there is nothing left to do.
2. It started the next iteration: it needs to reset the Bloom filter so that a fresh set of PTE tables can be recorded.
3. It ended the current iteration: it needs to reset the mm stats counters and tell its caller to increment max_seq.

Three consequences follow, and none of them are obvious from the design document:

• The generation is minted by finishing the list, not by scanning a page. iterate_mm_list() returns true only when the cursor wraps back to &mm_list->fifo, at which point try_to_inc_max_seq() calls inc_max_seq(). One full pass over every address space in the memcg equals exactly one new generation. A memcg with a thousand processes therefore produces generations more slowly per unit of scanning than one with ten — the generation boundary is an address-space boundary, not a page count.
• Newly-added address spaces are force-scanned. if (!mm_state->tail || mm_state->tail == mm_state->head) { mm_state->tail = mm_state->head->next; walk->force_scan = true; } — the tail cursor marks where the previous iteration ended, so any mm_struct added since then gets force_scan = true, which (as the next sections show) bypasses both the Bloom filter and the non-leaf young-bit shortcut. A process that has just started is scanned exhaustively once, because MGLRU has no history for it and its filter entries do not exist yet.
• The filter is reset one generation ahead. if (mm && first) reset_bloom_filter(mm_state, walk->seq + 1); — the reset happens at the start of an iteration and targets filter seq+1, the one that will be read next generation. This is the double-buffering, and it is the detail the Bloom-filter section below returns to.

The scheduler hook: one bit per node, per address space

get_next_mm() is where the design document’s claim — “this infrastructure also tracks the usage of mm_struct between context switches so that page table walkers can skip processes that have been sleeping since the last iteration” (kernel mm docs) — becomes three lines of code:

mm  = list_entry(mm_state->head, struct mm_struct, lru_gen.list);
key = pgdat->node_id % BITS_PER_TYPE(mm->lru_gen.bitmap);
 
if (!walk->force_scan && !test_bit(key, &mm->lru_gen.bitmap))
        return NULL;
 
clear_bit(key, &mm->lru_gen.bitmap);

The storage is a single unsigned long embedded in every mm_struct, under #ifdef CONFIG_LRU_GEN_WALKS_MMU (include/linux/mm_types.h at v6.12, lines 1019-1031), whose comment reads: “Set when switching to this mm_struct, as a hint of whether it has been used since the last time per-node page table walkers cleared the corresponding bits.” The setter is as cheap as it could possibly be:

static inline void lru_gen_use_mm(struct mm_struct *mm)
{
	/*
	 * When the bitmap is set, page reclaim knows this mm_struct has been
	 * used since the last time it cleared the bitmap. So it might be worth
	 * walking the page tables of this mm_struct to clear the accessed bit.
	 */
	WRITE_ONCE(mm->lru_gen.bitmap, -1);
}

Source: include/linux/mm_types.h at v6.12, lines 1087-1095. Called from context_switch() in kernel/sched/core.c at v6.12, line 5313, immediately after switch_mm_irqs_off(). When CONFIG_LRU_GEN_WALKS_MMU=n the whole function compiles to nothing.

Four details are worth extracting. First, the scheduler-side cost is one unconditional non-atomic store of -1 — no test, no read-modify-write, no branch. Reclaim’s observability into the scheduler costs the scheduler a single mov on the context-switch path, which is why nobody objects to it. Second, setting -1 sets every node’s bit at once: one context switch re-arms this address space for all NUMA nodes’ walkers simultaneously, and each node then clears only its own bit in get_next_mm(). Third, key = node_id % BITS_PER_TYPE(unsigned long) means node ids alias modulo 64 on a 64-bit machine; a system with more than 64 memory nodes would have nodes 0 and 64 sharing a bit, which is harmless (they would occasionally skip each other’s work or scan redundantly) but is a genuine approximation baked into the word size. Fourth, force_scan bypasses the check entirely, which is why a debugfs + command and a newly-added mm_struct both scan sleeping processes that the ordinary aging would skip.

A process that has not been scheduled since the last pass therefore contributes nothing and is skipped without its page tables being touched at all — which is the single largest saving on a machine with many mostly-idle processes, and something the rmap direction has no way to express.

The call stack, level by level

walk_mm() installs a struct mm_walk_ops and calls the generic walk_page_range() machinery:

static const struct mm_walk_ops mm_walk_ops = {
        .test_walk = should_skip_vma,
        .p4d_entry = walk_pud_range,
        .walk_lock = PGWALK_RDLOCK,
};

Note what is not there: no .pmd_entry, no .pte_entry. MGLRU hooks in at the P4D level and then descends by hand, because the generic walker’s per-entry callbacks would defeat the batching that is the entire point. The resulting stack:

Function	Level	What it does	Locking	Bail-out
walk_mm()	one mm_struct	mmap_read_trylock(), call walk_page_range(), then reset_batch_size() under lru_lock; loop while -EAGAIN	mem_cgroup_trylock_pages(), mmap_read_trylock()	-EBUSY if walk->seq != max_seq (someone else already advanced it), or if either trylock fails
should_skip_vma()	one VMA	the .test_walk filter — see the rejection table below	none	returns non-zero to skip the VMA entirely
walk_pud_range()	512 PUDs	loop PUDs, descend into walk_pmd_range(); get_next_vma(P4D_MASK, PUD_SIZE, ...) to jump to the next mapped VMA	none	if (need_resched() || walk->batched >= MAX_LRU_BATCH) → record walk->next_addr and return -EAGAIN
walk_pmd_range()	512 PMDs	the two-pass loop: Bloom-filter test, descend to PTEs, and batch non-leaf young-bit clears	none of its own	—
walk_pmd_range_locked()	≤ 66 PMDs	takes the PMD lock once and clears up to 1 + MIN_LRU_BATCH + 1 non-leaf accessed bits in one hold	spin_trylock(pmd_lockptr(...))	gives up silently if the trylock fails
walk_pte_range()	512 PTEs	the leaf sweep: test-and-clear every accessed bit, promote every young folio	spin_trylock(ptl) on the PTE table	returns false immediately if the trylock fails; returns suitable_to_scan() at the end

The MGLRU page-table walk, level by level, from mm/vmscan.c at v6.12. What it shows: every lock in the stack is a trylock. mmap_read_trylock, spin_trylock(ptl), spin_trylock(pmd_lockptr) — the walk never blocks waiting for a lock it does not already hold. The insight to take: this is a scanner that would rather produce an incomplete answer than stall. Missing a young PTE costs one generation of accuracy for one page; blocking on mmap_lock during reclaim would stall every fault in that address space. The same philosophy explains the -EAGAIN unwind in walk_pud_range(): rather than hold the walk open across a reschedule, it saves walk->next_addr, returns all the way to walk_mm(), lets that re-acquire mmap_read_trylock() from scratch, and resumes. next_addr is the only piece of state that has to survive.

sequenceDiagram
    autonumber
    participant AG as try_to_inc_max_seq
    participant IT as iterate_mm_list
    participant WM as walk_mm
    participant GW as walk_page_range
    participant PUD as walk_pud_range
    participant PMD as walk_pmd_range
    participant PTE as walk_pte_range
    participant BF as Bloom filter
    participant FL as folio flags and lru_gen lists

    AG->>IT: seq = max_seq, get the next mm_struct
    IT->>IT: skip processes that have not run since last pass
    IT-->>AG: one mm_struct, or last = true when the list wraps
    AG->>WM: walk_mm(mm, walk)
    WM->>WM: mem_cgroup_trylock_pages, then mmap_read_trylock
    WM->>GW: walk_page_range with p4d_entry set to walk_pud_range
    GW->>PUD: one P4D worth of address space
    PUD->>PMD: for each present PUD
    PMD->>PMD: pass 1, no PMD lock held
    PMD->>PMD: if should_clear_pmd_young and the PMD is not young, SKIP the whole 2 MiB
    PMD->>BF: test_bloom_filter(mm_state, seq, pmd + i)
    BF-->>PMD: miss, so skip this PTE table entirely
    BF-->>PMD: hit, so MM_NONLEAF_FOUND grows
    PMD->>PTE: walk_pte_range on the 512 leaf entries
    PTE->>PTE: spin_trylock the PTE table lock
    loop for each of up to 512 PTEs
        PTE->>PTE: get_pte_pfn rejects absent, zero, special, wrong-node
        PTE->>PTE: ptep_clear_young_notify tests AND clears the accessed bit
        PTE->>FL: folio_update_gen to the youngest generation
        PTE->>PTE: update_batch_size, deferring the lru_lock work
    end
    PTE-->>PMD: suitable_to_scan(total, young)
    PMD->>BF: if suitable, update_bloom_filter(seq + 1), MM_NONLEAF_ADDED grows
    PMD->>PMD: pass 2, walk_pmd_range_locked clears batched non-leaf young bits
    PUD-->>WM: EAGAIN once batched reaches MAX_LRU_BATCH or need_resched
    WM->>FL: reset_batch_size under lru_lock, applying all deferred counts at once
    WM->>WM: resume from walk->next_addr

One mm_struct’s walk, end to end. What it shows: three independent filters fire before a single PTE is read — the scheduler’s “has this process run?” check, the non-leaf young bit, and the Bloom filter — and only then does the leaf sweep happen. The insight to take: steps 25 and 26 are why this is cheap. Every promotion during the sweep is recorded in walk->nr_pages[gen][type][zone], an int array on the walker’s own stack-allocated state; lru_lock is taken exactly once, at the end, in reset_batch_size(). The classic scanner takes lru_lock per batch of isolated folios while also doing rmap lookups. MGLRU moved both costs out of the inner loop.

Sweeping one PTE table: walk_pte_range()

The leaf of the walk is walk_pte_range() (mm/vmscan.c, v6.12). It locks the PTE table, then loops over its entries. For each entry it extracts the page-frame number, resolves the folio, and then the load-bearing line:

if (!ptep_clear_young_notify(args->vma, addr, pte + i))
        continue;          /* not young: skip */
young++;
walk->mm_stats[MM_LEAF_YOUNG]++;
...
old_gen = folio_update_gen(folio, new_gen);   /* promote to youngest gen */
if (old_gen >= 0 && old_gen != new_gen)
        update_batch_size(walk, folio, old_gen, new_gen);

Source: walk_pte_range() in mm/vmscan.c, v6.12. ptep_clear_young_notify() is the atomic test-and-clear of the accessed bit: it returns whether the bit was set and clears it in the same operation (the _notify variant also informs the MMU-notifier subscribers, e.g. KVM, so a guest’s accessed bit is harvested too). If the bit was set, the page was touched since the last sweep, so it is “young”; the walker promotes it by setting its generation counter to new_gen = lru_gen_from_seq(max_seq) via folio_update_gen(), and batches the per-generation page-count delta. Pages whose accessed bit is clear are simply left where they are — they will age out as max_seq advances and min_seq eventually catches up.

Two refinements matter. First, the walk respects the VMA hints: should_skip_vma() skips regions that should not be aged, and the design assumes table-mapped accesses follow temporal locality “unless VM_SEQ_READ or VM_RAND_READ is present” (kernel mm docs) — i.e. madvise(MADV_SEQUENTIAL)/MADV_RANDOM change the policy. Second, after sweeping a table the function returns suitable_to_scan(total, young):

static bool suitable_to_scan(int total, int young)
{
        int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8);
        /* suitable if the average number of young PTEs per cacheline is >=1 */
        return young * n >= total;
}

Source: mm/vmscan.c, v6.12. n is how many PTEs fit in one cacheline (typically 8 on x86-64: a 64-byte line holds eight 8-byte PTEs). The test asks: was there on average at least one young PTE per cacheline read? If yes, the region was “dense” — worth walking again next time, so its address goes into the Bloom filter. If no, the region was mostly cold and re-scanning it would waste cache fetches.

What the sweep refuses to look at

Two filters decide which memory the walk even considers, and both are worth enumerating because “MGLRU walks the page tables” is a much broader statement than what the code actually does.

should_skip_vma() runs once per VMA as the generic walker’s .test_walk callback. Returning non-zero skips the whole region:

Test	Skipped because
!vma_is_accessible(vma)	PROT_NONE; nothing can have touched it
is_vm_hugetlb_page(vma)	hugetlbfs is not on the MGLRU lists at all
!vma_has_recency(vma)	VM_SEQ_READ or VM_RAND_READ — the application said via madvise() that recency does not predict reuse here
vma->vm_flags & (VM_LOCKED | VM_SPECIAL)	mlocked (unevictable) or special (VDSO, VM_PFNMAP, VM_IO)
vma == get_gate_vma(vma->vm_mm)	the architecture gate page
vma_is_anonymous(vma)	only if !walk->can_swap — with no swap, anonymous pages are unreclaimable, so reading their accessed bits is pure waste
shmem_mapping(mapping)	same, only if !walk->can_swap
mapping_unevictable(mapping)	e.g. ramfs
!mapping->a_ops->read_folio	“to exclude special mappings like dax, etc.” — nothing that cannot be faulted back in

get_pte_pfn() then runs once per PTE, and rejects with -1:

Test	Rejected because
!pte_present(pte)	swapped out, migrating, or a PROT_NONE NUMA-balancing entry — there is no page here now
is_zero_pfn(pfn)	the shared zero page is never reclaimed
pte_devmap(pte) || pte_special(pte)	device and special mappings (both wrapped in WARN_ON_ONCE — they should have been filtered by should_skip_vma())
!pte_young(pte) && !mm_has_notifiers(vma->vm_mm)	the fast path: the bit is already clear, so skip without even resolving the folio. The mm_has_notifiers() escape exists because an MMU notifier subscriber (KVM, for a guest) may hold accessed information the host PTE does not
!pfn_valid(pfn)	not real memory
pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)	wrong NUMA node — this reclaim pass is for a different node’s lruvec

and get_pfn_folio() rejects once more, on folio_nid(folio) != pgdat->node_id and folio_memcg_rcu(folio) != memcg — implementing the design document’s “page table walkers ignore any misplaced pages, e.g., if an mm_struct was migrated, pages left in the previous memcg will be ignored when the current memcg is under reclaim.”

The two rejection filters. What it shows: by the time a folio reaches ptep_clear_young_notify(), it has survived nine VMA tests and six PTE tests. The insight to take: the bolded !pte_young(pte) line is the one that makes the whole design viable. On a sparse or cold address space, the common case is that the accessed bit is already clear, and MGLRU detects that with a single non-atomic read of a PTE it was going to load into cache anyway — no folio lookup, no atomic, no lock. The expensive ptep_clear_young_notify() read-modify-write is paid only for PTEs that are actually young. This is precisely the asymmetry the rmap direction cannot exploit, because there the folio lookup happens first, by construction.

Two passes per PMD, and the non-leaf accessed bit

walk_pmd_range() is where the second architecture capability — the accessed bit in non-leaf PMD entries — earns its keep, and the structure is unusual enough that the source comments it:

Finish an entire PMD in two passes: the first only reaches to PTE tables to avoid taking the PMD lock; the second, if necessary, takes the PMD lock to clear the accessed bit in PMD entries.

In the first pass, for each of the 512 PMDs in a PUD:

if (!walk->force_scan && should_clear_pmd_young() && !mm_has_notifiers(args->mm)) {
        if (!pmd_young(val))
                continue;                       /* whole 2 MiB region is cold: SKIP */
        walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first);
}
 
if (!walk->force_scan && !test_bloom_filter(mm_state, walk->seq, pmd + i))
        continue;                               /* filter says sparse: SKIP */
 
walk->mm_stats[MM_NONLEAF_FOUND]++;
 
if (!walk_pte_range(&val, addr, next, args))
        continue;                               /* not cacheline-efficient: do not record */
 
walk->mm_stats[MM_NONLEAF_ADDED]++;
 
/* carry over to the next generation */
update_bloom_filter(mm_state, walk->seq + 1, pmd + i);

Read the first if carefully: when the hardware maintains a non-leaf accessed bit and LRU_GEN_NONLEAF_YOUNG is enabled, a single read of one PMD entry can retire 2 MiB of address space — 512 PTEs the walker never loads. The continue skips both the Bloom-filter test and walk_pte_range() entirely. That is the “negligible performance degradation” the admin guide attributes to turning bit 0x0004 off: negligible on average, but on an address space with large cold regions it is the difference between reading one cacheline and reading 512 PTEs’ worth.

Rather than take the PMD lock for each such entry, the walker batches. walk_pmd_range_locked() is a small state machine driven by two variables the caller owns — first (the address of the first PMD in the current batch) and a DECLARE_BITMAP(bitmap, MIN_LRU_BATCH):

1. First call with *first == -1: record *first = addr, zero the bitmap, return. Nothing is locked.
2. Subsequent calls within MIN_LRU_BATCH (64) PMD slots of *first: set bit i-1 in the bitmap and return. Still nothing locked.
3. A call beyond that range, or the flush call walk_pmd_range_locked(pud, -1, ...) at the end of the PMD loop: spin_trylock() the PMD lock once, iterate the recorded bits, and for each one either pmdp_test_and_clear_young() (a plain non-leaf clear) or — if the entry is a transparent huge page — the full pmdp_clear_young_notify() plus folio_update_gen() promotion. Then reset *first = -1.

The comment calls this “try to batch at most 1+MIN_LRU_BATCH+1 entries”, i.e. up to 66 PMD entries — up to 132 MiB of address space — cleared under one acquisition of the PMD spinlock. If the spin_trylock() fails, the whole batch is silently dropped (goto done) and those accessed bits stay set for another generation. Correctness is unaffected: a stale-set accessed bit makes a page look hotter than it is, which errs toward keeping it.

Two further things happen in that loop worth knowing:

• Transparent huge pages are handled here, not in walk_pte_range(). A pmd_trans_huge() entry is the leaf; get_pmd_pfn() and folio_update_gen() promote the 2 MiB folio directly, and MM_LEAF_YOUNG is incremented for it. So a THP-heavy workload’s young pages are discovered at the PMD level with 512x fewer entry reads. See Transparent Huge Pages.
• Dirty-bit propagation. Both walk_pte_range() and walk_pmd_range_locked() contain the same guarded folio_mark_dirty() call: if the hardware dirty bit is set in the entry but the folio is not yet marked dirty, the walk propagates it — unless the folio is anonymous, swap-backed and not in the swap cache, in which case marking it dirty would be meaningless. The accessed-bit sweep is therefore also, incidentally, a dirty-bit harvest.

flowchart TD
  A["walk_pmd_range: 512 PMD entries in one PUD"] --> B{"pmd_present and<br/>not the huge zero page?"}
  B -->|no| SKIP1["MM_LEAF_TOTAL++, next PMD"]
  B -->|yes| C{"pmd_trans_huge?<br/>(a 2 MiB THP leaf)"}
  C -->|yes| THP["walk_pmd_range_locked:<br/>batch this PMD for the locked pass<br/>promote the 2 MiB folio there"]
  C -->|no| D{"force_scan off AND<br/>should_clear_pmd_young AND<br/>no MMU notifiers?"}
  D -->|yes| E{"pmd_young?"}
  E -->|"NO — 2 MiB is cold"| SKIP2["SKIP the entire PTE table<br/>512 entries never read"]
  E -->|yes| F["batch the non-leaf clear<br/>via walk_pmd_range_locked"]
  D -->|no| G
  F --> G{"force_scan off AND<br/>test_bloom_filter(seq, pmd+i) misses?"}
  G -->|"miss — region was sparse last time"| SKIP3["SKIP this PTE table"]
  G -->|"hit or force_scan"| H["MM_NONLEAF_FOUND++<br/>walk_pte_range: sweep 512 PTEs"]
  H --> I{"suitable_to_scan(total, young)?<br/>young * 8 &gt;= total on x86-64"}
  I -->|no| J["do NOT record:<br/>the table drops out of the filter<br/>and is skipped next generation"]
  I -->|yes| K["MM_NONLEAF_ADDED++<br/>update_bloom_filter(seq+1, pmd+i)<br/>carry the table to the next generation"]
  SKIP1 --> Z["next PMD"]
  SKIP2 --> Z
  SKIP3 --> Z
  THP --> Z
  J --> Z
  K --> Z
  Z --> Y["after all 512: flush the PMD batch<br/>walk_pmd_range_locked(pud, -1, ...)"]

The per-PMD decision path, from walk_pmd_range() at v6.12. What it shows: three independent ways to avoid reading a PTE table — the entry is absent, the non-leaf accessed bit says the whole 2 MiB is cold, or the Bloom filter says the table was sparse last generation. The insight to take: force_scan disables the second and third of these. That is the entire meaning of the admin guide’s warning that setting force_scan=0 on the debugfs + verb “employs heuristics to reduce the overhead, which is likely to reduce the coverage as well” — and, read the other way, it is why force_scan defaults to 1 for working-set estimation: a job scheduler measuring cold memory cannot afford to have two optimizations hiding regions from it.

The rmap-Driven lru_gen_look_around()

The page-table walk is not the only way MGLRU clears accessed bits. The eviction path still occasionally walks the rmap for individual pages it is about to reclaim — and when it does, it would be wasteful to look at just the one PTE rmap pointed it at. So MGLRU exploits spatial locality with lru_gen_look_around() (mm/vmscan.c): “It scans the adjacent PTEs of a young PTE and promotes hot pages. If the scan was done cacheline efficiently, it adds the PMD entry pointing to the PTE table to the Bloom filter.” (kernel mm docs).

Concretely, given the PTE rmap landed on, lru_gen_look_around() first test-and-clears that PTE’s accessed bit, then expands to a window around it, clamped to the enclosing PMD and to the VMA:

start = max(addr & PMD_MASK, vma->vm_start);
end   = min(addr | ~PMD_MASK, vma->vm_end - 1) + 1;
...
if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { ... }   /* clamp to MIN_LRU_BATCH pages */
...
for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) {
        ...
        if (!ptep_clear_young_notify(vma, addr, pte + i))
                continue;
        young++;
        ...
}
/* feedback from rmap walkers to page table walkers */
if (mm_state && suitable_to_scan(i, young))
        update_bloom_filter(mm_state, max_seq, pvmw->pmd);

Source: lru_gen_look_around() in mm/vmscan.c, v6.12. MIN_LRU_BATCH is BITS_PER_LONG (64), so the window is at most ~64 PTEs — one PMD’s worth on the common path. The loop clears the accessed bit on every young PTE in the window and promotes those folios (either by updating the generation counter directly when a reclaim walk context exists, or via folio_activate()/folio_set_referenced() otherwise). The closing update_bloom_filter(..., pvmw->pmd) is the key feedback: if this PMD’s PTE table proved dense with hot pages, its PMD address is recorded so the aging path will later choose to walk that table directly rather than re-discovering it through rmap.

This is the closed loop the design doc summarizes: “Within the aging, rmap walks drive page table walks by inserting hot densely populated page tables to the Bloom filters.” (kernel mm docs). The eviction’s rmap walks discover which page tables are hot; the aging’s page-table walks then sweep exactly those tables.

Five early returns before any look-around happens

lru_gen_look_around() is called from the rmap walk in shrink_folio_list() for a folio that is about to be reclaimed, and it bails out in five distinct situations. The order matters because each one is a different judgement:

#	Condition	Returns	Why
1	!ptep_clear_young_notify(vma, addr, pte)	false	The PTE rmap pointed at is not young. This is the whole reason rmap called us: the answer is “not accessed”, so the folio proceeds toward reclaim and no look-around is warranted. This is the only path that returns false.
2	spin_is_contended(pvmw->ptl)	true	Someone else wants this PTE table’s lock. Take the one answer we have and get out; scanning 64 more PTEs while a fault waits is a bad trade.
3	vma->vm_flags & VM_SPECIAL	true	“exclude special VMAs containing anon pages from COW”
4	end - start == PAGE_SIZE	true	After clamping to the enclosing PMD and the VMA, the window is one page wide — there is nothing adjacent to look at.
5	!mem_cgroup_trylock_pages(memcg)	true	folio_update_gen() requires a stable folio_memcg(); a concurrent charge-move would make the promotion land in the wrong lruvec.

The window itself is clamped three ways, in this order: to the enclosing PMD (addr & PMD_MASK .. addr | ~PMD_MASK), to the VMA (vma->vm_start .. vma->vm_end), and then to MIN_LRU_BATCH = 64 pages. The third clamp is not a simple truncation — it centres the window on the hit address where it can:

if (end - start > MIN_LRU_BATCH * PAGE_SIZE) {
        if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2)
                end = start + MIN_LRU_BATCH * PAGE_SIZE;        /* near the low edge: extend right */
        else if (end - addr < MIN_LRU_BATCH * PAGE_SIZE / 2)
                start = end - MIN_LRU_BATCH * PAGE_SIZE;        /* near the high edge: extend left */
        else {
                start = addr - MIN_LRU_BATCH * PAGE_SIZE / 2;   /* in the middle: centre it */
                end = addr + MIN_LRU_BATCH * PAGE_SIZE / 2;
        }
}

So the look-around examines at most 64 PTEs — 256 KiB of virtual address space — positioned so the young PTE sits in the middle unless it is near an edge. The assumption being exploited is spatial locality: if this page was touched, its neighbours in virtual address space probably were too.

The promotion depends on who is calling

There is a fork in how lru_gen_look_around() promotes what it finds, and it is easy to miss:

walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL;
...
if (walk) {
        old_gen = folio_update_gen(folio, new_gen);
        if (old_gen >= 0 && old_gen != new_gen)
                update_batch_size(walk, folio, old_gen, new_gen);
        continue;
}
 
old_gen = folio_lru_gen(folio);
if (old_gen < 0)
        folio_set_referenced(folio);
else if (old_gen != new_gen)
        folio_activate(folio);

When a reclaim walk context exists — the common case, because the caller is reclaim — the promotion is batched into walk->nr_pages[] exactly as in walk_pte_range(), and lru_lock is taken later, once. When there is no walk context, the code falls back to the generic folio_activate() / folio_set_referenced() path. The comment above the fetch explains the priority: “avoid taking the LRU lock under the PTL when possible” — taking lru_lock while holding a page-table spinlock is a lock-ordering hazard the batched path sidesteps entirely.

Note also int young = 1; at the top of the function: the PTE rmap pointed at has already been counted before the loop begins, which is why the closing suitable_to_scan(i, young) is measuring the density of the window including the hit that triggered it.

The Bloom Filter — Skipping Sparse Regions

A naive page-table walk visits every PTE table in the address space, including the vast tracts that map nothing hot (or nothing at all). MGLRU prunes that with a Bloom Filter — a probabilistic set-membership structure that answers “is this element possibly in the set?” with no false negatives and a tunable false-positive rate (the general data structure is covered in Bloom Filter). MGLRU’s use is narrow and specific: the set is “PMD entries pointing to PTE tables that were recently found dense with young pages.”

The implementation is small. Verbatim from include/linux/mmzone.h at v6.12, lines 467-481, each lru_gen_mm_state carries two filters:

/* double-buffering Bloom filters */
#define NR_BLOOM_FILTERS	2
 
struct lru_gen_mm_state {
	/* synced with max_seq after each iteration */
	unsigned long seq;
	/* where the current iteration continues after */
	struct list_head *head;
	/* where the last iteration ended before */
	struct list_head *tail;
	/* Bloom filters flip after each iteration */
	unsigned long *filters[NR_BLOOM_FILTERS];
	/* the mm stats for debugging */
	unsigned long stats[NR_HIST_GENS][NR_MM_STATS];
};

head and tail are the mm_list cursors discussed above; stats is the MM_LEAF_TOTAL / MM_LEAF_YOUNG / MM_NONLEAF_FOUND / MM_NONLEAF_ADDED array that surfaces in /sys/kernel/debug/lru_gen_full.

And the filter primitives (mm/vmscan.c, v6.12) use a 2^15-bit bitmap with a 2-hash scheme:

#define BLOOM_FILTER_SHIFT 15           /* 2^15 = 32768 bits per filter */
 
static void get_item_key(void *item, int *key) {
        u32 hash = hash_ptr(item, BLOOM_FILTER_SHIFT * 2);
        key[0] = hash & (BIT(BLOOM_FILTER_SHIFT) - 1);  /* low 15 bits */
        key[1] = hash >> BLOOM_FILTER_SHIFT;            /* next 15 bits */
}
static bool test_bloom_filter(struct lru_gen_mm_state *mm_state, unsigned long seq, void *item) {
        int gen = seq % NR_BLOOM_FILTERS;
        unsigned long *filter = READ_ONCE(mm_state->filters[gen]);
        if (!filter) return true;                       /* no filter: scan everything */
        int key[2]; get_item_key(item, key);
        return test_bit(key[0], filter) && test_bit(key[1], filter);
}

Source: mm/vmscan.c, v6.12. A single 32-bit hash of the PMD pointer is split into two 15-bit indices; the element is “possibly present” only if both bits are set. update_bloom_filter() sets both bits. The seq % NR_BLOOM_FILTERS index implements double-buffering (detailed below). When the filter pointer is NULL, test_bloom_filter() returns true — fail open, scan the region — which is the safe default (a missed hot page just ages incorrectly for one cycle; it does not corrupt anything).

Getting the double-buffering the right way round

This is the easiest part of MGLRU to describe backwards, so it is worth naming the three call sites and their seq arguments explicitly. All are in mm/vmscan.c at v6.12:

Call	Site	seq passed	Filter index touched
test_bloom_filter(mm_state, walk->seq, pmd + i)	walk_pmd_range() — the aging	walk->seq, which equals max_seq	max_seq % 2 — read
update_bloom_filter(mm_state, max_seq, pvmw->pmd)	lru_gen_look_around() — the eviction	max_seq	max_seq % 2 — written
update_bloom_filter(mm_state, walk->seq + 1, pmd + i)	walk_pmd_range(), the “carry over to the next generation” line	max_seq + 1	(max_seq+1) % 2 — written
reset_bloom_filter(mm_state, walk->seq + 1)	iterate_mm_list(), on the first mm of an iteration	max_seq + 1	(max_seq+1) % 2 — cleared

So during generation max_seq, filter max_seq % 2 is the read filter and filter (max_seq+1) % 2 is the write filter. The eviction’s lru_gen_look_around() writes into the read filter — deliberately, so a PMD it discovers hot right now can be picked up by the aging walk that is running concurrently, in this same generation. The aging’s own discoveries go into the write filter, which is zeroed at the start of the iteration and becomes the read filter once max_seq advances. The source comment states the intent precisely:

Page table walkers use one of the two filters to reduce their search space. To get rid of non-leaf entries that no longer have enough leaf entries, the aging uses the double-buffering technique to flip to the other filter each time it produces a new generation. For non-leaf entries that have enough leaf entries, the aging carries them over to the next generation in walk_pmd_range(); the eviction also report them when walking the rmap in lru_gen_look_around().

The phrase “to get rid of non-leaf entries that no longer have enough leaf entries” is the whole reason there are two filters rather than one. A Bloom filter supports insertion but not deletion — you cannot clear one element’s bits without possibly clearing another’s. Double-buffering is the standard workaround: instead of deleting stale entries, wipe the whole spare filter and re-insert only the entries that are still valid. A PTE table that was dense last generation but sparse this one simply never gets re-inserted, and it vanishes from the filter when the buffers flip.

flowchart TD
  subgraph GEN_N["Generation N — while max_seq == N"]
    RF["filter[N mod 2]<br/>THE READ FILTER"]
    WF["filter[(N+1) mod 2]<br/>THE WRITE FILTER<br/>zeroed by reset_bloom_filter<br/>at the start of this iteration"]
  end
  AG["aging: walk_pmd_range<br/>for each of 512 PMDs"] -->|"test_bloom_filter(N, pmd)"| RF
  RF -->|"MISS -> skip this PTE table"| SK["512 PTEs never read<br/>no MM_NONLEAF_FOUND"]
  RF -->|"HIT -> MM_NONLEAF_FOUND++"| SW["walk_pte_range sweeps 512 PTEs"]
  SW --> ST{"suitable_to_scan?<br/>young PTEs per cacheline &gt;= 1"}
  ST -->|"no — table went cold"| DROP["record NOTHING<br/>the table silently expires<br/>when the buffers flip"]
  ST -->|yes| CARRY["MM_NONLEAF_ADDED++<br/>update_bloom_filter(N+1, pmd)"]
  CARRY --> WF
  EVI["eviction: lru_gen_look_around<br/>found a dense 64-PTE window"] -->|"update_bloom_filter(N, pmd)"| RF
  WF ==>|"max_seq becomes N+1<br/>roles swap"| RF2["filter[(N+1) mod 2]<br/>is now THE READ FILTER"]

The double-buffered feedback loop for one generation. What it shows: four arrows into two bitmaps. The eviction writes into the filter the aging is currently reading (immediate feedback, same generation); the aging writes into the spare filter (deferred feedback, next generation); and a table that fails suitable_to_scan() is dropped by simply not being written anywhere. The insight to take: deletion is impossible in a Bloom filter, so “forgetting” is implemented as not remembering — the spare filter is wiped wholesale at the start of each iteration and only currently-dense tables are re-inserted. That is why the filter tracks a workload that shifts its hot regions over time instead of saturating permanently.

The filter’s parameters, and what they cost

The kernel states its own Bloom parameters in the comment block above BLOOM_FILTER_SHIFT:

Parameter	Value	Where	Note
m — bits per filter	1 << 15 = 32,768	#define BLOOM_FILTER_SHIFT 15	allocated by bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT), ...)
bytes per filter	4,096 (one page)	32768 / 8
filters per lru_gen_mm_state	NR_BLOOM_FILTERS = 2	mmzone.h	so 8 KiB per memcg per NUMA node, and only when a walk has actually run — the pointers start NULL
k — hash functions	2	get_item_key()	one hash_ptr(item, 30) split into two 15-bit indices, not two independent hashes
hash	Knuth multiplicative (hash_ptr)	include/linux/hash.h	the comment lists “Jenkins’ hash function is an alternative to Knuth’s” as a future option
false-positive rate at n = 10,000	~1/5	source comment
false-positive rate at n = 20,000	~1/2	source comment
allocation flags	__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN	reset_bloom_filter()	best-effort: it can fail, and then the filter pointer stays NULL

MGLRU’s Bloom filter, parameterized. What it shows: an element is a PMD pointer — i.e. one entry represents one PTE table, covering 2 MiB of address space — so n = 10,000 entries corresponds to roughly 20 GiB of densely-mapped address space per memcg per node before the false-positive rate reaches 1/5. The insight to take: the two 15-bit keys come from splitting a single 32-bit hash (BUILD_BUG_ON(BLOOM_FILTER_SHIFT * 2 > BITS_PER_TYPE(u32)) enforces that they fit), so this is the cheap “double hashing” variant rather than two independent hash computations. Note the failure mode encoded in the allocation flags: if bitmap_zalloc() fails under memory pressure, filters[gen] stays NULL, test_bloom_filter() returns true unconditionally, and the walk degrades to scanning everything — slower, never wrong. A reclaim subsystem that allocates memory to do its job must fail open, and this one does.

The payoff: “In the aging path, set membership means that the PTE range will be scanned for young pages.” A negative result lets the walker skip the table entirely. Because the filter has no false negatives, a table that is genuinely hot is never wrongly skipped; a false positive merely costs “an additional scan of a range of PTEs, which may yield hot pages anyway” (kernel mm docs). The MM statistics MM_NONLEAF_FOUND and MM_NONLEAF_ADDED (in enum { ... NR_MM_STATS }, mmzone.h) count filter hits and insertions, surfacing in /sys/kernel/debug/lru_gen_full for tuning.

Architecture Support and the Runtime Switches

The previous sections showed what the walk does with the leaf and non-leaf accessed bits. This one answers the prior question: on which machines, and under which switches, does any of it happen at all?

Some architectures (notably x86 with recent Intel/AMD MMUs) set an accessed bit not just in leaf PTEs but in non-leaf entries (PMD/PUD) too, so the kernel can clear those in bulk to learn that a whole sub-tree is cold without descending. MGLRU exposes this as a capability, LRU_GEN_NONLEAF_YOUNG (in enum { LRU_GEN_CORE, LRU_GEN_MM_WALK, LRU_GEN_NONLEAF_YOUNG, NR_LRU_GEN_CAPS }, mmzone.h v6.12), gated at runtime through the bit-flag enabled knob. From the v6.12 admin-guide (multigen_lru.rst):

enabled bit	Component
0x0001	The main switch for the multi-gen LRU.
0x0002	Clearing the accessed bit in leaf PTEs in large batches, when the MMU sets it (e.g. x86). Disabling it causes a minor degradation for workloads that contiguously map hot pages.
0x0004	Clearing the accessed bit in non-leaf entries too, when the MMU sets it. Disabling it causes a negligible degradation.
[yYnN]	Apply to all components above.

So echo y > /sys/kernel/mm/lru_gen/enabled turns everything on and reads back 0x0007; echo 5 enables the main switch plus non-leaf clearing but not leaf batching and reads back 0x0005. Multi-Generational LRU owns this file as an ABI — including the important fact that what you write is not necessarily what you read back. What matters here is that the two upper bits are not read directly by the walk. They are static_keys consumed by exactly two one-line predicates:

static bool should_walk_mmu(void)
{
	return arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK);
}
 
static bool should_clear_pmd_young(void)
{
	return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG);
}

Source: mm/vmscan.c at v6.12, lines 2591-2599. get_cap(cap) is static_branch_likely(&lru_gen_caps[cap]) when CONFIG_LRU_GEN_ENABLED=y and static_branch_unlikely() otherwise — so the build-time default also chooses which way the branch predictor is biased, not merely the initial value.

Each predicate is a hardware check ANDed with a runtime switch, and the two halves come from different places:

	Build-time symbol	Runtime hardware predicate	Runtime switch	Used by
Leaf accessed bit	CONFIG_ARCH_HAS_HW_PTE_YOUNG	arch_has_hw_pte_young()	enabled bit 0x0002 (LRU_GEN_MM_WALK)	should_walk_mmu() — gates the entire mm_list walk
Non-leaf PMD accessed bit	CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG	arch_has_hw_nonleaf_pmd_young()	enabled bit 0x0004 (LRU_GEN_NONLEAF_YOUNG)	should_clear_pmd_young() — gates the 2 MiB skip in walk_pmd_range()
Compiling the walk in at all	CONFIG_LRU_GEN_WALKS_MMU — def_bool y, depends on LRU_GEN && ARCH_HAS_HW_PTE_YOUNG	—	—	#ifdef around get_mm_list(), get_next_mm(), iterate_mm_list()

The Kconfig help text for the two architecture symbols (arch/Kconfig at v6.12, lines 1620-1634) is worth quoting because it names the runtime-check escape hatch:

ARCH_HAS_HW_PTE_YOUNG — Architectures that select this option are capable of setting the accessed bit in PTE entries when using them as part of linear address translations. Architectures that require runtime check should select this option and override arch_has_hw_pte_young().

ARCH_HAS_NONLEAF_PMD_YOUNG — Architectures that select this option are capable of setting the accessed bit in non-leaf PMD entries when using them as part of linear address translations. Page table walkers that clear the accessed bit may use this capability to reduce their search space.

Who selects them, at v6.12 (read from each architecture’s Kconfig):

Architecture	ARCH_HAS_HW_PTE_YOUNG	ARCH_HAS_NONLEAF_PMD_YOUNG	Consequence
x86	selected unconditionally	selected if PGTABLE_LEVELS > 2	both optimizations available; enabled reads back 0x0007
arm64	selected, but overridden: #define arch_has_hw_pte_young cpu_has_hw_af (arch/arm64/include/asm/pgtable.h, line 1503)	not selected	the walk is compiled in, but is enabled per-CPU at runtime depending on whether the silicon implements hardware access-flag update; no non-leaf shortcut
powerpc, riscv, s390, loongarch	not selected	not selected	CONFIG_LRU_GEN_WALKS_MMU=n; the mm_list walk is compiled out entirely

Architecture support for MGLRU’s accessed-bit harvesting at v6.12. What it shows: the two capabilities are independent, and only x86 has both. arm64 is the interesting case: it selects ARCH_HAS_HW_PTE_YOUNG but redefines arch_has_hw_pte_young to cpu_has_hw_af, a runtime CPU-feature check. The insight to take: on arm64 hardware without the access-flag feature, CONFIG_LRU_GEN_WALKS_MMU is y, the code is compiled in, /sys/kernel/mm/lru_gen/enabled will accept a write of y — and should_walk_mmu() still returns false, so no mm_list walk ever happens. The readback tells you: enabled will show 0x0001, not 0x0007, because enabled_show() runs the same predicates. This is the concrete reason the admin guide says “Writing to enabled has no effect when a component is not supported by the hardware” — and the reason to always read the file back rather than trusting the write.

The rmap-or-walk decision, in one flowchart

try_to_inc_max_seq() contains the actual fork, and its comment is the clearest statement of MGLRU’s fallback policy anywhere in the tree:

/*
 * If the hardware doesn't automatically set the accessed bit, fallback
 * to lru_gen_look_around(), which only clears the accessed bit in a
 * handful of PTEs. Spreading the work out over a period of time usually
 * is less efficient, but it avoids bursty page faults.
 */
if (!should_walk_mmu()) {
        success = iterate_mm_list_nowalk(lruvec, seq);
        goto done;
}
 
walk = set_mm_walk(NULL, true);
if (!walk) {
        success = iterate_mm_list_nowalk(lruvec, seq);
        goto done;
}

Note the second fallback: even on x86 with everything enabled, set_mm_walk() can return NULL. It allocates a struct lru_gen_mm_walk with kzalloc(..., __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN) for non-kswapd callers, and under severe pressure that allocation fails. iterate_mm_list_nowalk() then advances mm_state->seq without walking anything, and inc_max_seq() still mints the generation — so the window keeps moving, but that generation’s promotions come only from whatever lru_gen_look_around() happens to find during eviction. kswapd is exempt: it uses the pre-allocated pgdat->mm_walk, so background reclaim never loses the walk to an allocation failure.

flowchart TD
  A["try_to_inc_max_seq(lruvec, seq, can_swap, force_scan)"] --> B{"get_mm_state(lruvec)<br/>returns NULL?"}
  B -->|"yes — CONFIG_LRU_GEN_WALKS_MMU=n"| Z1["inc_max_seq() directly:<br/>mint the generation, scan NOTHING"]
  B -->|no| C{"seq &lt;= mm_state-&gt;seq?"}
  C -->|"yes — someone already did this pass"| Z2["return false"]
  C -->|no| D{"should_walk_mmu()?<br/>arch_has_hw_pte_young AND bit 0x0002"}
  D -->|"NO"| E["iterate_mm_list_nowalk():<br/>advance mm_state->seq only"]
  D -->|yes| F["set_mm_walk(NULL, true)"]
  F --> G{"kswapd?"}
  G -->|yes| H["use the pre-allocated<br/>pgdat->mm_walk — cannot fail"]
  G -->|no| I["kzalloc with __GFP_NOWARN"]
  I -->|"allocation FAILED"| E
  I -->|ok| J
  H --> J["loop: iterate_mm_list() then walk_mm()<br/>until the list wraps"]
  E --> K["inc_max_seq():<br/>mint the generation anyway"]
  J --> K
  K --> L["promotions this generation come from<br/>the walk AND from lru_gen_look_around()"]
  E -.->|"walk skipped"| M["promotions this generation come<br/>ONLY from lru_gen_look_around()<br/>during eviction"]

The fallback ladder in try_to_inc_max_seq(), v6.12. What it shows: four distinct paths reach inc_max_seq(), and three of them skip the page-table walk. The generation is minted regardless — MGLRU never stalls the window because scanning was unavailable. The insight to take: “MGLRU walks page tables” is true on x86 with default settings and false in at least three other reachable configurations (no hardware accessed bit, bit 0x0002 cleared, or a failed kzalloc under pressure). In all of them MGLRU still works, because lru_gen_look_around() is a complete — if slower — accessed-bit harvester on its own. The walk is an optimization layered on top of a design that does not require it, which is exactly why the fallback comment says it “avoids bursty page faults”: clearing 64 accessed bits per eviction spreads the re-fault cost that clearing millions at once would concentrate.

Verified on This Machine

The code above is v6.12. This section checks the preconditions for the walk on a live host running a different tree — kernel 7.1.13-200.fc44 (Fedora 44) on an AMD Ryzen AI MAX+ 395, x86-64 — because the walk’s behaviour is determined almost entirely by architecture capabilities and two runtime bits, and those are things a reader can check on their own machine in ten seconds. The # multi-gen LRU { Kconfig block and the design document are byte-identical from v6.12 through v7.2 (verified by fetching and diffing both at both tags), so the configuration surface being checked has not moved.

$ uname -r
7.1.13-200.fc44.x86_64
 
$ grep -E "LRU_GEN|ARCH_HAS_HW_PTE_YOUNG|ARCH_HAS_NONLEAF_PMD_YOUNG" /boot/config-$(uname -r)
CONFIG_ARCH_HAS_HW_PTE_YOUNG=y
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y
CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y
# CONFIG_LRU_GEN_STATS is not set
CONFIG_LRU_GEN_WALKS_MMU=y
 
$ cat /sys/kernel/mm/lru_gen/enabled
0x0007

Working the two predicates through, term by term:

Term	Value here	Source of the value
arch_has_hw_pte_young()	true	x86 has no override, so the generic IS_ENABLED(CONFIG_ARCH_HAS_HW_PTE_YOUNG) applies, and the symbol is y
get_cap(LRU_GEN_MM_WALK)	true	bit 0x0002 is set in the 0x0007 readback
⇒ should_walk_mmu()	true	the full mm_list page-table walk is active
arch_has_hw_nonleaf_pmd_young()	true	CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=y, which x86 selects if PGTABLE_LEVELS > 2 — and CONFIG_PGTABLE_LEVELS=5 here
get_cap(LRU_GEN_NONLEAF_YOUNG)	true	bit 0x0004 is set
⇒ should_clear_pmd_young()	true	the 2 MiB non-leaf skip in walk_pmd_range() is active
CONFIG_LRU_GEN_WALKS_MMU	y	follows from LRU_GEN && ARCH_HAS_HW_PTE_YOUNG, both y
NR_HIST_GENS	1	CONFIG_LRU_GEN_STATS is not set

And the two hardware constants that parameterize suitable_to_scan():

$ getconf LEVEL1_DCACHE_LINESIZE
64
$ cat /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size
64

suitable_to_scan() computes n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8). With a 64-byte line and an 8-byte pte_t, that is 64 / 8 = 8, inside the clamp, so n = 8 on this machine and the density test is literally young * 8 >= total — “at least one young PTE per cacheline read, on average.” The clamp_t floor of 2 and ceiling of 8 mean the test degrades gracefully on architectures with unusual line sizes rather than becoming trivially true or trivially false.

Two consequences of the rest of the configuration:

• CONFIG_PGTABLE_LEVELS=5 but /proc/cpuinfo does not report the la57 flag, so this CPU runs 4-level paging and p4d is folded. The walk’s .p4d_entry = walk_pud_range hook therefore enters at what is effectively the PGD level; the generic walk_page_range() machinery handles the folding, which is exactly why MGLRU hooks at p4d_entry rather than writing its own top-level loop.
• Transparent huge pages are [madvise], and AnonHugePages is only 40,960 kB out of 125 GiB. So on this host the pmd_trans_huge() branch of walk_pmd_range() is almost never taken, and essentially all young-bit harvesting happens at the leaf. A host with transparent_hugepage/enabled = always and a large AnonHugePages would see a materially different split — most promotions happening inside walk_pmd_range_locked() at 2 MiB granularity rather than in walk_pte_range() at 4 KiB. See Transparent Huge Pages.

The walk's own statistics could not be read on this host

The MM_LEAF_TOTAL / MM_LEAF_YOUNG / MM_NONLEAF_FOUND / MM_NONLEAF_ADDED counters — the only direct observability the walk has — live in /sys/kernel/debug/lru_gen_full. /sys/kernel/debug is mode 0700 and root-owned, and no passwordless sudo was available to the account doing this write-up, so they were not read. Everything this note says about those counters comes from lru_gen_seq_show_full() in the v6.12 source. Note also that with CONFIG_LRU_GEN_STATS unset, NR_HIST_GENS is 1, so only the max_seq row would carry the uppercase TYFA values anyway. To resolve: sudo cat /sys/kernel/debug/lru_gen_full and read the four-number group at the end of each generation’s block. MGLRU Generations and Aging decodes that output format. uncertain

What can be shown without root is that MGLRU’s reclaim path is genuinely the one running. A 1,400 MiB anonymous working set swept three times inside a 512 MiB systemd-run --user --scope -p MemoryMax=512M cgroup produced pgscan_anon +1,686,568, pgsteal_anon +1,305,148 and workingset_refault_anon +1,076,090, with pgdeactivate remaining at exactly 0 — the signature of an engine that has no explicit active-to-inactive demotion step. That experiment and its full counter table live in Forcing generations to turn over, since the counters it moves are generation bookkeeping rather than walk statistics.

---

Failure Modes and Subtleties

Accessed-bit clearing forces minor faults later. Clearing the bit is not free downstream: the next access to a page whose bit MGLRU cleared will re-set it via the MMU, which on some configurations costs a microfault or a TLB update. The design tolerates this because the information gained (a fresh recency reading) is worth the re-set cost, and because unmapped pages need no TLB flush and clean pages need no writeback — MGLRU explicitly leans on these fast paths (kernel mm docs).

mmap_lock contention. Batched leaf clearing (bit 0x0002) can “theoretically worsen lock contention (mmap_lock)” per the admin doc, which is why it is a separable switch. A workload seeing reclaim-time mmap_lock stalls under MGLRU can try echo 5 > .../enabled to drop leaf batching.

Sparse address spaces defeat the walk. The design itself warns the page-table walk “can sweep all the young PTEs in an address space, but the address space can be too sparse to make a profit.” This is precisely the case the Bloom filter and suitable_to_scan() exist to catch — a process that maps a huge but mostly-cold region will have its tables dropped from the filter and skipped, falling back to targeted rmap look-around. If you ever observe MGLRU spending CPU walking tables that yield no young pages, the MM_LEAF_TOTAL vs MM_LEAF_YOUNG ratio in /sys/kernel/debug/lru_gen_full is the diagnostic.

False positives are harmless; false negatives are impossible — but the double-buffer flip is a deliberate forgetting. Because Bloom filters never produce false negatives, a hot table present in the filter is never wrongly skipped. The worst case from a stale or saturated filter is a wasted PTE-range scan — bounded, self-correcting, and never a correctness bug. What can look like a false negative is the buffer flip: a table that was dense in generation N but whose suitable_to_scan() came back false is not re-inserted into filter N+1, so from generation N+1 it is invisible until an rmap look-around rediscovers it. That is intentional expiry, not filter error, and it is the only mechanism MGLRU has for forgetting a region that went cold.

Every lock in the walk is a trylock, so coverage is best-effort. mmap_read_trylock() in walk_mm(), spin_trylock(ptl) in walk_pte_range(), spin_trylock(pmd_lockptr(...)) in walk_pmd_range_locked(), mem_cgroup_trylock_pages() in both walk_mm() and lru_gen_look_around(). Any of them failing silently drops that unit of work. Under heavy fault activity on the same address space the walk can therefore harvest substantially fewer accessed bits than it nominally covers, and there is no counter that records the misses — MM_LEAF_TOTAL counts entries the walker actually reached. When diagnosing “MGLRU is not promoting my hot pages”, contention is a hypothesis the available statistics cannot rule out.

force_scan turns off both optimizations at once. In walk_pmd_range(), walk->force_scan guards the non-leaf young-bit skip and the Bloom-filter test. It is set from two places: the debugfs + verb’s third optional argument (default 1), and automatically in iterate_mm_list() for any mm_struct added since the previous iteration. So a machine that is churning processes — a build farm, a CI runner — has a steady supply of force-scanned address spaces whose page tables are swept exhaustively. That is correct behaviour (MGLRU has no history for a new process) but it is a real cost that a long-lived-process workload never pays.

The walk cannot see unmapped page cache, at all. This is structural rather than a bug, but it surprises people: a file read through read()/write() with no mmap() produces page-cache folios that appear in no page table. walk_page_range() will never encounter them, get_pte_pfn() will never be called on them, and their recency is tracked entirely by the tier mechanism (PG_referenced / PG_workingset, see MGLRU Generations and Aging). If you are reasoning about why a read()-heavy workload behaves differently under MGLRU than an mmap()-heavy one, this is the reason: they are handled by two different halves of the design that share only the generation window.

Non-leaf clearing is disclaimed outside Intel and AMD. The admin guide says of bit 0x0004 that “this behavior was not verified on x86 varieties other than Intel and AMD” (v6.12 admin guide, text unchanged at v7.2). If you are running on a Zhaoxin, Hygon or emulated x86 and seeing implausible generation distributions, echo 3 > /sys/kernel/mm/lru_gen/enabled (main switch plus leaf batching, non-leaf off) is the bisect step — and the documentation grades the cost of leaving it off as “negligible”.

MMU notifiers disable both fast paths. walk_pmd_range() skips the non-leaf shortcut when mm_has_notifiers(args->mm), and get_pte_pfn() refuses to take the !pte_young() early exit for the same reason. An address space with MMU-notifier subscribers — most importantly a KVM guest’s memory, but also some GPU and RDMA drivers — is therefore scanned more expensively than an ordinary one, because the host PTE’s accessed bit is not the whole story: the notifier callee may hold access information the host cannot see. On a virtualization host, MGLRU’s page-table walk costs strictly more per guest page than per host page. See The Translation Lookaside Buffer and TLB Shootdowns for the related flushing costs.

See Also

• MGLRU Generations and Aging — what the walk feeds: the generation counter, max_seq/min_seq, and how promotion ages the LRU.
• Multi-Generational LRU — the umbrella concept and its merge history (Linux 6.1, Yu Zhao / Google).
• MGLRU vs the Classic LRU — why the walk-based scan replaces the rmap-heavy active/inactive scanner.
• Page Table Entries and Hardware Flags — the accessed/young bit this walk reads and clears, and the rest of the PTE flag layout.
• Bloom Filter — the probabilistic set-membership structure MGLRU uses to skip cold page-table regions.
• The Translation Lookaside Buffer and TLB Shootdowns — why clearing accessed bits and reclaiming mapped pages interacts with TLB cost.
• The LRU Lists · Memory Reclaim Overview — the surrounding reclaim machinery.
• Transparent Huge Pages — the pmd_trans_huge() branch of walk_pmd_range(), where a single PMD entry promotes 2 MiB.
• Direct Reclaim · kswapd and Background Reclaim — the two callers; kswapd gets a pre-allocated lru_gen_mm_walk, direct reclaim has to kzalloc one and can fail.
• The Memory Cgroup memcg — lruvec_memcg()->mm_list is per-memcg, and folio_memcg_rcu() mismatches are how the walk ignores migrated pages.
• MOC: Linux Memory Management MOC (§9, Multi-Generational LRU).

---

Sources

All fetched during this write-up. Kernel source is pinned to the tag named in the URL.

Primary — Linux v6.12:

• mm/vmscan.c — should_walk_mmu(), should_clear_pmd_young(), the Bloom-filter block (BLOOM_FILTER_SHIFT, get_item_key(), test_bloom_filter(), update_bloom_filter(), reset_bloom_filter()), get_mm_list(), get_next_mm(), iterate_mm_list(), iterate_mm_list_nowalk(), should_skip_vma(), get_next_vma(), get_pte_pfn(), get_pmd_pfn(), get_pfn_folio(), suitable_to_scan(), walk_pte_range(), walk_pmd_range_locked(), walk_pmd_range(), walk_pud_range(), walk_mm(), set_mm_walk(), try_to_inc_max_seq(), lru_gen_look_around().
• include/linux/mmzone.h — struct lru_gen_mm_state, struct lru_gen_mm_walk, enum { ... NR_MM_STATS }, NR_BLOOM_FILTERS, MIN_LRU_BATCH, MAX_LRU_BATCH, enum { LRU_GEN_CORE, LRU_GEN_MM_WALK, LRU_GEN_NONLEAF_YOUNG, NR_LRU_GEN_CAPS }.
• include/linux/mm_types.h — the per-mm_struct lru_gen sub-struct (list, bitmap, memcg), struct lru_gen_mm_list, and lru_gen_use_mm().
• kernel/sched/core.c — the single lru_gen_use_mm(next->mm) call site inside context_switch().
• include/linux/pgtable.h — the generic arch_has_hw_pte_young() and arch_has_hw_nonleaf_pmd_young() stubs and the comment explaining when an architecture must override them.
• arch/Kconfig — the help text for ARCH_HAS_HW_PTE_YOUNG and ARCH_HAS_NONLEAF_PMD_YOUNG.
• arch/x86/Kconfig, arch/arm64/Kconfig — which architecture selects which capability (powerpc, riscv, s390 and loongarch were checked too and select neither).
• arch/arm64/include/asm/pgtable.h — #define arch_has_hw_pte_young cpu_has_hw_af, the runtime override.
• mm/Kconfig — CONFIG_LRU_GEN_WALKS_MMU.
• Documentation/mm/multigen_lru.rst — “Rmap/PT walk feedback”, “Bloom filters”, “mm_struct list”.
• Documentation/admin-guide/mm/multigen_lru.rst — the enabled bit semantics and the force_scan argument.

Existence-checking (same paths fetched at later tags and compared):

• mm/Kconfig at v7.2 — the LRU_GEN block is byte-identical to v6.12 (also checked at v6.16, v6.17, v6.18, v7.0, v7.1: same 29 lines, same MD5).
• Documentation/mm/multigen_lru.rst at v7.2 — byte-identical to v6.12, so nothing in the walk’s documented design changed in that window.
• Documentation/admin-guide/mm/multigen_lru.rst at v7.2 — one paragraph differs from v6.12, and it concerns the debugfs - verb’s swappiness range, not the walk.

Secondary:

• Jonathan Corbet, “The multi-generational LRU”, 2 April 2021 — the scheduler hook and the rmap-cost framing.
• Jonathan Corbet, “Multi-generational LRU: the next generation”, 2021.
• Jonathan Corbet, “Merging the multi-generational LRU”, 2022.
• Jonathan Corbet, “Reconsidering the multi-generational LRU”, 5 March 2026 — Barry Song’s summary of look-around as one of MGLRU’s “strong design aspects”, and the wider 2026 status.

Measured on the host (7.1.13-200.fc44, x86-64, AMD Ryzen AI MAX+ 395, 2026-09-11): /boot/config-$(uname -r), /sys/kernel/mm/lru_gen/enabled, /proc/cpuinfo, getconf LEVEL1_DCACHE_LINESIZE, /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size, /sys/kernel/mm/transparent_hugepage/enabled, /proc/meminfo.

Not cited, deliberately

lore.kernel.org returns HTTP 403 to plain curl from this network (re-verified 2026-09-11, 146-byte body), so the MGLRU patch-series cover letters could not be read. No benchmark figure for the page-table walk versus the rmap scan appears anywhere in this note — the “two directions, costed” table above is a structural comparison derived from the code, not a measurement. Anyone wanting numbers should measure their own workload rather than trust a restated figure whose kernel version and benchmark are unknown.