Per-CPU Page Lists

The buddy allocator hands out physically-contiguous runs of pages, but every operation on it must take the per-zone lock — and the overwhelmingly common case is a single CPU allocating or freeing one page (an order-0 page) on a hot path like a page fault. Funnelling every such single-page operation through a shared zone lock would serialize all CPUs against each other and trash cache locality. The kernel’s answer is the per-CPU page list (the struct per_cpu_pages, universally abbreviated pcp or “pageset”): a small cache of pages held per CPU per zone — orders 0 through 3 and, on a CONFIG_TRANSPARENT_HUGEPAGE build, the PMD order as well — refilled and drained from the buddy allocator in batches under the zone lock, so that the steady-state fast path (alloc one page, free one page) touches only CPU-local data and never acquires zone->lock at all. It does take one lock, a per-(CPU, zone) spinlock inside the pageset itself, but that lock is essentially never contended; the phrase to use is zone-lock-free, not lock-free. The structure is defined in include/linux/mmzone.h (Linux v6.12 source).

This is one of the kernel’s most important scalability optimizations: it converts the page allocator’s hottest path from a globally-contended lock into a CPU-local, cache-hot operation, paying the zone lock only once per batch of pages rather than once per page. It is the per-CPU caching idea — also seen in the SLUB allocator’s per-CPU slabs — applied at the page granularity.

Scope and layering. This note sits in a crowded neighbourhood, so the boundaries are worth stating before anything else. Bottom to top: The Buddy Allocator owns the free_area free lists, the order range, buddy-finding, expand()/__free_one_page() splitting and coalescing, migratetype fallback stealing, and /proc/buddyinfo — none of that is re-explained here. Watermarks and the Allocation Fast Path owns the min/low/high watermarks and the decision to take the fast path at all. Memory Zones and Nodes owns which zone gets asked. This note owns exactly one layer: the per-CPU cache that sits between rmqueue() and zone->lock, and the single question that layer exists to answer — how does an order-0 allocation complete without acquiring the zone lock? Everything below is in service of that question: the structure that holds the cached pages, the arithmetic that sizes it, the auto-tuner that resizes it at runtime, and the drain paths where the abstraction leaks and pages have to go back.

The sibling one layer up is The Slab Allocator and SLUB, which applies the identical idea at object granularity; the two compose, because a SLUB cache refilling a slab allocates whole pages and those allocations come through here.

All source citations are pinned to Linux v6.12 (kernel.org lists 6.18, 6.12, 6.6, 6.1, 5.15 and 5.10 as longterm branches, mainline 7.3-rc2, stable 7.2.5, as of 2026-09-12 per releases.json; v6.12 is this vault’s pin, not the only LTS). Live measurements were taken on a 7.1.13 kernel and are labelled as such. The pcp structure is byte-for-byte identical between v6.12 and v6.18, verified by fetching include/linux/mmzone.h at both tags, so everything structural here holds across both current longterm branches.

Mental Model

Picture each CPU as having a small private “bucket” of loose pages for each memory zone — mostly single 4 KiB pages, but also small runs of 2, 4 and 8 pages and, where transparent huge pages are configured, whole 2 MiB blocks. When code wants one page, the CPU grabs it from its own bucket — no coordination with other CPUs. When code frees a page, it drops it back into the bucket. Only when the bucket runs empty (alloc) or overflows a high-water mark (free) does the CPU walk over to the big shared warehouse (the buddy allocator), unlock it, and either scoop up a batch of fresh pages or pour a batch of excess pages back — one lock acquisition amortized across many pages. Because the bucket is CPU-local and recently-touched, the pages it hands out are usually still warm in that CPU’s cache, which also speeds up the code that then uses the page.

flowchart TD
  subgraph CPU0["CPU 0 (local: no zone lock)"]
    A0["alloc 1 page →<br/>pop from pcp->lists[mt]"]
    F0["free 1 page →<br/>push to pcp->lists[mt]"]
  end
  subgraph PCP["struct per_cpu_pages (per CPU, per zone)"]
    L["lists[NR_PCP_LISTS]<br/>(one per migratetype + THP order)"]
    H["high / batch / count<br/>watermarks"]
  end
  A0 --> L
  F0 --> L
  L -- "empty on alloc:<br/>rmqueue_bulk(batch)" --> ZONE
  L -- "count >= high on free:<br/>free_pcppages_bulk(batch)" --> ZONE
  ZONE["Buddy allocator<br/>(zone->lock, shared by all CPUs)"]

The per-CPU page list as a write-back cache in front of the buddy allocator. What it shows: single-page alloc/free hits only the CPU-local pcp->lists[] (cheap); the expensive shared zone->lock is touched only on the refill (list empty) and drain (list over the high watermark) transitions, and then it moves a whole batch of pages at once. The insight to take: the design trades a little memory (pages parked in per-CPU buckets, invisible to other CPUs until drained) for an enormous reduction in zone-lock contention — the lock cost is divided by the batch size on the steady-state path.

The Mechanism Is Lock Avoidance

Everything else in this note is detail. The mechanism is this: an order-0 allocation on the hot path completes without ever acquiring zone->lock. Not “holds it briefly”, not “uses a fast-path variant” — does not acquire it at all. That is the entire reason the structure exists, and it is worth making the argument precisely, because the phrase “lock-free fast path” gets used loosely and the reality is a three-level hierarchy, not an absence of locking.

The three locks, and which one the fast path actually takes

Lock	Scope	Taken on the order-0 fast path?	Contention profile
zone->lock (spinlock_t, in struct zone)	one per zone, shared by every CPU on the machine	No — only on refill (rmqueue_bulk) and drain (free_pcppages_bulk), i.e. once per batch pages	The one that scales badly: every CPU in the system serialises here
pcp->lock (spinlock_t, in struct per_cpu_pages)	one per (CPU, zone) pair	Yes, via spin_trylock	Essentially never contended — the owning CPU is the only normal user; contenders are remote drainers and IRQ re-entry
pcpu_task_pin() (preempt_disable, or migrate_disable on PREEMPT_RT)	the calling task	Yes	Not a lock; it stops the task migrating between the per-CPU lookup and the lock acquisition

The pcpu_task_pin() step is not decoration. Locking a pcp is a lookup followed by a lock, and the kernel comment spells out the hazard (mm/page_alloc.c L109-L115): “Locking a pcp requires a PCP lookup followed by a spinlock. To avoid a migration causing the wrong PCP to be locked and remote memory being potentially allocated, pin the task to the CPU for the lookup+lock. preempt_disable is used on !RT because it is faster than migrate_disable. migrate_disable is used on RT because otherwise RT spinlock usage is interfered with and a high priority task cannot preempt the allocator.” Without the pin, a task could read this_cpu_ptr(zone->per_cpu_pageset), get migrated to another CPU, and then lock and drain a pageset belonging to a CPU it is no longer running on.

The composite helper is a trylock, not a lock (mm/page_alloc.c L138-L148):

#define pcpu_spin_trylock(type, member, ptr)				\
({									\
	type *_ret;							\
	pcpu_task_pin();			/* preempt_disable / migrate_disable */ \
	_ret = this_cpu_ptr(ptr);		/* now safe: we cannot migrate */ \
	if (!spin_trylock(&_ret->member)) {				\
		pcpu_task_unpin();					\
		_ret = NULL;			/* caller must have a fallback */ \
	}								\
	_ret;								\
})

Why trylock — and what happens when it fails

rmqueue_pcplist() calls pcp_spin_trylock() and, if it returns NULL, gives up on the pcp entirely and returns NULL to its caller (mm/page_alloc.c L3019-L3046); the comment says why the failure is possible at all: “spin_trylock may fail due to a parallel drain or IRQ reentrancy.” rmqueue() then falls through to rmqueue_buddy(), which does take zone->lock (L3065-L3090):

static inline
struct page *rmqueue(struct zone *preferred_zone, struct zone *zone,
		unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags,
		int migratetype)
{
	struct page *page;
 
	if (likely(pcp_allowed_order(order))) {
		page = rmqueue_pcplist(preferred_zone, zone, order,
				       migratetype, alloc_flags);
		if (likely(page))
			goto out;                 /* <- the hot path: no zone->lock, ever */
	}
 
	page = rmqueue_buddy(preferred_zone, zone, order, alloc_flags,
							migratetype);   /* <- takes zone->lock */
out:
	...
}

The free side mirrors it exactly. free_unref_page() tries the pcp and, on trylock failure, calls free_one_page() — which takes zone->lock and hands the page straight to __free_one_page() for coalescing (L2674-L2683):

	pcp = pcp_spin_trylock(zone->per_cpu_pageset);
	if (pcp) {
		free_unref_page_commit(zone, pcp, page, migratetype, order);
		pcp_spin_unlock(pcp);
	} else {
		free_one_page(zone, page, pfn, order, FPI_NONE);   /* zone->lock fallback */
	}

This “try the cheap lock, fall back to the expensive one” shape is the whole design in miniature: correctness never depends on the pcp being available, so the pcp is free to be unavailable — during a remote drain, during memory hot-unplug, when a page belongs to an isolated pageblock, or when an interrupt re-enters the allocator on a CPU that was already inside it. Every one of those cases simply takes the slow, correct path.

Which orders qualify

pcp_allowed_order() is the gate (mm/page_alloc.c L542-L551):

static inline bool pcp_allowed_order(unsigned int order)
{
	if (order <= PAGE_ALLOC_COSTLY_ORDER)     /* orders 0,1,2,3 -> 4 KiB..32 KiB */
		return true;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (order == HPAGE_PMD_ORDER)             /* order 9 on x86-64 -> 2 MiB */
		return true;
#endif
	return false;
}

PAGE_ALLOC_COSTLY_ORDER is 3 (include/linux/mmzone.h L46), defined as “the order at which allocations are deemed costly to service. That is between allocation orders which should coalesce naturally under reasonable reclaim pressure and those which will not.” So the pcp serves orders 0, 1, 2, 3 and 9 — and nothing else. An order-4 (64 KiB) allocation goes straight to zone->lock every single time. This is not the “order-0 only” cache most descriptions still call it; see the version history for when each of those orders was added and how that was dated.

flowchart TD
  RQ["rmqueue(zone, order, migratetype)"] --> ORD{"pcp_allowed_order(order)?<br/>order &le; 3, or order == 9 with THP"}
  ORD -->|"no (order 4..8, 10+)"| BUD
  ORD -->|yes| TRY["rmqueue_pcplist():<br/>pcpu_task_pin() then<br/>spin_trylock(&amp;pcp-&gt;lock)"]
  TRY -->|"trylock FAILS<br/>(remote drain in progress,<br/>or IRQ re-entry)"| BUD["rmqueue_buddy():<br/><b>spin_lock_irqsave(&amp;zone-&gt;lock)</b><br/>__rmqueue() / __rmqueue_smallest()"]
  TRY -->|"trylock succeeds"| PIDX["list = &amp;pcp-&gt;lists[<br/>order_to_pindex(migratetype, order)]"]
  PIDX --> EMPTY{"list_empty(list)?"}
  EMPTY -->|"no — <b>the hot path</b>"| POP["list_first_entry + list_del<br/>pcp-&gt;count -= 1 &lt;&lt; order<br/><b>zone-&gt;lock NEVER taken</b>"]
  EMPTY -->|"yes — refill"| NRA["nr_pcp_alloc(): batch =<br/>base_batch &lt;&lt; pcp-&gt;alloc_factor"]
  NRA --> BULK["rmqueue_bulk(zone, order, batch, ...)<br/><b>takes zone-&gt;lock ONCE for `batch` pages</b>"]
  BULK --> POP
  POP --> OUT["page returned<br/>PGALLOC++ , zone_statistics()"]
  BUD --> OUT

How rmqueue() decides whether zone->lock is touched, at v6.12. What it shows: three exits. The middle one — a non-empty per-CPU list — is the steady state on a busy machine and never touches the shared lock. The refill branch takes it once and amortises it over batch pages. The trylock-failure and wrong-order branches take it per allocation. The insight to take: the amortisation factor is exactly pcp->batch (measured at 63 for every zone above 1 GiB on the test machine below), so in steady state the zone lock is acquired roughly once per 63 order-0 allocations instead of once per allocation — and between refills, not at all.

The cost of making this work: pcp->lock itself

It is worth being honest that the per-pcp spinlock is not free. Mel Gorman’s commit introducing it measured the regression directly (4b23a68f9536, “mm/page_alloc: protect PCP lists with a spinlock”, 2022-06-24): “There is overhead in the fast path due to acquiring the spinlock even though the spinlock is per-cpu and uncontended in the common case.” The Page Fault Test (PFT) numbers in that commit show faults/sec moving between −1.34% and +0.61% across 1–8 threads, which he characterises as “a small hit in the number of faults per second but given that the results are more stable, it’s borderline noise.”

What the kernel bought for that noise was the ability to drain a remote CPU’s lists. Before the spinlock, the lists were protected by local_lock_irqsave, and the commit states the consequence: “Remote draining of the lists is impossible and a workqueue is required and every task allocation/free must disable then enable interrupts which is expensive.” The same commit reproduces pahole output showing the structure still fits in four cache lines:

struct per_cpu_pages {
        spinlock_t                 lock;                 /*     0     4 */
        int                        count;                /*     4     4 */
        int                        high;                 /*     8     4 */
        int                        batch;                /*    12     4 */
        short int                  free_factor;          /*    16     2 */
        short int                  expire;               /*    18     2 */
        /* XXX 4 bytes hole, try to pack */
        struct list_head           lists[13];            /*    24   208 */
        /* size: 256, cachelines: 4, members: 7 */
};

That layout is from the v6.0-era structure (13 lists, free_factor rather than free_count, no high_min/high_max); the v6.12 shape is below. The size matters because struct per_cpu_pages is ____cacheline_aligned_in_smp and there is one per (CPU, zone) — on the 32-CPU, 3-populated-zone machine measured later, that is 96 instances, or 24 KiB of metadata, which is nothing. What would matter is false sharing, and the alignment is what prevents it: two CPUs hammering their own pagesets must never dirty the same cache line.

The struct per_cpu_pages (v6.12)

Every struct zone has, per online CPU, one of these (it is ____cacheline_aligned_in_smp, so each CPU’s pageset sits in its own cache line to avoid false sharing):

struct per_cpu_pages {
	spinlock_t lock;	/* Protects lists field */
	int count;		/* number of pages in the list */
	int high;		/* high watermark, emptying needed */
	int high_min;		/* min high watermark */
	int high_max;		/* max high watermark */
	int batch;		/* chunk size for buddy add/remove */
	u8 flags;		/* protected by pcp->lock */
	u8 alloc_factor;	/* batch scaling factor during allocate */
#ifdef CONFIG_NUMA
	u8 expire;		/* When 0, remote pagesets are drained */
#endif
	short free_count;	/* consecutive free count */
 
	/* Lists of pages, one per migrate type stored on the pcp-lists */
	struct list_head lists[NR_PCP_LISTS];
} ____cacheline_aligned_in_smp;

Every field is either a page count, a threshold, or a piece of the auto-tuner. Grouping them that way:

Field	Type	Group	What it is
lock	spinlock_t	locking	The per-pcp lock discussed above. Not the zone lock. Added in v6.0; acquired with spin_trylock, so it can be skipped
count	int	state	Total pages currently parked in this pageset, summed across all its lists and counted in base pages — a cached order-3 block adds 8, not 1 (pcp->count += 1 << order)
high	int	threshold	The live, auto-tuned high watermark. count >= high after a free triggers a drain. Starts at 0 and is only clamped into [high_min, high_max] the first time nr_pcp_alloc() or nr_pcp_high() runs on this (CPU, zone) — which is why an unused zone shows high: 0 in /proc/zoneinfo
high_min	int	threshold	Floor for high. Computed from the zone’s low watermark divided by the number of node-local CPUs
high_max	int	threshold	Ceiling for high. Computed as if percpu_pagelist_high_fraction had been set to its minimum legal value, 8 — i.e. the largest value an operator could have asked for by hand
batch	int	threshold	Pages moved per zone-lock acquisition on refill/drain. Fixed at boot from zone size; not auto-tuned (the sysctl documentation states this explicitly)
flags	u8	auto-tuner	PCPF_PREV_FREE_HIGH_ORDER (bit 0) and PCPF_FREE_HIGH_BATCH (bit 1); both drive the high-order free_high heuristic
alloc_factor	u8	auto-tuner	Left-shift applied to batch on refill. Doubles on each consecutive order-0 allocation, halves on every free. Capped at CONFIG_PCP_BATCH_SCALE_MAX
expire	u8 (NUMA only)	drain	Countdown for remote-node draining; reset to 3 by the vmstat updater whenever this pageset’s zone stats moved (“3 seconds idle till flush”)
free_count	short	auto-tuner	Running count of consecutively freed pages. Halved on every allocation (pcp->free_count >>= 1), incremented by 1 << order on every free. Drives how many pages a drain moves
lists[NR_PCP_LISTS]	struct list_head[]	state	The page lists themselves — 14 of them on a CONFIG_TRANSPARENT_HUGEPAGE x86-64 build. Layout below

Two of these repay a second look because they are counter-intuitive.

count is in base pages, not entries. free_unref_page_commit() does pcp->count += 1 << order and __rmqueue_pcplist() does pcp->count -= 1 << order. So a pageset holding a single cached 2 MiB THP block reports count: 512. This matters when reading /proc/zoneinfo: the number there is directly comparable to nr_free_pages, not to a list length.

high starts at zero and stays there until used. per_cpu_pages_init() memsets the whole structure and then sets only high_min, high_max and batch to their boot-pageset values (mm/page_alloc.c L5646-L5668); pageset_update() later writes batch, high_min and high_max but deliberately never touches high. The first clamp(pcp->high, high_min, high_max) inside nr_pcp_alloc()/nr_pcp_high() is what lifts it. The observable consequence is in the live output later in this note: the DMA zone on the test machine shows high: 0 on all 32 CPUs, because nothing has ever allocated from ZONE_DMA on that box, while DMA32 and Normal show clamped, auto-tuned values.

The list array: order_to_pindex()

The lists are not “one per migratetype”. They are a flattened two-dimensional array indexed by (migratetype, order), plus two extras for transparent huge pages (include/linux/mmzone.h L656-L666):

/*
 * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists
 * are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list
 * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE.
 */
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define NR_PCP_THP 2
#else
#define NR_PCP_THP 0
#endif
#define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1))
#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP)

Substituting the real constants — MIGRATE_PCPTYPES is 3 (it is the enumerator immediately after MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_RECLAIMABLE, mmzone.h L48-L53) and PAGE_ALLOC_COSTLY_ORDER is 3 — gives NR_LOWORDER_PCP_LISTS = 3 × 4 = 12 and NR_PCP_LISTS = 12 + 2 = 14.

The index function is plain arithmetic (mm/page_alloc.c L509-L539):

static inline unsigned int order_to_pindex(int migratetype, int order)
{
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (order > PAGE_ALLOC_COSTLY_ORDER) {
		VM_BUG_ON(order != HPAGE_PMD_ORDER);        /* only order 9 gets here */
		movable = migratetype == MIGRATE_MOVABLE;
		return NR_LOWORDER_PCP_LISTS + movable;     /* 12 or 13 */
	}
#endif
	return (MIGRATE_PCPTYPES * order) + migratetype;  /* 0..11, order-major */
}
 
static inline int pindex_to_order(unsigned int pindex)
{
	int order = pindex / MIGRATE_PCPTYPES;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (pindex >= NR_LOWORDER_PCP_LISTS)
		order = HPAGE_PMD_ORDER;
#endif
	return order;
}

Note the asymmetry: the low-order lists are order-major, migratetype-minor, so the three migratetypes of a given order are adjacent; the two THP lists collapse three migratetypes into two buckets (movable / everything else). The full table:

pindex	Order	Size (x86-64, 4 KiB pages)	Migratetype
0	0	4 KiB	MIGRATE_UNMOVABLE
1	0	4 KiB	MIGRATE_MOVABLE
2	0	4 KiB	MIGRATE_RECLAIMABLE
3	1	8 KiB	MIGRATE_UNMOVABLE
4	1	8 KiB	MIGRATE_MOVABLE
5	1	8 KiB	MIGRATE_RECLAIMABLE
6	2	16 KiB	MIGRATE_UNMOVABLE
7	2	16 KiB	MIGRATE_MOVABLE
8	2	16 KiB	MIGRATE_RECLAIMABLE
9	3	32 KiB	MIGRATE_UNMOVABLE
10	3	32 KiB	MIGRATE_MOVABLE
11	3	32 KiB	MIGRATE_RECLAIMABLE
12	9 (HPAGE_PMD_ORDER)	2 MiB	unmovable or reclaimable
13	9 (HPAGE_PMD_ORDER)	2 MiB	movable

classDiagram
    class zone {
        +char* name
        +spinlock_t lock  «shared by ALL CPUs»
        +free_area free_area[MAX_ORDER+1]
        +per_cpu_pages __percpu* per_cpu_pageset
        +per_cpu_zonestat __percpu* per_cpu_zonestats
        +int pageset_high_min
        +int pageset_high_max
        +int pageset_batch
    }
    class per_cpu_pages {
        «one per (CPU, zone), ____cacheline_aligned_in_smp»
        +spinlock_t lock
        +int count  «in base pages»
        +int high  «auto-tuned, 0 until first use»
        +int high_min
        +int high_max
        +int batch  «fixed at boot»
        +u8 flags
        +u8 alloc_factor
        +u8 expire  «NUMA only»
        +short free_count
        +list_head lists[14]
    }
    class lists_0_11 {
        «NR_LOWORDER_PCP_LISTS = MIGRATE_PCPTYPES * 4 = 12»
        pindex = 3*order + migratetype
        order 0..3  ·  4 KiB .. 32 KiB
        UNMOVABLE / MOVABLE / RECLAIMABLE
    }
    class lists_12_13 {
        «NR_PCP_THP = 2»
        pindex = 12 + (migratetype == MOVABLE)
        order 9  ·  2 MiB
        12: unmovable+reclaimable, 13: movable
    }
    zone "1" --> "N CPUs" per_cpu_pages : per_cpu_pageset
    per_cpu_pages --> lists_0_11 : lists[0..11]
    per_cpu_pages --> lists_12_13 : lists[12..13]

The pcp’s place in struct zone, and the shape of its list array on a CONFIG_TRANSPARENT_HUGEPAGE x86-64 build. What it shows: one zone owns one zone->lock shared by every CPU, plus a per-CPU array of per_cpu_pages, each holding 14 independent lists keyed by (order, migratetype). The insight to take: the segregation is what lets a refill or an allocation touch one list without disturbing the others — and it is also why a burst of high-order frees can evict order-0 pages, since count and high are shared across all 14 lists even though the lists themselves are not.

Resolved

An earlier revision flagged as unverified the claim that the pcp caches orders 0 through 3 rather than order-0 only. It is now confirmed against order_to_pindex(), pindex_to_order() and pcp_allowed_order() in mm/page_alloc.c at the v6.12 tag, all fetched during this revision, and cross-checked against /proc/zoneinfo on a live system.

Mechanical Walk-through: Alloc, Free, Refill, Drain

Before the page paths, one disambiguation. A struct zone carries two per-CPU things that are easy to confuse, and only one of them holds pages:

struct per_cpu_zonestat {        /* per-CPU per-zone COUNTERS, not pages */
	s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS];
	s8 stat_threshold;
	unsigned long vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];   /* CONFIG_NUMA */
};
struct per_cpu_nodestat {        /* per-CPU per-node COUNTERS */
	s8 stat_threshold;
	s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS];
};

These batch up VM statistics (e.g. NR_FREE_PAGES) the same way the page lists batch pages — folding per-CPU deltas into the global counters only when a threshold is crossed, so that hot counters don’t bounce a shared cache line between CPUs. They matter to this note for one reason: the periodic vmstat updater that folds them is also the timer that drives pcp decay and remote draining, described below. They hold no pages.

Allocation: rmqueue_pcplist → __rmqueue_pcplist → rmqueue_bulk

rmqueue_pcplist() takes the pcp trylock, does one piece of auto-tuner bookkeeping, and delegates (mm/page_alloc.c L3019-L3046):

	pcp = pcp_spin_trylock(zone->per_cpu_pageset);
	if (!pcp) { pcp_trylock_finish(UP_flags); return NULL; }   /* fall back to the buddy */
 
	/*
	 * On allocation, reduce the number of pages that are batch freed.
	 * See nr_pcp_free() where free_factor is increased for subsequent frees.
	 */
	pcp->free_count >>= 1;                                     /* (a) decay the free streak */
	list = &pcp->lists[order_to_pindex(migratetype, order)];
	page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, list);
	pcp_spin_unlock(pcp);
	...
	if (page) {
		__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);  /* (b) */
		zone_statistics(preferred_zone, zone, 1);
	}

Line (a) is half of the alloc/free symmetry that makes the auto-tuner work: an allocation halves free_count, and (in free_unref_page_commit) a free halves alloc_factor. The two counters therefore measure streaks rather than totals, and decay to nothing as soon as the workload changes direction. Line (b) is what makes pgalloc_normal in /proc/vmstat usable as a pcp fast-path counter — it is incremented here, on the pcp path, as well as in rmqueue_buddy.

__rmqueue_pcplist() is the pop, with the refill inlined (L2987-L3015):

	do {
		if (list_empty(list)) {
			int batch = nr_pcp_alloc(pcp, zone, order);   /* how many to pull */
			int alloced;
 
			alloced = rmqueue_bulk(zone, order, batch, list,
					migratetype, alloc_flags);    /* TAKES zone->lock ONCE */
 
			pcp->count += alloced << order;
			if (unlikely(list_empty(list)))
				return NULL;                          /* zone genuinely empty */
		}
 
		page = list_first_entry(list, struct page, pcp_list);
		list_del(&page->pcp_list);
		pcp->count -= 1 << order;
	} while (check_new_pages(page, order));               /* retry if a page is corrupt */

rmqueue_bulk() is the only place on the allocation path that touches zone->lock. It grabs it once, loops calling __rmqueue() batch times, and appends each block with list_add_tail() — deliberately, so that pages come off the list in ascending PFN order, with the in-tree rationale being that “this is useful for IO devices that can merge IO requests if the physical pages are ordered properly” (L2300-L2329).

How many pages to pull: nr_pcp_alloc()

This is the alloc-side auto-tuner (mm/page_alloc.c L2936-L2983):

	base_batch = READ_ONCE(pcp->batch);
	high_min   = READ_ONCE(pcp->high_min);
	high_max   = READ_ONCE(pcp->high_max);
	high = pcp->high = clamp(pcp->high, high_min, high_max);   /* (1) first-use clamp */
 
	if (unlikely(high < base_batch))
		return 1;                                          /* (2) pcp disabled / boot pageset */
 
	if (order)
		batch = base_batch;
	else
		batch = (base_batch << pcp->alloc_factor);         /* (3) scale ONLY order-0 */
 
	/* If we had larger pcp->high, we could avoid to allocate from zone. */
	if (high_min != high_max && !test_bit(ZONE_BELOW_HIGH, &zone->flags))
		high = pcp->high = min(high + batch, high_max);    /* (4) grow high on refill */
 
	if (!order) {
		max_nr_alloc = max(high - pcp->count - base_batch, base_batch);
		/* Double the number of pages allocated each time there is
		 * subsequent allocation of order-0 pages without any freeing. */
		if (batch <= max_nr_alloc &&
		    pcp->alloc_factor < CONFIG_PCP_BATCH_SCALE_MAX)
			pcp->alloc_factor++;                       /* (5) ramp up */
		batch = min(batch, max_nr_alloc);
	}
 
	/* Scale batch relative to order if batch implies free pages can be stored on the PCP. */
	if (batch > 1)
		batch = max(batch >> order, 2);                    /* (6) fewer, bigger blocks */
 
	return batch;

Step by step. (1) is where a never-used pageset’s high of 0 gets lifted into range — the mechanism behind high: 0 in /proc/zoneinfo for an idle zone. (2) high < batch means the pcp has been administratively disabled (zone_pcp_disable() sets high to 0, see the drain section) or is still the boot pageset; refill exactly one block and do not cache. (3) the alloc_factor shift applies only to order-0: a high-order refill uses the base batch, because 63 THPs is 126 MiB. (5) each consecutive order-0 refill with no intervening free bumps alloc_factor by one, so the batch doubles: 63, 126, 252, 504, 1008, 2016 — capped by CONFIG_PCP_BATCH_SCALE_MAX, whose in-tree default is 5 (range 0–6), i.e. a maximum multiplier of 32 (mm/Kconfig L701-L710). The Kconfig help text states the trade directly: “The batch number is scaled automatically to improve page allocation/free throughput. But too large scale factor may hurt latency. This option sets the upper limit of scale factor to limit the maximum latency.” (6) finally divides by the order so the batch is measured in pages, not blocks, with a floor of 2.

Freeing: free_unref_page → free_unref_page_commit

The free path decides three things: is this page even eligible for the pcp, which list does it go on, and has the pageset now gone over high. Eligibility first (mm/page_alloc.c L2644-L2686):

	if (!pcp_allowed_order(order)) {
		__free_pages_ok(page, order, FPI_NONE);      /* order 4..8, 10+: straight to buddy */
		return;
	}
	if (!free_pages_prepare(page, order))
		return;
 
	migratetype = get_pfnblock_migratetype(page, pfn);
	if (unlikely(migratetype >= MIGRATE_PCPTYPES)) {
		if (unlikely(is_migrate_isolate(migratetype))) {
			free_one_page(page_zone(page), page, pfn, order, FPI_NONE);
			return;                                  /* ISOLATE must NEVER be cached */
		}
		migratetype = MIGRATE_MOVABLE;                   /* HIGHATOMIC and CMA -> MOVABLE */
	}

The comment above that block is the design statement: “We only track unmovable, reclaimable and movable on pcp lists. Place ISOLATE pages on the isolated list because they are being offlined but treat HIGHATOMIC and CMA as movable pages so we can get those areas back if necessary. Otherwise, we may have to free excessively into the page allocator.” Note the consequence: a MIGRATE_CMA or MIGRATE_HIGHATOMIC page freed through the pcp is relabelled movable for caching purposes and will be handed back out as a movable page. The real migratetype is re-read from the pageblock on drain (free_pcppages_bulk calls get_pfnblock_migratetype() again before __free_one_page()), so the buddy lists stay correct; it is only the cached copy that is approximate.

Then the commit (L2593-L2637):

	pcp->alloc_factor >>= 1;                              /* mirror of free_count >>= 1 */
	__count_vm_events(PGFREE, 1 << order);
	pindex = order_to_pindex(migratetype, order);
	list_add(&page->pcp_list, &pcp->lists[pindex]);       /* LIFO: cache-hot page first */
	pcp->count += 1 << order;
 
	batch = READ_ONCE(pcp->batch);
	if (order && order <= PAGE_ALLOC_COSTLY_ORDER) {
		free_high = (pcp->free_count >= batch &&
			     (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) &&
			     (!(pcp->flags & PCPF_FREE_HIGH_BATCH) ||
			      pcp->count >= READ_ONCE(batch)));
		pcp->flags |= PCPF_PREV_FREE_HIGH_ORDER;
	} else if (pcp->flags & PCPF_PREV_FREE_HIGH_ORDER) {
		pcp->flags &= ~PCPF_PREV_FREE_HIGH_ORDER;
	}
	if (pcp->free_count < (batch << CONFIG_PCP_BATCH_SCALE_MAX))
		pcp->free_count += (1 << order);
	high = nr_pcp_high(pcp, zone, batch, free_high);
	if (pcp->count >= high) {
		free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high),
				   pcp, pindex);                  /* TAKES zone->lock */
		...
	}

Note list_add, not list_add_tail — frees push onto the head, so the page a CPU just released is the next one it will be handed. That is the cache-locality half of the design, and it is the opposite of the refill path’s list_add_tail, which wants PFN order.

The free_high heuristic exists because caching high-order blocks is a fragmentation risk, and the in-tree comment says so: “As high-order pages other than THP’s stored on PCP can contribute to fragmentation, limit the number stored when PCP is heavily freeing without allocation.” When a CPU is on a sustained streak of freeing orders 1–3 with no allocations in between, free_high becomes true and the drain switches to a much more aggressive mode. The two flags that gate it are documented in mmzone.h (L668-L681): PCPF_PREV_FREE_HIGH_ORDER means “a high-order page is freed in the previous page freeing. To avoid to drain PCP for an accident high-order page freeing”, and PCPF_FREE_HIGH_BATCH means “preserve pcp->batch pages in PCP before draining PCP for consecutive high-order pages freeing without allocation if data cache slice of CPU is large enough.”

Freeing many folios at once: free_unref_folios()

There is a third entry point worth naming because reclaim and truncation use it heavily. free_unref_folios() takes a folio_batch (up to 31 folios) and frees the whole batch, holding one pcp->lock for as long as consecutive folios belong to the same zone and switching locks only when the zone changes (mm/page_alloc.c L2691-L2800). The saving is not the zone lock — that is already amortised — but the repeated pcpu_task_pin() / this_cpu_ptr() / spin_trylock sequence, which free_unref_page() would pay once per folio. Everything else, including the free_high heuristic and the drain decision, is the same free_unref_page_commit() code.

The two drain-sizing functions

nr_pcp_high() answers “what is the threshold right now”, and nr_pcp_free() answers “how many pages do I move” (mm/page_alloc.c L2520-L2590):

static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free_high)
{
	/* Free as much as possible if batch freeing high-order pages. */
	if (unlikely(free_high))
		return min(pcp->count, batch << CONFIG_PCP_BATCH_SCALE_MAX);   /* up to 63*32 */
 
	if (unlikely(high < batch))
		return 1;                          /* PCP disabled or boot pageset */
 
	min_nr_free = batch;                       /* always leave at least `batch` behind */
	max_nr_free = high - batch;
 
	/* Increase the batch number to the number of the consecutive
	 * freed pages to reduce zone lock contention. */
	batch = clamp_t(int, pcp->free_count, min_nr_free, max_nr_free);
	return batch;
}

So the drain size is clamp(free_count, batch, high - batch): a CPU that has just freed thousands of pages in a row drains thousands at once (one zone-lock acquisition), while a CPU that trickled over the threshold drains only batch. And min_nr_free = batch guarantees the pageset is never emptied — there is always at least one batch left so the next allocation still hits the fast path.

nr_pcp_high() layers three policies on top of the clamp:

	high = pcp->high = clamp(pcp->high, high_min, high_max);
	if (unlikely(!high)) return 0;                     /* pcp disabled: drain everything */
 
	if (unlikely(free_high)) {
		pcp->high = max(high - (batch << CONFIG_PCP_BATCH_SCALE_MAX), high_min);
		return 0;                                  /* (A) high-order streak: drain all */
	}
 
	if (test_bit(ZONE_RECLAIM_ACTIVE, &zone->flags)) { /* (B) kswapd is scanning */
		int free_count = max_t(int, pcp->free_count, batch);
		pcp->high = max(high - free_count, high_min);
		return min(batch << 2, pcp->high);
	}
 
	if (high_min == high_max) return high;             /* (C) sysctl set by hand: no tuning */
 
	if (test_bit(ZONE_BELOW_HIGH, &zone->flags)) {     /* (D) zone under its high watermark */
		int free_count = max_t(int, pcp->free_count, batch);
		pcp->high = max(high - free_count, high_min);
		high = max(pcp->count, high_min);
	} else if (pcp->count >= high) {
		int need_high = pcp->free_count + batch;
		if (pcp->high < need_high)
			pcp->high = clamp(need_high, high_min, high_max);   /* (E) grow */
	}
	return high;

(B) and (D) are the memory-pressure feedback: when the zone is below its high watermark or kswapd is actively scanning, the pcp stops growing and starts shrinking toward high_min, returning hoarded pages to the buddy allocator where reclaim can see them. (C) is how the sysctl disables auto-tuning — setting percpu_pagelist_high_fraction makes high_min == high_max, and this early return is the switch. (E) is the free-side growth rule, the mirror of step (4) in nr_pcp_alloc().

stateDiagram-v2
    direction LR
    [*] --> Empty: per_cpu_pages_init<br/>count 0, high 0, batch from zone size
    Empty --> Filling: alloc finds list_empty<br/>nr_pcp_alloc gives batch = base shifted by alloc_factor<br/>rmqueue_bulk takes zone lock ONCE
    Filling --> Serving: count += alloced blocks
    Serving --> Serving: alloc — list_del from head<br/>count down, free_count halved<br/>NO zone lock
    Serving --> Serving: free — list_add to head<br/>count up, alloc_factor halved<br/>NO zone lock
    Serving --> Empty: allocated down to an empty list
    Serving --> Draining: a free pushed count past nr_pcp_high
    Draining --> Serving: free_pcppages_bulk of nr_pcp_free pages<br/>takes zone lock ONCE<br/>always leaves at least batch behind
    Serving --> Decaying: 1 s vmstat tick calls decay_pcp_high
    Decaying --> Serving: high = max3 of count minus 32 batches,<br/>high minus high/8, and high_min<br/>then drain count minus high
    Serving --> Purged: drain_all_pages, CPU offline,<br/>zone_pcp_disable, or remote NUMA expiry
    Purged --> Empty: drain_pages_zone loops until count is 0

The pcp’s steady-state cycle plus its three exits. What it shows: two self-loops in Serving are the hot path and neither touches zone->lock; every transition that does take the zone lock is labelled, and each moves many pages. The insight to take: there are three distinct ways pages leave a pageset — a watermark drain (bounded, leaves batch behind), the 1-second idle decay (geometric, floored at high_min), and a forced purge (unbounded, loops to zero). Only the first is part of the fast path; the other two are where the per-CPU abstraction leaks.

Sizing: zone_batchsize() and zone_highsize()

The two thresholds are computed at zone init and recomputed on CPU hotplug and on memory hotplug. Both are short enough to reproduce in full, and both were verified against a live /proc/zoneinfo below.

batch — how much one zone-lock acquisition is worth

static int zone_batchsize(struct zone *zone)
{
	int batch;
 
	/*
	 * The number of pages to batch allocate is either ~0.1%
	 * of the zone or 1MB, whichever is smaller. The batch
	 * size is striking a balance between allocation latency
	 * and zone lock contention.
	 */
	batch = min(zone_managed_pages(zone) >> 10, SZ_1M / PAGE_SIZE);
	batch /= 4;		/* We effectively *= 4 below */
	if (batch < 1)
		batch = 1;
 
	/*
	 * Clamp the batch to a 2^n - 1 value. Having a power
	 * of 2 value was found to be more likely to have
	 * suboptimal cache aliasing properties in some cases.
	 * ...
	 */
	batch = rounddown_pow_of_two(batch + batch/2) - 1;
 
	return batch;
}

(mm/page_alloc.c L5523-L5568)

Symbol by symbol, for a zone with managed pages:

1. zone_managed_pages(zone) >> 10 is managed / 1024, i.e. ~0.1% of the zone.
2. SZ_1M / PAGE_SIZE is 1048576 / 4096 = 256 pages on x86-64. The min() caps the batch at 1 MiB worth.
3. batch /= 4 → at most 64.
4. rounddown_pow_of_two(batch + batch/2) - 1 → rounddown_pow_of_two(64 + 32) - 1 = rounddown_pow_of_two(96) - 1 = 64 - 1 = 63.

So any zone larger than 1 GiB gets batch = 63, full stop, on every x86-64 machine. That is not an inference: Huang Ying’s commit restricting the batch scale factor states it outright — “the default batch size is chosen carefully (in zone_batchsize(), the value is 63 for zone > 1GB)” (52166607ecc9, 2023-10-16).

The 2^n − 1 shape is deliberate anti-aliasing. The in-tree comment explains: “Having a power of 2 value was found to be more likely to have suboptimal cache aliasing properties in some cases. For example if 2 tasks are alternately allocating batches of pages, one task can end up with a lot of pages of one half of the possible page colors and the other with pages of the other colors.”

Uncertain

Verify: what the comment batch /= 4; /* We effectively *= 4 below */ refers to in v6.12. There is no multiplication by 4 anywhere later in zone_batchsize() — the only remaining arithmetic is rounddown_pow_of_two(batch + batch/2) - 1, which multiplies by at most 1.5 before rounding down. The nearest surviving × 4 is high = max(high, batch << 2) in zone_highsize(), a different function. The comment therefore looks like a leftover from an earlier revision where high was derived inside this function. Reason: this is a reading of a stale in-tree comment, and the commit that orphaned it was not traced. To resolve: git log -L on the batch /= 4 line in mm/page_alloc.c would name the commit that moved the high computation out. uncertain

high — how much memory a CPU may hoard

static int zone_highsize(struct zone *zone, int batch, int cpu_online, int high_fraction)
{
	int high, nr_split_cpus;
	unsigned long total_pages;
 
	if (!high_fraction) {
		/* By default, the high value of the pcp is based on the zone
		 * low watermark so that if they are full then background
		 * reclaim will not be started prematurely. */
		total_pages = low_wmark_pages(zone);
	} else {
		/* If percpu_pagelist_high_fraction is configured, the high
		 * value is based on a fraction of the managed pages in the zone. */
		total_pages = zone_managed_pages(zone) / high_fraction;
	}
 
	nr_split_cpus = cpumask_weight(cpumask_of_node(zone_to_nid(zone))) + cpu_online;
	if (!nr_split_cpus)
		nr_split_cpus = num_online_cpus();
	high = total_pages / nr_split_cpus;
 
	/* Ensure high is at least batch*4. The multiple is based on the
	 * historical relationship between high and batch. */
	high = max(high, batch << 2);
 
	return high;
}

(mm/page_alloc.c L5572-L5620)

The key design choice is in the first comment: by default high is derived from the zone’s low watermark, not from its size. The reasoning is that the sum of all per-CPU hoards should be small enough that filling them cannot by itself push the zone below the point where background reclaim starts. Dividing by nr_split_cpus — the number of CPUs local to this zone’s NUMA node, plus one if a CPU is currently coming online — is what makes that sum work out: high_min × nr_cpus ≈ low_wmark_pages(zone).

zone_set_pageset_high_and_batch() then computes both bounds (L5685-L5715):

	new_batch = max(1, zone_batchsize(zone));
	if (percpu_pagelist_high_fraction) {
		new_high_min = zone_highsize(zone, new_batch, cpu_online,
					     percpu_pagelist_high_fraction);
		/* PCP high is tuned manually, disable auto-tuning via
		 * setting high_min and high_max to the manual value. */
		new_high_max = new_high_min;
	} else {
		new_high_min = zone_highsize(zone, new_batch, cpu_online, 0);
		new_high_max = zone_highsize(zone, new_batch, cpu_online,
					     MIN_PERCPU_PAGELIST_HIGH_FRACTION);
	}

with MIN_PERCPU_PAGELIST_HIGH_FRACTION = 8 (L93). So:

$\text{high\_min}=\max \left(\frac{\text{low\_wmark\_pages}(zone)}{\text{nr\_split\_cpus}},4\cdot \text{batch}\right)$

$\text{high\_max}=\max \left(\frac{\text{zone\_managed\_pages}(zone)/8}{\text{nr\_split\_cpus}},4\cdot \text{batch}\right)$

high_max is defined as “the PCP high value when percpu_pagelist_high_fraction sysctl knob is set to MIN_PERCPU_PAGELIST_HIGH_FRACTION. That is, the maximal pcp->high that can be set via sysctl knob by hand” (90b41691b988, “mm: add framework for PCP high auto-tuning”, Huang Ying, 2023-10-16). The auto-tuner may therefore grow a pageset up to — but never beyond — what an administrator could have configured manually.

The arithmetic, checked against a live machine

Predicted values from the formulas above versus /proc/zoneinfo on the test box (Fedora 44, kernel 7.1.13, 32 CPUs on one NUMA node, vm.percpu_pagelist_high_fraction = 0):

Zone	managed	low (incl. boost)	boost	Predicted batch	Observed batch	Predicted high_min	Observed min high	Predicted high_max	Observed max high
DMA	3,841	4	0	1	1 ✓	4	0 (never used — see below)	15	0
DMA32	457,117	658	0	63	63 ✓	252	252 ✓	1,785	1,485
Normal	32,326,596	49,505	512	63	63 ✓	1,531	1,531 ✓	126,275	4,589

Working the Normal row by hand: managed >> 10 = 31,568, capped at 256; 256 / 4 = 64; rounddown_pow_of_two(64 + 32) − 1 = 63. ✓. For high_min, /proc/zoneinfo prints low as low_wmark_pages(), which includes zone->watermark_boost — currently 512 — so the un-boosted low watermark in effect when the pagesets were sized was 49,505 − 512 = 48,993; 48,993 / 32 = 1,531 (integer division), and max(1531, 63 << 2 = 252) = 1,531, which is exactly the minimum high observed across all 32 CPUs. For DMA32: 658 / 32 = 20, and max(20, 252) = 252, again exactly the observed minimum. The formulas reproduce the live values to the page.

The DMA row is the high-starts-at-zero behaviour described earlier: high_min would be 4, but nothing has ever allocated from ZONE_DMA on this machine, so clamp() has never run and pcp->high is still its memset value of 0. The observed maxima (1,485 and 4,589) are below high_max simply because no CPU had been pushed that hard at sampling time — the measurement section below drives one to 19,927.

Auto-Tuning: high Floats, batch Does Not

Until v6.7, pcp->high was a constant computed at boot. Huang Ying’s series made it a runtime variable, and the motivation in the merge commit is worth quoting because it is the general argument for auto-tuning anything in mm/ (90b41691b988): “The page allocation performance requirements of different workloads are usually different. … Now, we have a system wide sysctl knob (percpu_pagelist_high_fraction) to tune PCP high by hand. But, it’s hard to find out the best value by hand. And one global configuration may not work best for the different workloads that run on the same system.”

The algorithm has three stated targets (51a755c56dc0, “mm: tune PCP high automatically”, 2023-10-16): “Minimize allocation/freeing from/to shared zone; Minimize idle pages in PCP; Minimize pages in PCP if the system free pages is too few.” It reaches them with three rules — grow on refill, decay on the vmstat tick, shrink under reclaim:

Direction	Trigger	Rule	Where
Grow	a refill from the zone	high = min(high + batch, high_max) — “if we had larger pcp->high, we could avoid to allocate from zone”	nr_pcp_alloc()
Grow	a free that pushed count past high	high = clamp(free_count + batch, high_min, high_max) — ensure high can hold the batch about to be freed	nr_pcp_high()
Decay	every vmstat tick (default 1 s)	high = max3(count − (batch << 5), high − (high >> 3), high_min), then drain count − high	decay_pcp_high()
Shrink	ZONE_BELOW_HIGH or ZONE_RECLAIM_ACTIVE set	high = max(high − max(free_count, batch), high_min)	nr_pcp_high()
Disabled	percpu_pagelist_high_fraction != 0	high_min == high_max, so every rule above is a no-op	zone_set_pageset_high_and_batch()

The decay constant is arbitrary and admitted to be so: “1/8 (>> 3) of PCP high will be decreased periodically. The value 1/8 is kind of arbitrary. Just to make sure that the idle PCP pages will be freed eventually.” The decay runs from refresh_cpu_vm_stats(do_pagesets=true), i.e. from the same periodic worker that folds per-CPU vmstat deltas, at vm.stat_interval seconds (default 1) (mm/vmstat.c L836-L840, mm/page_alloc.c L2335-L2364).

The payoff was measured by the author on a 224-CPU two-socket Intel server running “8 kbuild instances in parallel (each with make -j 28) in 8 cgroup”, simulating the 0-Day kbuild service. With the series applied: build time −3.5%; “the cycles% of the spinlock contention (mostly for zone lock) decreases from 11.0% to 0.5%”; free_high drains −65.6%; and “the number of pages allocated from zone (instead of from PCP) decreases 83.9%.” An 11% → 0.5% move in zone-lock contention is the single clearest statement of what this whole mechanism is for.

flowchart TD
  HMIN["high_min = max(low_wmark_pages(zone) / nr_node_local_cpus,<br/>4 * batch)<br/><i>computed at boot and on hotplug</i>"]
  HMAX["high_max = max((managed_pages / 8) / nr_node_local_cpus,<br/>4 * batch)<br/><i>= what percpu_pagelist_high_fraction=8 would give</i>"]
  HIGH(["pcp-&gt;high<br/>floats in [high_min, high_max]"])
  HMIN -.->|floor| HIGH
  HMAX -.->|ceiling| HIGH

  REFILL["refill from the zone<br/>nr_pcp_alloc()"] -->|"high = min(high + batch, high_max)"| HIGH
  OVER["a free pushed count past high<br/>nr_pcp_high()"] -->|"high = clamp(free_count + batch,<br/>high_min, high_max)"| HIGH
  TICK["vmstat tick, every vm.stat_interval (1 s)<br/>decay_pcp_high()"] -->|"high = max3(count - 32*batch,<br/>high - high/8, high_min)"| HIGH
  PRESS["ZONE_BELOW_HIGH or ZONE_RECLAIM_ACTIVE<br/>nr_pcp_high()"] -->|"high = max(high - max(free_count, batch),<br/>high_min)"| HIGH
  FH["free_high: a streak of order 1..3 frees<br/>with no allocations"] -->|"high = max(high - 32*batch, high_min)<br/>and drain EVERYTHING"| HIGH

  SYSCTL["vm.percpu_pagelist_high_fraction != 0"] -->|"sets high_min == high_max:<br/>every rule above becomes a no-op"| OFF["auto-tuning DISABLED<br/>high is fixed"]

  ALLOCF["pcp-&gt;alloc_factor<br/>++ on each consecutive order-0 refill,<br/>&gt;&gt;= 1 on every free,<br/>capped at CONFIG_PCP_BATCH_SCALE_MAX = 5"] --> RB["refill size = batch &lt;&lt; alloc_factor<br/>63 -&gt; 126 -&gt; 252 -&gt; 504 -&gt; 1008 -&gt; 2016"]
  FREEC["pcp-&gt;free_count<br/>+= 1&lt;&lt;order on every free,<br/>&gt;&gt;= 1 on every alloc"] --> DS["drain size = clamp(free_count,<br/>batch, high - batch)"]

The auto-tuner, all five feedback rules on one page (v6.7 and later). What it shows: high is pulled up by evidence that the cache is too small (a refill happened; a free overflowed it) and pushed down by evidence that it is too large or that memory is scarce (the 1 s idle tick; the zone dropping below its high watermark; a high-order free streak). The two streak counters at the bottom are independent of high — they size individual refills and drains rather than the ceiling. The insight to take: alloc_factor and free_count are deliberately opposed — every allocation halves free_count and every free halves alloc_factor — so both decay to nothing the moment a workload stops being one-directional. That is what keeps a mixed alloc/free workload from ratcheting the batch size up indefinitely.

Note the asymmetry in the table’s last column: batch is never auto-tuned. The sysctl documentation is explicit — “The batch value of each per-cpu page list remains the same regardless of the value of the high fraction so allocation latencies are unaffected” (Documentation/admin-guide/sysctl/vm.rst L897-L898). batch bounds the latency of a single refill or drain (how long the zone lock is held); high bounds the memory a CPU hoards. Only the second is safe to move at runtime. The alloc_factor shift does scale the effective refill size, which is why CONFIG_PCP_BATCH_SCALE_MAX exists to cap it.

Tuning: percpu_pagelist_high_fraction

The one user-facing tunable is vm.percpu_pagelist_high_fraction, documented in the kernel admin guide. It bounds how large pcp->high may grow: it sets the fraction of a zone’s pages that may be stored across the per-CPU page lists, expressed as a divisor and further divided by the number of online CPUs. The default value is 0, which means “use the kernel’s default sizing” (derived from zone size and CPU count as above) rather than an operator-imposed cap. Raising the fraction (a smaller effective divisor caps high higher) lets each CPU hoard more pages — fewer zone-lock trips, but more free memory parked per-CPU and invisible to allocators on other CPUs; lowering it does the reverse. Operators rarely touch this; it matters mainly on machines with very many CPUs and small zones, where the sum of all per-CPU pagesets can lock away a meaningful slice of free memory.

Three things about the knob are worth pinning down against the source rather than the folklore.

Its units are inverted, and the minimum is 8. The value is a divisor: “A user can specify a number like 100 to allocate 1/100th of each zone between per-cpu lists” (Documentation/admin-guide/sysctl/vm.rst L886-L903). Larger number → smaller cache. The handler rejects anything below MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) and restores the old value (mm/page_alloc.c L6276-L6305), because 1/8 of a zone per-CPU is already the documented ceiling: “The min value for this is 8 which means that we do not allow more than 1/8th of pages in each zone to be stored on per-cpu page lists.”

Setting it switches the auto-tuner off. This is the single most important thing to know before touching it, and it is not in the sysctl documentation — it is in zone_set_pageset_high_and_batch(), where a non-zero fraction makes new_high_max = new_high_min with the comment “PCP high is tuned manually, disable auto-tuning via setting high_min and high_max to the manual value.” Every growth and decay rule in nr_pcp_alloc()/nr_pcp_high()/decay_pcp_high() is gated on high_min != high_max, so they all become no-ops. Writing this sysctl on a v6.7-or-later kernel therefore replaces an adaptive cache with a fixed one — which the auto-tuning commit anticipated: “It’s possible that PCP high auto-tuning doesn’t work well for some workloads. So, when PCP high is tuned by hand via the sysctl knob, the auto-tuning will be disabled.”

It does not change batch, so it does not change latency. “The batch value of each per-cpu page list remains the same regardless of the value of the high fraction so allocation latencies are unaffected” (vm.rst L897-L898). It is purely a footprint knob.

Writing 0 reverts to the default, watermark-derived, auto-tuned behaviour, and the write takes pcp_batch_high_lock and re-runs zone_set_pageset_high_and_batch() for every zone.

Draining: Where the Per-CPU Abstraction Leaks

A page parked in a pageset is free — it is counted in NR_FREE_PAGES — but it is invisible to every other CPU, and, more importantly, it is invisible to the buddy allocator’s coalescing. While it sits on a pcp list it cannot merge with its buddy, so it cannot contribute to a higher-order block. That is the cost of the whole design, and draining is how the kernel pays it back. There are six distinct drain paths in v6.12, and they differ in who runs them, how much they move, and whether they are allowed to fail.

Path	Who calls it	How much it moves	Can it be skipped?
Watermark drain — free_pcppages_bulk(nr_pcp_free(...)) from free_unref_page_commit	the freeing CPU itself, inline	clamp(free_count, batch, high − batch); up to batch << 5 when free_high	No — it is the steady state
Idle decay — decay_pcp_high()	the periodic vmstat worker, once per vm.stat_interval (default 1 s)	count − high, after high is cut by up to high >> 3	Yes — it returns a “more work to do” flag and runs again next tick
drain_local_pages(zone)	the current CPU, for one zone or all	loops min(count, batch << 5) until count == 0	No
drain_all_pages(zone) → __drain_all_pages(zone, false)	any CPU, remotely, for every CPU whose pcp->count != 0	everything on every such pageset	The racy pcp->count check may miss a CPU (see below)
__drain_all_pages(zone, true) via zone_pcp_disable()	memory hot-unplug, alloc_contig_range()/CMA	everything, on every online CPU, with no emptiness check	No — callers “need the drain to be complete when the call returns”
drain_zone_pages() (NUMA only)	the vmstat worker on the CPU that owns the pageset	min(count, batch)	Yes — gated on the expire countdown

flowchart LR
  subgraph SELF["Self-drain — the owning CPU, on its own pageset"]
    FU["free_unref_page_commit()<br/>count &ge; nr_pcp_high()"] --> FPB
    DPH["decay_pcp_high()<br/>1 s vmstat tick"] --> FPB
    DZP["drain_zone_pages()<br/>NUMA expire hits 0"] --> FPB
    DLP["drain_local_pages(zone)"] --> DPZ
  end
  subgraph REMOTE["Remote drain — some other CPU, or a hotplug thread"]
    DAP["drain_all_pages(zone)<br/>= __drain_all_pages(zone, false)<br/>skips CPUs whose count reads 0"] --> DPZ
    ZPD["zone_pcp_disable()<br/>high_min = high_max = 0, then<br/>__drain_all_pages(zone, TRUE)"] --> DPZ
    PACD["page_alloc_cpu_dead()<br/>CPU going offline"] --> DPG["drain_pages(cpu)<br/>all zones"]
  end
  DPG --> DPZ["drain_pages_zone(cpu, zone)<br/>loops min(count, batch &lt;&lt; 5)<br/>until count == 0"]
  DPZ --> FPB["free_pcppages_bulk(zone, n, pcp, pindex)<br/>spin_lock_irqsave(zone-&gt;lock)<br/>round-robin over all 14 lists<br/>__free_one_page() coalesces each block"]
  FPB --> BUDDY["buddy free_area lists<br/>pages can now merge with their buddies"]
  CALLERS["callers of drain_all_pages():<br/>allocation slow path,<br/>compaction, CMA,<br/>memory hot-unplug"] --> DAP
  CMA["alloc_contig_range() / CMA,<br/>memory hot-unplug"] --> ZPD

Every path by which a page leaves a per-CPU list, at v6.12. What it shows: two families. Self-drains (left) are cheap and routine; remote drains (right) are the ones that became possible only when pcp->lock replaced local_lock_irqsave in v6.0 — before that, the right-hand column had to be implemented as a per-CPU workqueue and could be delayed indefinitely on a tickless or realtime CPU. Every path funnels into the same free_pcppages_bulk(), which is the only place pcp pages re-enter the buddy allocator. The insight to take: free_pcppages_bulk() walks the 14 lists round-robin, starting one before the pindex that triggered the drain, so a drain caused by freeing an order-3 block will also release order-0 pages — the eviction interaction described in the failure modes.

The steady-state drain, and why it never empties the list

Covered above: nr_pcp_free() returns clamp(pcp->free_count, batch, high − batch), and the min_nr_free = batch floor exists to “Leave at least pcp->batch pages on the list”. A drain that emptied the pageset would force the very next allocation to take the zone lock for a refill, which is exactly what the cache exists to prevent.

The idle decay — how a hot CPU gives memory back

decay_pcp_high() is the counterweight to the auto-tuner’s growth rules (mm/page_alloc.c L2335-L2364):

	if (pcp->high > high_min) {
		pcp->high = max3(pcp->count - (batch << CONFIG_PCP_BATCH_SCALE_MAX),
				 pcp->high - (pcp->high >> 3), high_min);
		if (pcp->high > high_min)
			todo++;              /* tell the caller to schedule another round */
	}
 
	to_drain = pcp->count - pcp->high;
	if (to_drain > 0) {
		spin_lock(&pcp->lock);
		free_pcppages_bulk(zone, to_drain, pcp, 0);
		spin_unlock(&pcp->lock);
		todo++;
	}

The three-way max3() is a rate limiter in both directions: high − (high >> 3) is the ×7/8 geometric decay, count − (batch << 5) caps how many pages a single tick may release (2,016 on a zone with batch = 63, “avoid to free too many pages to control latency”), and high_min is the floor. This is directly observable, and the measurement section below watches a pageset fall from high = 17,962 to high = 3,106 over 18 seconds of idle.

Remote draining — the reason pcp->lock exists

This is the part that changed most recently and the part most write-ups get wrong. Historically, draining another CPU’s pcp lists was impossible: they were protected by local_lock_irqsave, which by construction only the owning CPU could hold. The kernel therefore queued a per-CPU workqueue item and asked each CPU to drain itself. Jonathan Corbet described the failure mode (LWN, Remote per-CPU page list draining, 15 February 2022): “At best, it causes context switches on each target CPU to run the list-draining callback. But if a target CPU is running in the tickless mode, or if it is running a high-priority realtime task, then the workqueue entry may not run at all for a long time. So any free pages on that CPU will remain locked up in its local lists and, as luck would have it, that’s probably where most of the free pages have ended up.”

That article describes an RCU-based fix from Nicolas Saenz Julienne (two sets of lists per CPU, swapped by compare-and-swap, old set reaped after a grace period) and notes that an earlier spinlock attempt “worked, but it added just the sort of overhead that the per-CPU lists were created to avoid. So those patches did not make it into the kernel.” The spinlock approach is nevertheless what shipped: Mel Gorman’s 4b23a68f9536 landed for v6.0, five months after that article, having measured the overhead as “borderline noise”. Once each pageset had its own spinlock_t, any CPU could take it, and the workqueue disappeared. In v6.12, __drain_all_pages() simply calls drain_pages_zone(cpu, zone) for other CPUs, from the calling CPU, with no IPI and no workqueue (mm/page_alloc.c L2442-L2508):

	for_each_online_cpu(cpu) {
		...
		} else if (zone) {
			pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu);
			if (pcp->count)
				has_pcps = true;      /* racy on purpose — see force_all_cpus */
		}
		...
	}
 
	for_each_cpu(cpu, &cpus_with_pcps) {
		if (zone)
			drain_pages_zone(cpu, zone);   /* runs HERE, on the calling CPU */
		else
			drain_pages(cpu);
	}

Two subtleties. The pcp->count pre-check is deliberately racy, and the function comment says so: “drain_all_pages() is optimized to only execute on cpus where pcplists are not empty. The check for non-emptiness can however race with a free to pcplist that has not yet increased the pcp->count from 0 to 1. Callers that need the guarantee that every CPU has drained can disable the optimizing racy check.” Those callers pass force_all_cpus = true. And the whole operation is serialised by a mutex with a trylock twist: a zone-specific drain will wait for an in-progress drain, while a whole-system drain will silently give up, “Do not drain if one is already in progress unless it’s specific to a zone. Such callers are primarily CMA and memory hotplug and need the drain to be complete when the call returns.”

This is the single point where pcp->lock is actually contended: the draining CPU takes a lock the owning CPU is trying to take on its fast path. It is also why the owning CPU’s pcp_spin_trylock() can fail — “spin_trylock may fail due to a parallel drain or IRQ reentrancy” — and why the fallback to zone->lock has to exist.

CPU hotplug

A CPU going offline runs page_alloc_cpu_dead() from the hotplug state machine (mm/page_alloc.c L5882-L5911):

static int page_alloc_cpu_dead(unsigned int cpu)
{
	lru_add_drain_cpu(cpu);
	mlock_drain_remote(cpu);
	drain_pages(cpu);              /* (1) all zones, this CPU's pagesets */
	vm_events_fold_cpu(cpu);       /* (2) fold its event counters into ours */
	cpu_vm_stats_fold(cpu);        /* (3) fold its zone/node stat deltas */
 
	for_each_populated_zone(zone)
		zone_pcp_update(zone, 0);  /* (4) recompute high/batch for the new CPU count */
 
	return 0;
}

Step (1) is the obvious one — hoarded pages would otherwise be unreachable until the CPU came back. Step (4) is the one people miss: high_min and high_max are both divided by the number of node-local CPUs, so removing a CPU makes every surviving pageset’s ceiling larger. Hotplug therefore resizes the caches as well as flushing one. The matching page_alloc_cpu_online() does the same recompute with cpu_online = 1, which is the + cpu_online term in zone_highsize() — the incoming CPU is not yet in cpumask_of_node() when the recompute runs, so the code adds it by hand.

MIGRATE_ISOLATE — the pages that must never be cached

The other place the abstraction leaks is page isolation, and it leaks in the opposite direction: some pages must be kept out of the cache entirely. MIGRATE_ISOLATE marks a pageblock that is being taken away from the allocator — for memory hot-unplug, for alloc_contig_range()/CMA, or for compaction’s isolation step. If such a page were parked on a pcp list it would be handed straight back out to the next allocation, defeating the isolation. So free_unref_page() checks and bypasses (mm/page_alloc.c L2674-L2683):

	migratetype = get_pfnblock_migratetype(page, pfn);
	if (unlikely(migratetype >= MIGRATE_PCPTYPES)) {
		if (unlikely(is_migrate_isolate(migratetype))) {
			free_one_page(page_zone(page), page, pfn, order, FPI_NONE);
			return;                   /* straight to the buddy under zone->lock */
		}
		migratetype = MIGRATE_MOVABLE;
	}

But a per-page check is not enough, because a page can be freed concurrently with the pageblock being marked isolated. The complete answer is zone_pcp_disable(), which turns the cache off for the whole zone for the duration (mm/page_alloc.c L6772-L6790):

/*
 * Effectively disable pcplists for the zone by setting the high limit to 0
 * and draining all cpus. A concurrent page freeing on another CPU that's about
 * to put the page on pcplist will either finish before the drain and the page
 * will be drained, or observe the new high limit and skip the pcplist.
 *
 * Must be paired with a call to zone_pcp_enable().
 */
void zone_pcp_disable(struct zone *zone)
{
	mutex_lock(&pcp_batch_high_lock);
	__zone_set_pageset_high_and_batch(zone, 0, 0, 1);   /* high_min = high_max = 0 */
	__drain_all_pages(zone, true);                      /* force: no racy skip */
}

The comment is the correctness argument in two clauses: set high to 0 first, then drain with force_all_cpus. A racing free either completes before the drain (and gets drained) or sees high == 0 and, via nr_pcp_high() returning 0, immediately drains itself. There is no window. This is also the origin of the high < batch → return 1 early exits in nr_pcp_alloc() and nr_pcp_free(): with high = 0 and batch = 1, the pcp still functions, it just refuses to cache anything.

Remote NUMA expiry

On CONFIG_NUMA, a CPU can end up holding pages that belong to a remote node’s zone — for instance after an allocation that fell back off-node. Those pages are doubly wasteful: they are invisible to their home node’s allocator and they are slow for whoever eventually gets them. The expire field bounds how long that can last. The vmstat worker sets expire = 3 whenever it folds a non-zero stat delta for that zone (“3 seconds idle till flush”), and then, once per tick (mm/vmstat.c L840-L872):

		if (!__this_cpu_read(pcp->expire) || !__this_cpu_read(pcp->count))
			continue;                        /* nothing to age or nothing to drain */
 
		if (zone_to_nid(zone) == numa_node_id()) {
			__this_cpu_write(pcp->expire, 0);
			continue;                        /* "We never drain zones local to this processor." */
		}
 
		if (__this_cpu_dec_return(pcp->expire)) { changes++; continue; }
 
		if (__this_cpu_read(pcp->count)) {
			drain_zone_pages(zone, this_cpu_ptr(pcp));   /* min(count, batch) */
			changes++;
		}

So a remote-node pageset that goes idle is drained batch pages at a time after roughly three seconds; a local-node pageset is never aged this way (its expire is reset to 0 immediately). Note this is the only drain path that is NUMA-conditional; every other one above is unconditional. On the single-node test machine below, it never fires.

Reading the Per-CPU Section of /proc/zoneinfo

/proc/zoneinfo is the only userspace window onto the pcp, and it is a narrow one: exactly three numbers per (CPU, zone). The printing code is short enough to settle every ambiguity (mm/vmstat.c L1788-L1807):

	seq_printf(m, "\n  pagesets");
	for_each_online_cpu(i) {
		pcp = per_cpu_ptr(zone->per_cpu_pageset, i);
		seq_printf(m,
			   "\n    cpu: %i"
			   "\n              count: %i"
			   "\n              high:  %i"
			   "\n              batch: %i",
			   i, pcp->count, pcp->high, pcp->batch);
#ifdef CONFIG_SMP
		pzstats = per_cpu_ptr(zone->per_cpu_zonestats, i);
		seq_printf(m, "\n  vm stats threshold: %d", pzstats->stat_threshold);
#endif
	}

Field	Kernel expression	Units	What it tells you
cpu:	the loop index over for_each_online_cpu	—	Offline CPUs are not printed, so the row count varies with hotplug
count:	pcp->count	base pages across all 14 lists	How much memory this CPU is currently hoarding in this zone. count × PAGE_SIZE summed over all CPUs is the size of the hoard
high:	pcp->high	base pages	The live auto-tuned threshold, not high_min. 0 means this pageset has never been used (see above) or the pcp is administratively disabled
batch:	pcp->batch	base pages	Fixed at boot from zone size. 63 for any zone over 1 GiB on x86-64
vm stats threshold:	per_cpu_zonestat.stat_threshold	counter units	Not a pcp field — this is the vmstat folding threshold, printed inside the same loop. Easy to misattribute

What is not exported: high_min, high_max, alloc_factor, free_count, flags, expire, and the per-list breakdown. So you can see how much a CPU is holding and what its current ceiling is, but not which orders or migratetypes, and not where in its tuning range it sits. Reconstructing high_min and high_max requires computing them from managed, low, boost and the CPU count with the formulas above — which is what the validation table does.

There is no per-pcp counter in /proc/vmstat either, but two global counters are incremented on the pcp path specifically and are the best available proxies: pgalloc_<zone> is bumped in rmqueue_pcplist() (as well as in rmqueue_buddy()), and pgfree is bumped in free_unref_page_commit() (as well as in the non-pcp free paths). Their difference over an interval is the net page movement; their magnitude is the churn.

Measuring It on a Live System

All of the following is unprivileged and reproducible. Machine: Fedora 44, kernel 7.1.13-200.fc44.x86_64, AMD Ryzen AI MAX+ 395 (16 cores / 32 SMT threads, one NUMA node), 125 GiB RAM, CONFIG_PCP_BATCH_SCALE_MAX=5, vm.percpu_pagelist_high_fraction = 0, vm.stat_interval = 1.

Baseline: 200 MiB is parked in per-CPU lists right now

Summing count across all 32 CPUs per zone on an idle-ish desktop:

zone DMA:    32 cpus, batch={1},  high all 0,          count sum = 0      (0.0 MiB)
zone DMA32:  32 cpus, batch={63}, high 252..1485,      count sum = 4666   (18.2 MiB)
zone Normal: 32 cpus, batch={63}, high 1531..4589,     count sum = 51396  (200.8 MiB)

219 MiB of “free” memory is sitting in per-CPU page lists on this machine at rest — counted in MemFree and nr_free_pages, invisible to any other CPU, and unable to coalesce. That is the concrete size of the trade this note describes. It is also about 0.17% of RAM, which is why nobody notices.

Driving one CPU and watching its pageset move

Pin a process to CPU 7, snapshot /proc/zoneinfo and /proc/vmstat, churn 4 GiB of order-0 anonymous pages through mmap → write one byte per page → munmap, and snapshot again — all from inside the pinned process so the sampling happens on the same CPU:

pinned to CPU 7 (sched_getcpu()=7)

--- BEFORE workload ---
  zone DMA     cpu7  count=0       high=0        batch=1
  zone DMA32   cpu7  count=235     high=252      batch=63
  zone Normal  cpu7  count=258     high=1531     batch=63
  vmstat: pgalloc_normal=8343719324  pgfree=8370665252  nr_free_pages=3454580

faulted and freed 1048576 order-0 anonymous pages (4.0 GiB total churn)
  on CPU 7, still pinned (sched_getcpu()=7)

--- AFTER workload ---
  zone DMA     cpu7  count=0       high=0        batch=1
  zone DMA32   cpu7  count=235     high=252      batch=63
  zone Normal  cpu7  count=18286   high=19927    batch=63
  vmstat: pgalloc_normal=8344774024  pgfree=8371720325  nr_free_pages=3437459

Reading it out. pgalloc_normal moved by 1,054,700 and pgfree by 1,055,073 against 1,048,576 faults — the ~6,000-event excess is other activity on the machine during the 0.5 s run, and the near-equality confirms every fault was an order-0 alloc/free pair. CPU 7’s Normal pageset went from count = 258 to count = 18,286: 71 MiB parked on a single CPU’s lists, all of it pages the workload freed and nobody has taken back yet. And high rose from exactly high_min (1,531) to 19,927 — the auto-tuner in action, still far below high_max = 126,275. DMA and DMA32 did not move at all, because an anonymous page fault allocates from ZONE_NORMAL; note DMA’s high is still 0 after all this, which is the never-clamped behaviour predicted above.

Watching high decay back

Same workload, then sample CPU 7’s Normal pageset every 2 seconds while otherwise idle:

t=0 (end of churn)     zone Normal cpu7: count=17140   high=17962    batch=63
t=2s idle              zone Normal cpu7: count=13799   high=13871    batch=63
t=4s idle              zone Normal cpu7: count=10606   high=10684    batch=63
t=6s idle              zone Normal cpu7: count=6715    high=8498     batch=63
t=8s idle              zone Normal cpu7: count=5959    high=6570     batch=63
t=10s idle             zone Normal cpu7: count=5076    high=5086     batch=63
t=12s idle             zone Normal cpu7: count=3937    high=3958     batch=63
t=14s idle             zone Normal cpu7: count=2900    high=3157     batch=63
t=16s idle             zone Normal cpu7: count=66      high=3358     batch=63
t=18s idle             zone Normal cpu7: count=924     high=3106     batch=63
t=20s idle             zone Normal cpu7: count=4083    high=5626     batch=63

This is decay_pcp_high() running once per vm.stat_interval = 1 s. high falls from 17,962 to 3,106 over 18 seconds and count tracks it down, because each tick sets high = max3(count − 2016, high − high/8, 1531) and then drains count − high. The average ratio works out near ×0.85 per second, between the pure ×7/8 = 0.875 geometric term and the faster count − (batch << 5) term that dominates while count is large. The t=16s/t=18s/t=20s rows are noise from unrelated activity on the machine touching CPU 7 — high ticking back up at t=20s is the growth rule in nr_pcp_alloc() firing on somebody else’s refill, which is exactly the behaviour you would want and a reminder that these are live shared counters, not a controlled experiment. Note that high never approaches high_min = 1,531 during this window; a fully idle CPU would get there in another ~10 s.

Concurrent order-0 fault throughput

The lock-avoidance claim invites a scaling test. N independent processes (separate mm_structs, so mmap_lock is not the bottleneck), each pinned to its own CPU, repeatedly writing one byte per page across a 32 MiB region and then madvise(MADV_DONTNEED)ing it — pure order-0 fault-in and free:

procs=1    elapsed=  0.166s  order-0 faults=327680       1972807 pages/s   1973 kpages/s/cpu
procs=2    elapsed=  0.174s  order-0 faults=655360       3770625 pages/s   1885 kpages/s/cpu
procs=4    elapsed=  0.194s  order-0 faults=1310720      6744676 pages/s   1686 kpages/s/cpu
procs=8    elapsed=  0.245s  order-0 faults=2621440     10692755 pages/s   1337 kpages/s/cpu
procs=16   elapsed=  0.390s  order-0 faults=5242880     13433465 pages/s    840 kpages/s/cpu
procs=32   elapsed=  1.012s  order-0 faults=10485760    10357665 pages/s    324 kpages/s/cpu

Throughput scales to 13.4 million order-0 faults per second across 16 CPUs and then falls back. It is tempting to read the fall-off as zone-lock contention, and that reading would be wrong. Every page fault here includes a 4 KiB clear_page, so 13,433,465 pages/s is 55.0 GB/s of memory being zeroed (13,433,465 × 4,096), and a single CPU manages 8.1 GB/s. The machine has 16 physical cores with 2 SMT threads each, so the 16→32 step adds no execution resources, and 55 GB/s is in the neighbourhood of what this LPDDR5X part sustains. In other words: the page allocator is not the bottleneck in this test — the memset is. That is a positive result for the pcp, but it is a weaker one than a clean lock-contention measurement would be.

Uncertain

Verify: that the 16→32 fall-off above is SMT plus memory bandwidth rather than zone->lock contention. The arithmetic (55 GB/s of clear_page at peak, 8.1 GB/s single-threaded, 16 physical cores) is strongly suggestive but not conclusive, because no direct measurement of lock contention was possible. Reason: perf is not installed on this machine, /sys/kernel/debug is mode 0700, and there is no passwordless sudo, so neither perf lock contention, nor perf record -e lock:contention_begin, nor the mm_page_alloc_zone_locked tracepoint was reachable. To resolve: sudo perf record -e lock:contention_begin -g -- ./scalep 32 40 and check whether zone->lock appears, or compare against a run with vm.percpu_pagelist_high_fraction set very high (sudo sysctl -w vm.percpu_pagelist_high_fraction=10000000, which floors high at 4 × batch = 252) to see how much throughput a near-disabled pcp actually costs. Both need root. uncertain

What could not be measured, and the exact commands that would

• Zone-lock contention directly. perf, bpftool, strace and ltrace are not installed; /sys/kernel/debug is 0700. The resolving command is sudo perf lock contention -b -- ./scalep 32 40, or sudo perf stat -e 'kmem:mm_page_alloc_zone_locked' -- ./scalep 32 40, which counts exactly the allocations that fell through to the buddy allocator under zone->lock.
• Per-list occupancy. Nothing exports pcp->lists[pindex] lengths. sudo drgn -c /proc/kcore -e 'for cpu in for_each_online_cpu(prog): ...' walking zone->per_cpu_pageset would print them; so would a one-line bpftrace kprobe on __rmqueue_pcplist recording arg1 (the order).
• A forced drain. drain_all_pages() has no userspace trigger. echo 1 > /proc/sys/vm/compact_memory (root) calls it via compaction and is the closest available handle; echo 3 > /proc/sys/vm/drop_caches does not drain pcp lists.
• alloc_factor ramping. Not exported. Inferable only indirectly, from how fast count climbs relative to batch during a refill burst.

Version History: What Changed and When

Every date below was established by existence-checking — fetching include/linux/mmzone.h at successive release tags and recording whether a symbol was present — and then confirmed against a dated commit found through git.kernel.org’s cgit log search (?qt=grep&q=…, which works with plain curl; lore.kernel.org returns 403).

Release	Change	How dated
2.6.32 (2009)	One list per migratetype instead of a single list with a linear search. “The following two patches remove searching in the page allocator fast-path by maintaining multiple free-lists in the per-cpu structure. At the time the search was introduced, increasing the per-cpu structures would waste a lot of memory as per-cpu structures were statically allocated at compile-time. This is no longer the case.”	5f8dcc21211a, Mel Gorman, 2009-09-21
v5.14	High-order support. NR_PCP_THP and NR_PCP_LISTS = MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1 + NR_PCP_THP) appear. Before this the pcp held order-0 only, so “all THP and ‘cheap’ high-order allocations including SLUB contends on the zone->lock.” The commit also records the size cost: “struct per_cpu_pages increases in size to 256 bytes (4 cache lines) on x86-64.”	Absent in v5.13’s mmzone.h, present in v5.14’s. 44042b449872, Mel Gorman, 2021-06-28
v5.14	ZONE_RECLAIM_ACTIVE shrinks the pcp when kswapd is scanning	c49c2c47dab6, Mel Gorman
v6.0	spinlock_t lock added to struct per_cpu_pages, replacing local_lock_irqsave. This is what made remote draining possible and removed the per-allocation IRQ disable/enable	Absent in v5.19’s mmzone.h, present in v6.0’s. 4b23a68f9536, Mel Gorman, 2022-06-24
v6.0	List index space reorganised: NR_LOWORDER_PCP_LISTS introduced and the THP lists cut from three (one per migratetype) to one	NR_LOWORDER_PCP_LISTS absent in v5.19, present in v6.0. 5d0a661d808f, “use only one PCP list for THP-sized allocations”, Mel Gorman, 2022-07-17
v6.7	high auto-tuning. high_min, high_max, alloc_factor and free_count appear; free_factor is removed. CONFIG_PCP_BATCH_SCALE_MAX added (default 5). Measured effect on a 224-CPU kbuild workload: zone-lock contention 11.0% → 0.5% of cycles	All four absent in v6.6’s mmzone.h, present in v6.7’s; free_factor present through v6.6 and gone in v6.7. 90b41691b988 + 51a755c56dc0 + 52166607ecc9, all Huang Ying, 2023-10-16
v6.10	NR_PCP_THP goes from 1 to 2 — the single THP list is split into movable and non-movable	#define NR_PCP_THP 1 in v6.9, 2 in v6.10. bf14ed81f571, yangge, 2024-06-24
v6.12 → v6.18	No change. struct per_cpu_pages is byte-for-byte identical at both tags	Diffed include/linux/mmzone.h at v6.12 and v6.18

Two corrections this dating produced, both against claims that circulate widely (and one of which an earlier revision of this note repeated):

1. The pcp is not an “order-0 cache”. It has served orders 0–3 and the THP order since v5.14 (June 2021). Descriptions that say “per-CPU lists of single pages” have been out of date for four years.
2. The per-pcp spinlock is a v6.0 change, not a v5.19 one. The commit is dated 2022-06-24, which is inside the v5.19 development window, but it was merged for v6.0 — include/linux/mmzone.h at the v5.19 tag has no spinlock_t in struct per_cpu_pages, and at v6.0 it does. Dating kernel changes by commit date rather than by release tag is a reliable way to be off by one release.

Failure Modes and Diagnosis

• “Free” memory that isn’t allocatable. Pages sitting in per-CPU pagesets count as free (NR_FREE_PAGES) but cannot satisfy a higher-order allocation on another CPU until drained. On many-CPU machines this can surface as high-order allocation failures despite plenty of reported free memory — the fix is the drain that the slow path already performs, but it explains transient order-N allocation stalls.
• Fragmentation pressure. Because pcp pages skip buddy coalescing while parked, a workload with huge per-CPU caches can degrade the buddy allocator’s ability to form high-order blocks. This is one reason compaction and the slow path drain the pcp lists.
• NUMA hoarding. Without the expire-driven remote drain, a CPU that allocated remote-node pages and then went quiet would hold that remote memory indefinitely; the expiry mechanism bounds this.
• High-order frees evicting order-0 pages. count and high are shared across all 14 lists even though the lists are separate, so a burst of order-3 frees can push count over high and trigger a drain that free_pcppages_bulk() services round-robin across every list — including the order-0 ones. Mel Gorman flagged this in the commit that added high-order support (44042b449872): “this is not necessarily a universal performance win because of how it is implemented. High-order pages can cause pcp->high to be exceeded prematurely for lower-orders so for example, a large number of THP pages being freed could release order-0 pages from the PCP lists. Hence, much depends on the allocation/free pattern as observed by a single CPU.” The free_high heuristic and PCPF_* flags are the mitigation, not a fix.
• pcp->high reading 0 in /proc/zoneinfo is usually not a problem. It means either “this (CPU, zone) pageset has never been used” (common — see ZONE_DMA in the measurements above) or “the pcp is administratively disabled for this zone”, which is a transient state held by zone_pcp_disable() during memory hot-unplug or a CMA allocation. Only the second is worth investigating, and it resolves itself when the operation completes.
• Order 4 and above never get cached. pcp_allowed_order() allows orders 0–3 and 9 only. A driver doing sustained order-4 (64 KiB) or order-5 allocations takes zone->lock on every single one and will show zone-lock contention no matter how the pcp is tuned. The fix is on the caller’s side — use a kmem_cache, a mempool, or the THP order — not in the pcp.
• Diagnosis, in order of usefulness:
  1. /proc/zoneinfo per-CPU count summed per zone — how much memory is currently hoarded. Compare to MemFree.
  2. /proc/zoneinfo high versus the high_min you compute from low, boost and the node-local CPU count — tells you whether the auto-tuner has grown this pageset or it is sitting at its floor.
  3. /proc/vmstat pgalloc_<zone> and pgfree deltas — total churn. High churn with flat MemFree is the pcp doing its job.
  4. /proc/buddyinfo and /proc/pagetypeinfo for whether high-order blocks actually exist (see The Buddy Allocator for how to read them).
  5. perf lock contention or the kmem:mm_page_alloc_zone_locked tracepoint for direct evidence of fall-through to the buddy allocator — both need root.

Alternatives and Relationship to Sibling Mechanisms

The per-CPU page list is the page-granularity analogue of the SLUB allocator’s per-CPU freelist (kmem_cache_cpu): both keep a CPU-local cache in front of a globally-locked structure, both refill/drain in batches, and both trade a little hoarded memory for a lot of avoided contention. They compose: SLUB sits on top of the buddy allocator, so a SLUB cache refilling its slab ultimately pulls order-N pages — often through the pcp lists. The pcp is not a replacement for the buddy allocator (it caches only small orders and depends on the buddy for refill, coalescing, and all large/contiguous allocations); it is a front-end cache. It is also distinct from zone/node structure — there is one pageset per (CPU, zone), so a 64-CPU box with 4 zones has 256 pagesets.

Placing it among the other ways the kernel keeps allocation off a shared lock:

Mechanism	Granularity	Cache unit	Refill source	Bounded by	When it is the right layer
Per-CPU page lists (this note)	whole pages, orders 0–3 and 9	struct list_head lists[14] per (CPU, zone)	rmqueue_bulk() from the buddy free lists	pcp->high, auto-tuned from the zone’s low watermark	Anything that allocates and frees pages at a high rate: page faults, page cache, network buffers, SLUB slab refills
SLUB per-CPU freelist (The Slab Allocator and SLUB)	objects smaller than a page	kmem_cache_cpu — one active slab plus a percpu partial list	a partial slab, or alloc_pages() (which goes through the pcp)	cpu_partial per cache	Many objects of one fixed size and type
mempool	fixed-size objects, pre-reserved	a private array of pre-allocated elements	the backing allocator, at pool creation	the pool’s min_nr	Paths that must not fail under memory pressure (block I/O submission, filesystem writeback)
alloc_pages_bulk*()	whole order-0 pages, many at once	none — it is a bulk interface, not a cache	the pcp lists directly (__rmqueue_pcplist in a loop under one pcp->lock)	the caller’s requested count	Filling a large array of pages in one call: networking receive rings, vmalloc
zone->lock directly	any order	none	the buddy free lists	nothing	Orders 4–8 and 10+, and anything the pcp refuses (MIGRATE_ISOLATE)

The bulk allocator is the closest relative and deserves a sentence, because it is often mistaken for a competitor. alloc_pages_bulk_*() does not add a cache; it uses this one, calling __rmqueue_pcplist() in a loop while holding pcp->lock once for the whole batch (mm/page_alloc.c L4657-L4670). Its value is amortising the entry cost — the zone iteration, watermark check, and lock acquisition — across many pages, which is the same idea one level up (LWN, Bulk memory-allocation APIs; Bulk memory allocation without a new allocator).

Production Notes

The pcp lists are largely invisible in production precisely because they work: the steady-state page-allocator path on a busy server is dominated by order-0 fault-in and free, and the pcp keeps that path off the contended zone lock. The measurement above puts a number on “invisible”: 219 MiB of a 125 GiB machine — 0.17% of RAM — parked across 96 pagesets at rest, and 13.4 million order-0 faults per second sustained across 16 CPUs without the allocator becoming the bottleneck.

When it actually shows up. Three situations, in rough order of how often they are the real problem:

1. High-order allocation failures with plenty of MemFree. The classic symptom: a driver or a THP fault fails an order-4+ allocation while /proc/meminfo shows tens of gigabytes free. Part of that free memory is on pcp lists, where it cannot coalesce. The allocation slow path already calls drain_all_pages() before giving up, so a persistent failure is almost always genuine fragmentation rather than pcp hoarding — but a transient one, or one that resolves on retry, often is the pcp. Check by summing count per zone in /proc/zoneinfo and comparing to the shortfall in /proc/buddyinfo.
2. Very many CPUs against a small zone. high_min is low_wmark_pages(zone) / nr_node_local_cpus, floored at 4 × batch. On a machine with enough CPUs, the floor wins and the sum of the floors can exceed the zone’s low watermark. ZONE_DMA32 on the test machine is already close: high_min = 252 × 32 CPUs = 8,064 pages, against a low watermark of 658. The kernel is deliberately willing to overshoot here rather than make the cache useless; the consequence is that on a many-CPU box, DMA32 can be almost entirely resident in per-CPU lists. This is the one case where lowering vm.percpu_pagelist_high_fraction’s effect (i.e. raising the number) is a real tool — bearing in mind it also switches the auto-tuner off.
3. Latency-sensitive workloads and the batch scale. A drain moves up to batch << CONFIG_PCP_BATCH_SCALE_MAX = 63 × 32 = 2,016 pages under zone->lock in one go. For a PREEMPT_RT or hard-latency workload that is a measurable hold time, and it is exactly what CONFIG_PCP_BATCH_SCALE_MAX exists to bound: “too large scale factor may hurt latency. This option sets the upper limit of scale factor to limit the maximum latency.” Lowering it to 0 disables the scaling entirely at a throughput cost. Note this is a build-time option, not a sysctl.

The historical arc is worth knowing because it explains why the code looks the way it does. The mechanism started in 2.6.32 as a single per-CPU list per migratetype, protected by interrupt disabling. In v5.14 it gained multi-order support, which made it useful to SLUB and THP but introduced the high-order-eviction interaction above. In v6.0 it gained a per-pcp spinlock — a small, measured cost on the fast path bought in exchange for remote draining, which removed the tickless/realtime-CPU failure mode that the workqueue approach had. In v6.7 the fixed high became an auto-tuned one, and that is the change with the largest published numbers attached: on a 224-CPU kbuild workload, zone-lock contention fell from 11.0% to 0.5% of cycles and allocations that fell through to the zone dropped 83.9%. Each of those three steps traded a little fast-path cost or a little complexity for a large reduction in tail behaviour, which is the recurring shape of scalability work in mm/.

A note on what “lock-free” means here. It is common to see the pcp described as a lock-free fast path. It is not lock-free; it takes a per-CPU spinlock via trylock, and before v6.0 it disabled interrupts. What it is is zone-lock-free, and that is the property that matters, because zone->lock is the one lock every CPU on the machine contends for. Getting this distinction right is the difference between understanding why pcp->lock was an acceptable addition and thinking it was a regression.

See Also

• The Buddy Allocator — the shared, zone-locked allocator the pcp caches in front of; pcp refills/drains go here.
• Watermarks and the Allocation Fast Path — the fast path serves order-0 from the pcp; the slow path drains the pcp under pressure.
• Memory Zones and Nodes — there is one per_cpu_pages per (CPU, zone).
• Page Order and Fragmentation — per-migratetype pcp lists keep page types segregated; parked pcp pages skip coalescing.
• The Slab Allocator and SLUB — the per-CPU-cache idea applied to kernel objects, layered on top of the buddy allocator.
• Memory Compaction — drains the pcp to free contiguous runs.
• NUMA Memory Model — the expire field drains remote-node pcp pages.
• Per-CPU Variables — the alloc_percpu / this_cpu_ptr machinery zone->per_cpu_pageset is built on, and the general per-CPU-data pattern.
• GFP Flags and Allocation Contexts — the migratetype the pcp indexes by is derived from the caller’s GFP flags.
• Transparent Huge Pages — the reason pcp_allowed_order() makes a special exception for order 9, and the consumer of lists[12]/lists[13].
• The Zero Page and Lazy Allocation — the order-0 allocation a copy-on-write break from the zero page performs comes through here.
• UP: Linux Memory Management MOC (§4 The Page Allocator).
• UP: Performance Engineering MOC — filed with Per-CPU Variables and Split Page Table Locks under “removing shared state so cores stop contending”; this note is the worked example of that pattern at page granularity.