vault

Per-CPU Page Lists

14 min read

Per-CPU Page Lists

The buddy allocator hands out physically-contiguous runs of pages, but every operation on it must take the per-zone lock — and the overwhelmingly common case is a single CPU allocating or freeing one page (an order-0 page) on a hot path like a page fault. Funnelling every such single-page operation through a shared zone lock would serialize all CPUs against each other and trash cache locality. The kernel’s answer is the per-CPU page list (the struct per_cpu_pages, universally abbreviated pcp or “pageset”): a small cache of order-0 (and, optionally, transparent-huge-page-order) pages held per CPU per zone, refilled and drained from the buddy allocator in batches under the zone lock, so that the steady-state fast path — alloc one page, free one page — touches only CPU-local data and takes no zone lock at all (it takes only a lightweight per-pcp spinlock). The structure is defined in include/linux/mmzone.h (Linux v6.12 source).

This is one of the kernel’s most important scalability optimizations: it converts the page allocator’s hottest path from a globally-contended lock into a CPU-local, cache-hot operation, paying the zone lock only once per batch of pages rather than once per page. It is the per-CPU caching idea — also seen in the SLUB allocator’s per-CPU slabs — applied at the page granularity.

Mental Model

Picture each CPU as having a small private “bucket” of loose order-0 pages for each memory zone. When code wants one page, the CPU grabs it from its own bucket — no coordination with other CPUs. When code frees a page, it drops it back into the bucket. Only when the bucket runs empty (alloc) or overflows a high-water mark (free) does the CPU walk over to the big shared warehouse (the buddy allocator), unlock it, and either scoop up a batch of fresh pages or pour a batch of excess pages back — one lock acquisition amortized across many pages. Because the bucket is CPU-local and recently-touched, the pages it hands out are usually still warm in that CPU’s cache, which also speeds up the code that then uses the page.

flowchart TD
  subgraph CPU0["CPU 0 (local, lock-free fast path)"]
    A0["alloc 1 page →<br/>pop from pcp->lists[mt]"]
    F0["free 1 page →<br/>push to pcp->lists[mt]"]
  end
  subgraph PCP["struct per_cpu_pages (per CPU, per zone)"]
    L["lists[NR_PCP_LISTS]<br/>(one per migratetype + THP order)"]
    H["high / batch / count<br/>watermarks"]
  end
  A0 --> L
  F0 --> L
  L -- "empty on alloc:<br/>rmqueue_bulk(batch)" --> ZONE
  L -- "count >= high on free:<br/>free_pcppages_bulk(batch)" --> ZONE
  ZONE["Buddy allocator<br/>(zone->lock, shared by all CPUs)"]

The per-CPU page list as a write-back cache in front of the buddy allocator. What it shows: single-page alloc/free hits only the CPU-local pcp->lists[] (cheap); the expensive shared zone->lock is touched only on the refill (list empty) and drain (list over the high watermark) transitions, and then it moves a whole batch of pages at once. The insight to take: the design trades a little memory (pages parked in per-CPU buckets, invisible to other CPUs until drained) for an enormous reduction in zone-lock contention — the lock cost is divided by the batch size on the steady-state path.

The struct per_cpu_pages (v6.12)

Every struct zone has, per online CPU, one of these (it is ____cacheline_aligned_in_smp, so each CPU’s pageset sits in its own cache line to avoid false sharing):

struct per_cpu_pages {
	spinlock_t lock;	/* Protects lists field */
	int count;		/* number of pages in the list */
	int high;		/* high watermark, emptying needed */
	int high_min;		/* min high watermark */
	int high_max;		/* max high watermark */
	int batch;		/* chunk size for buddy add/remove */
	u8 flags;		/* protected by pcp->lock */
	u8 alloc_factor;	/* batch scaling factor during allocate */
#ifdef CONFIG_NUMA
	u8 expire;		/* When 0, remote pagesets are drained */
#endif
	short free_count;	/* consecutive free count */
 
	/* Lists of pages, one per migrate type stored on the pcp-lists */
	struct list_head lists[NR_PCP_LISTS];
} ____cacheline_aligned_in_smp;

Walking the fields:

  • lock — a per-pcp spinlock, not the zone lock. It was added (in the 5.19-era rework) so the pcp could be manipulated without disabling preemption/IRQs the old way; it is cheap because it is almost never contended (only this CPU normally touches its own pageset, the exception being remote draining, below).
  • count — how many pages are currently parked in this pageset across all its lists.
  • high — the high watermark: when count reaches high after a free, the pageset is over-full and a batch of pages is drained back to the buddy allocator. This is the knob that bounds how much memory the pageset can hoard.
  • high_min / high_max — the floor and ceiling for the adaptive high value. high is not constant: it floats between high_min and high_max depending on recent free activity, so a CPU doing a burst of frees can temporarily hold more pages (fewer drains), while an idle CPU’s high decays back toward high_min to release memory.
  • batch — the chunk size: how many pages are moved to/from the buddy allocator in one refill or drain. This is the amortization factor — the zone lock is taken once per batch pages.
  • alloc_factor — scales the effective batch up during sustained allocation, so a CPU on a heavy alloc streak refills more pages per zone-lock acquisition.
  • free_count — a running count of consecutive frees, used to drive the adaptive growth of high (a long free streak raises high toward high_max).
  • expire (NUMA builds only) — a countdown for remote pageset draining: a pageset holding pages from a remote NUMA node is periodically aged, and when expire hits zero those remote pages are drained so they can return to their home node. This keeps a CPU from indefinitely hoarding another node’s memory.
  • lists[NR_PCP_LISTS] — the actual page lists, one per migratetype (so that, e.g., MOVABLE and UNMOVABLE pages stay segregated to fight fragmentation), plus extra lists for transparent-huge-page-order pages.

The list count

#define NR_PCP_THP 2  /* or 0 if !CONFIG_TRANSPARENT_HUGEPAGE */
#define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1))
#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP)

MIGRATE_PCPTYPES is the number of migratetypes cached on the pcp; PAGE_ALLOC_COSTLY_ORDER is 3. So the “low-order” lists cover migratetypes × orders 0..3 — i.e. the pcp caches not just order-0 single pages but small contiguous orders up to order 3, segregated by migratetype, with two extra lists for THP-order pages (when transparent huge pages are configured). This per-order, per-migratetype segregation lets the pcp serve a small higher-order allocation locally too, not only single pages.

Uncertain

Verify: that the pcp caches orders 0 through PAGE_ALLOC_COSTLY_ORDER (3) and not only order-0. The NR_LOWORDER_PCP_LISTS macro arithmetic (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) strongly implies orders 0..3 are each cached per migratetype, and the multi-order pcp work landed around 5.x, but I did not fetch the alloc/free code (__rmqueue_pcplist, free_unref_page_commit) at v6.12 to confirm the order indexing and that all four orders are actually populated. Reason: source fetch of mm/page_alloc.c was blocked (backend overload). To resolve: read order_to_pindex()/pindex_to_order() and __rmqueue_pcplist in mm/page_alloc.c at the v6.12 tag. uncertain

Mechanical Walk-through: Alloc, Free, Refill, Drain

The two sibling per-CPU stats structures the zone also carries are distinct from the page lists and hold statistics, not pages:

struct per_cpu_zonestat {        /* per-CPU per-zone counters */
	s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS];
	s8 stat_threshold;
	unsigned long vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];   /* CONFIG_NUMA */
};
struct per_cpu_nodestat {        /* per-CPU per-node counters */
	s8 stat_threshold;
	s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS];
};

These batch up VM statistics (e.g. NR_FREE_PAGES) the same way the page lists batch pages — folding per-CPU deltas into the global counters only when a threshold is crossed, so that hot counters don’t bounce a shared cache line between CPUs.

The page-movement paths themselves work as follows (the function names below are the v6.12 mm/page_alloc.c entry points):

Allocation (rmqueue_pcplist → __rmqueue_pcplist). An order-0 (or small-order) allocation that does not demand special placement is routed to the local pageset. The allocator takes pcp->lock, pops a page off the appropriate lists[] entry for the requested migratetype/order, decrements count, and returns — no zone lock. If the relevant list is empty, it first calls into the buddy allocator (rmqueue_bulk) under the zone lock to pull a batch of pages (scaled by alloc_factor) onto the list, then satisfies the allocation from the freshly-filled list.

Free (free_unref_page → free_unref_page_commit). Freeing a single page that is eligible for pcp caching pushes it onto the local lists[] for its migratetype/order, increments count and free_count. If count now meets or exceeds pcp->high, the pageset is over its watermark and free_pcppages_bulk() drains a batch of pages back to the buddy allocator under the zone lock — coalescing them with their buddies as it goes.

Uncertain

Verify: the exact control flow and function names above (rmqueue_pcplist, __rmqueue_pcplist, rmqueue_bulk, free_unref_page, free_unref_page_commit, free_pcppages_bulk) and the precise condition that triggers a drain (count >= high vs count >= high + batch), plus how alloc_factor/free_count adjust the batch and high adaptively. These are reconstructed from the field comments and prior kernel knowledge, NOT quoted from the v6.12 source (the mm/page_alloc.c fetch was blocked during research). Reason: source fetch blocked (backend overload). To resolve: read these functions and nr_pcp_high()/nr_pcp_free() in mm/page_alloc.c at the v6.12 tag. uncertain

Sizing high and batch

The default high and batch are computed at zone init from the zone’s size and the number of online CPUs (functions zone_batchsize() and zone_highsize() in mm/page_alloc.c). The intent is: batch large enough to amortize the zone lock but small enough that draining doesn’t cause a latency spike; high large enough to absorb bursts but bounded so the per-CPU pagesets across all CPUs don’t lock away a large fraction of the zone’s free memory.

Uncertain

Verify: the actual zone_batchsize() and zone_highsize() formulas in v6.12 (historically batch was derived as roughly min(zone_pages / 1024, some_cap) rounded to a power-of-two-minus-one, and high as a multiple of batch scaled by percpu_pagelist_high_fraction and CPU count — but the multi-order/adaptive-high rework changed this). I could NOT fetch mm/page_alloc.c at v6.12 to quote the exact arithmetic. Reason: source fetch blocked (backend overload). To resolve: read zone_batchsize, zone_highsize, and nr_pcp_high at the v6.12 tag and reproduce the formulas symbol-by-symbol. uncertain

Tuning: percpu_pagelist_high_fraction

The one user-facing tunable is vm.percpu_pagelist_high_fraction, documented in the kernel admin guide. It bounds how large pcp->high may grow: it sets the fraction of a zone’s pages that may be stored across the per-CPU page lists, expressed as a divisor and further divided by the number of online CPUs. The default value is 0, which means “use the kernel’s default sizing” (derived from zone size and CPU count as above) rather than an operator-imposed cap. Raising the fraction (a smaller effective divisor caps high higher) lets each CPU hoard more pages — fewer zone-lock trips, but more free memory parked per-CPU and invisible to allocators on other CPUs; lowering it does the reverse. Operators rarely touch this; it matters mainly on machines with very many CPUs and small zones, where the sum of all per-CPU pagesets can lock away a meaningful slice of free memory.

Draining: When Pages Must Come Back

Pages parked in a pcp are free but unavailable to any other CPU and to the buddy allocator’s coalescing — so several events force a drain (drain_pages, drain_all_pages, drain_zone_pages):

  • Over the high watermark on free — the steady-state batch drain described above.
  • Memory pressure / low watermarks — when the allocation slow path cannot meet a watermark, it drains the pcp lists (drain_all_pages) to return those free pages to the buddy allocator where they can coalesce into higher orders and be seen system-wide. This is why a system under pressure flushes per-CPU caches: hoarded pcp pages would otherwise block compaction and high-order allocation.
  • CPU hotplug / offline — a CPU going offline drains its pagesets so its hoarded pages aren’t orphaned.
  • Remote NUMA expiry — the expire countdown drains a pageset’s remote-node pages so they migrate back toward their home node.
  • Explicit — echo to /proc/sys/vm/... paths and drain_all_pages() from various subsystems.

Draining is the counterweight to caching: the more aggressively pages are held per-CPU (high high), the more must be reclaimed at drain time, and the more high-order/contiguous allocations can be temporarily starved.

Failure Modes and Diagnosis

  • “Free” memory that isn’t allocatable. Pages sitting in per-CPU pagesets count as free (NR_FREE_PAGES) but cannot satisfy a higher-order allocation on another CPU until drained. On many-CPU machines this can surface as high-order allocation failures despite plenty of reported free memory — the fix is the drain that the slow path already performs, but it explains transient order-N allocation stalls.
  • Fragmentation pressure. Because pcp pages skip buddy coalescing while parked, a workload with huge per-CPU caches can degrade the buddy allocator’s ability to form high-order blocks. This is one reason compaction and the slow path drain the pcp lists.
  • NUMA hoarding. Without the expire-driven remote drain, a CPU that allocated remote-node pages and then went quiet would hold that remote memory indefinitely; the expiry mechanism bounds this.
  • Diagnosis: /proc/zoneinfo reports per-CPU pageset count/high/batch per zone per CPU; /proc/vmstat and the per-CPU stat counters show alloc/free rates; perf on zone->lock contention reveals whether the pcp is effectively absorbing the hot path (it should: heavy zone->lock contention usually means batch/high are mis-sized or draining is too frequent).

Alternatives and Relationship to Sibling Mechanisms

The per-CPU page list is the page-granularity analogue of the SLUB allocator’s per-CPU freelist (kmem_cache_cpu): both keep a CPU-local cache in front of a globally-locked structure, both refill/drain in batches, and both trade a little hoarded memory for a lot of avoided contention. They compose: SLUB sits on top of the buddy allocator, so a SLUB cache refilling its slab ultimately pulls order-N pages — often through the pcp lists. The pcp is not a replacement for the buddy allocator (it caches only small orders and depends on the buddy for refill, coalescing, and all large/contiguous allocations); it is a front-end cache. It is also distinct from zone/node structure — there is one pageset per (CPU, zone), so a 64-CPU box with 4 zones has 256 pagesets.

Production Notes

The pcp lists are largely invisible in production precisely because they work: the steady-state page allocator path on a busy server is dominated by order-0 fault-in and free, and the pcp keeps that path off the contended zone lock. The visible artifacts are (1) the per-CPU pageset rows in /proc/zoneinfo, which operators occasionally inspect when chasing high-order allocation failures, and (2) the rare need to drop percpu_pagelist_high_fraction on extreme-core-count, small-zone systems where the summed per-CPU hoard becomes non-trivial. The 5.19-era addition of the per-pcp spinlock (replacing the older local_irq_save/local_lock discipline) and the adaptive high/high_min/high_max + alloc_factor/free_count machinery were both contention/latency optimizations: the adaptive high lets a CPU on a free-heavy burst hold more pages (fewer drains) while still releasing memory when idle, smoothing the worst-case zone-lock traffic without permanently over-provisioning every pageset.

Uncertain

Verify: the “5.19-era” attribution for the per-pcp spinlock introduction and the adaptive-high (high_min/high_max/alloc_factor/free_count) rework — these are from memory, not pinned to a merge-window LWN summary or commit during this task. Reason: LWN/source confirmation fetch was blocked (backend overload). To resolve: check the git history of struct per_cpu_pages in include/linux/mmzone.h and the associated LWN merge-window summaries. uncertain

See Also

Graph View

Backlinks