Automatic NUMA Balancing

Automatic NUMA balancing is the kernel feature that, on a NUMA machine, continuously moves a task’s memory toward the node whose CPU is running it (and, in tiering mode, hot pages toward fast memory) so that most memory accesses stay local. It works by a deliberately sneaky trick: a per-task scanner periodically marks some of the task’s pages inaccessible (PROT_NONE), so the next access traps a minor page fault; the fault handler (do_numa_page) treats that trap as a NUMA hinting fault, samples which CPU and which node touched the page, feeds the sample to the scheduler via task_numa_fault, and — when the evidence is strong enough — migrates the page to the faulting CPU’s node (per the v6.12 Documentation/admin-guide/sysctl/kernel.rst: “the kernel samples what task thread is accessing memory by periodically unmapping pages and later trapping a page fault”). The scheduler reciprocally uses the same fault statistics to pull the task toward the node holding most of its memory. It is controlled by /proc/sys/kernel/numa_balancing and is enabled by default on NUMA hardware in Linux 6.12 LTS.

Everything below is pinned to Linux 6.12 LTS (released 2024-11-17), with mechanism quoted from the v6.12 source tree. The feature is gated by CONFIG_NUMA_BALANCING, which “adds support for automatic NUMA aware memory/task placement” and depends on SMP && NUMA && MIGRATION && !PREEMPT_RT (v6.12 init/Kconfig). A companion option, CONFIG_NUMA_BALANCING_DEFAULT_ENABLED, has default y and, per its help text, “automatic NUMA balancing will be enabled if running on a NUMA machine” — so on a typical multi-socket Linux box it is on out of the box. It is a no-op on uniform-memory (UMA) systems.

Version context, as of 2026-09-04: v6.12 is a maintained longterm series (6.12.108 was the current point release on 2026-09-02 per kernel.org/releases.json), with mainline at 7.3-rc1 and 6.18 the newer LTS. Reading v6.12 here is reading a supported LTS rather than the leading edge; the two places where later kernels differ are flagged inline.

A Short, Verified History

The merge history matters more than most, because the feature arrived in two distinct instalments and the second one — task grouping — is the part that makes it work on threaded applications. Both are now pinned to primary or near-primary sources rather than recollection.

Linux 3.8 (released 2013-02-18) shipped the foundation. The KernelNewbies changelog lists “Automatic NUMA balancing” among the prominent features and describes it exactly as a foundation rather than a finished policy: “The Linux NUMA implementation had some deficiencies. This release includes a new NUMA foundation which will allow to build smarter NUMA policies in the next releases” (KernelNewbies, Linux 3.8). LWN’s contemporaneous account confirms the attribution and the shape of the debate: “NUMA balancing was a topic of fierce debate through much of 2012; that discussion culminated with the merging of Mel Gorman’s NUMA balancing infrastructure patch set into the 3.8 kernel. Those patches provided the basic structure upon which a NUMA balancing solution could be built, but did not attempt to solve the problem in a comprehensive way” (Corbet, NUMA scheduling progress, LWN, 2013-10-01).

Linux 3.13 (released 2014-01-19) shipped the policy. This is where the scheduler side, task grouping, and the cpupid encoding landed — a 63-changeset series from Mel Gorman. KernelNewbies describes it as adding “many of such policies that attempt to put a process near its memory, and can handle cases such as shared pages between processes or transparent huge pages” (KernelNewbies, Linux 3.13). Everything in the “Task Grouping” section below dates from here.

Linux 5.13 (2021) moved the scan-rate knobs out of /proc/sys — see the configuration section; this is the single most common stale fact about the subsystem.

Linux 5.18 (2022) added NUMA_BALANCING_MEMORY_TIERING. The mode constant first appears in include/linux/sched/sysctl.h at the v5.18 tag and is absent at v5.17 (checked by fetching both), from commit c574bbe91703 “NUMA balancing: optimize page placement for memory tiering system”. Linux 6.1 added the companion numa_balancing_promote_rate_limit_MBps sysctl (absent from kernel/sysctl.c at v6.0, present at v6.1), from commit c6833e10008f “memory tiering: rate limit NUMA migration throughput”.

timeline
    title Automatic NUMA balancing, feature by release
    3.8 (2013-02-18) : Mel Gorman's "balancenuma" foundation merged
                     : PROT_NONE scanning + hinting faults + page migration
                     : no scheduler policy yet
    3.13 (2014-01-19) : 63-changeset scheduler-support series
                      : preferred node, task migration, task swapping
                      : cpupid packed into page flags, NUMA groups
                      : read-only and shared-library pages excluded
    5.13 (2021) : scan_delay_ms / scan_period_min_ms / scan_period_max_ms / scan_size_mb
                : MOVED from /proc/sys/kernel to /sys/kernel/debug/sched/numa_balancing/
                : commit 8a99b6833c88, "debug only" knobs de-promoted
    5.18 (2022) : NUMA_BALANCING_MEMORY_TIERING mode added
                : hot/cold promotion by hint-fault latency, for PMEM and CXL
    6.1 (2022) : numa_balancing_promote_rate_limit_MBps sysctl added
    6.12 LTS (2024-11-17) : the tree this note reads
                          : folio-based (migrate_misplaced_folio), sched_ext-aware

Twelve years of one subsystem. What it shows: the feature was not designed and merged in one piece — 3.8 shipped a sampling mechanism with no policy, 3.13 shipped the policy, and the 2021–2022 work repurposed the whole thing for tiered memory. The insight to take: the two most commonly wrong facts about this subsystem are both dated. “The scan knobs are sysctls” was true until 5.13 and is repeated in countless tuning guides; “NUMA balancing is about multi-socket servers” was true until 5.18 and is now only half the story, since NUMA_BALANCING_MEMORY_TIERING uses the same PTE-scanning machinery to move hot pages onto fast memory on machines with only one socket.

Mental Model — Sample by Faulting, Then Migrate

The problem local allocation alone cannot solve: memory is placed once, when allocated, on the node of the CPU that ran the allocating thread. But threads migrate, working sets shift, and fork/exec scramble placement. After enough churn, a thread can end up running on node 1 while most of its pages sit on node 0, paying the interconnect tax on every access — and nothing in the static allocator ever fixes it.

Automatic NUMA balancing closes that gap with a feedback loop. It cannot afford to observe every memory access (hardware gives no cheap per-page access log), so it samples by forcing faults. Periodically it walks the task’s address space and rewrites a chunk of its page-table entries to PROT_NONE — not really revoking access, just arming a trap. When the task next touches such a page, the MMU raises a normal minor fault; the kernel recognizes it as a hinting fault (the page is present in RAM, only the PTE was poisoned), records “CPU on node X touched a page on node Y,” restores the PTE, and — on sufficient evidence — migrates the page from Y to X. The scheduler watches the accumulated per-node fault counts and tries to run the task where its memory already is. The two pressures converge: pages drift toward the task, the task drifts toward its pages.

flowchart TB
  SCAN["task_numa_work (scanner)<br/>runs off the scheduler tick"]
  PROT["change_prot_numa()<br/>mark PTEs PROT_NONE"]
  ACCESS["task touches page → MMU fault"]
  FAULT["do_numa_page()<br/>recognize hinting fault, restore PTE"]
  CHECK["numa_migrate_check / should_numa_migrate_memory<br/>two-stage filter"]
  MIG["migrate_misplaced_folio()<br/>move page to faulting node"]
  STAT["task_numa_fault()<br/>update per-node fault stats"]
  PLACE["task_numa_placement + numa_migrate_preferred<br/>pull TASK toward its memory"]
  SCAN --> PROT --> ACCESS --> FAULT
  FAULT --> CHECK
  CHECK -->|"strong relation"| MIG
  FAULT --> STAT --> PLACE
  PLACE -.->|"scheduler moves task"| SCAN

The automatic NUMA balancing feedback loop in v6.12. What it shows: the scanner arms traps (PROT_NONE), an access springs one, do_numa_page handles it and both (a) considers migrating the page and (b) feeds the scheduler a sample that nudges the task. The insight to take: it is a sampled, statistical mechanism — one fault is just one data point; migration happens only when repeated faults agree, and the scan rate self-tunes so the sampling overhead stays bounded (the code caps it at ~3% of a task’s CPU time).

Mechanical Walk-through

Before the code, the timeline. The whole subsystem is one loop that alternates between two contexts — the scheduler tick that arms traps, and the fault handler that springs them — with the crucial property that they are separated in time by however long it takes the task to next touch the page. That gap is the sampling interval, and it is what turns a mechanism with no hardware support into a usable access profile.

sequenceDiagram
    autonumber
    participant Tick as scheduler tick<br/>(task_tick_numa)
    participant Work as task_numa_work<br/>(runs in the task's own context)
    participant PT as page tables<br/>(change_prot_numa)
    participant App as the task, in userspace
    participant MMU as MMU / fault entry
    participant Fault as do_numa_page<br/>(mm/memory.c)
    participant Filter as should_numa_migrate_memory<br/>(kernel/sched/fair.c)
    participant Mig as migrate_misplaced_folio
    participant Sched as task_numa_fault →<br/>task_numa_placement

    Note over Tick,Work: ARMING — periodic, self-throttled to ≤3% of task CPU time
    Tick->>Work: queue work if jiffies ≥ mm->numa_next_scan
    Work->>PT: walk 256 MB of VMAs from mm->numa_scan_offset
    PT-->>PT: rewrite present PTEs to the PROT_NONE<br/>"NUMA hinting" encoding
    Note over PT: access is NOT actually revoked —<br/>the page stays resident, only the PTE is poisoned

    Note over App,Fault: SPRINGING — happens whenever the task next touches the page
    App->>MMU: ordinary load or store
    MMU->>Fault: minor fault — pte_protnone() and vma_is_accessible()
    Fault->>Fault: take PTL, re-read live PTE, bail if it changed
    Fault->>Fault: numa_migrate_check(): count NUMA_HINT_FAULTS,<br/>set TNF_FAULT_LOCAL if folio_nid == numa_node_id(),<br/>read+swap the folio's last cpupid
    Fault->>Filter: mpol_misplaced() → should_numa_migrate_memory()
    alt evidence is strong enough
        Filter-->>Fault: target_nid
        Fault->>Mig: isolate folio, migrate to target_nid
        Mig-->>Fault: TNF_MIGRATED
    else weak / shared / rate-limited
        Filter-->>Fault: NUMA_NO_NODE
        Note over Fault: goto out_map — page stays put
    end
    Fault->>Sched: task_numa_fault(last_cpupid, nid, nr_pages, flags)
    Sched-->>Sched: accumulate p->numa_faults[] per node,<br/>per MEM/CPU, per private/shared
    Sched-->>Sched: task_numa_placement() picks numa_preferred_nid<br/>numa_migrate_preferred() pulls the TASK there
    Fault->>PT: restore the real protection (out_map)
    PT-->>App: the original load or store completes

One full turn of the NUMA hinting-fault cycle in v6.12. What it shows: the arming and the springing are separated in time and in context — the scanner runs off the tick and poisons PTEs in bulk; the fault handler runs synchronously inside the task’s own memory access. Everything the kernel learns about where memory is being touched comes from that one fault, which yields two independent facts at once: which node holds the page and which node is running the task. The insight to take: the cost is paid by the application, in-line, on a normal load or store. There is no background profiler. Each hinting fault is a real minor fault with a page-table lock acquisition, and a migration adds a page copy plus a TLB shootdown to the critical path of an ordinary memory access. That is why every design decision downstream — the 3% scan budget, the two-stage filter, the promotion rate limit — is about not doing this too often, and why a workload with hand-placed memory should switch the whole thing off.

1. The scanner arms the traps (task_numa_work)

A per-task work item, task_numa_work, is queued from the scheduler tick (task_tick_numa) and runs in the task’s own context (v6.12 kernel/sched/fair.c). It does not scan the whole address space at once. It scans sysctl_numa_balancing_scan_size megabytes worth of pages per pass (default 256 MB), tracking a rolling offset mm->numa_scan_offset so successive passes sweep the address space:

pages = sysctl_numa_balancing_scan_size;
pages <<= 20 - PAGE_SHIFT;   /* MB → pages */
virtpages = pages * 8;       /* skip-ahead budget over empty/already-marked regions */

For each eligible VMA it calls change_prot_numa(vma, start, end) (which lives in mm/mprotect.c and ultimately calls change_protection with the MM_CP_PROT_NUMA flag) to rewrite the present PTEs to the PROT_NONE “NUMA hinting” encoding. The scanner is careful about which VMAs it touches (fair.c):

• It skips VMAs that are not vma_migratable, that are hugetlb (is_vm_hugetlb_page), or VM_MIXEDMAP.
• It skips read-only file-backed mappings and the vDSO — “Shared library pages mapped by multiple processes are not migrated as it is expected they are cache replicated. Avoid hinting faults in read-only file-backed mappings or the vDSO as migrating the pages will be of marginal benefit.”
• It skips inaccessible VMAs “to avoid any confusion between PROT_NONE and NUMA hinting PTEs.”
• Per-VMA state (vma->numab_state) tracks scan sequences so a VMA the task hasn’t actually accessed recently is skipped (NUMAB_SKIP_PID_INACTIVE) — unless no other candidate remains, in which case the scan is forced to guarantee forward progress.

Critically, the scanner self-throttles to keep overhead bounded. After scanning, it accounts the CPU time spent: “Make sure tasks use at least 32x as much time to run other code than they used here, to limit NUMA PTE scanning overhead to 3% max” (fair.c).

2. The scan period adapts

The interval between scans is not fixed. The defaults bound it (fair.c):

unsigned int sysctl_numa_balancing_scan_period_min = 1000;   /* ms */
unsigned int sysctl_numa_balancing_scan_period_max = 60000;  /* ms */
unsigned int sysctl_numa_balancing_scan_size = 256;          /* MB per scan */
unsigned int sysctl_numa_balancing_scan_delay = 1000;        /* ms initial delay */

A task’s per-task numa_scan_period starts near the minimum and is recomputed by task_numa_placement as faults accumulate: a task whose accesses are already mostly local gets its scan period lengthened (less need to look), while a task with many remote faults is scanned more aggressively. The numa_scan_delay (1 s) is the grace period before a freshly-started task is scanned at all — newly forked tasks shouldn’t pay scanning cost before they’ve settled.

The bounds are not used raw. task_nr_scan_windows() divides the task’s resident set size by the 256 MB scan size to get the number of passes needed to sweep it, and both task_scan_min() and task_scan_max() divide the sysctl values by that count. A task with a 2 GB RSS needs 8 windows, so its effective minimum period is 1000/8 = 125 ms — meaning the “1 second minimum” is a bound on the time to scan the whole address space, not on the interval between individual scans. Big processes are therefore scanned more frequently in wall-clock terms, which is what keeps the sampling rate proportional to the thing being sampled. task_scan_min() also imposes a floor derived from MAX_SCAN_WINDOW (2560 MB) so the division cannot drive the period to zero.

The adaptation itself lives in update_task_scan_period() (v6.12 fair.c) and works on two ratios computed over the faults since the last placement pass, each expressed in tenths (NUMA_PERIOD_SLOTS is 10, NUMA_PERIOD_THRESHOLD is 7):

• lr_ratio = local * 10 / (local + remote) — what fraction of faults were already node-local.
• ps_ratio = private * 10 / (private + shared) — what fraction were private rather than shared.

Condition	Meaning	Action	Why
local + shared == 0	The task took no hinting faults at all	Double the period, up to numa_scan_period_max	Task is idle, or its memory is all in regions the scanner skips. Stop paying to look
numa_faults_locality[2] nonzero	A migration failed	Double the period	“we are migrating too quickly or the local node is overloaded” — back off rather than retry harder
ps_ratio >= 7	≥70% of faults are private	Lengthen by (ps_ratio − 7) period-slots	“Most memory accesses are local. There is no need to do fast NUMA scanning, since memory is already local”
lr_ratio >= 7	≥70% of faults are local	Lengthen by (lr_ratio − 7) slots	“Most memory accesses are shared with other tasks. There is no point in continuing fast NUMA scanning, since other tasks may just move the memory elsewhere”
otherwise	Mostly private faults, but not local	Shorten by (7 − max(lr, ps)) slots	“Speed up NUMA scanning to get the memory moved over” — this is the only branch that increases cost, and it fires exactly when there is a fixable problem

The result is clamped to [task_scan_min(p), task_scan_max(p)] and the locality counters are zeroed for the next window. Note the shape of the control law: four of the five branches make the feature cheaper and only one makes it more expensive, and the expensive one is gated on evidence that migration will actually help. This is the mechanism behind the claim that idle or well-placed workloads converge to near-zero overhead — the scan period walks up to the 60-second maximum and stays there. A workload that is thrashing, by contrast, hits the migration-failure branch and also backs off, which is a deliberate choice to fail quiet rather than fail loud.

One more subtlety in task_scan_max(): for a task in a NUMA group, the maximum period is scaled by the group’s size and by how shared its faults are —

		period *= refcount_read(&ng->refcount);
		period *= shared + 1;
		period /= private + shared + 1;
		smax = max(smax, period);

so a large group whose faults are mostly shared gets a much longer ceiling. The reasoning is the same as the lr_ratio branch above: when many tasks touch the same pages, no single task’s scanning tells you much, and scanning harder just multiplies the cost by the group size.

3. The hinting fault is handled (do_numa_page)

When the task touches an armed page, the main fault path detects the special PTE and dispatches to do_numa_page (v6.12 mm/memory.c):

if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
	return do_numa_page(vmf);

do_numa_page takes the page-table lock, re-reads the live PTE (bailing if it changed under it), reconstructs the real protection with pte_modify, finds the struct folio, and calls numa_migrate_check to decide the target node. numa_migrate_check bumps the NUMA_HINT_FAULTS statistic, flags whether the fault was already local (folio_nid(folio) == numa_node_id() ⇒ TNF_FAULT_LOCAL and NUMA_HINT_FAULTS_LOCAL), records the accessing PID into the page’s cpupid field, and finally calls mpol_misplaced to ask “given this task’s memory policy, is this page on the wrong node?” — returning the target node id, or NUMA_NO_NODE to leave it put (memory.c).

If a target node comes back, do_numa_page isolates the folio and calls migrate_misplaced_folio to move it (see Page Migration); on success it sets TNF_MIGRATED. Either way it calls task_numa_fault to record the sample, then restores the PTE to normal protection (out_map:) so the access can complete:

target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags, writable, &last_cpupid);
if (target_nid == NUMA_NO_NODE)
	goto out_map;
if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
	flags |= TNF_MIGRATE_FAIL;
	goto out_map;
}
if (!migrate_misplaced_folio(folio, vma, target_nid)) {
	nid = target_nid;
	flags |= TNF_MIGRATED;
	task_numa_fault(last_cpupid, nid, nr_pages, flags);
	return 0;
}

4. The two-stage migration filter — “migrate on the second fault”

The migration decision is deliberately conservative. A single fault is weak evidence: a page touched once by a remote CPU may never be touched again, and migrating it is wasted work plus a TLB shootdown. So should_numa_migrate_memory applies a two-stage filter that, in effect, requires the same task–page relationship to show up twice before acting (v6.12 fair.c). The code’s own comment explains the statistics:

Multi-stage node selection is used in conjunction with a periodic migration fault to build a temporal task↔page relation. By using a two-stage filter we remove short/unlikely relations. Using P(p) ~ n_p / n_t as per frequentist probability … getting the same result twice in a row … is then given by P(n)². This quadric squishes small probabilities, making it less likely we act on an unlikely task↔page relation.

Each page stores the last CPU+PID that faulted it (its cpupid); a new fault compares against it. A private fault (same PID touching the page again) migrates readily; a shared fault (different tasks) is held to the higher bar of the two-stage test and may instead trigger task grouping so related tasks are co-located. There is an explicit early-life exception — “Allow first faults or private faults to migrate immediately early in the lifetime of a task. The magic number 4 is based on waiting for two full passes of the ‘multi-stage node selection’ test” (p->numa_scan_seq <= 4) — so a fresh task converges quickly rather than waiting out the filter.

The full decision, read top to bottom from should_numa_migrate_memory(), v6.12 fair.c, is a sequence of six gates, and any one of them can end the story:

1. Memoryless destination — if (!node_state(dst_nid, N_MEMORY)) return false. You cannot migrate onto a CPU-only node.
2. Tiering fork — if (folio_use_access_time(folio)) diverts the whole decision to the hot/cold path described in the tiering section below. In tiering mode the page’s cpupid field is repurposed to store an access timestamp, so the private/shared logic below cannot apply.
3. Slow-tier guard — a page on a non-top-tier node with no valid cpupid is left alone unless tiering mode is on.
4. Early-life exemption — (p->numa_preferred_nid == NUMA_NO_NODE || p->numa_scan_seq <= 4) combined with a first or private fault migrates immediately. The comment names the constant: “The magic number 4 is based on waiting for two full passes of the ‘multi-stage node selection’ test.”
5. The two-stage filter proper — if (!cpupid_pid_unset(last_cpupid) && cpupid_to_nid(last_cpupid) != dst_nid) return false. The previous fault must have come from the same node we are now proposing to migrate to. This is what makes it a two-sample test.
6. Private vs. shared — a private fault (cpupid_match_pid) migrates. A shared fault falls through to the group heuristics: migrate if the destination node is ACTIVE_NODE_FRACTION (3) times more CPU-active for this group than the source, or if the group’s CPU-vs-memory ratio favours the move with 3/4 hysteresis:

	/*
	 * Distribute memory according to CPU & memory use on each node,
	 * with 3/4 hysteresis to avoid unnecessary memory migrations:
	 *
	 * faults_cpu(dst)   3   faults_cpu(src)
	 * --------------- * - > ---------------
	 * faults_mem(dst)   4   faults_mem(src)
	 */
	return group_faults_cpu(ng, dst_nid) * group_faults(p, src_nid) * 3 >
	       group_faults_cpu(ng, src_nid) * group_faults(p, dst_nid) * 4;

Walking that inequality symbol by symbol: group_faults_cpu(ng, N) is how much CPU activity the group has on node N (faults weighted by the faulting task’s runtime share, so a busy thread counts for more than an idle one); group_faults(p, N) is how much of the group’s memory is being faulted on node N. The ratio faults_cpu / faults_mem per node is therefore “how CPU-heavy this node is relative to how much of our memory lives there.” The test asks whether the destination is more CPU-heavy-per-byte than the source — i.e. whether moving the page follows the compute — and the factor 3/4 on the left demands the destination win by at least 33% rather than by a hair. Cross-multiplied into integers to avoid division, that is exactly the code above. Without the hysteresis, two nodes with near-equal ratios would migrate the page back and forth on alternating faults.

stateDiagram-v2
    direction TB
    [*] --> Resident : page allocated on node Y

    Resident --> Armed : task_numa_work rewrites the PTE<br/>to the PROT_NONE hinting encoding
    Armed --> Resident : PTE changed under us<br/>(pte_same check fails, or COW, or unmapped)

    Armed --> Sampled : task on node X touches it<br/>do_numa_page runs

    state Sampled {
        [*] --> CheckDst
        CheckDst --> Reject : dst node has no memory
        CheckDst --> Tiering : folio_use_access_time()<br/>(slow-tier page, tiering mode)
        CheckDst --> Early : numa_scan_seq <= 4<br/>and first-or-private fault
        CheckDst --> TwoStage : otherwise
        TwoStage --> Reject : last cpupid's node != X<br/>(the two samples disagree)
        TwoStage --> Accept : private fault<br/>(same PID touched it last)
        TwoStage --> GroupTest : shared fault
        GroupTest --> Accept : dst 3x more CPU-active than src,<br/>OR cpu-per-mem ratio wins by 4 to 3
        GroupTest --> Reject : otherwise
        Tiering --> Accept : hint-fault latency < nbp_threshold<br/>AND under the promotion rate limit
        Tiering --> Reject : page is cold, or rate-limited
        Early --> Accept
    }

    Sampled --> Migrating : Accept, target_nid returned
    Sampled --> Resident : Reject — NUMA_NO_NODE<br/>PTE restored, page stays on Y<br/>(cpupid still updated — this fault<br/>becomes the next one's evidence)

    Migrating --> ResidentX : migrate_misplaced_folio() succeeds<br/>TNF_MIGRATED, now on node X
    Migrating --> Resident : isolation or copy fails<br/>TNF_MIGRATE_FAIL

    ResidentX --> Armed : next scan pass arms it again
    ResidentX --> [*] : freed

    note right of Sampled
      Every path — accept OR reject —
      records the sample. A rejected
      fault is not wasted: it stores
      this CPU+PID as the page's
      last_cpupid, which is exactly
      what the NEXT fault tests against.
    end note

The lifecycle of one page under automatic NUMA balancing. What it shows: a page cycles between Resident and Armed under the scanner, and every fault runs a six-gate decision whose default answer is “leave it alone.” The insight to take: the two-stage filter is not a threshold on a counter — it is a state machine over two consecutive samples, and the state is stored in the page itself. The first fault from a new node almost always rejects, and its only lasting effect is to overwrite last_cpupid; only if the next fault on that page also comes from that node does the filter accept. The kernel’s own comment justifies this statistically: sampling gives P(p) ≈ n_p/n_t, and “getting the same result twice in a row … is then given by P(n)². This quadric squishes small probabilities, making it less likely we act on an unlikely task↔page relation.” Squaring a probability is how you buy confidence with no extra memory.

5. The scheduler side (task_numa_fault → task_numa_placement)

task_numa_fault is the bridge from the MM fault path into the scheduler (v6.12 fair.c). It first checks the global static key — if (!static_branch_likely(&sched_numa_balancing)) return; — so when the feature is disabled the cost is a single predicted branch. It then accumulates per-task, per-node fault counts in p->numa_faults, distinguishing memory faults (where the page is, NUMA_MEMBUF) from CPU faults (which node ran the task, NUMA_CPUBUF) and private vs. shared. Periodically it calls task_numa_placement, which scans those counts to pick the task’s numa_preferred_nid (the node holding most of its memory), and numa_migrate_preferred / task_numa_migrate, which try to migrate the task itself onto that node. So the scheduler pulls the task toward its memory while do_numa_page pulls the memory toward the task — the loop converges from both ends.

6. Task grouping — making threads that share pages converge

Everything so far treats “a task’s memory” as if it were exclusively owned. It usually is not. Threads in one process share an address space by definition; separate processes routinely share a large mapping that neither of them “owns.” If the balancer only ever pulls each task toward its own faults, two threads hammering the same 2 MiB region can end up on opposite nodes, each dragging the pages back and forth. Task grouping is the fix, and it is the piece that turns a page-placement mechanism into something that works for real threaded applications.

The mechanism starts from a very tight data budget. To decide whether two tasks share a page, the kernel must know who touched it last — but there is no room in struct page for a task pointer. LWN’s account of the 3.13 series describes the compromise precisely: the node ID already stored in the page’s flags field was widened into a cpupid, packing the last-accessing CPU number together with “the process ID of the last process to access the page … only the bottom eight bits of the process ID are used, with the understanding that some collisions will be unavoidable” (Corbet, LWN, 2013-10-01). Eight bits of PID is a 1-in-256 chance of a false match, which is why the code applies further sanity tests rather than trusting it.

Given a cpupid, task_numa_group() (v6.12 fair.c) runs on a shared fault. It cannot look up the other task by PID — it only has eight bits — so it does something indirect: it looks at whichever task is currently running on the CPU recorded in the cpupid and checks whether that task’s low PID bits match.

	tsk = READ_ONCE(cpu_rq(cpu)->curr);
	if (!cpupid_match_pid(tsk, cpupid))
		goto no_join;
	grp = rcu_dereference(tsk->numa_group);
	if (!grp || grp == my_grp)
		goto no_join;
	/* Only join the other group if its bigger; if we're the bigger group,
	 * the other task will join us. */
	if (my_grp->nr_tasks > grp->nr_tasks)
		goto no_join;
	/* Tie-break on the grp address. */
	if (my_grp->nr_tasks == grp->nr_tasks && my_grp > grp)
		goto no_join;
	/* Always join threads in the same process. */
	if (tsk->mm == current->mm)
		join = true;
	/* Simple filter to avoid false positives due to PID collisions */
	if (flags & TNF_SHARED)
		join = true;

Four details in that fragment carry the design:

• Smaller joins bigger, always. The nr_tasks comparison plus the pointer-address tie-break gives a total order, which is what stops two tasks from simultaneously deciding to join each other and deadlocking or oscillating. It is the same trick as lock-ordering by address.
• Same mm is an automatic join. Threads of one process are grouped unconditionally — no statistical evidence required, because they demonstrably share an address space.
• TNF_SHARED gates the cross-process case. That flag is set back in numa_migrate_check() only when folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED) (v6.12 mm/memory.c) — genuinely shared memory, not a coincidental PID collision.
• Read-only pages are excluded from grouping entirely via TNF_NO_GROUP, set when the PTE is not writable. The comment explains: “Avoid grouping on RO pages in general. RO pages shouldn’t hurt as much anyway since they can be in shared cache state.” The historical reason is blunter — LWN records that without this, “since every process shares access to, for example, the C library, all processes in the system tended to get pulled together into a single NUMA group,” and Peter Zijlstra added the read-only exclusion to stop it.

Once tasks are grouped, the fault statistics are accumulated twice — once in p->numa_faults[] and once in ng->faults[] — and the group’s numbers take precedence in task_numa_placement(): else if (group_faults > max_faults) selects the node by group faults rather than per-task faults, then preferred_group_nid() refines it. The numa_group structure holds a flexible array split into two regions, and the comment on it names the asymmetry that makes the whole thing work:

	/*
	 * faults[] array is split into two regions: faults_mem and faults_cpu.
	 *
	 * Faults_cpu is used to decide whether memory should move
	 * towards the CPU. As a consequence, these stats are weighted
	 * more by CPU use than by memory faults.
	 */
	unsigned long faults[];

That is the crux: memory faults answer “where is the memory?”, CPU faults answer “where is the work?”, and the weighting in task_numa_placement() normalises CPU faults by the task’s runtime share — f_weight = div64_u64(runtime << 16, period + 1) — so that, as the comment says, “tasks with little runtime have little over-all impact on throughput, and thus their faults are less important.” A mostly-idle thread in a group does not get to vote as loudly as the one doing the work.

flowchart TB
  F["A NUMA hinting fault fires<br/>task P on node X, page on node Y"]
  F --> C{"cpupid says the last<br/>faulter was... ?"}
  C -->|"same PID (private)"| PRIV["PRIVATE fault<br/>TNF_FAULT_LOCAL if X == Y"]
  C -->|"a different PID (shared)"| SH["SHARED fault"]
  C -->|"unset — first ever fault"| FIRST["First fault<br/>record cpupid, usually migrate<br/>if early in task lifetime"]

  PRIV --> MOVEPAGE["MOVE THE PAGE<br/>migrate_misplaced_folio to X"]

  SH --> G{"Groupable?"}
  G -->|"TNF_NO_GROUP (read-only page)"| NOG["No grouping.<br/>Shared libraries would otherwise<br/>merge every process into one group"]
  G -->|"same mm — threads of one process"| JOIN["JOIN the group unconditionally"]
  G -->|"TNF_SHARED — genuinely shared VMA"| JOIN
  JOIN --> ORDER["Smaller group joins bigger<br/>tie-break on struct address<br/>— gives a total order, no oscillation"]
  ORDER --> GRP["numa_group: faults_mem[] and faults_cpu[]<br/>accumulated across ALL member tasks"]

  GRP --> DEC{"Group's memory and CPU<br/>agree on a node?"}
  DEC -->|yes| MOVETASK["MOVE THE TASKS<br/>sched_setnuma sets numa_preferred_nid<br/>task_numa_migrate pulls them there —<br/>the whole group converges on one node"]
  DEC -->|no| RATIO["Weigh faults_cpu / faults_mem per node<br/>with 4-to-3 hysteresis, then decide"]
  RATIO --> MOVEPAGE
  RATIO --> MOVETASK

  MOVEPAGE --> CONV["Convergence from both ends"]
  MOVETASK --> CONV
  NOG --> CONV
  FIRST --> CONV

Task-versus-page placement, and where grouping enters. What it shows: the cpupid on the page routes the fault down one of three paths. A private fault is simple — move the page to the task. A shared fault is the hard case, and grouping is what converts it from an unanswerable question (“which of these two tasks owns this page?”) into an answerable one (“where should this whole set of tasks live?”). The insight to take: the balancer has two levers of very different cost, and it prefers the cheap one. Moving a page is a 4 KiB copy plus a TLB shootdown. Moving a task is just a run-queue change — but it drags an entire working set’s worth of future locality with it, so for a threaded application the high-value move is almost always MOVE THE TASKS. Grouping exists precisely so the scheduler has a coherent entity to move. Without it, per-task placement on a shared working set produces exactly the ping-pong the two-stage filter is trying to prevent, one layer down.

Configuration

The master switch: numa_balancing

/proc/sys/kernel/numa_balancing takes the bitwise-OR of these modes (v6.12 kernel.rst):

0   NUMA_BALANCING_DISABLED
1   NUMA_BALANCING_NORMAL
2   NUMA_BALANCING_MEMORY_TIERING

• NUMA_BALANCING_NORMAL (1) “optimize[s] page placement among different NUMA nodes to reduce remote accessing” — the classic behavior described above. This is the default value when balancing is enabled (set_numabalancing_state(true) sets the mode to NUMA_BALANCING_NORMAL, v6.12 kernel/sched/core.c).
• NUMA_BALANCING_MEMORY_TIERING (2) “optimize[s] page placement among different types of memory (represented as different NUMA nodes) to place the hot pages in the fast memory” — the tiered-memory mode for systems with slow/far memory (CXL, persistent memory). Here the migration decision in should_numa_migrate_memory switches from private/shared to hot/cold: it uses the hint-fault latency (numa_hint_fault_latency) against a per-node adaptive threshold (pgdat->nbp_threshold) and a promotion rate limit (v6.12 fair.c).

The two bits are independent and OR-able, giving four legal values:

Value	Modes active	What the migration decision uses	Typical machine
0	Disabled	Nothing. sched_numa_balancing static key off, task_numa_fault() returns on a single predicted branch	Anything hand-placed with numactl; also the automatic reading on a single-node box
1	NORMAL	private/shared cpupid + two-stage node-agreement filter	Multi-socket server, general purpose. The default when balancing is enabled
2	MEMORY_TIERING	hot/cold hint-fault latency vs. adaptive nbp_threshold	Tiered memory only; slow-tier pages promoted, but no cross-socket locality work
3	Both	Whichever applies per page — folio_use_access_time() selects	Multi-socket and tiered: the common shape of a CXL-equipped server

The mode is stored in sysctl_numa_balancing_mode and read by should_numa_migrate_memory() and by the scanner’s top-tier check. Note that value 2 alone leaves the classic socket-locality behaviour off, which is rarely what you want on a machine that has both multiple sockets and a slow tier — 3 is the setting for that case.

Writing the sysctl requires CAP_SYS_ADMIN; toggling the static key is what __set_numabalancing_state does (core.c). Examples:

# Inspect current mode
$ cat /proc/sys/kernel/numa_balancing
1
 
# Disable entirely (e.g. for a workload already pinned with numactl --membind)
$ echo 0 | sudo tee /proc/sys/kernel/numa_balancing
 
# Enable normal balancing
$ sudo sysctl kernel.numa_balancing=1
 
# Enable both normal balancing and memory tiering (1 | 2)
$ sudo sysctl kernel.numa_balancing=3

Tiering rate limit

For tiering mode there is a throughput governor, numa_balancing_promote_rate_limit_MBps: “The per-node max promotion throughput in MB/s will be limited to be no more than the set value… A rule of thumb is to set this to less than 1/10 of the PMEM node write bandwidth” (v6.12 kernel.rst). Uncontrolled promotion/demotion churn between memory types can hurt application latency more than the locality wins help, so this bound matters on real tiered systems.

Scan-rate tunables — they are not sysctls any more

This is the single most commonly stale fact about the subsystem, and it is worth stating flatly: numa_balancing_scan_delay_ms, numa_balancing_scan_period_min_ms, numa_balancing_scan_period_max_ms and numa_balancing_scan_size_mb have not been /proc/sys/kernel/ entries since Linux 5.13. Tuning guides that tell you to sysctl -w kernel.numa_balancing_scan_size_mb=... are quoting a kernel from 2021 or earlier, and the write will simply fail.

The move is verifiable by reading kernel/sysctl.c across tags. All four .procname entries are present at v5.10, v5.11 and v5.12 and gone at v5.13 and everything after. The commit is 8a99b6833c88, “sched: Move SCHED_DEBUG sysctl to debugfs”, by Peter Zijlstra (2021-03-24), whose changelog states the intent in one sentence: “Stop polluting sysctl with undocumented knobs that really are debug only, move them all to /debug/sched/ along with the existing /debug/sched_ files that already exist.”* The same patch deletes the four ctl_table entries from kernel/sysctl.c and adds them to kernel/sched/debug.c as debugfs files. At v6.12 that registration reads:

	numa = debugfs_create_dir("numa_balancing", debugfs_sched);
	debugfs_create_u32("scan_delay_ms",      0644, numa, &sysctl_numa_balancing_scan_delay);
	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
	debugfs_create_u32("scan_size_mb",       0644, numa, &sysctl_numa_balancing_scan_size);
	debugfs_create_u32("hot_threshold_ms",   0644, numa, &sysctl_numa_balancing_hot_threshold);

— v6.12 kernel/sched/debug.c

So the live paths are:

# All five knobs, root-only, gated on debugfs being mounted:
/sys/kernel/debug/sched/numa_balancing/scan_delay_ms       # 1000
/sys/kernel/debug/sched/numa_balancing/scan_period_min_ms  # 1000
/sys/kernel/debug/sched/numa_balancing/scan_period_max_ms  # 60000
/sys/kernel/debug/sched/numa_balancing/scan_size_mb        # 256
/sys/kernel/debug/sched/numa_balancing/hot_threshold_ms    # 1000 (tiering; see below)

Two consequences follow from where they now live, and both matter more than the path change itself. First, debugfs is 0700 on its mount point and is not mounted at all in many hardened or containerised environments, so these knobs may be genuinely unavailable rather than merely relocated. Second, and more importantly, the relocation is a statement of support policy: sysctl knobs are a stable user-space interface, debugfs knobs explicitly are not. Peter Zijlstra’s changelog calls them “debug only.” Building a production tuning story on them is building on something upstream has said it may change. Verified on the reference machine (Fedora kernel 7.1.8): ls /proc/sys/kernel/ | grep numa returns exactly two entries, numa_balancing and numa_balancing_promote_rate_limit_MBps, and nothing else.

The defaults — a 1 s to 60 s adaptive period, 256 MB per scan, 1 s initial delay — are backstopped by the self-throttling 3% overhead cap and the five-branch control law described earlier, and in practice they suit almost every workload. The realistic tuning decision is not “which scan period” but the binary one: leave numa_balancing on, or turn it off.

Memory Tiering — Why This Subsystem Matters Again

For most of its life, automatic NUMA balancing answered one question: which socket should this page live on? Since Linux 5.18 it answers a second and now more consequential one: which kind of memory should this page live in? That is NUMA_BALANCING_MEMORY_TIERING, and it is the reason the subsystem is being actively developed in the CXL era rather than quietly maintained.

The framing is a reuse of existing machinery. Persistent memory (Intel Optane DC PMEM, historically) and now Compute Express Link (CXL) attached memory are presented to Linux as NUMA nodes with no CPUs — nodes that have N_MEMORY but not N_CPU. That is not a hack; it is how Documentation/admin-guide/mm/numaperf.rst describes the model, and it means every mechanism that already knew how to move a page between nodes automatically knows how to move a page between tiers. What tiering mode changes is only the decision rule, not the plumbing.

The rule change is stated in one line of should_numa_migrate_memory():

The pages in slow memory node should be migrated according to hot/cold instead of private/shared. — v6.12 kernel/sched/fair.c

Everything about the private/shared, two-stage, cpupid-based filter is replaced for slow-tier pages, because the question is different. On a two-socket machine, “should this page move?” means “is it on the wrong socket?” — a locality question, answered by comparing node IDs. On a tiered machine it means “is this page hot enough to deserve fast memory?” — a frequency question, which node IDs cannot answer.

So the kernel repurposes the field. In tiering mode, a slow-memory folio’s cpupid no longer stores a CPU and PID at all; it stores an access timestamp. numa_migrate_check() says so directly — “For memory tiering mode, cpupid of slow memory page is used to record page access time. So use default value” — and hands back a dummy last_cpupid. The hotness measure then falls out of subtracting two timestamps:

static int numa_hint_fault_latency(struct folio *folio)
{
	int last_time, time;
	time = jiffies_to_msecs(jiffies);
	last_time = folio_xchg_access_time(folio, time);
	return (time - last_time) & PAGE_ACCESS_TIME_MASK;
}

Hint-fault latency is the interval between two consecutive hinting faults on the same page. A short interval means the page is being touched often — hot. A long interval means cold. This is an elegant inversion: the scanner’s sampling period, which is a cost in the classic mode, becomes the measuring stick in tiering mode.

Three gates then decide a promotion, all in should_numa_migrate_memory():

1. Free-space shortcut. pgdat_free_space_enough() checks whether the destination (fast) node has headroom above its promotion watermark plus a slack of max(1 GB, node_present_pages / 16). If it does, the function returns true immediately and resets pgdat->nbp_threshold to 0 with the comment “workload changed, reset hot threshold.” When fast memory is plentiful, do not be clever — just promote.
2. Adaptive hotness threshold. Otherwise the latency is compared against pgdat->nbp_threshold, seeded from sysctl_numa_balancing_hot_threshold (default MSEC_PER_SEC, i.e. 1000 ms, exposed at /sys/kernel/debug/sched/numa_balancing/hot_threshold_ms). numa_promotion_adjust_threshold() re-tunes it once per scan_period_max (60 s) using a hysteresis band: if the observed promotion-candidate rate exceeds 110% of the reference rate the threshold is tightened (fewer pages qualify as hot); below 90% it is loosened, up to a ceiling of 2 × ref_th, in steps of ref_th × 2 / NUMA_MIGRATION_ADJUST_STEPS where NUMA_MIGRATION_ADJUST_STEPS is 16. This is a closed-loop controller whose setpoint is the promotion rate, not the hotness itself.
3. Rate limit. numa_promotion_rate_limit() counts candidates into the PGPROMOTE_CANDIDATE node counter and refuses if more than numa_balancing_promote_rate_limit_MBps worth have been proposed in the last second. The documentation explains the units and gives a sizing rule: “The per-node max promotion throughput in MB/s will be limited to be no more than the set value… A rule of thumb is to set this to less than 1/10 of the PMEM node write bandwidth” (v6.12 kernel.rst). On the reference machine the default reads back as 65536 MB/s — effectively unlimited, which is the right default for a machine with no slow tier.

flowchart TB
  subgraph CLASSIC["numa_balancing = 1 — NUMA_BALANCING_NORMAL"]
    direction TB
    Q1["Question: is this page on the WRONG SOCKET?"]
    E1["Evidence: folio_nid vs numa_node_id,<br/>plus the page's last cpupid"]
    R1["Rule: private vs shared,<br/>two-stage node-agreement filter"]
    A1["Action: migrate page toward the faulting CPU<br/>AND migrate the task toward its memory"]
    Q1 --> E1 --> R1 --> A1
  end
  subgraph TIER["numa_balancing = 2 — NUMA_BALANCING_MEMORY_TIERING"]
    direction TB
    Q2["Question: is this page HOT ENOUGH<br/>to deserve fast memory?"]
    E2["Evidence: hint-fault LATENCY.<br/>cpupid field repurposed to hold a<br/>timestamp instead of CPU+PID"]
    R2["Rule: latency &lt; nbp_threshold,<br/>threshold retuned every 60 s by a<br/>closed loop on promotion RATE"]
    A2["Action: PROMOTE the page to the fast node.<br/>Tasks are not moved — the slow node has no CPUs"]
    Q2 --> E2 --> R2 --> A2
  end
  SHARED["Shared plumbing:<br/>task_numa_work PTE scanner,<br/>do_numa_page hinting fault,<br/>migrate_misplaced_folio"]
  SHARED --> CLASSIC
  SHARED --> TIER
  A1 --> V1["Counters: numa_hint_faults,<br/>numa_hint_faults_local,<br/>numa_pages_migrated"]
  A2 --> V2["Counters: pgpromote_candidate,<br/>pgpromote_success, pgpromote_candidate_nrl,<br/>and pgdemote_* on the way back down"]
  A2 -.->|"the other direction"| DEMO["Demotion is NOT done here.<br/>Cold pages fall to the slow tier via<br/>page reclaim — see Memory Reclaim Overview"]

The same machinery answering two different questions. What it shows: classic balancing and tiering share the scanner, the fault handler and the migration primitive, and diverge only at the decision rule — including a genuine field overload, where a slow-tier page’s cpupid stores a timestamp rather than a CPU and PID. The insight to take: the two modes are OR-able (numa_balancing=3) precisely because they are disjoint decisions on disjoint page populations — the tiering rule applies only where folio_use_access_time() is true. Note also the dotted arrow: NUMA balancing only ever moves pages up the hierarchy. Demotion of cold pages to the slow tier is reclaim’s job, not this subsystem’s, which is why the pgdemote_kswapd / pgdemote_direct counters are grouped with reclaim statistics and not with the numa_* ones. A complete tiering story is promotion here plus demotion in Memory Reclaim Overview, and the two are tuned independently.

The practical significance is worth being explicit about. Automatic NUMA balancing has always been optional on a two-socket server — a nice optimisation that a well-tuned database would disable. On a machine with a CXL memory tier there is no static placement that is correct, because hotness changes over time and static tools cannot follow it. numactl --membind can tell you which tier a workload’s memory starts in; nothing but a feedback loop can keep the hot 10% of a 1 TB working set resident in the fast 128 GB as the working set shifts. That is the argument for why this subsystem, twelve years old and long considered a mixed blessing, is being invested in again.

Uncertain

Verify: the quantitative benefit of NUMA_BALANCING_MEMORY_TIERING on real CXL hardware, and whether the 1000 ms default hot_threshold_ms is appropriate for CXL latencies (roughly 2–3× local DRAM) as opposed to the Optane PMEM latencies (closer to 3–5×) the threshold was originally tuned against. Reason: the mechanism is fully sourced from the v6.12 tree, but no measurement was consulted, and the reference machine for these notes has a single NUMA node with no slow tier, so nothing could be measured locally. To resolve: cite a published CXL tiering evaluation naming its hardware, or measure pgpromote_success versus application latency on a real tiered system while sweeping hot_threshold_ms. uncertain

Measuring It — the Counters That Actually Move

Automatic NUMA balancing is unusually easy to observe, because every stage of the loop increments a named counter. The CONFIG_NUMA_BALANCING-gated events are declared in include/linux/vm_event_item.h and named for /proc/vmstat in mm/vmstat.c:

/proc/vmstat counter	Incremented by	Reading it
numa_pte_updates	change_prot_numa() — the scanner, per PTE armed	The cost side. Rising steadily means the scanner is working
numa_huge_pte_updates	Same, for PMD-level (transparent huge page) entries	A THP arms one PMD covering 2 MiB — see Transparent Huge Pages
numa_hint_faults	numa_migrate_check(), once per hinting fault taken	The sampling side. The ratio to numa_pte_updates tells you what fraction of armed pages were actually touched
numa_hint_faults_local	Same, when folio_nid(folio) == numa_node_id()	The health metric. local / total approaching 1.0 means placement has converged
numa_pages_migrated	On a successful migrate_misplaced_folio()	The action side. Should spike after a workload change and then decay
pgpromote_candidate / pgpromote_success	Tiering mode: proposed vs. actually promoted	A large gap means the promotion rate limit is binding
pgpromote_candidate_nrl	Candidates rejected by the rate limiter specifically	Direct evidence to raise numa_balancing_promote_rate_limit_MBps
pgdemote_kswapd / pgdemote_direct / pgdemote_khugepaged	Reclaim demoting pages to a slower tier	The other half of tiering; belongs to reclaim, not to this subsystem

The per-node allocation counters — numa_hit, numa_miss, numa_foreign, numa_interleave, numa_local, numa_other — are a different family: they describe the page allocator’s behaviour at allocation time, not the balancer’s. numastat prints exactly these six per node, which is why numastat alone cannot tell you whether balancing is working; it tells you whether allocation was local.

Read on the reference machine — 32 CPUs, AMD Ryzen AI MAX+ 395, Fedora kernel 7.1.8, weeks of uptime:

$ lscpu | grep -i numa
NUMA node(s):        1
NUMA node0 CPU(s):   0-31
 
$ cat /proc/sys/kernel/numa_balancing
0
 
$ grep -E '^(numa_|pgpromote|pgdemote)' /proc/vmstat
numa_hit 12355049614
numa_miss 0
numa_foreign 0
numa_interleave 3787
numa_local 12355049614
numa_other 0
pgpromote_success 0
pgpromote_candidate 0
pgpromote_candidate_nrl 0
pgdemote_kswapd 0
pgdemote_direct 0
pgdemote_khugepaged 0
numa_pte_updates 0
numa_huge_pte_updates 0
numa_hint_faults 0
numa_hint_faults_local 0
numa_pages_migrated 0

That output is a complete worked example of the feature’s off state, and every line of it is informative. CONFIG_NUMA=y, CONFIG_NUMA_BALANCING=y and CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y are all set in this kernel’s /boot/config-* — and yet numa_balancing reads 0. The reason is in the Kconfig help text quoted at the top of this note: the default-enabled option turns balancing on “if running on a NUMA machine,” and this box has exactly one node (/sys/devices/system/node/online is 0, and node0/distance is the single value 10). One node means every access is local by construction, so set_numabalancing_state() never enables the static key and all five numa_* balancing counters are permanently zero. Correspondingly numa_miss, numa_foreign and numa_other are zero while numa_local equals numa_hit at 12.36 billion — 100% local allocation, trivially, because there is nowhere else to allocate.

This is the diagnostic worth internalising: a numa_balancing of 0 on a NUMA-capable kernel usually means “one node,” not “someone turned it off.” Check numactl --hardware (or lscpu | grep -i NUMA, which needs no extra package) before concluding anything. The interesting counters on a real multi-node box are the two ratios:

• numa_hint_faults / numa_pte_updates — the sampling yield. Low means the scanner is arming pages the task never touches, i.e. paying cost for no signal; the VMA-skipping heuristics (NUMAB_SKIP_PID_INACTIVE) exist to raise it.
• numa_hint_faults_local / numa_hint_faults — the convergence metric, and the one number to graph. A workload that is placed well sits near 1.0. A workload that never converges — the number oscillating, with numa_pages_migrated climbing continuously — is thrashing and is a candidate for numa_balancing=0 plus manual binding.

Per-process detail comes from /proc/<pid>/numa_maps, which prints one line per VMA with the per-node page counts (N0=, N1=, …) plus anon=, dirty= and mapped=. It is the tool for answering “did the pages actually move?” as opposed to “did the kernel try?”.

Failure Modes and When It Hurts

Already-pinned workloads pay pure overhead. The documentation is explicit: “The unmapping of pages and trapping faults incur additional overhead… If the target workload is already bound to NUMA nodes then this feature should be disabled” (v6.12 kernel.rst). If you have already placed memory and tasks correctly with numactl --membind/--cpunodebind, balancing can only churn PTEs and take faults for no benefit — turn it off (numa_balancing=0).

Migration thrash on shared pages. Pages shared by tasks on different nodes have no single right home; naive migration would ping-pong them. The two-stage filter and task grouping exist precisely to damp this, and the scanner skips read-only file/library mappings, but pathological sharing patterns can still cause migration churn visible as elevated numa_pages_migrated and TNF_MIGRATE_FAIL counts.

Latency spikes from the fault and migration cost. Each hinting fault is a real minor fault (page-table lock, fault handling) and a migration is a copy plus a TLB shootdown (The Translation Lookaside Buffer and TLB Shootdowns). For latency-sensitive tasks the sampling itself can be the problem; the 3% cap bounds the average but not individual spikes.

Big-page interactions. The scanner skips hugetlb VMAs outright. Transparent huge pages (Transparent Huge Pages) participate but a hinting fault on a THP can force a page to be handled at huge granularity, and migration of a 2 MiB folio is a larger unit of work — another reason the filter is conservative.

The PREEMPT_RT incompatibility. CONFIG_NUMA_BALANCING depends on SMP && NUMA && MIGRATION && !PREEMPT_RT (v6.12 init/Kconfig). On a real-time kernel the feature is not merely discouraged, it cannot be built. The reason is the failure mode above taken to its conclusion: a hinting fault is unbounded latency injected into an ordinary memory access, which is exactly what PREEMPT_RT exists to eliminate. This is worth knowing because it also means a machine tuned for CPU isolation on an RT kernel has no automatic NUMA balancing at all, and must place memory by hand.

Interleaved workloads get actively fought. This is the sharpest failure mode and it is not in the documentation. MPOL_INTERLEAVE deliberately spreads a mapping across nodes to maximise aggregate bandwidth — every access is remote by design. Automatic balancing reads that same distribution as misplacement and tries to correct it. mpol_misplaced() does consult the policy, so an explicit MPOL_INTERLEAVE on the VMA is respected; the trap is a workload that achieves interleaving incidentally (for example by allocating from many threads spread across nodes) without ever setting a policy. There, balancing will migrate pages toward whichever thread happens to fault first, quietly destroying the bandwidth spread. Symptom: numa_pages_migrated climbing steadily with no improvement in numa_hint_faults_local.

Diagnosis. Per-task and per-node NUMA stats surface the behavior: /proc/<pid>/numa_maps shows per-VMA node placement, numastat aggregates numa_hit/numa_miss/numa_foreign/numa_other per node, and the numa_pte_updates, numa_hint_faults, numa_hint_faults_local, and numa_pages_migrated counters in /proc/vmstat directly reflect the scanner and migration activity. The counter table in the previous section says what each one means; the decision tree below says what to do about each pattern.

flowchart TB
  S["Symptom: NUMA balancing is<br/>suspected of hurting"]
  S --> A{"numa_pte_updates<br/>rising?"}
  A -->|no| A1["The scanner is not running.<br/>Check: numa_balancing != 0,<br/>more than one online node,<br/>not a PREEMPT_RT kernel<br/>(CONFIG_NUMA_BALANCING depends on !PREEMPT_RT)"]
  A -->|yes| B{"numa_hint_faults<br/>rising too?"}
  B -->|"no — updates without faults"| B1["Low sampling yield.<br/>The scanner is arming pages the task<br/>never touches. Pure cost, no signal.<br/>Common with huge sparse mappings"]
  B -->|yes| C{"numa_hint_faults_local<br/>/ numa_hint_faults"}
  C -->|"climbing toward 1.0"| C1["WORKING AS INTENDED.<br/>Placement is converging.<br/>Leave it alone"]
  C -->|"flat and low"| D{"numa_pages_migrated<br/>climbing continuously?"}
  D -->|yes| D1["THRASHING.<br/>Pages have no single right home —<br/>heavy cross-node sharing, or an<br/>interleaved workload being 'corrected'.<br/>Fix: MPOL_INTERLEAVE explicitly,<br/>or numa_balancing = 0 + numactl"]
  D -->|"no, migrations rare"| D2["Migration is being REFUSED.<br/>Check TNF_MIGRATE_FAIL: destination<br/>node full, folio not isolatable,<br/>or hugetlb VMAs the scanner skips"]
  C -->|"oscillating"| D1
  A1 --> Z["Decide: fix the cause,<br/>or set numa_balancing = 0<br/>and place by hand"]
  B1 --> Z
  D1 --> Z
  D2 --> Z

Turning four /proc/vmstat counters into a diagnosis. What it shows: the three counters form a pipeline — pages armed, faults taken, pages moved — so comparing consecutive stages localises the problem to one stage rather than blaming the feature as a whole. The insight to take: “NUMA balancing is slowing us down” is four different problems with four different fixes, and they are distinguishable in about ten seconds of reading counters. The one that most often gets misdiagnosed is the numa_pte_updates-rising-but-numa_hint_faults-flat case: that is the scanner paying full cost for zero information, and no amount of tuning the migration policy touches it — the fix is either narrower mappings or turning the feature off. Note also that numa_hint_faults_local / numa_hint_faults climbing is the only branch that means “working,” which is why it is the one number to put on a dashboard.

Alternatives and When to Choose Them

Automatic balancing is the zero-configuration option — good for general-purpose servers and workloads whose placement you can’t predict. When you can predict placement, the static tools are better because they avoid the sampling overhead entirely:

• Explicit binding — numactl --cpunodebind/--membind (or cgroup cpusets) nails a workload to nodes; combine with numa_balancing=0. Best for dedicated, well-understood workloads (databases, HPC).
• Memory policies — set_mempolicy/mbind with MPOL_BIND/MPOL_PREFERRED/MPOL_INTERLEAVE give per-allocation control without fully pinning CPUs; see NUMA Memory Policies. Interleaving deliberately spreads memory to maximize aggregate bandwidth — the opposite goal from balancing’s locality.
• Doing nothing — on a UMA box balancing is a no-op anyway; on a NUMA box with a workload that fits in one node’s memory and rarely migrates, default local allocation may already be enough.

The full policy vocabulary, from Documentation/admin-guide/mm/numa_memory_policy.rst, v6.12, is wider than the three modes usually cited, and picking the right one matters more than tuning the balancer:

Mode	Behaviour	Use it when
MPOL_DEFAULT	No policy — fall back to the next most specific policy in the hierarchy (VMA → task → system default local allocation)	Removing a policy you set earlier
MPOL_BIND	Allocate only from the given node set; fail (or reclaim hard) rather than go elsewhere	Hard partitioning, where a remote allocation is worse than pressure
MPOL_PREFERRED	Prefer one node, fall back freely	Soft affinity with no failure risk
MPOL_PREFERRED_MANY	As MPOL_PREFERRED but with a node mask rather than a single node	A workload that should prefer a socket pair, or the fast tier as a set
MPOL_INTERLEAVE	Round-robin allocations across the node set	Bandwidth-bound workloads. Deliberately the opposite goal to balancing — maximise aggregate bandwidth, accept remote latency
MPOL_WEIGHTED_INTERLEAVE	Interleave with per-node weights	Tiered or heterogeneous-bandwidth systems, where equal round-robin would over-fill the slow tier

Two flags modify how a policy survives a cpuset change: MPOL_F_STATIC_NODES keeps the node mask literal, and MPOL_F_RELATIVE_NODES re-maps it relative to the cpuset’s allowed nodes. They are mutually exclusive. There is also set_mempolicy_home_node(), which sets a “home node” hint on an existing VMA policy — useful for telling the kernel which node a mapping belongs to without binding allocations to it.

The system-call layer, per the same document, is: set_mempolicy() for the whole task, mbind() for an address range (and, since 2.6.16, mbind() can also migrate the existing pages in that range, not just set policy for future allocations), plus move_pages() and migrate_pages() for explicit relocation of already-allocated memory. numactl is a thin, convenient wrapper over these — nothing it does is unavailable to a program that wants to place its own memory precisely.

flowchart TB
  START["I have a NUMA or tiered machine.<br/>What should place my memory?"]
  START --> N{"How many NUMA nodes<br/>does the box have?"}
  N -->|"one"| ONE["Nothing to do.<br/>numa_balancing is a no-op and reads 0;<br/>the static key is never enabled"]
  N -->|"two or more"| T{"Is one of them a<br/>CPU-less slow tier?<br/>(CXL, PMEM)"}

  T -->|yes| TIER["numa_balancing = 2 or 3<br/>MEMORY_TIERING.<br/>No static tool can follow hotness<br/>as the working set shifts"]
  T -->|no| K{"Do you KNOW the right<br/>placement in advance?"}

  K -->|"no — general purpose,<br/>unpredictable, multi-tenant"| AUTO["numa_balancing = 1.<br/>The zero-configuration default.<br/>Bounded at ~3% of task CPU time"]
  K -->|"yes — dedicated workload,<br/>you own the machine"| BW{"Latency-bound<br/>or bandwidth-bound?"}

  BW -->|"latency-bound<br/>(OLTP, in-memory KV)"| BIND["numactl --cpunodebind --membind,<br/>or cpuset cpus + mems.<br/>THEN set numa_balancing = 0 —<br/>otherwise you pay scan cost for nothing"]
  BW -->|"bandwidth-bound<br/>(HPC streaming, analytics scans)"| INTER["MPOL_INTERLEAVE or<br/>MPOL_WEIGHTED_INTERLEAVE.<br/>Spread deliberately.<br/>Balancing would UNDO this — disable it"]

  K -->|"partly — per-region"| MB["mbind() per address range.<br/>Hot index MPOL_BIND local,<br/>cold heap interleaved"]

  BIND --> CHECK["Verify: numa_hint_faults should stop rising"]
  INTER --> CHECK
  MB --> CHECK
  AUTO --> CHECK2["Verify: numa_hint_faults_local / numa_hint_faults<br/>should climb toward 1.0 and stay"]
  TIER --> CHECK3["Verify: pgpromote_success rising,<br/>pgpromote_candidate_nrl near zero"]

Choosing a placement mechanism. What it shows: four terminal answers, selected by two questions — do you know the placement in advance, and are you optimising for latency or bandwidth — plus a distinct branch for tiered memory. The insight to take: the two right-hand branches both end with “and then turn balancing off,” and the reason is different in each case. With --membind the balancer has nothing to fix, so its scanning is pure cost. With MPOL_INTERLEAVE it is actively counterproductive: interleaving deliberately places memory remotely to gain aggregate bandwidth, and balancing will read exactly that as misplacement and try to undo it. Those two mechanisms have opposite objective functions, and running them together means paying for a fight. Note also that every terminal state names a counter to check — a placement decision you have not verified against /proc/vmstat is a hypothesis.

The honest summary: automatic balancing trades a small, bounded CPU overhead for adaptive locality you don’t have to think about. It pays off when placement is dynamic or unknown, it is dead weight when you’ve already done the placement by hand, and it is a genuine conflict when your hand-placement was MPOL_INTERLEAVE. On a tiered machine the calculus flips entirely — there, adaptivity is the only thing that works, because the property being tracked (hotness) is one no static tool can express.

See Also

• NUMA Memory Model — the topology, pg_data_t, zonelists, and default local allocation that this feature dynamically corrects.
• Page Migration — migrate_misplaced_folio and the underlying machinery that moves a page’s contents between nodes.
• NUMA Memory Policies — mbind/set_mempolicy; mpol_misplaced consults the policy when deciding the migration target.
• The Page Fault Handler — handle_mm_fault/do_numa_page is the path that intercepts hinting faults.
• Minor and Major Faults — a NUMA hinting fault is a minor fault (page is in RAM; only the PTE was poisoned).
• The Translation Lookaside Buffer and TLB Shootdowns — migration and the PROT_NONE rewrite both cost TLB invalidations.
• Transparent Huge Pages — THP folios participate in hinting faults and migration at huge granularity.
• Memory Reclaim Overview — the other half of memory tiering: NUMA balancing only promotes pages up to fast memory; cold pages fall back down through reclaim’s demotion path (pgdemote_kswapd, pgdemote_direct).
• The EEVDF Scheduler — the fair scheduler whose tick drives task_tick_numa() and whose run-queue placement task_numa_migrate() overrides; sched_setnuma() is this subsystem writing into the scheduler.
• Scheduler Load Balancing — the periodic balancer that NUMA placement must cooperate with; pulling a task toward its memory can fight pulling it toward an idle CPU.
• MOC: Linux Memory Management MOC (§13, NUMA and Memory Placement).