NUMA Memory Policies

On a Non-Uniform Memory Access (NUMA) machine, which node a page is allocated on is a performance decision — local memory is faster than remote — and NUMA memory policy is the mechanism by which userspace (and the kernel’s defaults) make that decision. A policy is a triple: a mode (MPOL_DEFAULT, MPOL_BIND, MPOL_INTERLEAVE, MPOL_WEIGHTED_INTERLEAVE, MPOL_PREFERRED, MPOL_PREFERRED_MANY, or MPOL_LOCAL), an optional set of nodes the mode acts on, and optional mode flags that modify how the node set is interpreted under cpuset changes (per the kernel numa_memory_policy.rst). Crucially, a policy is attached at one of several scopes — a hard-coded system default, a per-task default set with set_mempolicy(2), or a per-range (per-VMA) policy set with mbind(2) — and the kernel resolves the effective policy for any allocation by falling back from most-specific to least-specific scope. This note covers the system-call interface (set_mempolicy, mbind, get_mempolicy), each mode, the scope hierarchy, and the numactl(8) command-line front end, as documented for the Linux 6.12 LTS kernel (released 2024-11-17) and man-pages 6.18 (the current man-pages release at the time of writing). Memory policy is only present when the kernel is built with CONFIG_NUMA (per numa(7)). One caution up front, established below: the authority on the set of modes is include/uapi/linux/mempolicy.h, not the kernel’s prose documentation — numa_memory_policy.rst still announces “the following 4 behavioral modes” while the header’s enum has seven, and the .rst omits MPOL_LOCAL entirely.

Mental Model — Policy Resolution by Scope

The single most important idea is that there is no one “the policy” for a process. The kernel keeps policies at three (conceptually four) nested scopes, and resolves the effective policy for a specific allocation by starting at the most specific scope that covers it and falling back outward when that scope is “default.”

flowchart TD
  ALLOC["A page must be allocated<br/>(fault on a virtual address)"]
  VMA{"Does the VMA covering<br/>this address have an<br/>explicit policy?<br/>(set by mbind)"}
  TASK{"Does the task have an<br/>explicit policy?<br/>(set by set_mempolicy)"}
  SYS["System Default Policy<br/>(hard-coded: local allocation<br/>once the system is up)"]
  ALLOC --> VMA
  VMA -->|"yes"| USEVMA["Use VMA policy"]
  VMA -->|"no / MPOL_DEFAULT"| TASK
  TASK -->|"yes"| USETASK["Use task policy"]
  TASK -->|"no / MPOL_DEFAULT"| SYS

How the kernel picks the policy that governs one allocation. What it shows: MPOL_DEFAULT at any scope does not mean “local allocation” — it means “fall through to the next, more general scope.” Only the system default, at the bottom, is concretely “local allocation” (allocate on the node of the CPU that faulted). The insight to take: to reason about where a page lands, you must know which scope installed the governing policy. A per-VMA MPOL_BIND overrides a per-task MPOL_INTERLEAVE, which overrides the system local-allocation default — and get_mempolicy(2) with MPOL_F_ADDR is how you ask the kernel which one actually applies to a given address.

The Scopes in Detail

The kernel’s numa_memory_policy.rst describes the scopes “from most general to most specific.”

System Default Policy is “hard coded into the kernel.” Once the system is up and running it is local allocation — allocate on the node of the CPU that triggered the fault. (During early boot it is temporarily interleave-across-all-nodes, “so as not to overload the initial boot node with boot-time allocations.”) This is the policy every allocation falls back to when nothing more specific applies.

Task/Process Policy is the optional per-task default set with set_mempolicy(2). It “applies to the entire address space of a task” and governs every allocation not covered by a more specific scope. It is inherited across fork()/clone() (without CLONE_VM) and preserved across execve() — which lets a wrapper (like numactl) set a policy and then exec a NUMA-unaware program that inherits it. In a multithreaded process, a task policy applies only to the thread that installs it and to threads it later creates; sibling threads that already existed keep their own. And it is not retroactive: “A task policy applies only to pages allocated after the policy is installed. Any pages already faulted in… remain where they were allocated.”

VMA Policy is a per-range policy set with mbind(2) on [addr, addr+len). It governs allocations backing that range only; ranges with no explicit VMA policy fall back to the task policy. VMA policy has sharp edges spelled out in the kernel doc:

• It “applies ONLY to anonymous pages” — anonymous mappings (MAP_ANONYMOUS), the stack and heap, and the copy-on-write pages of a MAP_PRIVATE file mapping (the policy fires at the COW write). A VMA policy on a MAP_SHARED file mapping is ignored; those page-cache pages follow task/system policy instead.
• VMA policies are shared between threads of a process (they live in the shared address space) and inherited across fork(), but NOT across execve() — the address space is torn down and rebuilt, so “only NUMA-aware applications may use VMA policies.”
• Installing a policy on a sub-range of an existing mapping causes the kernel to split the VMA into 2 or 3 pieces, each with its own policy (the VMA split mechanism).

Shared Policy is a fourth scope for shared memory objects (shmget()/mmap(MAP_ANONYMOUS|MAP_SHARED)): the policy attaches to the object, not to one task’s address-space range, so every task attaching to it obeys the same policy regardless of who faults the page.

The Four Scopes Side by Side

The scopes differ on far more than precedence — they have different installation calls, different inheritance behavior across fork and exec, and different answers to “which pages does this actually govern?” Getting a policy to stick requires matching the scope to the lifetime you need:

	System default	Task policy	VMA policy	Shared policy
Precedence	lowest (4th)	3rd	highest for anon (1st)	attaches to the object, not the range
Installed by	hard-coded in the kernel	set_mempolicy(2)	mbind(2)	mbind(2) on a shared object
Governs	every allocation nothing else covers	the task’s whole address space	one [addr, addr+len) range	every mapping of that object, in every process
Concrete meaning when unset	is local allocation — the floor	falls through to system default	falls through to task policy	falls through to task policy
Inherited across fork()	n/a	yes (without CLONE_VM)	yes (address space is copied)	yes (object outlives the fork)
Survives execve()	n/a	yes	no — address space is torn down	yes (object is independent)
Applies to MAP_SHARED file pages	yes	yes	no — silently ignored	n/a
Shared between threads	n/a	no — per-thread, and only threads created after installation	yes (one address space)	yes
Who can use it	everyone	NUMA-unaware programs too, via a numactl wrapper	“only NUMA-aware applications” (kernel doc)	NUMA-aware, cooperating processes

The four policy scopes compared on every axis that changes how you would use them. What it shows: the two rows that trip people up are execve() and threads. A task policy survives exec, which is the entire reason numactl prog works on a program that knows nothing about NUMA; a VMA policy does not survive exec, which is why the kernel documentation says flatly that only NUMA-aware applications may use VMA policies. Meanwhile a task policy is per-thread, not per-process, and is not retroactively applied to already-existing sibling threads. The insight to take: pick the scope by asking “what must outlive what?” — if the policy must outlive an exec, it has to be a task policy; if it must apply to one buffer and not the rest of the heap, it has to be a VMA policy; if it must apply to every process touching a segment, it has to be a shared policy.

Note the asymmetry in the “highest precedence” row: a VMA policy wins over a task policy for anonymous pages, which is the only kind of page it governs. For a MAP_SHARED file mapping the VMA policy is not merely outranked, it is not consulted at all, so the task policy is the effective one even though the VMA has a policy attached. That is the single most common reason a carefully-mbind-ed region does not behave as expected.

The scopes’ different survival rules across fork() and execve() are the part that decides whether a policy can be applied to a program you did not write, so they are worth tracing as a lifecycle:

sequenceDiagram
    autonumber
    participant N as numactl (wrapper)
    participant T as task policy<br/>(in task_struct)
    participant V as VMA policy<br/>(in mm_struct)
    participant P as target program
    N->>T: set_mempolicy(MPOL_INTERLEAVE, {0,1})
    Note over T: installed on THIS thread only
    N->>P: execve("./prog")
    Note over T: SURVIVES exec -<br/>address space is replaced,<br/>task_struct is not
    Note over V: any VMA policy would be<br/>DESTROYED here (mm torn down)
    P->>V: mbind(buf, MPOL_BIND, {1})
    Note over V: now buf is governed by BIND,<br/>rest of the heap still INTERLEAVE
    P->>P: fork()
    Note over T: inherited by child
    Note over V: inherited by child<br/>(mm is copied)
    P->>P: child calls execve()
    Note over T: still inherited
    Note over V: GONE in the child
    Note over T,V: A NUMA-unaware program can only ever<br/>be given a TASK policy from outside.

How a policy at each scope survives (or does not survive) exec and fork. What it shows: the task policy lives in task_struct and so outlives the replacement of the address space at execve(); VMA policies live in mm_struct and die with it. The insight to take: this asymmetry is the entire reason numactl prog is possible. There is no way to mbind a range of a program that has not started yet, so any policy you impose on a third-party binary from outside must be a task policy — which in turn means it applies to the whole address space, and you cannot use this route to give one buffer a different policy from the rest. That requires the program’s cooperation, which is why the kernel documentation says “only NUMA-aware applications may use VMA policies.”

Components: Modes, Node Sets, and Mode Flags

A policy is “a mode, optional mode flags, and an optional set of nodes” (kernel doc). The nodemask is a bitmask of node IDs (bit n set ⇒ node n is in the set), and maxnode bounds how many bits the kernel reads. The modes (per set_mempolicy(2) / mbind(2), with version provenance):

The Complete Mode Set — Reference Table

The authoritative list of modes is the UAPI header, include/uapi/linux/mempolicy.h, not the prose documentation. At v6.12 that header declares a single anonymous enum with seven modes plus a terminator (mempolicy.h:19-28, v6.12):

/* Policies */
enum {
	MPOL_DEFAULT,              /* 0 */
	MPOL_PREFERRED,            /* 1 */
	MPOL_BIND,                 /* 2 */
	MPOL_INTERLEAVE,           /* 3 */
	MPOL_LOCAL,                /* 4 */
	MPOL_PREFERRED_MANY,       /* 5 */
	MPOL_WEIGHTED_INTERLEAVE,  /* 6 */
	MPOL_MAX,	/* always last member of enum */
};

Because the modes are an unadorned enum with no explicit initializers, the numeric value of a mode is its position in that list — which is why the values look arbitrary relative to the conceptual grouping (MPOL_LOCAL is 4, wedged between MPOL_INTERLEAVE and MPOL_PREFERRED_MANY, purely because that is when it was added). MPOL_MAX is 7 and is the validation bound: set_mempolicy(7, ...) returns EINVAL, verified live below.

Mode	enum	nodemask required?	How nodemask is interpreted	Strict or soft	First release
MPOL_DEFAULT	0	must be empty	not used	n/a — removes the policy	2.6.7 (original API)
MPOL_PREFERRED	1	optional	one node — the first node in the mask; an empty mask means local allocation	soft (falls back by increasing distance)	2.6.7 (original API)
MPOL_BIND	2	required, non-empty	the only legal set of nodes	strict — never goes off-mask	2.6.7 (original API)
MPOL_INTERLEAVE	3	required, non-empty	the round-robin ring, walked in ascending node-ID order	soft per page (each page still obeys that node’s zonelist fallback)	2.6.7 (original API)
MPOL_LOCAL	4	must be empty	not used	soft	3.8
MPOL_PREFERRED_MANY	5	required, non-empty	the set of preferred nodes; spill to any other node under pressure across all of them	soft	5.15
MPOL_WEIGHTED_INTERLEAVE	6	required, non-empty	the ring, but each node is visited in proportion to its sysfs weight	soft per page	6.9

The seven modes as of v6.12. What it shows: the mode set is not a flat taxonomy — it is two axes crossed. One axis is strictness (only MPOL_BIND is strict; everything else falls back under pressure), the other is cardinality (one node, a set walked round-robin, or a set treated as a preference pool). The insight to take: nodemask does not mean the same thing in any two rows. For MPOL_BIND it is a whitelist, for MPOL_INTERLEAVE it is an ordered ring, for MPOL_PREFERRED only its first bit is read, and for MPOL_DEFAULT/MPOL_LOCAL passing one at all is an error. Reading nodemask as “the nodes to use” is right only for MPOL_BIND.

The same seven modes arranged by the two axes that actually distinguish them, which is a more useful way to hold them in your head than the enum order:

flowchart TD
  Q0["Which mode do I want?"]
  Q0 --> Q1{"Must allocation be<br/>REFUSED rather than<br/>placed on a wrong node?"}
  Q1 -->|"yes — strict"| BIND["<b>MPOL_BIND</b> (2)<br/>allocate only from nodemask;<br/>reclaim or OOM rather than spill"]
  Q1 -->|"no — soft fallback is fine"| Q2{"How many nodes<br/>do I care about?"}
  Q2 -->|"wherever I am running"| LOCAL["<b>MPOL_LOCAL</b> (4)<br/>node of the faulting CPU<br/>(empty nodemask)"]
  Q2 -->|"one specific node"| PREF["<b>MPOL_PREFERRED</b> (1)<br/>reads only the FIRST bit<br/>of your nodemask"]
  Q2 -->|"a set, as a preference pool"| PM["<b>MPOL_PREFERRED_MANY</b> (5)<br/>spill only when ALL are pressured<br/>(since v5.15)"]
  Q2 -->|"a set, spread for bandwidth"| Q3{"Are the nodes<br/>equally fast?"}
  Q3 -->|"yes — uniform DRAM"| IL["<b>MPOL_INTERLEAVE</b> (3)<br/>round-robin, one page each"]
  Q3 -->|"no — tiered / CXL memory"| WIL["<b>MPOL_WEIGHTED_INTERLEAVE</b> (6)<br/>round-robin in proportion<br/>to sysfs weights (since v6.9)"]
  Q0 --> Q4{"Actually, I want to<br/>REMOVE a policy"}
  Q4 --> DEF["<b>MPOL_DEFAULT</b> (0)<br/>not a placement policy —<br/>deletes this scope's policy"]

Choosing a mode by the questions that actually discriminate between them. What it shows: the first question is strictness, because only MPOL_BIND will fail an allocation; every other mode degrades to “somewhere else” under pressure. The second is cardinality and intent — one node for latency, a set for bandwidth, a weighted set for heterogeneous bandwidth. The insight to take: MPOL_DEFAULT hangs off to the side deliberately. It is not a choice among placements, it is the absence of one, and putting it in the same list as the other six is what makes people believe it means “the default placement, i.e. local.” It does not — MPOL_LOCAL is that. The pairs to keep straight are MPOL_PREFERRED vs MPOL_PREFERRED_MANY (one bit read vs the whole mask) and MPOL_DEFAULT vs MPOL_LOCAL (delete vs assert).

The two version claims in that last column were established by existence-checking the header at successive tags rather than from memory — fetching include/uapi/linux/mempolicy.h at each tag and grepping for the constant:

Tag	MPOL_PREFERRED_MANY	MPOL_WEIGHTED_INTERLEAVE	header size
v5.14	absent	absent	2,560 B
v5.15	present	absent	2,582 B
v6.8	present	absent	2,582 B
v6.9	present	present	2,609 B
v6.12	present	present	2,609 B
v6.18	present	present	2,930 B

Existence-checking the UAPI header across tags, all HTTP 200. What it shows: MPOL_PREFERRED_MANY first appears in v5.15 and MPOL_WEIGHTED_INTERLEAVE in v6.9, each bracketed by an adjacent tag where it is absent. The insight to take: this is how to date an API addition without trusting a changelog — a 200-with-the-symbol at tag N and a 200-without it at tag N-1 pins the release exactly. Note the header keeps growing after v6.12 (2,609 → 2,930 B by v6.18), so v6.12’s seven modes are not the final word.

The kernel's own prose documentation is stale — trust the header

Documentation/admin-guide/mm/numa_memory_policy.rst at v6.12 opens its mode section with the line “NUMA memory policy supports the following 4 behavioral modes” (line 173, v6.12) — and then proceeds to document six of them (MPOL_DEFAULT, MPOL_BIND, MPOL_PREFERRED, MPOL_INTERLEAVED, MPOL_PREFERRED_MANY, MPOL_WEIGHTED_INTERLEAVE). The count was never updated as modes were appended. Two further defects in the same file: it spells the interleave mode MPOL_INTERLEAVED, which is not the name of any constant (the real one has no trailing “D”), and it never documents MPOL_LOCAL at all — instead describing local allocation only obliquely, as a MPOL_PREFERRED policy with an internal MPOL_F_LOCAL flag set, which is the implementation rather than the userspace-visible mode. Use the .rst for the semantics of the modes it does cover (it is far more detailed than the man pages on interleave indexing and cpuset rebinding) but take the mode set, the names, and the numeric values from include/uapi/linux/mempolicy.h.

• MPOL_DEFAULT — Remove any non-default policy at this scope; fall back to the next more general scope (see the mental model). nodemask must be empty (NULL/zero maxnode).

• MPOL_BIND — A strict policy: allocations come only from nodes in nodemask; the kernel will not go off-mask even under pressure (which is what makes it strict — it will reclaim or fail rather than spill to an unlisted node). Since Linux 2.6.26 the in-mask selection picks “the node with sufficient free memory that is closest to the node where the allocation takes place” rather than strict lowest-ID-first ordering.

• MPOL_INTERLEAVE — Spread page allocations round-robin across the nodes in nodemask, in node-ID order. This “optimizes for bandwidth instead of latency by spreading out pages and memory accesses… across multiple nodes.” The caveat: a single page’s accesses still hit one node’s bandwidth; interleave only helps a large region with a spread-out access pattern (the man page suggests “at least 1 MB”). It is the classic policy for a large shared database buffer pool.

• MPOL_WEIGHTED_INTERLEAVE (since Linux 6.9 — present in 6.12) — Like MPOL_INTERLEAVE, but allocations are distributed across nodes in proportion to per-node weights set in /sys/kernel/mm/mempolicy/weighted_interleave/nodeN. The kernel doc’s example: nodes [0,1] weighted [5,2] ⇒ “5 pages will be allocated on node0 for every 2 pages allocated on node1.” Weights are 1–255, default-resettable by writing 0 or an empty string (per the sysfs ABI doc at v6.12). This exists for heterogeneous-memory systems (e.g. CXL-attached memory with different bandwidth than local DRAM), where uniform round-robin would underuse the higher-bandwidth tier. Weights affect only new allocations and never migrate already-placed pages.

  Automatic weight derivation exists, but not in v6.12 — it landed in v6.16. The v6.12 sysfs ABI documents only manual nodeN weights. Existence-checking mm/mempolicy.c across tags for the auto-weighting machinery gives a clean boundary:

  Tag	auto-weighting symbols in mm/mempolicy.c
  v6.12	none
  v6.14	none
  v6.15	none
  v6.16	present (mode_auto, weighted_interleave_auto_show/store, wi_auto_attr)
  v6.18	present

  At v6.16 the header grows a bool mode_auto in the weighted-interleave state and a sysfs attribute declared as __ATTR(auto, 0664, weighted_interleave_auto_show, weighted_interleave_auto_store) (mm/mempolicy.c, v6.16), with a comment noting that “A null weighted_interleave_state is interpreted as having .mode=auto” — so auto is the default once the feature exists. The v6.16 ABI documentation dates the auto file to May 2025 (versus January 2024 for the nodeN files) and describes the semantics: in auto mode “all node weights are re-calculated and overwritten (visible via the nodeN interfaces) whenever new bandwidth data is made available during either boot or hotplug events,” and — the detail that catches people — “writing a new weight to a node directly via the nodeN interface will also automatically switch the system to manual mode.” So you cannot hand-tune one node and leave the rest on auto; the first manual write takes the whole system out of auto.

  On the measurement machine (running a 7.1 kernel, well past v6.16) both files are present and auto is on:

  $ for f in /sys/kernel/mm/mempolicy/weighted_interleave/*; do echo -n "$f = "; cat $f; done
  /sys/kernel/mm/mempolicy/weighted_interleave/auto = true
  /sys/kernel/mm/mempolicy/weighted_interleave/node0 = 1

  The weight is 1 because there is one node and, per the ABI doc, nodes “onlined with missing bandwidth data” fall back to “a default weight of 1” — a single-node machine has no bandwidth ratio to derive. For the v6.12 pin, treat weights as manual-only; the nodeN files exist, the auto file does not.

• MPOL_PREFERRED — A soft preference for one node: allocate there first, but “fall back to ‘near by’ nodes if the preferred node is low on free memory.” An empty nodemask makes it equivalent to local allocation. This is the policy to use when you want locality but cannot tolerate allocation failure.

• MPOL_PREFERRED_MANY (since Linux 5.15 — present in 6.12) — MPOL_PREFERRED generalized to a set of preferred nodes: prefer any node in nodemask, and only on pressure across all of them spill to other nodes. “Intended to benefit page allocations where specific memory types (i.e., non-volatile, high-bandwidth, or accelerator memory) are of greater importance than node location.”

• MPOL_LOCAL (since Linux 3.8) — Explicitly request local allocation (the node of the faulting CPU), with soft fallback to other nodes under pressure. nodemask must be empty. It differs from MPOL_DEFAULT in that MPOL_DEFAULT removes the policy (falling back to a possibly-non-local outer scope), whereas MPOL_LOCAL is a concrete local-allocation policy at this scope.

Mode Flags

A mode may be OR-ed with one optional flag, governing what happens when the task’s allowed-node set (its cpuset, see cpusets) changes underneath it:

• MPOL_F_STATIC_NODES (since 2.6.26) — Treat nodemask as physical node IDs; do not remap it when the cpuset’s allowed nodes change. Applied to the intersection of the user mask and the cpuset; if they do not overlap, default behavior is used.
• MPOL_F_RELATIVE_NODES (since 2.6.26) — Treat nodemask as relative to the cpuset’s allowed set, remapping it as that set changes (preserving the relative positions). Mutually exclusive with MPOL_F_STATIC_NODES.
• MPOL_F_NUMA_BALANCING (since Linux 5.12, valid only with MPOL_BIND) — Enable automatic NUMA balancing for the task. Using it with any other mode returns EINVAL.

Every MPOL_* Flag, With Its Bit Position

The word passed as mode to set_mempolicy/mbind is an int carrying the mode in its low bits and the mode flags in its high bits — which is why the mode flags start at bit 15 and count down, leaving room for the mode enum to grow upward. mbind and get_mempolicy each take a separate flags argument with its own, unrelated numbering that starts at bit 0. All four families live in the same header (mempolicy.h, v6.12), and confusing them is easy because the names all begin MPOL_F_:

Constant	Value	Passed in	Meaning
Mode flags — OR-ed into the mode argument; the legal set is MPOL_MODE_FLAGS
MPOL_F_STATIC_NODES	1 << 15 (32768)	mode of set_mempolicy/mbind	nodemask is physical node IDs; never remap it when the cpuset’s allowed nodes change. Applied to the intersection with the cpuset; empty intersection ⇒ Default behavior.
MPOL_F_RELATIVE_NODES	1 << 14 (16384)	mode of set_mempolicy/mbind	nodemask is relative to the cpuset’s allowed set and is remapped as that set changes. Mutually exclusive with MPOL_F_STATIC_NODES.
MPOL_F_NUMA_BALANCING	1 << 13 (8192)	mode of set_mempolicy	Run automatic NUMA balancing within the policy. Valid only with MPOL_BIND — EINVAL otherwise (verified live below).
get_mempolicy flags — the flags argument of get_mempolicy only
MPOL_F_NODE	1 << 0	flags of get_mempolicy	With MPOL_F_ADDR, return the node ID where addr currently resides; without it, return the next interleave node.
MPOL_F_ADDR	1 << 1	flags of get_mempolicy	Look up the policy via the VMA covering addr — i.e. return the effective policy, not the task policy.
MPOL_F_MEMS_ALLOWED	1 << 2	flags of get_mempolicy	Return the set of nodes the task is permitted to use (its cpuset mems), ignoring policy entirely. Cannot be combined with the two above.
mbind flags — the flags argument of mbind only; MPOL_MF_VALID is the legal set
MPOL_MF_STRICT	1 << 0	flags of mbind	Verification mode: fail with EIO if pages already in the range violate the new policy.
MPOL_MF_MOVE	1 << 1	flags of mbind	Migrate existing pages mapped only by this process to conform.
MPOL_MF_MOVE_ALL	1 << 2	flags of mbind	Migrate existing pages even when shared with other processes. Requires CAP_SYS_NICE.
MPOL_MF_LAZY	1 << 3	—	Declared in the header but commented “UNSUPPORTED FLAG” and deliberately excluded from MPOL_MF_VALID, so passing it yields EINVAL. A reserved bit, not an API.
MPOL_MF_INTERNAL	1 << 4	—	Marks where kernel-internal mbind flags begin; never passed from userspace.
Internal-only flags — never OR-ed into an API argument; they share the struct mempolicy flags word
MPOL_F_SHARED	1 << 0	kernel-internal	Identifies a shared policy (the fourth scope).
MPOL_F_MOF	1 << 3	kernel-internal	“This policy wants migrate on fault” — set by MPOL_F_NUMA_BALANCING.
MPOL_F_MORON	1 << 4	kernel-internal	“Migrate On protnone Reference On Node” — the balancing hook proper.

All MPOL_* flag constants at v6.12, grouped by which argument they belong to. What it shows: three completely separate numbering spaces, two of which both start at bit 0 (MPOL_F_NODE and MPOL_MF_STRICT are both 1), plus a fourth set that userspace must never send. The insight to take: the mode flags are high bits precisely so that mode | MPOL_F_STATIC_NODES can be decoded unambiguously from one int; the get_mempolicy and mbind flag sets are low bits and are not interchangeable, so passing MPOL_MF_MOVE (2) where MPOL_F_ADDR (2) was meant will silently do the wrong thing rather than error. Note also that MPOL_F_LOCAL, which the kernel’s .rst discusses at length, does not appear here at all — it was an internal flag that MPOL_LOCAL (mode 4) superseded as the userspace-visible spelling.

How Interleave Actually Picks a Node

“Round-robin across the nodes” is a description of the aggregate result, not of the mechanism. The kernel never keeps a single global rotating cursor — that would be a contended shared counter on every fault. Instead there are two different node-selection functions, chosen by what is being allocated, and the distinction has a visible consequence: one is deterministic per virtual address and the other is order-of-allocation dependent.

The kernel documentation states the split explicitly. For anonymous and shared-memory pages, “Interleave mode indexes the set of nodes specified by the policy using the page offset of the faulting address into the segment [VMA] containing the address modulo the number of nodes specified by the policy.” For page-cache pages, it “indexes the set of nodes specified by the policy using a node counter maintained per task” (numa_memory_policy.rst, v6.12).

In the v6.12 source these are interleave_nid() and interleave_nodes() respectively (mm/mempolicy.c:1900-1914 and :2024-2039, v6.12):

/* Offset-based: deterministic for a given interleave index (ilx). */
static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx)
{
	nodemask_t nodemask;
	unsigned int target, nnodes;
	int i;
	int nid;
 
	nnodes = read_once_policy_nodemask(pol, &nodemask);
	if (!nnodes)
		return numa_node_id();
	target = ilx % nnodes;              /* which slot in the ring */
	nid = first_node(nodemask);
	for (i = 0; i < target; i++)        /* walk that many set bits */
		nid = next_node(nid, nodemask);
	return nid;
}
 
/* Cursor-based: advances a per-task cursor on every call. */
static unsigned int interleave_nodes(struct mempolicy *policy)
{
	unsigned int nid;
	unsigned int cpuset_mems_cookie;
 
	/* to prevent miscount, use tsk->mems_allowed_seq to detect rebind */
	do {
		cpuset_mems_cookie = read_mems_allowed_begin();
		nid = next_node_in(current->il_prev, policy->nodes);
	} while (read_mems_allowed_retry(cpuset_mems_cookie));
 
	if (nid < MAX_NUMNODES)
		current->il_prev = nid;     /* the cursor lives in task_struct */
	return nid;
}

Walking interleave_nid symbol by symbol: ilx is the interleave index, derived from the page’s offset within its VMA, so page 0 of the mapping gets ilx = 0, page 1 gets ilx = 1, and so on. nnodes is the population count of the policy’s nodemask — how many nodes are actually set, not the highest node ID. target = ilx % nnodes reduces the index to a slot number in [0, nnodes). The for loop then converts that slot number into a node ID by walking target set bits forward from the lowest set bit, because the nodemask is sparse — slot 1 of the mask {0, 3} is node 3, not node 1. The cost is a bit-walk proportional to target, which is why this is fine for a handful of nodes and would not be for thousands.

interleave_nodes is the other shape entirely: next_node_in(current->il_prev, policy->nodes) advances from wherever this task landed last, wrapping at the top of the mask, and stores the result back into current->il_prev — a field in task_struct, so the cursor is per task, not per policy and not global. The read_mems_allowed_begin/read_mems_allowed_retry pair is a seqlock retry loop guarding against a concurrent cpuset rebind changing policy->nodes mid-selection; the comment says why — “to prevent miscount.”

flowchart TD
  START["Interleave policy governs<br/>this allocation"]
  START --> KIND{"What kind of page?"}
  KIND -->|"anonymous page,<br/>or shared memory (shmem)"| OFF["interleave_nid(pol, ilx)<br/>ilx = page offset into the VMA"]
  KIND -->|"page cache page<br/>(file read-ahead, etc.)"| CUR["interleave_nodes(pol)<br/>uses current-&gt;il_prev cursor"]
  OFF --> MOD["target = ilx % nnodes<br/>nnodes = popcount(nodemask)"]
  MOD --> WALK["nid = first_node(nodemask)<br/>then advance 'target' set bits"]
  WALK --> DET["Deterministic:<br/>the SAME virtual address always<br/>maps to the SAME node"]
  CUR --> ADV["nid = next_node_in(il_prev, nodes)<br/>wraps at top of mask"]
  ADV --> STORE["store nid back into<br/>current-&gt;il_prev"]
  STORE --> NONDET["Order-dependent:<br/>which node a page gets depends on<br/>how many pages this TASK<br/>allocated before it"]
  DET --> ZL["Chosen node is then used<br/>as if it were MPOL_PREFERRED:<br/>allocation follows that node's zonelist"]
  NONDET --> ZL

The two interleave node-selection paths and what distinguishes them. What it shows: interleave is not one algorithm. Anonymous and shmem pages use an offset-derived index, so placement is a pure function of the virtual address — reproducible across runs and identical in every process mapping the same offset. Page-cache pages use a per-task cursor, so placement depends on allocation history and is not reproducible. The insight to take: this is why an interleaved anonymous buffer gives you a predictable, evenly striped layout you can reason about (page n is on node n mod nnodes), while interleaved file-cache pages only spread out statistically. It also explains the final box: interleave picks a starting node and then defers to that node’s normal zonelist fallback, so interleave is soft per page — if the chosen node is full, the page still gets allocated, just elsewhere. Only MPOL_BIND refuses.

Weighted Interleave — Walking the Weight Table

MPOL_WEIGHTED_INTERLEAVE replaces “modulo the number of nodes” with “modulo the sum of the weights,” then walks the ring subtracting each node’s weight until the remainder falls inside one (weighted_interleave_nid(), mm/mempolicy.c:1980-2016, v6.12):

	rcu_read_lock();
	table = rcu_dereference(iw_table);
	/* calculate the total weight */
	for_each_node_mask(nid, nodemask) {
		weight = table ? table[nid] : 1;   /* system default usage */
		weight = weight ? weight : 1;      /* 0 is never a real weight */
		weight_total += weight;
	}
 
	/* Calculate the node offset based on totals */
	target = ilx % weight_total;
	nid = first_node(nodemask);
	while (target) {
		weight = table ? table[nid] : 1;
		weight = weight ? weight : 1;
		if (target < weight)
			break;
		target -= weight;
		nid = next_node_in(nid, nodemask);
	}
	rcu_read_unlock();
	return nid;

Note three details the prose documentation does not give you. First, weight_total is recomputed on every allocation by summing the weights of the nodes in the mask — the table is not pre-reduced. Second, iw_table is read under RCU (rcu_dereference), which is how a sysfs write can swap the whole weight table without stopping allocators. Third, the two weight = weight ? weight : 1 lines mean a stored weight of zero is silently treated as one, so there is no way to express “never allocate here” through weights — that is what leaving the node out of the nodemask is for.

Applying it to the kernel doc’s own example — nodes {0, 1} with weights [5, 2], so weight_total = 7:

ilx	target = ilx % 7	Walk	Chosen node
0	0	loop body never runs (target == 0)	node 0
1	1	1 < 5 ⇒ break	node 0
2	2	2 < 5 ⇒ break	node 0
3	3	3 < 5 ⇒ break	node 0
4	4	4 < 5 ⇒ break	node 0
5	5	5 < 5 false ⇒ target = 0, nid = 1; target == 0 ⇒ exit	node 1
6	6	6 < 5 false ⇒ target = 1, nid = 1; 1 < 2 ⇒ break	node 1
7	0	pattern repeats	node 0

Hand-executing weighted_interleave_nid() for weights [5, 2]. What it shows: the first five indices of every seven land on node 0 and the next two on node 1 — exactly the documented “5 pages will be allocated on node0 for every 2 pages allocated on node1.” The insight to take: the distribution is not a random 5:2 draw, it is a fixed repeating stripe of five pages then two pages, aligned to ilx = 0. For an anonymous mapping, where ilx is the page offset, that means the layout is deterministic and contiguous in runs — a 5-page run on the fast node followed by a 2-page run on the slow one — not finely alternated. Whether that matters depends on whether your access pattern is sequential over those runs.

Uncertain

Verify: whether the per-node weight is applied at page granularity in all cases or can be applied to a whole folio (multiple contiguous pages) when THP or large folios are in play — the ilx passed to weighted_interleave_nid() is an interleave index, and how it is derived for a large folio was not traced here. Reason: only the selection function was read, not all of its callers. To resolve: trace every caller of weighted_interleave_nid() and interleave_nid() in mm/mempolicy.c at v6.12 and record how each computes ilx for orders above 0. uncertain

The System Calls

#include <numaif.h>
 
long set_mempolicy(int mode, const unsigned long *nodemask, unsigned long maxnode);
long mbind(void *addr, unsigned long len, int mode,
           const unsigned long *nodemask, unsigned long maxnode, unsigned int flags);
long get_mempolicy(int *mode, unsigned long *nodemask, unsigned long maxnode,
                   void *addr, unsigned long flags);

set_mempolicy installs the task policy. mode is one of the modes above, optionally OR-ed with a flag; nodemask/maxnode give the node set. It “defines the default policy for the thread”; the policy applies only “when a new page is allocated” — for anonymous memory, on first touch. All modes except MPOL_DEFAULT require a non-empty nodemask (and MPOL_LOCAL/MPOL_DEFAULT require an empty one).

mbind installs the VMA policy on [addr, addr+len) (addr must be page-aligned). Its flags argument controls what happens to pages already present in the range — by default mbind only affects future allocations:

• MPOL_MF_STRICT — Fail with EIO if existing pages already violate the policy (a verification mode).
• MPOL_MF_MOVE — Attempt to migrate existing (non-shared) pages to conform to the new policy.
• MPOL_MF_MOVE_ALL — Migrate existing pages even if shared with other processes; requires CAP_SYS_NICE.

The migration here is exactly page migration (available via mbind since 2.6.16). Note the shared-mapping rules from the man page: MAP_SHARED file mappings ignore the mbind policy entirely; a MAP_PRIVATE file mapping’s policy fires only at COW; and SHM_HUGETLB segments only honor the policy when the page is faulted by the very process that called mbind.

get_mempolicy is the introspection call. With flags == 0 it returns the calling thread’s task policy (addr must be NULL). With MPOL_F_ADDR and a non-NULL addr, it returns the effective policy governing that address — which may be a VMA policy different from the task policy. With MPOL_F_NODE | MPOL_F_ADDR it returns the node ID where addr is currently allocated (faulting a page in if none exists). With MPOL_F_MEMS_ALLOWED (since 2.6.24) it returns the set of nodes the thread is allowed to use (its cpuset’s mems), ignoring mode. These flags have mutual-exclusion rules (MPOL_F_MEMS_ALLOWED cannot combine with MPOL_F_ADDR or MPOL_F_NODE), enforced with EINVAL.

The Whole Surface in One Table

Between the three syscalls, the two extra placement syscalls, numactl(8)’s options, and the procfs/sysfs files, the policy interface is wider than it first appears. Mapping every entry point to the scope it touches:

Entry point	Scope it installs or reads	Notes
set_mempolicy(mode, nodemask, maxnode)	task	Survives execve; per-thread; future allocations only.
mbind(addr, len, mode, nodemask, maxnode, flags)	VMA (or shared object)	addr page-aligned; may split the VMA; flags controls existing pages.
get_mempolicy(&mode, nodemask, maxnode, addr, flags)	reads task or effective	flags = 0 ⇒ task policy; MPOL_F_ADDR ⇒ effective policy at addr; +MPOL_F_NODE ⇒ the node a page is on.
set_mempolicy_home_node(start, len, home_node, flags)	VMA (refines)	Sets a “home node” to refine MPOL_BIND/MPOL_PREFERRED_MANY fallback ordering; added Linux 5.17. Does not change the mode.
migrate_pages(pid, maxnode, old_nodes, new_nodes)	none — moves pages	Bulk-relocates a process’s pages between node sets. Orthogonal to policy.
move_pages(pid, count, pages, nodes, status, flags)	none — moves pages	Per-page relocation and per-page node query. Orthogonal to policy.
numactl -i / --interleave=<nodes>	task, MPOL_INTERLEAVE	Then execs the target — which is why the task scope’s exec survival matters.
numactl -w / --weighted-interleave=<nodes>	task, MPOL_WEIGHTED_INTERLEAVE	Uses the sysfs weights.
numactl -m / --membind=<nodes>	task, MPOL_BIND	The strict one — can OOM with free memory elsewhere.
numactl -p / --preferred=<node>	task, MPOL_PREFERRED	Single node.
numactl -P / --preferred-many=<nodes>	task, MPOL_PREFERRED_MANY	Node set.
numactl -l / --localalloc	task, MPOL_LOCAL
numactl -N / --cpunodebind, -C / --physcpubind	not a memory policy	CPU affinity. Commonly paired with --membind, but a different kernel mechanism.
numactl --balancing	adds MPOL_F_NUMA_BALANCING	Only meaningful alongside --membind.
numactl --hardware	reads topology	Nodes, sizes, distance matrix.
/proc/<pid>/numa_maps	reads per-range effective policy + actual placement	The Nn=<count> fields are where pages really are.
/sys/devices/system/node/	reads topology	possible, online, nodeN/meminfo, nodeN/distance. The fallback when numactl is not installed.
/sys/kernel/mm/mempolicy/weighted_interleave/nodeN	weights for mode 6	1–255; writing switches the system to manual mode.
/sys/kernel/mm/mempolicy/weighted_interleave/auto	weight source for mode 6	true ⇒ kernel derives weights from firmware bandwidth data. Added v6.16 (see below).

Every way to set or observe a memory policy, and which scope each touches. What it shows: only three calls install policy; everything else either reads it, moves pages irrespective of it, or is a userspace convenience that ends in one of those three calls. The insight to take: migrate_pages/move_pages and --cpunodebind are the two things most often mistaken for policy. Neither installs a policy: the first two relocate pages the policy already placed, and the third constrains CPUs, which only indirectly affects placement by changing which node counts as “local.”

set_mempolicy_home_node — The Fourth Call, With No Man Page

There is a fourth policy syscall that the man pages do not cover: set_mempolicy_home_node(2) is genuinely undocumented in man-pages — https://man7.org/linux/man-pages/man2/set_mempolicy_home_node.2.html returns HTTP 404, so the source is the only authority. Existence-checking arch/x86/entry/syscalls/syscall_64.tbl pins it:

Tag	set_mempolicy_home_node in the x86-64 syscall table
v5.16	absent
v5.17	present
v6.12	present, as syscall number 450

Its signature and validation come from SYSCALL_DEFINE4 (mm/mempolicy.c:1536-1560, v6.12):

SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, len,
		unsigned long, home_node, unsigned long, flags)
{
	...
	start = untagged_addr(start);
	if (start & ~PAGE_MASK)
		return -EINVAL;
	/*
	 * flags is used for future extension if any.
	 */
	if (flags != 0)
		return -EINVAL;
	/*
	 * Check home_node is online to avoid accessing uninitialized
	 * NODE_DATA.
	 */
	if (home_node >= MAX_NUMNODES || !node_online(home_node))
		return -EINVAL;

So: start must be page-aligned, flags must be zero (it is reserved — passing anything is EINVAL), and home_node must be a currently-online node, not merely a legal node number. Unlike mbind, it takes a single node rather than a mask, and it does not change the mode of the policy already on the range — it annotates it.

What the annotation does is visible in policy_nodemask() (mm/mempolicy.c:2045-2070, v6.12), the function that turns a policy into the (preferred node, allowed nodemask) pair the page allocator wants:

	case MPOL_PREFERRED_MANY:
		nodemask = &pol->nodes;
		if (pol->home_node != NUMA_NO_NODE)
			*nid = pol->home_node;
		break;
	case MPOL_BIND:
		/* Restrict to nodemask (but not on lower zones) */
		...
		if (pol->home_node != NUMA_NO_NODE)
			*nid = pol->home_node;

The home node overrides the starting node (*nid) while leaving the allowed set (nodemask) alone, and only for MPOL_BIND and MPOL_PREFERRED_MANY. That is precisely the gap it fills: both of those modes take a set of nodes but, by themselves, give you no way to say which member of the set should be tried first. Without a home node, MPOL_BIND over {0,1,2} starts from the faulting CPU’s node and works outward by distance; with set_mempolicy_home_node(..., 1, 0) it always starts from node 1 regardless of where the thread is running. For a thread pool spread across sockets that all share one bound region, that turns per-thread-varying placement into one consistent layout. Note that home_node defaults to NUMA_NO_NODE (policy->home_node = NUMA_NO_NODE at mm/mempolicy.c:338), which is the “not set” sentinel — this feature is opt-in and invisible unless used.

The same switch statement incidentally confirms two claims from the mode table above. MPOL_PREFERRED does *nid = first_node(pol->nodes) — it reads only the lowest set bit of your nodemask and ignores the rest, so passing a multi-node mask to MPOL_PREFERRED is silently not an error and silently not what you meant (that is what MPOL_PREFERRED_MANY is for). And MPOL_BIND sets nodemask only when apply_policy_zone() agrees, with a WARN_ON_ONCE(gfp & __GFP_THISNODE) guarding against a caller combining bind with a flag that would contradict it.

Worked Examples

From userspace with the raw syscalls (via libnuma’s thin wrappers)

The snippet below uses the conventional libnuma headers, which is how most code reaches these calls. Note that libnuma is a separate package, not part of glibc, and none of these three calls has a glibc wrapper — if it is not installed, call them through syscall(2) directly, as the measurement section below does.

#include <numaif.h>
#include <numa.h>
 
/* Interleave this process's future anonymous allocations across nodes 0 and 1. */
unsigned long mask = (1UL << 0) | (1UL << 1);   /* nodes 0,1 */
set_mempolicy(MPOL_INTERLEAVE, &mask, 8 * sizeof(mask));
 
/* Bind one large buffer strictly to node 1, and migrate any pages already there. */
void *buf = mmap(NULL, len, PROT_READ|PROT_WRITE,
                 MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
unsigned long n1 = (1UL << 1);
mbind(buf, len, MPOL_BIND, &n1, 8 * sizeof(n1), MPOL_MF_MOVE);
 
/* Ask: which node is the first page of buf actually on? */
int node;
get_mempolicy(&node, NULL, 0, buf, MPOL_F_NODE | MPOL_F_ADDR);

Line by line: the first set_mempolicy makes node-0/1 round-robin the task default, so any later malloc/stack/heap fault spreads across both nodes. The mbind then overrides that for the buf range with a strict MPOL_BIND to node 1, and MPOL_MF_MOVE migrates pages that were already faulted in (e.g. under the inherited interleave policy) onto node 1. The final get_mempolicy with MPOL_F_NODE|MPOL_F_ADDR reports the node the first page of buf physically sits on (faulting it in if needed). The 8 * sizeof(mask) is maxnode — the number of bits in the mask the kernel should read.

From the command line with numactl(8)

numactl is the standard front end; it sets the policy and execs the target program, which inherits the task policy (per numactl(8)):

numactl --hardware                              # show nodes, sizes, and the distance matrix
numactl --interleave=all bigdatabase args       # round-robin across all nodes
numactl --weighted-interleave=all db args       # weighted round-robin (uses sysfs weights)
numactl --cpunodebind=0 --membind=0,1 process   # run on node 0's CPUs; allocate only on 0 or 1
numactl --preferred=1 ./prog                     # soft-prefer node 1
numactl --preferred-many=0x3 ./prog              # soft-prefer the set {0,1}
numactl --localalloc ./prog                      # MPOL_LOCAL

--interleave/-i ⇒ MPOL_INTERLEAVE; --weighted-interleave/-w ⇒ MPOL_WEIGHTED_INTERLEAVE; --membind/-m ⇒ MPOL_BIND; --preferred/-p ⇒ MPOL_PREFERRED; --preferred-many/-P ⇒ MPOL_PREFERRED_MANY; --localalloc/-l ⇒ MPOL_LOCAL. The separate --cpunodebind/-N and --physcpubind/-C options set CPU affinity (which CPUs the task runs on) — a distinct knob from memory policy, though commonly paired with --membind so a task runs and allocates on the same node. --balancing enables NUMA balancing alongside the policy.

You can inspect the result without code via /proc/<pid>/numa_maps, which shows, per memory range, “the effective memory policy for that memory range and on which nodes the pages have been allocated” — e.g. N0=12 N1=11 for an interleaved range (per numa(7)).

Measured on a Single-Node Machine — and Why That Matters

Everything above describes placement across nodes. It is worth being explicit about what you can and cannot observe when there is only one node, because the great majority of developer machines and small cloud instances are single-node even when they have many cores.

The test machine for this note has 32 CPUs and ~125 GiB of RAM, which sounds like a NUMA machine. It is not. Reading sysfs directly (numactl is not installed here, so the topology comes from /sys/devices/system/node/):

$ ls /sys/devices/system/node/
has_cpu  has_generic_initiator  has_memory  has_normal_memory
node0/  online  possible  power/  uevent
 
$ cat /sys/devices/system/node/possible /sys/devices/system/node/online
0
0
 
$ head -3 /sys/devices/system/node/node0/meminfo
Node 0 MemTotal:       131150216 kB
Node 0 MemFree:        14362256 kB
Node 0 MemUsed:        116787960 kB
 
$ cat /sys/devices/system/node/node0/distance
10

A single node0/ directory, possible and online both reading just 0, and a distance matrix that is the single value 10 — the self-distance, which is by convention always 10 (the ACPI SLIT normalizes local access to 10, so a value of 20 conventionally means “twice the local cost”). This is a one-node machine. All 125 GiB is node 0, every CPU is node 0’s CPU, and there is no remote memory to be slower.

Consequently, on this machine every policy is behaviorally identical. MPOL_BIND to {0}, MPOL_INTERLEAVE over {0}, MPOL_PREFERRED node 0, and MPOL_LOCAL all place every page on node 0, because there is nowhere else. Interleave “spreads” across a ring of one. Weighted interleave’s weights are irrelevant when weight_total comes from a single node. This note’s performance claims therefore cannot be validated here — they require at least two nodes with a real distance between them. What can be validated here is the API surface: which modes the kernel accepts, which argument combinations it rejects, and whether scope precedence works as documented. Those are exactly the facts most often gotten wrong, and they are node-count-independent.

The kernel is built with policy support even though the hardware has one node:

$ grep -E '^CONFIG_NUMA|^CONFIG_NUMA_BALANCING' /boot/config-$(uname -r)
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA=y
CONFIG_NUMA_KEEP_MEMINFO=y
CONFIG_NUMA_MIGRATION=y

This is the normal state of a distribution kernel: CONFIG_NUMA=y so one binary boots on both one-node laptops and 8-socket servers. It means the set_mempolicy/mbind/get_mempolicy syscalls are present and functional here — they just have one node to choose from.

Exercising all seven modes with raw syscalls

libnuma (and therefore <numaif.h>) is not installed on this machine, so the test program below calls the syscalls directly through syscall(2) with the mode constants transcribed from the v6.12 UAPI header. This is worth knowing as a technique: the policy syscalls have no glibc wrappers at all — libnuma is a separate package, and syscall(SYS_set_mempolicy, ...) is the dependency-free way to reach them.

static long set_mempolicy_(int mode, const unsigned long *nm, unsigned long mn)
{ return syscall(SYS_set_mempolicy, mode, nm, mn); }
static long get_mempolicy_(int *mode, unsigned long *nm, unsigned long mn,
                           void *a, unsigned long fl)
{ return syscall(SYS_get_mempolicy, mode, nm, mn, a, fl); }
static long mbind_(void *a, unsigned long len, int mode, const unsigned long *nm,
                   unsigned long mn, unsigned flags)
{ return syscall(SYS_mbind, a, len, mode, nm, mn, flags); }

Real output:

== baseline task policy ==
  mode=0 (MPOL_DEFAULT) nodemask=0x0
  MPOL_F_MEMS_ALLOWED -> allowed nodes bitmask = 0x1
 
== try each mode with nodemask={0} on this 1-node box ==
  set_mempolicy(MPOL_DEFAULT            , NULL/0) -> OK
  set_mempolicy(MPOL_PREFERRED          , {node0}) -> OK
  set_mempolicy(MPOL_BIND               , {node0}) -> OK
  set_mempolicy(MPOL_INTERLEAVE         , {node0}) -> OK
  set_mempolicy(MPOL_LOCAL              , NULL/0) -> OK
  set_mempolicy(MPOL_PREFERRED_MANY     , {node0}) -> OK
  set_mempolicy(MPOL_WEIGHTED_INTERLEAVE, {node0}) -> OK

All seven modes from the table are accepted by this kernel, confirming the enum is complete and that MPOL_PREFERRED_MANY and MPOL_WEIGHTED_INTERLEAVE are genuinely present rather than merely declared. The baseline task policy is MPOL_DEFAULT with an empty nodemask — the expected “no task policy installed, fall through to the system default” state. MPOL_F_MEMS_ALLOWED returns 0x1: bit 0 set, one allowed node, which independently corroborates the sysfs topology.

The argument-validation rules, confirmed by rejection

This is the part worth running, because the rules are easy to state and easy to get wrong:

== negative tests (argument validation) ==
  MPOL_DEFAULT with non-empty nodemask -> Invalid argument
  MPOL_BIND to node 5 (does not exist)  -> Invalid argument
  MPOL_INTERLEAVE|MPOL_F_NUMA_BALANCING -> Invalid argument
  MPOL_BIND|MPOL_F_NUMA_BALANCING        -> OK
  mode 7 (== MPOL_MAX, invalid)          -> Invalid argument

Four documented rules, each confirmed by an actual EINVAL:

1. MPOL_DEFAULT must have an empty nodemask. The kernel doc says “It is an error for the set of nodes specified for this policy to be non-empty,” and it is.
2. A nodemask may not name a node that is not in the task’s allowed set. Node 5 does not exist on a one-node machine, so MPOL_BIND to {5} is rejected — the mask is validated against allowed nodes, not merely against MAX_NUMNODES.
3. MPOL_F_NUMA_BALANCING is valid only with MPOL_BIND. Combined with MPOL_INTERLEAVE it returns EINVAL; combined with MPOL_BIND it succeeds. This confirms the flag restriction against the running kernel rather than against the man page alone.
4. MPOL_MAX is a bound, not a mode. mode = 7 is rejected, which is exactly what makes MPOL_MAX’s “always last member of enum” comment load-bearing: adding a mode moves the validation boundary automatically.

Scope precedence, confirmed end to end

The last part of the program installs a task policy, then mbinds a narrower policy onto one mapping, then asks the kernel which policy actually governs that address:

== mbind a VMA, then read back effective policy at that address ==
  task policy now: MPOL_INTERLEAVE
  mbind(buf, MPOL_BIND, {0}) -> OK
  get_mempolicy(MPOL_F_ADDR) at buf -> MPOL_BIND  (VMA policy overrides task policy)
  get_mempolicy(MPOL_F_NODE|MPOL_F_ADDR) -> page is on node 0
 
== /proc/self/numa_maps for that range ==
55a68a6d8000 interleave:0 heap anon=10 dirty=10 active=0 N0=10 kernelpagesize_kB=4
7ffd4d238000 interleave:0 stack anon=5 dirty=5 active=0 N0=5 kernelpagesize_kB=4

This is the mental-model diagram executed for real. The task policy is MPOL_INTERLEAVE; get_mempolicy with no flags would report that. With MPOL_F_ADDR against the mbind-ed buffer it reports MPOL_BIND instead — the VMA scope winning over the task scope, observed rather than asserted. Adding MPOL_F_NODE switches the question from “what policy?” to “where is this page?”, and returns node 0.

The numa_maps output shows the other half. The heap and stack ranges inherited the task policy and are labelled interleave:0 — “interleave over the node set {0}”. anon=10 is ten anonymous pages present, dirty=10 all written, and N0=10 says all ten are on node 0. On a multi-node machine with an interleave over {0,1} this same line would read something like N0=5 N1=5, and that Nn= breakdown is the ground truth for whether a policy did what you wanted — it reports where pages are, not where policy said they should go.

Uncertain

Verify: the actual performance consequences of each mode — local-vs-remote latency ratios, interleave’s bandwidth gain, and the size threshold (the man page suggests “at least 1 MB”) above which interleave pays off. Reason: the measurement machine has exactly one NUMA node, so no remote access exists and every mode is behaviorally identical here; none of the performance claims in this note were measured. To resolve: rerun on a multi-socket host (/sys/devices/system/node/online reading e.g. 0-1) and compare a numactl --membind=0 --cpunodebind=0 run against --membind=1 --cpunodebind=0 on a memory-bandwidth benchmark; the nodeN/distance matrix gives the firmware’s own estimate of the ratio to expect. uncertain

Failure Modes and Misunderstandings

“MPOL_DEFAULT means local allocation.” No — it means remove the policy at this scope and fall back outward. Only the system default is concretely local allocation. If you mbind(..., MPOL_DEFAULT, ...) a range, that range starts following the task policy, which might be MPOL_INTERLEAVE. To force local allocation regardless of outer scope, use MPOL_LOCAL.

MPOL_BIND is strict and can OOM a node. Because MPOL_BIND refuses to spill off-mask, an allocation that exhausts the bound nodes triggers reclaim and, failing that, the OOM killer — even when other nodes have free memory. This is by design (it is the “strict” mode) but is a frequent production surprise. If you want locality without that risk, use MPOL_PREFERRED/MPOL_PREFERRED_MANY.

The MPOL_BIND failure mode is the one that causes real production incidents, so it is worth drawing the divergence between what MPOL_BIND does and what MPOL_PREFERRED_MANY does when the bound nodes fill up:

flowchart TD
  F["Allocation on a 2-node machine.<br/>Policy names node 0 only.<br/>node 0 is full, node 1 has 40 GiB free."]
  F --> M{"Which mode?"}
  M -->|"MPOL_BIND {0}"| B1["policy_nodemask() returns<br/>nodemask = {0}<br/>allocator may not look outside it"]
  B1 --> B2["direct reclaim on node 0"]
  B2 --> B3{"reclaim freed<br/>enough?"}
  B3 -->|yes| B4["allocation succeeds<br/>(after a latency spike)"]
  B3 -->|no| B5["<b>OOM killer fires</b><br/>constraint = CONSTRAINT_MEMORY_POLICY<br/>scored only over the policy's nodes"]
  B5 --> B6["a process is killed<br/><b>while 40 GiB sits free on node 1</b>"]
  M -->|"MPOL_PREFERRED_MANY {0}"| P1["nodemask = {0} is a PREFERENCE"]
  P1 --> P2["try node 0, fail"]
  P2 --> P3["fall back to all other nodes"]
  P3 --> P4["allocation succeeds on node 1<br/>(slower memory, no kill)"]

The same out-of-memory situation under a strict versus a soft policy. What it shows: MPOL_BIND genuinely means “this node or nothing,” and “nothing” escalates through reclaim to the OOM killer even though the machine as a whole has plenty of free memory. Note the OOM constraint: oom_kill.c distinguishes CONSTRAINT_MEMORY_POLICY, so the victim is chosen by badness computed over the policy’s nodes, not the whole system. The insight to take: choose MPOL_BIND only when a remote allocation is genuinely worse for you than a dead process — a hard-real-time deadline, or a measurement you must not contaminate. For the far more common goal “please keep this near the CPU,” MPOL_PREFERRED_MANY (or MPOL_PREFERRED) expresses the same intent and degrades instead of failing. A large fraction of “the OOM killer fired but free showed memory available” reports on multi-socket machines are a --membind that should have been a --preferred.

Policy is not retroactive and not remembered across swap. All scopes apply “only to pages allocated after the policy is installed”; to relocate existing pages you need mbind with MPOL_MF_MOVE(_ALL) or migrate_pages(2)/move_pages(2). And per the man-page note, “Memory policy is not remembered if the page is swapped out” — a swapped-in page is placed by the policy in effect at swap-in time.

Interleave on a MAP_SHARED file does nothing. VMA policy “applies ONLY to anonymous pages.” A common mistake is mbind-ing an interleave policy over an mmap’d shared data file and expecting the page cache to spread; it will not — those pages follow task/system policy. Use shared-memory segments (shmget/MAP_ANONYMOUS|MAP_SHARED) for shared policy to take effect.

cpuset interaction silently remaps your mask. Without MPOL_F_STATIC_NODES, the kernel remaps your nodemask when the task’s cpuset mems change — so a policy “bound to node 2” can quietly start using a different node after a container is moved between cpusets. Use MPOL_F_STATIC_NODES if you mean physical node IDs.

Alternatives and When to Choose Them

Memory policy is the explicit, static placement tool. Its main alternative is Automatic NUMA Balancing, which migrates pages toward the CPUs actually touching them at runtime by sampling minor page faults — zero configuration, adapts to changing access patterns, but adds fault and migration overhead and reacts rather than dictates. Use explicit policy when you know the placement you want (a database pinning its buffer pool, a real-time task avoiding remote latency); rely on automatic balancing for general workloads whose access patterns you do not know ahead of time. The two can coexist: MPOL_F_NUMA_BALANCING with MPOL_BIND runs balancing within the bound node set. For relocating already-placed pages outside the policy framework, migrate_pages(2) (move all of a process’s pages between node sets) and move_pages(2) (move specific pages) are the lower-level primitives.

See Also

• NUMA Memory Model — what NUMA nodes and inter-node distance are; the hardware reality these policies steer allocation across.
• Automatic NUMA Balancing — the runtime, fault-driven alternative/complement to explicit policy.
• Memory Zones and Nodes — the pg_data_t/zonelist machinery that policy reshapes; how fallback ordering works underneath a policy.
• Page Migration — the mechanism behind MPOL_MF_MOVE and migrate_pages/move_pages.
• Virtual Memory Areas — VMAs, which mbind splits to attach per-range policy.
• The Page Fault Handler — where policy is consulted (first-touch allocation).
• GFP Flags and Allocation Contexts — the per-allocation flags policy works alongside.
• Memory Overcommit and Accounting — the other half of “can this allocation proceed?”: policy decides where a page goes, overcommit accounting decides whether the promise is accepted at all.
• Transparent Huge Pages — large folios interact with interleave’s per-page index; a 2 MiB page cannot be split across nodes.
• The Buddy Allocator — what actually satisfies the allocation once policy has named a starting node and a permitted nodemask.
• MOC: Linux Memory Management MOC (§13, NUMA and Memory Placement).