Cpusets and CPU Partitioning

A cpuset is the cgroup controller that confines the tasks in a control group to a chosen set of CPUs and memory (NUMA) nodes. Where [[CPU Affinity and sched_setaffinity|sched_setaffinity]] pins one task at a time, a cpuset draws a durable, inheritable boundary around an entire cgroup subtree: every task in the group — and every task it forks — is restricted to the cpuset’s cpuset.cpus and cpuset.mems, and the kernel silently clamps any narrower per-task affinity to lie within it. On large NUMA machines this is how you co-locate a job with the memory it touches; on latency-sensitive systems it is how you carve exclusive slices of CPU out of the general scheduler. The cgroup-v2 controller adds the key modern capability: cpuset.cpus.partition, which promotes a cpuset to a partition root and gives it sole ownership of a set of CPUs, optionally with load balancing disabled (“isolated”), causing the scheduler to rebuild its scheduling domains so those CPUs form their own domain. This note is verified against the 6.12 LTS kernel (released 2024-11-17, Documentation/admin-guide/cgroup-v2.rst and kernel/cgroup/cpuset.c); the v2 partition interface is also current in 6.18 LTS.

Mental Model

A cpuset is a box around a chunk of the machine. The box names some CPUs (cpuset.cpus) and some memory nodes (cpuset.mems); any task placed in the box can use only those resources. Boxes nest: a child box can only ever name resources its parent already owns — the controller is strictly hierarchical, so a child can never escape its parent’s CPU set (cgroup-v2 §Cpuset).

The second, harder idea is the partition. A plain cpuset shares its CPUs with the rest of the system — load balancing still moves other tasks across them. Marking a cpuset as a partition root makes its CPUs exclusive: they are removed from every other cpuset and become a self-contained scheduling domain. An isolated partition goes further and turns load balancing off inside that domain, so the scheduler never spontaneously migrates anything between those CPUs — you place tasks there by hand. This is the cgroup-native equivalent of the old [[CPU Isolation isolcpus and nohz_full|isolcpus=]] boot parameter, but reconfigurable at runtime.

flowchart TD
  ROOT["root cpuset (always a partition root)<br/>cpuset.cpus = 0-7"]
  ROOT --> GEN["/general (member)<br/>cpuset.cpus = 0-3<br/>shares CPUs, load-balanced"]
  ROOT --> RT["/rt (partition root)<br/>cpuset.cpus = 4-7<br/>cpuset.cpus.partition = root"]
  RT --> ISO["/rt/poller (isolated)<br/>cpuset.cpus = 6-7<br/>partition = isolated<br/>NO load balancing"]
  RT -.exclusive.-> EXCL["CPUs 4-7 removed from /general<br/>own scheduling domain"]
  ISO -.->|"generate_sched_domains()"| SD["partition_sched_domains()<br/>rebuilds sched domains:<br/>{0-3} balanced, {4-5} balanced, {6,7} singletons"]

A three-level cpuset hierarchy carving exclusive CPUs. What it shows: the root owns all CPUs; /general is a plain member cpuset sharing CPUs 0–3; /rt is a partition root claiming 4–7 exclusively (so they vanish from /general); /rt/poller is isolated, taking 6–7 with load balancing off. A change to any partition triggers generate_sched_domains() → partition_sched_domains(), which rebuilds the scheduler’s domains so the isolated CPUs become singleton domains. The insight to take: “member” cpusets only filter where tasks may run; “partition root/isolated” cpusets additionally restructure the scheduler’s load-balancing topology — that domain rebuild is the mechanism that makes the isolation real, not just advisory.

The Interface Files

Each cpuset-enabled cgroup exposes these files (cgroup-v2.rst v6.12):

• cpuset.cpus (read-write) — the requested CPUs for this cgroup, as comma-separated numbers/ranges (0-4,6,8-10). The actual list granted is “subjected to constraints imposed by its parent and can differ from the requested CPUs.” Empty means inherit from the nearest non-empty ancestor (or all CPUs). Its value is stable across CPU hotplug — unplugging a CPU does not rewrite what you asked for.
• cpuset.cpus.effective (read-only) — the onlined CPUs actually granted by the parent. This is what tasks may really use; it is a subset of cpuset.cpus (or the parent’s effective set if cpuset.cpus is empty), and it does track hotplug.
• cpuset.mems (read-write) — the requested memory (NUMA) nodes. Writing a new value migrates the cgroup’s pages toward the named nodes; the doc warns “there is a cost for this memory migration… set cpuset.mems properly before spawning new tasks.”
• cpuset.mems.effective (read-only) — the granted, online memory nodes.
• cpuset.cpus.exclusive (read-write) — CPUs reserved for forming a partition; unused unless the cgroup becomes a partition root. Must obey the exclusivity rule — no CPU may appear in two sibling cgroups’ exclusive sets.
• cpuset.cpus.exclusive.effective (read-only) — the effective exclusive set available to become a partition.
• cpuset.cpus.partition (read-write, single value) — the partition state machine, detailed below. Notably “owned by the parent cgroup and is not delegatable.”
• cpuset.cpus.isolated (read-only, root cgroup only) — all CPUs currently in isolated partitions system-wide.

The requested-vs-effective distinction is the same shape as the task-side user-mask-vs-effective-mask split: you state intent (cpuset.cpus), and the kernel grants you the intersection of intent with what your parent and hotplug allow (cpuset.cpus.effective).

The Partition State Machine

cpuset.cpus.partition accepts three input values and reports up to five on read (cgroup-v2.rst v6.12):

Written value	Meaning
member	Non-root member of a partition (the default for every non-root cgroup)
root	Partition root — owns an exclusive set of CPUs, with load balancing
isolated	Partition root without load balancing, also excluded from unbound workqueues

On read it can additionally show root invalid (<reason>) or isolated invalid (<reason>), where the parenthesized text explains the degradation. A cpuset partition is “a collection of cpuset-enabled cgroups with a partition root at the top of the hierarchy and its descendants except those that are separate partition roots themselves… A partition has exclusive access to the set of exclusive CPUs allocated to it. Other cgroups outside of that partition cannot use any CPUs in that set.”

Setting root makes the cgroup “the root of a new partition or scheduling domain,” with its exclusive CPUs drawn from cpuset.cpus.exclusive.effective. Setting isolated puts the partition’s CPUs “in an isolated state without any load balancing from the scheduler and excluded from the unbound workqueues” — and the documentation warns that “tasks placed in such a partition with multiple CPUs should be carefully distributed and bound to each of the individual CPUs for optimal performance,” because the scheduler will no longer balance them for you.

Validity. A partition root is valid only if certain conditions hold; otherwise it degrades to “invalid,” retaining some state but “behav[ing] more like a member.” For a local partition root (parent is itself a valid partition root) the conditions are: (1) the parent is a valid partition root; (2) cpuset.cpus.exclusive.effective is non-empty (may contain offline CPUs); (3) cpuset.cpus.effective is non-empty unless there are no tasks. A remote partition root (parent is not a partition root) needs conditions 2 and 3 but not 1. External events — CPU hotplug, or edits to cpuset.cpus/cpuset.cpus.exclusive — “can cause a valid partition root to become invalid and vice versa,” and a poll/inotify event fires on the file each time the state flips, so a management agent can watch for degradation without polling.

Local vs remote partitions. A local partition’s parent is also a valid partition root; a remote partition’s parent is not. For a local partition, cpuset.cpus.exclusive is optional — it implicitly equals cpuset.cpus. For a remote partition you must write cpuset.cpus.exclusive down the hierarchy “before the target partition root,” because the exclusive CPUs have to be threaded through intermediate non-partition ancestors. As of 6.12, “a remote partition cannot be created under a local partition,” and all ancestors of a remote root except the system root cgroup must not be partition roots.

Mechanical Walk-through: How a Cpuset Constrains a Task

When a task is in a non-root cpuset, the scheduler learns the cpuset’s allowed CPUs through cpuset_cpus_allowed() in kernel/cgroup/cpuset.c (cpuset.c v6.12):

void cpuset_cpus_allowed(struct task_struct *tsk, struct cpumask *pmask)
{
	spin_lock_irqsave(&callback_lock, flags);
	rcu_read_lock();
	cs = task_cs(tsk);
	if (cs != &top_cpuset)
		guarantee_online_cpus(tsk, pmask);   /* the cpuset's online CPUs */
	if ((cs == &top_cpuset) || cpumask_empty(pmask)) {
		const struct cpumask *possible_mask = task_cpu_possible_mask(tsk);
		cpumask_andnot(pmask, possible_mask, subpartitions_cpus); /* drop partition CPUs */
		if (!cpumask_intersects(pmask, cpu_online_mask))
			cpumask_copy(pmask, possible_mask);  /* last-resort fallback */
	}
	rcu_read_unlock();
	spin_unlock_irqrestore(&callback_lock, flags);
}

For a task in a real (non-top) cpuset, guarantee_online_cpus() writes the cpuset’s online allowed CPUs into pmask. For a task in the top cpuset, the allowed set is “all possible CPUs minus the CPUs that have been handed to subpartitions” (cpumask_andnot(pmask, possible_mask, subpartitions_cpus)) — this is exactly how creating an exclusive partition removes those CPUs from everyone else: the global subpartitions_cpus mask is subtracted from the top cpuset’s allowed set. This cpuset_cpus_allowed() result is precisely the mask that [[CPU Affinity and sched_setaffinity|__sched_setaffinity()]] intersects every user affinity request against, which is why a per-task sched_setaffinity can never escape the cpuset.

Rebuilding scheduling domains

A plain affinity change touches one task; a cpuset partition change restructures the scheduler’s scheduling domains for the whole machine. The trigger is rebuild_sched_domains_locked() (cpuset.c v6.12):

void rebuild_sched_domains_locked(void)
{
	...
	/* Generate domain masks and attrs */
	ndoms = generate_sched_domains(&doms, &attr);
	/* Have scheduler rebuild the domains */
	partition_and_rebuild_sched_domains(ndoms, doms, attr);
}

generate_sched_domains() builds a “partial partition of the system’s CPUs” — a set of non-overlapping CPU subsets — by scanning every cpuset that has load balancing enabled and merging those whose allowed CPUs overlap (a union-find), while skipping cpusets where load balancing is off. The comment in the source spells out the goal: produce “as many such domains as possible, each as small as possible.” The resulting array of cpumasks is handed to partition_sched_domains() in kernel/sched/core.c, which tears down and rebuilds the scheduler’s load-balancing domains to match. This is the load-bearing mechanism: an isolated partition’s CPUs are omitted from the load-balanced set, so generate_sched_domains() emits them as singleton (or no) domains, and partition_sched_domains() makes the load balancer stop touching them. Isolation is not a flag the balancer politely consults — it is the absence of those CPUs from the domain structure the balancer iterates.

rebuild_sched_domains_locked() also guards against racing CPU hotplug: if it detects that a partition root’s effective CPUs are no longer a subset of cpu_active_mask, it bails early and lets cpuset_handle_hotplug() redo the rebuild, avoiding handing an offline CPU to partition_sched_domains().

Configuration Walk-through

A complete example carving CPUs 4–7 into an exclusive partition with cgroup v2:

# 1. Enable the cpuset controller in the parent (root) cgroup
echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control
 
# 2. Create a child cgroup that will become a partition root
mkdir /sys/fs/cgroup/rt
echo "+cpuset" > /sys/fs/cgroup/rt/cgroup.subtree_control
 
# 3. Give it CPUs 4-7 and (for a local partition) optionally mark them exclusive
echo "4-7"  > /sys/fs/cgroup/rt/cpuset.cpus
echo "4-7"  > /sys/fs/cgroup/rt/cpuset.cpus.exclusive   # optional for a local partition
 
# 4. Promote it to a partition root — CPUs 4-7 now vanish from every other cpuset
echo root   > /sys/fs/cgroup/rt/cpuset.cpus.partition
 
# 5. Verify
cat /sys/fs/cgroup/rt/cpuset.cpus.partition   # -> "root"  (or "root invalid (...)")
cat /sys/fs/cgroup/rt/cpuset.cpus.effective   # -> "4-7"
cat /sys/fs/cgroup/cpuset.cpus.effective      # -> "0-3"  (4-7 removed from root)
 
# 6. For the lowest jitter, make a sub-partition isolated (no load balancing)
mkdir /sys/fs/cgroup/rt/poller
echo "+cpuset" > /sys/fs/cgroup/rt/cgroup.subtree_control
echo "6-7"     > /sys/fs/cgroup/rt/poller/cpuset.cpus
echo isolated  > /sys/fs/cgroup/rt/poller/cpuset.cpus.partition
 
# 7. Place a task and pin it within the isolated set by hand
echo $PID > /sys/fs/cgroup/rt/poller/cgroup.procs
taskset -pc 6 $PID      # isolated => no auto-balancing, so bind each thread yourself

Line by line: step 1 turns on the controller in the parent so children get cpuset.* files (cgroup v2’s explicit enablement model). Steps 2–3 create the cgroup and request CPUs 4–7. Step 4 is the pivotal write: promoting to root makes 4–7 exclusive — cpuset_cpus_allowed() for tasks elsewhere now subtracts subpartitions_cpus, and generate_sched_domains() splits 4–7 into their own domain. Step 5 confirms the partition is root (not root invalid) and that the root cgroup’s effective CPUs dropped to 0–3. Steps 6–7 add an isolated sub-partition with load balancing off; because the scheduler will not balance within it, you must taskset-pin each thread to a specific CPU yourself — exactly what the documentation warns about.

Interaction with isolcpus

The boot-time [[CPU Isolation isolcpus and nohz_full|isolcpus=]] parameter removes CPUs from the general load-balancing pool at boot, statically and irreversibly until reboot. Cpusets are the dynamic, finer-grained alternative — and the two interoperate by an explicit rule: “A user can pre-configure certain CPUs to an isolated state with load balancing disabled at boot time with the isolcpus kernel boot command line option. If those CPUs are to be put into a partition, they have to be used in an isolated partition” (cgroup-v2.rst v6.12). In other words, CPUs already isolated by isolcpus cannot be put into a load-balanced (root) partition — they can only join an isolated partition, because their boot-time state already says “no load balancing.” The cgroup-v1 default behavior was that “load balancing is done across all CPUs, except those marked isolated using the kernel boot time isolcpus= argument” (cpuset(7)). The modern guidance is to prefer cpuset partitions over isolcpus for new deployments because they are reconfigurable, but isolcpus still wins for the most static, lowest-jitter setups and is frequently combined with nohz_full.

Cgroup v1 vs v2

The cpuset controller exists in both cgroup hierarchies, but the interface differs. In v1 (Documentation/admin-guide/cgroup-v1/cpusets.rst) the relevant knobs are cpuset.cpus, cpuset.mems, cpuset.cpu_exclusive/cpuset.mem_exclusive (sibling-exclusivity flags), and cpuset.sched_load_balance/cpuset.sched_relax_domain_level for controlling balancing per cpuset (cpuset(7)). In v2 the per-cpuset sched_load_balance flag is gone; its job is subsumed by the cleaner cpuset.cpus.partition state machine (isolated = load balancing off) plus cpuset.cpus.exclusive. The cpu_exclusive/mem_exclusive flags are likewise folded into the partition/exclusivity model. Modern systemd-managed hosts use v2; the partition interface described here is v2-only.

Uncertain

Verify: the exact set of cgroup-v1 cpuset files and which were dropped vs renamed in v2. Reason: this note read the v2 cgroup-v2.rst and the cpuset(7) man page (which still documents v1-era flags like sched_load_balance, cpu_exclusive) but did not fetch Documentation/admin-guide/cgroup-v1/cpusets.rst at v6.12 directly. To resolve: read the v1 cpusets doc at the v6.12 tag and diff the file list against §Cpuset of cgroup-v2.rst. uncertain

Failure Modes and Common Misunderstandings

• root invalid (...). The most common surprise: you write root to cpuset.cpus.partition and it reads back root invalid (Parent is not a partition root) or (...exclusive CPUs is empty). The partition silently degrades to member-like behavior. Read the parenthesized reason and satisfy the validity conditions (parent must be a valid root for a local partition; cpuset.cpus.exclusive.effective must be non-empty).
• Exclusivity-rule write errors. Setting cpuset.cpus.exclusive to a CPU already exclusive in a sibling is rejected with a write error — “an exclusive CPU appearing in two or more of its child cgroups is not allowed.”
• Hotplug invalidation. Offlining a CPU that belongs to a partition root can flip it to root invalid. Watch the poll/inotify event on cpuset.cpus.partition rather than assuming a partition stays valid.
• Tasks idle on an isolated partition. Because an isolated partition has no load balancing, two threads dumped into a two-CPU isolated partition may both pile onto one CPU — the scheduler will not spread them. You must taskset-bind each one. This is by design, not a bug.
• Forgetting cpuset.mems causes cross-node thrash. A cpuset constrains CPUs but if cpuset.mems is left wide, tasks allocate memory on the wrong NUMA node and pay remote-access penalties — defeating the locality the cpuset was meant to provide. See NUMA Balancing and Task Placement.
• Confusing “member” with isolation. A member cpuset still shares its CPUs with everyone; it filters where its own tasks run, it does not keep other tasks out. Exclusivity requires a partition.

Alternatives and When to Choose Them

• [[CPU Affinity and sched_setaffinity|sched_setaffinity/taskset]] — per-task pinning, no memory-node control, no exclusivity, not inheritable. Use for a handful of threads inside an already-carved cpuset.
• Cpuset member — confine a cgroup of tasks (and their memory) to a CPU set without taking the CPUs from anyone. Use for NUMA locality and soft partitioning.
• Cpuset partition root — exclusive CPUs with load balancing. Use to dedicate a pool of cores to a workload while still letting the scheduler balance within that pool.
• Cpuset partition isolated — exclusive CPUs without load balancing. Use for the lowest-jitter latency-critical threads, hand-pinned.
• [[CPU Isolation isolcpus and nohz_full|isolcpus=]] boot parameter — static, boot-time isolation. Use for the most demanding static setups, often with nohz_full; cpuset partitions are the runtime-reconfigurable successor.

Production Notes

Container runtimes and Kubernetes drive cpusets heavily: the kubelet’s CPU Manager with the static policy assigns exclusive whole cores to Guaranteed-QoS pods by writing cpuset.cpus on the pod’s cgroup — the kernel mechanism is exactly this controller (how Kubernetes decides the assignment lives in Kubernetes MOC, not here; see also cgroups Integration). On NUMA hardware, HPC and database operators pair cpuset.cpus with cpuset.mems so a job’s threads and its pages live on the same node, avoiding the remote-memory penalty. Telecom and real-time deployments use isolated partitions (or isolcpus) to give dataplane poll loops uncontended cores. A recurring operational lesson: a partition silently going root invalid after a hotplug or a sibling’s exclusive-CPU edit is a classic “my isolation evaporated” incident — monitor cpuset.cpus.partition via its poll event in production.

See Also

• CPU Affinity and sched_setaffinity — per-task masks that the cpuset clamps via cpuset_cpus_allowed()
• CPU Isolation isolcpus and nohz_full — boot-time isolation; isolated partitions are the cgroup-native successor
• Housekeeping CPUs and Tickless Isolation — where displaced timers and unbound work go when CPUs are isolated
• Scheduler Load Balancing — what partition_sched_domains() reconfigures
• Scheduling Domains and CPU Topology — the domain structure cpusets rebuild
• NUMA Balancing and Task Placement — why cpuset.mems matters alongside cpuset.cpus
• The cgroup v2 CPU Controller — the sibling cpu controller (weight/bandwidth); cpuset is the placement controller
• cgroups Integration — how container runtimes wire cgroups together
• Linux Process Scheduling MOC — parent map (§F Affinity, Isolation, and Partitioning)