CPU Affinity and sched_setaffinity

CPU affinity is the kernel-enforced constraint that a task may run only on a specified subset of the machine’s CPUs. Each task carries an affinity mask, task_struct::cpus_mask, a bitmap of CPU numbers it is allowed to occupy; the scheduler honors it everywhere it decides where a task runs — at wakeup placement, at load balancing, at fork. Userspace sets the mask with the sched_setaffinity(2) system call (or the taskset(1) wrapper) and reads it back with sched_getaffinity(2). The subtlety that dominates this note is that since Linux 5.x–6.x the kernel keeps two masks: the user-requested mask (what you asked for, stored in user_cpus_ptr) and the effective mask (cpus_mask/cpus_ptr, what the task is actually allowed on after intersecting your request with the constraints of its cpuset). This split — landed by Will Deacon’s user_cpus_ptr introduction (commit b90ca8badbd1, 2021) and Waiman Long’s “Always preserve the user requested cpumask” (commit 8f9ea86fdf99, 2022) — is why an affinity request can survive a cpuset that temporarily takes CPUs away and reappear when they come back. Everything below is read at the v6.12 tag — this vault’s pin, and one of five longterm branches as of 2026-09-12 per kernel.org/releases.json (6.18.51, 6.12.109, 6.6.156, 6.1.187, 5.15.220). The affinity machinery has not moved since: existence-checking the two load-bearing lines forward through the tags, the cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr) masking block in __set_cpus_allowed_ptr() and the cpumask_and(mask, &p->cpus_mask, cpu_active_mask) in sched_getaffinity() are both present verbatim at v6.12, v6.18 and v7.1. Every runtime measurement in this note was taken on this host’s running kernel, 7.1.13-200.fc44.x86_64, a stable release — source claims are pinned to v6.12, measurements to 7.1.13, and the two are distinguished throughout.

Mental Model

Think of affinity as a filter applied to the set of CPUs the scheduler is willing to consider for a task. The task’s task_struct holds the filter as a cpumask_t bitmap. When any scheduler code asks “may this task run on CPU n?”, it consults this mask. The filter is enforced at three moments: when a sleeping task wakes up and the scheduler picks a CPU for it (wakeup placement via select_task_rq), when the periodic load balancer considers pulling a task to a less-busy CPU, and when a CPU goes offline and the task must be evicted somewhere legal.

The crucial mental refinement for modern kernels is that the mask you set is not necessarily the mask that takes effect. Your request is filtered a second time through the task’s cgroup cpuset: the effective mask is the intersection of what you asked for and what the cpuset permits. The kernel remembers your raw request separately so it can recompute the effective mask whenever the cpuset boundary moves.

flowchart TD
  U["userspace: sched_setaffinity(pid, mask)<br/>'I want CPUs {2,3,4,5}'"] --> SYS["SYSCALL_DEFINE3(sched_setaffinity)<br/>copy_from_user → new_mask"]
  SYS --> SSA["sched_setaffinity()<br/>save request → user_cpus_ptr<br/>flags = SCA_USER"]
  SSA --> US["__sched_setaffinity()<br/>cpuset_cpus_allowed(p) → {2,3,4,5,6,7}<br/>new_mask = request ∩ cpuset"]
  US --> EFF["effective mask {2,3,4,5}<br/>written to p->cpus_mask"]
  EFF --> PTR["p->cpus_ptr normally aliases &p->cpus_mask"]
  PTR --> STR["select_task_rq / load balance<br/>only ever pick a CPU in cpus_ptr"]
  CS["cpuset shrinks to {4,5}"] -.recompute.-> US2["effective recomputed:<br/>request {2,3,4,5} ∩ {4,5} = {4,5}<br/>request preserved in user_cpus_ptr"]

Two-stage filtering of an affinity request. What it shows: the syscall stores the raw request in user_cpus_ptr, then __sched_setaffinity() intersects it with the task’s cpuset-allowed set to produce the effective cpus_mask that the scheduler actually obeys. The insight to take: the request and the effective mask are distinct; when a cpuset later shrinks (dashed path), the effective mask is recomputed from the preserved request, so the task gives up CPUs gracefully and reclaims them if the cpuset expands again. The recomputation has a hole, though: it only happens when the intersection is non-empty, and when it is empty the task silently gets the entire cpuset instead. That case is the subject of Where the re-intersection actually happens below, and it is where this mechanism actually hurts people.

The Central Mechanism — A Mask Is a Constraint, Not a Placement

The single most common misreading of sched_setaffinity() is that it puts a thread somewhere. It does not. It removes CPUs from the set the scheduler is allowed to consider, and then the scheduler makes exactly the same decision it would have made anyway, restricted to the survivors. Two consequences follow immediately and both bite in practice: a mask with more than one CPU in it gives you no control at all over which of them you get, and a mask can be narrowed underneath you by machinery you did not invoke.

Both are directly observable. The program below reads its mask, optionally sets a new one, then spins for 300 ms calling sched_getcpu() between short bursts of work and tallies which CPU it was actually on. All runs are on the host used throughout this note: AMD Ryzen AI MAX+ 395, 16 physical cores / 32 logical CPUs, single NUMA node, kernel 7.1.13-200.fc44.x86_64.

=== 1. unpinned, 300ms ===
before: sched_getaffinity -> {0,...,31}  count=32
  61757 samples; observed CPUs: cpu2=19541(31.6%) cpu4=2894(4.7%) cpu5=1(0.0%)
                                cpu6=2614(4.2%) cpu7=206(0.3%) cpu18=36321(58.8%) cpu31=180(0.3%)

=== 2. sched_setaffinity to {4,5} ===
after:  sched_getaffinity -> {4,5}  count=2
  61776 samples; observed CPUs: cpu4=58074(94.0%) cpu5=3702(6.0%)

=== 3. sched_setaffinity to {6} ===
after:  sched_getaffinity -> {6}  count=1
  61823 samples; observed CPUs: cpu6=61823(100.0%)

=== 4. taskset -c 8-11 ./affin 300000   (no internal setaffinity call) ===
before: sched_getaffinity -> {8,9,10,11}  count=4
  58194 samples; observed CPUs: cpu8=58194(100.0%)

Run 1 is the baseline: with no constraint at all a single busy thread visited seven different CPUs in 300 ms. Run 2 is the point of the whole section — the mask {4,5} was honoured perfectly (the thread never appeared on any other CPU) and yet the split between the two allowed CPUs, 94% to 6%, is something nobody asked for and nobody controls. Run 4 is the same lesson in its starkest form: four CPUs were permitted and the scheduler used exactly one of them, all 58,194 samples on cpu8, because nothing ever made it worth migrating. Only run 3, where the mask has a single member, does the mask actually determine placement — and that is not affinity working differently, it is select_task_rq() short-circuiting because nr_cpus_allowed == 1 leaves nothing to decide.

The same thing is visible without any custom program, in the processor field of /proc/<pid>/stat — field 39, documented in proc_pid_stat(5) as “CPU number last executed on”, and written by seq_put_decimal_ll(m, " ", task_cpu(task)) in fs/proc/array.c at v6.12 (line 642). ps -o psr reads the same value.

=== /proc/<tid>/stat field 39 (processor) while pinned with taskset -c 12 ===
  pid 2192998 field39(psr)=12  field2(comm)=(sh) state=R
  pid 2192998 field39(psr)=12  field2(comm)=(sh) state=R
  pid 2192998 field39(psr)=12  field2(comm)=(sh) state=R
  ps -o psr: 12

=== unpinned busy loop: field 39 sampled every 10 ms for 1 s ===
  cpu18: 35 samples   cpu2: 32   cpu4: 15   cpu3: 5   cpu5: 3   cpu20: 2
  cpu19: 2            cpu17: 2   cpu6: 1    cpu29: 1  cpu13: 1  cpu0: 1

Sample, do not snapshot

The unpinned loop above touched twelve CPUs in one second. A single cat /proc/<pid>/stat would have returned 18 and looked exactly like a pinned thread. psr is a gauge read at one instant, not a summary — any claim of the form “the task is on CPU N” made from one read is a claim about your sampling rate, not about the task. Every placement number in this note is a histogram over at least 100 samples for this reason.

flowchart TD
  REQ["userspace request<br/>sched_setaffinity(pid, mask) or taskset -c"] --> SYS["SYSCALL_DEFINE3(sched_setaffinity)<br/>kernel/sched/syscalls.c:1320"]
  SYS --> SSA["sched_setaffinity()<br/>snapshot request into user_mask<br/>flags = SCA_USER"]
  SSA --> ISA["__sched_setaffinity()<br/>new_mask = request AND cpuset_cpus_allowed(p)"]
  ISA --> DL{"SCHED_DEADLINE task<br/>and admission control on?"}
  DL -- "mask loses a root-domain CPU" --> EB["-EBUSY, nothing installed"]
  DL -- "no, or mask still covers rd->span" --> INST["__set_cpus_allowed_ptr()<br/>-> set_cpus_allowed_common()"]
  INST --> MASK["p->cpus_mask = new_mask<br/>p->nr_cpus_allowed = weight(new_mask)<br/>SCA_USER: swap into p->user_cpus_ptr"]
  MASK --> PTR["p->cpus_ptr aliases &p->cpus_mask<br/>(unless inside migrate-disable)"]
  PTR --> SEL{"nr_cpus_allowed > 1<br/>and not migration-disabled?"}
  SEL -- "no: exactly one legal CPU" --> ONE["cpumask_any(p->cpus_ptr)<br/>MASK == PLACEMENT, the only such case"]
  SEL -- "yes: two or more legal CPUs" --> CLS["p->sched_class->select_task_rq()<br/>EEVDF wakeup balancer picks<br/>by cache warmth, idleness, sched domains"]
  CLS --> GUARD{"is_cpu_allowed(p, cpu)?"}
  GUARD -- "no" --> FB["select_fallback_rq()"]
  GUARD -- "yes" --> RUN["task runs there<br/>the SCHEDULER chose, you only filtered"]
  ONE --> RUN
  FB --> RUN

From a user request to an actual CPU. What it shows: the request is filtered twice before it becomes cpus_mask (cpuset intersection, deadline admission), and then cpus_mask is only ever consulted as a veto — select_task_rq() still runs the scheduling class’s chooser over whatever survives. The insight to take: there is exactly one branch in this whole diagram where your mask determines the answer, the nr_cpus_allowed == 1 short-circuit. Everywhere else you have narrowed a search space, not made a placement, which is why the measured split across a two-CPU mask was 94/6 rather than anything you asked for.

The SMT trap: two CPUs in a mask may be one core

Because a mask is expressed in logical CPU numbers, {4,20} and {4,5} look identical to sched_getaffinity — two CPUs each. On this machine they are not remotely equivalent:

$ cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
4,20
$ cat /sys/devices/system/cpu/cpu5/topology/thread_siblings_list
5,21

CPU 4 and CPU 20 are the two SMT threads of one physical core. A throughput-bound microbenchmark (eight independent integer accumulator chains, so it is issue-port limited rather than latency limited) run as two pinned processes:

=== solo on cpu4 (sibling 20 idle) ===
  cpu4      90.70 ms    17.64 Gops/s
=== pair on 4+5 (TWO physical cores) ===
  cpu4      82.17 ms    19.47 Gops/s
  cpu5      82.15 ms    19.48 Gops/s      aggregate 38.95 Gops/s
=== pair on 4+20 (ONE physical core, SMT siblings) ===
  cpu4     143.18 ms    11.17 Gops/s
  cpu20    143.07 ms    11.18 Gops/s      aggregate 22.35 Gops/s

Same mask cardinality, 43% less aggregate throughput and 1.74× the per-thread latency. The scheduler would not normally have made this mistake — select_task_rq() consults the SMT scheduling domain and prefers an idle physical core — but an explicit affinity mask overrides that judgement, because a mask is a hard constraint and the topology preference is only a heuristic. Read thread_siblings_list before writing any mask; see Scheduling Domains and CPU Topology for how the scheduler models this when you let it.

The generalisation is uncomfortable but correct: pinning replaces the scheduler’s topology-aware placement with your own, and you are then responsible for knowing the topology. On a machine with more than one NUMA node the same trap appears again at a coarser grain, where a “two CPU” mask can straddle two memory nodes — see NUMA Balancing and Task Placement.

Do not call TSC ticks "cycles" when reasoning about pinned threads

A natural next step after pinning is to measure with rdtsc, and on this machine that silently mis-scales every result. Measured here by timing a chain of dependent single-cycle add instructions against CLOCK_MONOTONIC and rdtscp over the same interval: the TSC runs at a fixed 2.9944–2.9945 GHz (identical on every core, under every load — it is constant_tsc and nonstop_tsc, both present in /proc/cpuinfo), while the cores under that load ran at 4.96–5.03 GHz on CPUs 4 and 5. Treating TSC ticks as cycles therefore under-reports by 39.7–40.5% on those cores. Two further traps: the ratio is not a constant — the same benchmark on CPU 8 reached only 4.35 GHz (TSC under-reporting 31.1%), because boost clocks vary per core and with thermal state — and a first attempt using a plain C loop rather than an explicit asm dependency chain measured 4.82 GHz and 37.9%, close enough to look right and wrong enough to matter. The TSC is an excellent clock and a bad cycle counter. See The Time Stamp Counter and TSC Reliability.

The task_struct Fields

In include/linux/sched.h at v6.12, the affinity state of every task is four adjacent fields (sched.h v6.12, lines 877–883):

int                     nr_cpus_allowed;   /* popcount of cpus_mask, cached */
const cpumask_t        *cpus_ptr;          /* the mask the scheduler reads */
cpumask_t              *user_cpus_ptr;     /* the user-requested mask, or NULL */
cpumask_t               cpus_mask;         /* the effective allowed mask */
void                   *migration_pending;
...
unsigned short          migration_disabled;

Walking the fields:

• cpus_mask is the effective affinity — the actual bitmap of CPUs the task may run on right now, after every constraint has been applied. It is a cpumask_t embedded directly in task_struct (not a pointer), so it always exists.
• cpus_ptr is a const cpumask_t * that the scheduler reads through whenever it needs the allowed set. In the normal case cpus_ptr == &cpus_mask — the pointer simply aliases the embedded mask. The indirection exists so the kernel can temporarily redirect the task to a single-CPU mask during a migrate-disable critical section (see below) without clobbering cpus_mask. This is the load-bearing detail: scheduler hot paths dereference cpus_ptr, not cpus_mask, precisely so migrate-disable can swap the pointer cheaply and restore it afterward.
• user_cpus_ptr is NULL until userspace calls sched_setaffinity(), at which point it points at a heap-allocated copy of the raw user request. This is the mask the user typed, before cpuset intersection. It lets the kernel recompute cpus_mask if the cpuset boundary moves.
• nr_cpus_allowed is the cached population count of cpus_mask. The scheduler checks nr_cpus_allowed > 1 as a fast gate before bothering to choose a CPU — a task pinned to exactly one CPU skips placement logic entirely.

Mechanical Walk-through: Setting Affinity

When userspace calls sched_setaffinity(pid, cpusetsize, mask), the kernel enters SYSCALL_DEFINE3(sched_setaffinity, …) in kernel/sched/syscalls.c (syscalls.c v6.12, line 1320). It allocates a kernel-side cpumask_var_t, copies the user bitmap in via get_user_cpu_mask(), and calls sched_setaffinity(pid, new_mask).

The inner sched_setaffinity() does the permission and bookkeeping work:

long sched_setaffinity(pid_t pid, const struct cpumask *in_mask)
{
	struct affinity_context ac;
	struct cpumask *user_mask;
	...
	if (p->flags & PF_NO_SETAFFINITY)        /* kernel per-CPU threads refuse */
		return -EINVAL;
	if (!check_same_owner(p)) {              /* not your task? */
		if (!ns_capable(__task_cred(p)->user_ns, CAP_SYS_NICE))
			return -EPERM;               /* need CAP_SYS_NICE */
	}
	...
	user_mask = alloc_user_cpus_ptr(NUMA_NO_NODE);
	if (user_mask)
		cpumask_copy(user_mask, in_mask);    /* snapshot the raw request */
	...
	ac = (struct affinity_context){
		.new_mask  = in_mask,
		.user_mask = user_mask,
		.flags     = SCA_USER,               /* "this is a user request" */
	};
	retval = __sched_setaffinity(p, &ac);
	kfree(ac.user_mask);
	return retval;
}

Line by line: PF_NO_SETAFFINITY is set on per-CPU kernel threads that must stay on their CPU; affinity changes to them are refused with -EINVAL. If the caller does not own the target task, it needs the CAP_SYS_NICE capability or the call fails -EPERM. The function then snapshots the raw request into a freshly allocated user_mask, packages everything into a struct affinity_context with the SCA_USER flag set — that flag is how the lower layers know this is an explicit user request (versus an internal kernel rebalance) and therefore should update user_cpus_ptr — and hands off to __sched_setaffinity().

The struct affinity_context (defined in kernel/sched/sched.h) and its SCA_* flags are the internal vocabulary of every affinity change:

struct affinity_context {
	const struct cpumask	*new_mask;   /* the mask to install */
	struct cpumask		*user_mask;  /* new user_cpus_ptr, if SCA_USER */
	unsigned int		flags;
};
#define SCA_CHECK		0x01   /* honor PF_NO_SETAFFINITY etc. */
#define SCA_MIGRATE_DISABLE	0x02   /* entering a migrate-disable region */
#define SCA_MIGRATE_ENABLE	0x04   /* leaving one */
#define SCA_USER		0x08   /* a userspace request; update user_cpus_ptr */

Those four bits are the kernel’s entire answer to “who is asking, and how much do we trust them?” — and the answer determines both whether user_cpus_ptr is written and whether it is read back:

Flag	Set by	Writes user_cpus_ptr?	Re-intersects new_mask with user_cpus_ptr?	Also implies
(none)	internal callers: cpuset’s update_tasks_cpumask(), hotplug, restrict_cpus_allowed_ptr()	no	yes, when the result is non-empty	—
SCA_CHECK 0x01	added by __sched_setaffinity()	no	(orthogonal)	honour PF_NO_SETAFFINITY, validate against online CPUs
SCA_MIGRATE_DISABLE 0x02	migrate_disable_switch()	no	no — skipped	redirect cpus_ptr to a one-CPU mask, leave cpus_mask alone
SCA_MIGRATE_ENABLE 0x04	migrate_enable()	no	no — skipped	restore cpus_ptr = &cpus_mask
SCA_USER 0x08	sched_setaffinity() only	yes, swap(p->user_cpus_ptr, ctx->user_mask)	no — this is the new request	—

The “re-intersects” column is the one that repays attention, because it is inverted from what the names suggest: the flag that means “a user asked for this” is precisely the flag that disables consulting the saved user request. That is correct — a fresh request replaces the old one rather than being filtered through it — but it means the saved request is only ever consulted on paths the user did not initiate, which is exactly the cpuset case examined later in this note.

The intersection: __sched_setaffinity

This is where the user request becomes the effective mask (syscalls.c v6.12, line 1194):

int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
{
	cpumask_var_t cpus_allowed, new_mask;
	...
	cpuset_cpus_allowed(p, cpus_allowed);            /* what the cpuset permits */
	cpumask_and(new_mask, ctx->new_mask, cpus_allowed);  /* request ∩ cpuset */
 
	ctx->new_mask = new_mask;
	ctx->flags |= SCA_CHECK;
 
	retval = dl_task_check_affinity(p, new_mask);    /* deadline-task admission */
	if (retval) goto out;
 
	retval = __set_cpus_allowed_ptr(p, ctx);         /* install the mask */
	...
	cpuset_cpus_allowed(p, cpus_allowed);
	if (!cpumask_subset(new_mask, cpus_allowed)) {   /* raced a cpuset change? */
		cpumask_copy(new_mask, cpus_allowed);    /* fall back to cpuset's set */
		...
		__set_cpus_allowed_ptr(p, ctx);
		retval = -EINVAL;
	}
	...
}

cpuset_cpus_allowed() (in kernel/cgroup/cpuset.c) returns the set of CPUs the task’s cpuset currently permits. cpumask_and() intersects that with the user’s request to form new_mask — this intersection is the entire reason the effective mask can differ from what you asked for. If the task is a SCHED_DEADLINE task, dl_task_check_affinity() enforces that the new mask still covers the task’s deadline root domain (you cannot strand a deadline task off its admitted CPUs). Reading that function at v6.12 (kernel/sched/syscalls.c, lines 1171–1191) pins the details the man page leaves vague: it is a no-op unless both task_has_dl_policy(p) and dl_bandwidth_enabled() hold, the test is cpumask_subset(task_rq(p)->rd->span, mask) — the new mask must be a superset of the whole root domain, not merely overlap it — and the error returned is -EBUSY, not the EINVAL you get from every other affinity rejection. __set_cpus_allowed_ptr() then writes new_mask into the task and, because SCA_USER is set, swaps the saved user_mask into p->user_cpus_ptr. The closing re-check handles a race where the cpuset changed concurrently: if the just-installed mask is no longer a subset of the cpuset’s allowed CPUs, the kernel resets to the cpuset’s set and returns -EINVAL.

Installing the mask: set_cpus_allowed_common

__set_cpus_allowed_ptr() ultimately calls set_cpus_allowed_common() (core.c v6.12, line 2644):

void set_cpus_allowed_common(struct task_struct *p, struct affinity_context *ctx)
{
	if (ctx->flags & SCA_MIGRATE_DISABLE) {
		__do_set_cpus_allowed(p, ctx);
		return;
	}
	cpumask_copy(&p->cpus_mask, ctx->new_mask);  /* the effective mask */
	p->nr_cpus_allowed = cpumask_weight(ctx->new_mask);
	if (ctx->flags & SCA_USER)
		swap(p->user_cpus_ptr, ctx->user_mask);  /* persist the request */
}

The effective bitmap lands in p->cpus_mask, nr_cpus_allowed is recomputed, and — only when SCA_USER is set — the user’s raw request is swapped into p->user_cpus_ptr. The old user_cpus_ptr (now in ctx->user_mask) is freed by the caller. This is the precise point where the “two masks” design is realized: internal kernel callers (load balancing, hotplug) set only cpus_mask; userspace requests additionally update user_cpus_ptr.

How Affinity Constrains Placement

Affinity is meaningless unless the scheduler obeys it. The single chokepoint is select_task_rq() (core.c v6.12):

int select_task_rq(struct task_struct *p, int cpu, int *wake_flags)
{
	if (p->nr_cpus_allowed > 1 && !is_migration_disabled(p)) {
		cpu = p->sched_class->select_task_rq(p, cpu, *wake_flags);
		*wake_flags |= WF_RQ_SELECTED;
	} else {
		cpu = cpumask_any(p->cpus_ptr);          /* pinned: just pick the one */
	}
	if (unlikely(!is_cpu_allowed(p, cpu)))
		cpu = select_fallback_rq(task_cpu(p), p);
	return cpu;
}

If nr_cpus_allowed == 1, the task is pinned and the scheduler short-circuits to cpumask_any(p->cpus_ptr) — no placement decision is made. Otherwise the scheduling class’s own select_task_rq (for fair tasks, the EEVDF wakeup balancer) chooses among CPUs, and that chooser is itself constrained to p->cpus_ptr — it never proposes a CPU outside the mask. As a final guard, is_cpu_allowed() re-checks the chosen CPU; if it is somehow illegal (offline, not in the mask), select_fallback_rq() finds a legal one. is_cpu_allowed() begins with task_allowed_on_cpu(), which tests membership in the mask — affinity is the first gate. The same cpus_ptr constraint governs the periodic load balancer: it will not migrate a task to a CPU outside its mask.

Reading affinity back, sched_getaffinity() returns cpumask_and(mask, &p->cpus_mask, cpu_active_mask) — the effective mask intersected with currently-active CPUs. Note this returns cpus_mask, not user_cpus_ptr: you read back what is in effect, not what you originally requested. If a cpuset has shrunk your effective set, sched_getaffinity reflects the shrunken set even though the kernel still remembers your broader request internally.

Migrate-Disable and the cpus_ptr Indirection

The cpus_ptr/cpus_mask split exists for migrate-disable, a PREEMPT_RT-era mechanism letting a task pin itself to its current CPU without holding a non-preemptible section. When a task enters migrate-disable, migrate_disable_switch() redirects p->cpus_ptr to a single-CPU mask (the current CPU) via __do_set_cpus_allowed(p, {.flags = SCA_MIGRATE_DISABLE}), leaving cpus_mask untouched. On migrate_enable(), the code restores p->cpus_ptr = &p->cpus_mask and, if the real affinity changed meanwhile, re-applies it (core.c v6.12, lines 2296–2362). Because every scheduler read goes through cpus_ptr, the task is transparently glued to one CPU for the duration with a single pointer swap. This is why the scheduler dereferences cpus_ptr rather than &cpus_mask directly.

stateDiagram-v2
    direction LR
    [*] --> Normal
    Normal: Normal operation
    Normal: cpus_ptr == &cpus_mask
    Normal: nr_cpus_allowed == weight(cpus_mask)
    Normal: scheduler may place anywhere in the mask
    Pinned: Inside migrate_disable()
    Pinned: cpus_ptr -> a single-CPU mask
    Pinned: cpus_mask UNCHANGED underneath
    Pinned: select_task_rq short-circuits
    Normal --> Pinned: migrate_disable_switch()<br/>__do_set_cpus_allowed(p, SCA_MIGRATE_DISABLE)
    Pinned --> Normal: migrate_enable()<br/>cpus_ptr = &cpus_mask<br/>re-apply if cpus_mask changed meanwhile
    Normal --> Normal: sched_setaffinity(SCA_USER)<br/>rewrites cpus_mask and user_cpus_ptr
    Pinned --> Pinned: sched_setaffinity during the region<br/>updates cpus_mask only<br/>takes effect on migrate_enable()

The cpus_ptr indirection as a two-state machine. What it shows: migrate-disable never touches cpus_mask — it retargets the pointer the scheduler reads through, so the task’s real affinity is preserved untouched and restored by a single assignment. The insight to take: this is why an affinity change issued while a task is inside a migrate-disable region appears not to take effect immediately; it lands in cpus_mask, which nothing is reading, and becomes visible only at migrate_enable(). It is also why SCA_MIGRATE_DISABLE/SCA_MIGRATE_ENABLE are listed alongside SCA_USER in the __set_cpus_allowed_ptr() masking guard — all three mean “do not re-intersect with the saved user request.”

Persistence Across cpuset Changes

The reason for user_cpus_ptr is captured by Waiman Long’s commit 8f9ea86fdf99 “sched: Always preserve the user requested cpumask” (2022-09-22): “user provided CPU affinity via sched_setaffinity(2) is preserved even if the task is being moved to a different cpuset” (LWN 909123, commit 8f9ea86fdf99). Concretely: you pin a task to CPUs {2,3,4,5}; later the task’s cpuset is narrowed to {4,5}. Without the saved request the kernel would have to overwrite cpus_mask to {4,5} and forget you ever wanted {2,3}. With user_cpus_ptr holding {2,3,4,5}, the kernel recomputes the effective mask as the intersection {4,5} but keeps your request; if the cpuset later widens back to {2,3,4,5,6,7}, the task’s effective mask is recomputed as {2,3,4,5} again — you reclaim exactly the CPUs you asked for, no more, no less.

The kernel functions that drive this are force_compatible_cpus_allowed_ptr() (restrict a task’s affinity to a subset and stash the old mask in user_cpus_ptr) and relax_compatible_cpus_allowed_ptr() (restore the saved request once the constraint lifts), both in core.c. The original use case, from Will Deacon’s commit b90ca8badbd1 (2021), was asymmetric arm64: some 32-bit applications can only run on the subset of cores that support the AArch32 instruction set, so the kernel must transparently restrict their affinity while remembering the broader 64-bit request (commit b90ca8badbd1). The cpuset use case was layered on top later.

Where the re-intersection actually happens — and when it silently gives up

The prose above describes the intent. The implementation is one if at the top of __set_cpus_allowed_ptr() in kernel/sched/core.c at v6.12 (lines 3100–3116), and reading it carefully changes the picture:

int __set_cpus_allowed_ptr(struct task_struct *p, struct affinity_context *ctx)
{
	rq = task_rq_lock(p, &rf);
	/*
	 * Masking should be skipped if SCA_USER or any of the SCA_MIGRATE_*
	 * flags are set.
	 */
	if (p->user_cpus_ptr &&
	    !(ctx->flags & (SCA_USER | SCA_MIGRATE_ENABLE | SCA_MIGRATE_DISABLE)) &&
	    cpumask_and(rq->scratch_mask, ctx->new_mask, p->user_cpus_ptr))
		ctx->new_mask = rq->scratch_mask;
 
	return __set_cpus_allowed_ptr_locked(p, ctx, rq, &rf);
}

Three conditions must all hold for the saved user request to be re-applied. p->user_cpus_ptr must be non-NULL — i.e. userspace has called sched_setaffinity() at least once; a task that never did has nothing to preserve and the whole block is skipped. The caller must not be a user request or a migrate-disable transition: SCA_USER means “this is the new request, do not re-intersect it with the old one.” And — the load-bearing clause — cpumask_and() must return true, which it does only when the result is non-empty.

That last condition is the failure mode nobody expects. Suppose a task asks for {2,3} and is later moved into a cpuset limited to {8-15}. The cpuset path calls set_cpus_allowed_ptr(task, {8-15}) with flags == 0, so the masking block is entered, cpumask_and({8-15}, {2,3}) is empty and returns false — and the block is skipped. ctx->new_mask stays {8-15}, which is what gets installed. The user’s request is not honoured, not partially honoured, and not reported: the task is simply given the whole cpuset. sched_getaffinity() will report {8-15}, the same as every other task in that cpuset, with no indication that {2,3} was ever requested. The request does survive in user_cpus_ptr, so if the cpuset later widens back to include CPU 2 or 3 the intersection becomes non-empty again and the request reasserts itself — but in the meantime the task is running unpinned across eight CPUs while its author believes it is pinned to two.

sequenceDiagram
  autonumber
  participant U as Your code
  participant K as kernel/sched
  participant CS as cgroup cpuset
  participant T as task_struct
  U->>K: sched_setaffinity(tid, {2,3})
  K->>CS: cpuset_cpus_allowed(p)
  CS-->>K: {0-31}, the root cpuset
  K->>T: cpus_mask = {2,3} AND {0-31} = {2,3}
  K->>T: user_cpus_ptr = {2,3} (SCA_USER was set)
  U->>K: sched_getaffinity(tid)
  K-->>U: {2,3} -- looks correct
  Note over CS: an orchestrator moves the task into a cpuset limited to {8-15}
  CS->>K: update_tasks_cpumask -> set_cpus_allowed_ptr(task, {8-15}), flags = 0
  K->>K: cpumask_and({8-15}, user_cpus_ptr {2,3}) is EMPTY, returns false
  Note right of K: masking block skipped, no error path taken
  K->>T: cpus_mask = {8-15}
  U->>K: sched_getaffinity(tid)
  K-->>U: {8-15} -- your pinning is gone, silently
  Note over CS: later the cpuset widens back to {0-31}
  CS->>K: set_cpus_allowed_ptr(task, {0-31}), flags = 0
  K->>K: cpumask_and({0-31}, {2,3}) = {2,3}, non-empty
  K->>T: cpus_mask = {2,3} -- the request reasserts itself

How a cpuset silently overrides an affinity request, and how it later comes back. What it shows: the override is not a conflict the kernel resolves in your favour or reports — it is an internal caller writing cpus_mask through a path that only consults user_cpus_ptr when the intersection happens to be non-empty. The insight to take: sched_setaffinity() returning 0 is a statement about the mask at that instant, not a durable guarantee. If your pinning must hold, either own the cpuset too or re-assert the mask after any cgroup migration — and note that step 12 is also why “it started working again on its own” is a real and confusing symptom.

Two corollaries worth stating plainly. First, sched_getaffinity() is the only thing userspace can read, and it returns cpus_mask & cpu_active_mask — never user_cpus_ptr. There is no interface at all for reading back what you originally asked for; the kernel remembers it and will not tell you. Second, the ordering means a container runtime’s cpuset is a hard ceiling on anything a process inside the container does with taskset, which is the mundane explanation for the overwhelming majority of “affinity is being ignored” reports.

Demonstration blocked: cpuset is not delegated to user sessions on this host

I could measure everything else in this note directly, but not the cpuset override, because the cpuset controller is not delegated to unprivileged users here. cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers returns cpu io memory pids — no cpuset — and mkdir under that path returns Permission denied. The failure is instructive in its own right: systemd-run --user --scope -p AllowedCPUs=4,5 … accepts the property, starts the unit, and does nothing. The resulting scope’s cgroup directory has no cpuset.cpus.effective file at all, so there is no cpuset to violate and no error to report. A property that is silently inert is the same hazard as a mask that is silently overridden, arriving from the opposite direction. The narrative above is therefore read from the v6.12 source (__set_cpus_allowed_ptr(), update_tasks_cpumask(), cpuset_cpus_allowed()) rather than executed. To resolve: re-run as root with cgroup_no_v1=all, create a cgroup with cpuset.cpus = 8-15, move a task pinned to {2,3} into it, and read sched_getaffinity before and after. uncertain

Userspace: cpu_set_t and taskset

The C API in <sched.h> represents a mask as the opaque cpu_set_t, manipulated by macros (sched_setaffinity(2)):

#define _GNU_SOURCE
#include <sched.h>
 
cpu_set_t set;
CPU_ZERO(&set);              /* clear all bits */
CPU_SET(2, &set);            /* allow CPU 2 */
CPU_SET(3, &set);            /* allow CPU 3 */
if (sched_setaffinity(0, sizeof(set), &set) == -1)  /* 0 = calling thread */
	perror("sched_setaffinity");
 
CPU_ZERO(&set);
sched_getaffinity(0, sizeof(set), &set);
if (CPU_ISSET(2, &set)) puts("running set may include CPU 2");

CPU_ZERO/CPU_SET/CPU_CLR/CPU_ISSET are the bitmap accessors; the glibc cpu_set_t is a fixed 128-byte structure covering CPUs 0–1023. On machines with more than 1024 CPUs the static type is too small and sched_setaffinity returns EINVAL; you must allocate a dynamic mask with CPU_ALLOC(3) and pass its actual byte size as cpusetsize. The kernel side uses variable-length unsigned long * bitmaps, so the syscall’s cpusetsize/len argument is the negotiation point between the two.

The taskset(1) command is the CLI wrapper (taskset(1)):

taskset -c 2,3 ./myprog          # launch myprog pinned to CPUs 2 and 3
taskset -pc 0-3 1234             # repin running PID 1234 to CPUs 0-3 (-c = list form)
taskset -pc 1234                 # read PID 1234's current affinity as a list
taskset -apc 0,1 1234            # -a: apply to ALL threads of PID 1234

-c/--cpu-list accepts human ranges (0-3, 0,5,8-11, stride 0-10:2) instead of a hex bitmask; -p operates on an existing PID; -a applies to every thread of the process rather than the single thread named by the PID. Internally taskset is just sched_setaffinity/sched_getaffinity.

The whole affinity API surface, and who wins

Four different interfaces can constrain which CPUs a thread runs on. They are not alternatives at the same level — they compose, and they compose asymmetrically. The table below is the thing to internalize before debugging any “my pinning didn’t work” report.

Interface	Granularity	Underlying mechanism	Survives fork/exec?	Position in the precedence chain
sched_setaffinity(2)	one thread (the pid argument is a TID)	writes cpus_mask with SCA_USER, stores the raw request in user_cpus_ptr	inherited by fork and preserved across exec	lowest — it is intersected with everything above it
pthread_setaffinity_np(3)	one thread	thin glibc wrapper: resolves the pthread_t to a kernel TID and calls sched_setaffinity. Returns the error number directly instead of setting errno; the man page notes it migrates the thread immediately if it is not already on an allowed CPU	same as above	identical to sched_setaffinity — it is sched_setaffinity
taskset(1)	one thread, or with -a every thread of a process	sched_setaffinity, nothing more	as above	identical — a CLI, not a policy layer
cgroup v2 cpuset (cpuset.cpus)	every task in a cgroup subtree, inherited by new members	update_tasks_cpumask() in kernel/cgroup/cpuset.c iterates the cgroup’s tasks and calls set_cpus_allowed_ptr() on each — an internal caller, so no SCA_USER	applies to any task in the cgroup, including ones created later	highest — a task can never run outside cpuset.cpus.effective

“Highest” is not a figure of speech. __sched_setaffinity() begins by calling cpuset_cpus_allowed(p, cpus_allowed) and then cpumask_and(new_mask, ctx->new_mask, cpus_allowed), so a user request is born intersected. And because the cpuset path never sets SCA_USER, a cpuset change does not merely win the argument, it never has the argument — it rewrites cpus_mask through a different door.

There is one more constraint below all four, easy to forget because it never fails loudly: the mask is also intersected with the CPUs that exist. sched_getaffinity() returns cpumask_and(mask, &p->cpus_mask, cpu_active_mask) — the effective mask AND the currently active CPUs, so an offlined CPU silently disappears from what you read back. And on the way in, a request containing CPUs that do not exist is quietly trimmed rather than rejected, as long as something survives:

=== A. mask containing only a non-existent CPU (99) ===
taskset: failed to set pid 2199640's affinity: Invalid argument
=== B. mask {4,99} -> intersection with online ===
Cpus_allowed_list:	4
=== C. per-CPU kernel thread (PF_NO_SETAFFINITY), migration/0 ===
pid 19's current affinity list: 0
taskset: affinity cannot be set due to PF_NO_SETAFFINITY flag set
taskset: failed to set pid 19's affinity: Invalid argument
=== D. another user's task (pid 1, root-owned systemd) ===
taskset: failed to set pid 1's affinity: Operation not permitted
=== F. /proc/self/status under taskset -c 8-11 ===
Cpus_allowed:	00000f00
Cpus_allowed_list:	8-11

Run B is the one to remember: asking for {4, 99} on a 32-CPU machine succeeds and silently yields {4}. Only when the intersection is empty (run A) do you get EINVAL. Silent narrowing is the rule, hard failure the exception — the same philosophy that makes the cpuset interaction so easy to miss. Runs C and D confirm the two refusal cases in prose below: PF_NO_SETAFFINITY on a per-CPU kernel thread (util-linux is helpful enough to name the flag), and EPERM without CAP_SYS_NICE on a task you do not own. Run F shows the two /proc/<pid>/status fields that report the effective mask in both hex and list form — the quickest way to check a running process without writing code.

Hard vs Soft Affinity — and Why There Is No SCHED_FLAG For It

The affinity described here is hard: a CPU outside the mask is forbidden, full stop, even if every allowed CPU is saturated and other CPUs sit idle. A long-proposed alternative is soft affinity, where the mask is a preference — the task runs on its preferred CPUs when they are available but is permitted to spill onto idle CPUs elsewhere under contention. Jonathan Corbet’s 2019 LWN write-up describes a patch set adding a sched_setaffinity2() syscall with SCHED_HARD_AFFINITY and SCHED_SOFT_AFFINITY flags (LWN 792502).

That work was never merged. As of 6.12 LTS there is no soft-affinity flag — not in sched_setaffinity, not as a SCHED_FLAG_* on sched_setattr(2). The reviewers found the performance case unconvincing (gains under 7%, noisy data) and questioned whether improving AutoNUMA would be the better lever. So the only affinity primitive userspace has is the hard mask of sched_setaffinity. The closest thing to a “preference” the kernel actually ships is the wakeup-placement heuristics inside the fair-class balancer, which prefer cache-warm CPUs but are not user-controllable per task.

Uncertain

Verify: that no soft-affinity SCHED_FLAG_* or sched_setaffinity2() exists in 6.18 LTS (this note confirms its absence in 6.12 and that the 2019 proposal was rejected, but does not re-scan 6.18’s uapi/linux/sched.h flag list). Reason: soft affinity is a recurring proposal that could in principle reappear; checked 6.12 source and 2019 LWN, not 6.18 uapi headers. To resolve: grep SCHED_FLAG_ in include/uapi/linux/sched.h at the v6.18 tag. uncertain

Failure Modes and Common Misunderstandings

Every outcome of an affinity change, in one place. The “observed” column marks the rows reproduced live on this host (kernel 7.1.13); the rest are read from the v6.12 source and marked as such.

Symptom	sched_setaffinity result	Cause	Observed?
Request contains only absent/offline CPUs	EINVAL, mask unchanged	intersection with cpu_active_mask/cpuset is empty	yes — taskset -c 99 on a 32-CPU box
Request contains some absent CPUs	success, silently trimmed	same intersection, non-empty result	yes — taskset -c 4,99 gave Cpus_allowed_list: 4
Mask “ignored”, getaffinity shows the cgroup’s set	success, then silently overridden	cpuset clamp, or the empty-intersection hole in __set_cpus_allowed_ptr()	no — cpuset not delegated here, see the callout above
Target is a per-CPU kernel thread	EINVAL	PF_NO_SETAFFINITY	yes — taskset -pc 0-31 19 (migration/0)
Target belongs to another user	EPERM	no CAP_SYS_NICE, check_same_owner() fails	yes — taskset -pc 0-3 1 (root’s systemd)
Target is a SCHED_DEADLINE task, mask does not cover rd->span	EBUSY	dl_task_check_affinity() — note the different errno	source only
Static cpu_set_t on a >1024-CPU machine	EINVAL	glibc’s fixed 128-byte mask cannot express CPU ≥ 1024; use CPU_ALLOC	source/man only — this host has 32 CPUs
Cpuset changed concurrently with the call	EINVAL, but the mask is reset to the cpuset’s set	the closing re-check in __sched_setaffinity()	source only
Call succeeds, thread still wanders among the allowed CPUs	success, working as designed	a mask is a constraint, not a placement	yes — 94/6 split across {4,5}

• EINVAL from an empty intersection. If your requested mask shares no CPU with the task’s cpuset-allowed set (or with online CPUs), the intersection is empty and sched_setaffinity returns EINVAL (“contains no processors that are currently physically on the system and permitted to the thread”). The affinity is left unchanged.
• “My affinity didn’t take” — silent cpuset clamping. You ask for {0-15} but the task is in a cpuset limited to {8-15}; the syscall succeeds and sched_getaffinity reports {8-15}. The kernel “silently imposes” the cpuset restriction (cpuset(7)). This is not a bug — it is the intersection by design. The full request still lives in user_cpus_ptr.
• EPERM setting another user’s task. Changing affinity on a task you do not own requires CAP_SYS_NICE; without it you get EPERM.
• EINVAL on >1024-CPU machines with static cpu_set_t. The glibc fixed mask cannot express CPU numbers above 1023; use CPU_ALLOC.
• Pinning a kernel per-CPU thread fails. Threads with PF_NO_SETAFFINITY (per-CPU kthreads, the migration thread) refuse affinity changes with EINVAL.
• Confusing affinity with isolation. Pinning a task to a CPU does not keep other tasks off it. To get a CPU to yourself you must also keep the rest of the system away — that is what isolcpus, housekeeping CPUs, and exclusive cpuset partitions are for. Affinity is one half of isolation, never the whole.

Alternatives and When to Choose Them

• sched_setaffinity / taskset — per-task pinning, the lightest tool. Use for ad-hoc pinning of a few threads. Does not stop others from using the CPU.
• cpusets — confine a whole cgroup of tasks (and their memory nodes) to a CPU set, and with v2 partitions carve exclusive CPUs. Use when you want a durable, hierarchical, inheritable boundary rather than per-task masks.
• isolcpus= boot parameter — remove CPUs from the general scheduler’s load-balancing pool at boot. Use for the most demanding low-jitter workloads, in combination with affinity to place your task on the cleared CPUs.

These compose: a common production pattern is isolcpus to clear the cores, an exclusive cpuset partition to own them, and sched_setaffinity to place the specific threads. Side by side:

	sched_setaffinity / taskset	cgroup v2 cpuset	isolcpus= boot parameter
Scope	one thread (one TID)	a cgroup subtree, inherited by new members	machine-wide, fixed at boot
When applied	any time, immediately	any time, immediately, to every task in the subtree	boot only — requires a reboot to change
Keeps others off the CPU?	no	yes, with an exclusive partition	yes, removes the CPU from load balancing
Privilege needed	none for your own threads	write access to the cgroup — typically root or a delegated subtree	root, plus a reboot
Survives being moved between cgroups?	request survives in user_cpus_ptr, effect may not	it is the cgroup, so yes	yes
Failure style	EINVAL/EPERM, or silent trimming	silent clamping of everything below it	none — it simply is not there
Use it for	ad-hoc placement of a few threads	durable, hierarchical, inheritable boundaries	the lowest-jitter workloads, as a foundation for the other two

Reading down the “keeps others off” row is the single most useful thing in this table: it is the one property affinity does not have and the one people most often assume it does.

Production Notes

High-frequency-trading, telecom dataplane (DPDK), and real-time control workloads routinely pin polling threads to dedicated cores so the thread never migrates and its cache and TLB stay warm. The standard recipe is not affinity alone — it is affinity layered on isolated CPUs so nothing else competes. The cpus_ptr indirection matters here: under PREEMPT_RT, a thread holding a per-CPU resource uses migrate-disable to stay put without disabling preemption, which is why the pointer is the read path and not the embedded mask. A frequent operational gotcha is forgetting that container runtimes set cpusets: a thread that “ignores” its sched_setaffinity is usually being clamped by the cgroup cpuset its container lives in.

See Also

What this note owns. The sched_setaffinity/sched_getaffinity syscall pair and the mask itself: the cpus_mask/cpus_ptr/user_cpus_ptr/nr_cpus_allowed fields, the struct affinity_context and SCA_* vocabulary, the intersection rules in __sched_setaffinity() and __set_cpus_allowed_ptr(), the full userspace API surface and its precedence order, and the measured claim that a mask constrains without placing. It does not own: how a task is physically moved once the mask changes — a running task needs the stop-class migration thread while a sleeping one is merely relinked on wakeup, which is Migration the Stop Class and the Migration Thread’s subject; the run-queue structures the mask filters over (Per-CPU Run Queues and struct rq); the cpuset controller’s own semantics, partitions and cpuset.cpus.exclusive (Cpusets and CPU Partitioning, The cgroup v2 CPU Controller); the boot-time removal of CPUs from load balancing (CPU Isolation isolcpus and nohz_full); or the SMT/NUMA topology model the scheduler uses when you do not pin (Scheduling Domains and CPU Topology).

• Scheduling Domains and CPU Topology — what thread_siblings_list means to the scheduler, and why an SMT-sibling mask defeats a placement heuristic rather than a rule
• The Time Stamp Counter and TSC Reliability — why rdtsc on a pinned thread is a clock and not a cycle counter
• Cpusets and CPU Partitioning — the cgroup controller that supplies cpuset_cpus_allowed() and clamps the effective mask
• CPU Isolation isolcpus and nohz_full — the boot-time way to clear CPUs of other work; affinity places tasks onto them
• Scheduler Load Balancing — obeys cpus_ptr when migrating tasks
• Wakeup Balancing and select_task_rq — where affinity gates wakeup placement
• Housekeeping CPUs and Tickless Isolation — where displaced work goes
• The task_struct Process Descriptor — home of cpus_mask/cpus_ptr/user_cpus_ptr
• Migration the Stop Class and the Migration Thread — PF_NO_SETAFFINITY, the migration kthread
• SCHED_DEADLINE and Earliest Deadline First — why deadline tasks resist affinity restriction
• Linux Process Scheduling MOC — parent map (§F Affinity, Isolation, and Partitioning)