Migration, the Stop Class and the Migration Thread

Linux keeps a separate run queue per CPU, so a runnable task lives on exactly one CPU’s queue at a time. Moving a task from one CPU to another — migration — is therefore not a free pointer swap: it must dequeue the task from the source run queue, change its recorded CPU, and enqueue it on the destination, all under the right locks. For a task that is merely queued and not currently executing, the kernel does this directly with move_queued_task() while holding the source rq->lock. But a task that is currently running on its CPU cannot be relocated out from under itself — you cannot rewrite a task’s CPU while its registers and kernel stack are live on that CPU. Linux solves this with the stop scheduling class (stop_sched_class), the highest-priority class in the system, embodied per CPU by the migration/N kthread (the cpu_stopper.thread). To move a running task, the kernel queues work to that CPU’s stopper; the stopper, having absolute priority, instantly preempts the victim, which forces it off the CPU — and now migration_cpu_stop() can safely move the no-longer-running task (core.c v6.12 L2409–2419, “This is how migration works”). This note traces that machinery as of Linux 6.12 (released 2024-11-17; still a supported longterm branch at 6.12.109 as of 2026-09-12, per kernel.org/releases.json).

The One Idea

If you take away a single sentence, take this one: the stop class is the kernel’s mechanism for “stop everything, briefly.” It is not a priority level that competes with other priority levels; it sits outside the ordinary priority contest. A task in the stop class beats a SCHED_DEADLINE task, beats a SCHED_FIFO priority-99 task, beats everything — by construction, not by being a bigger number. That single property is reused at two very different scales:

One CPU. stop_one_cpu() monopolises one CPU for the duration of one callback. This is how a running task gets migrated, and it costs a couple of context switches on one CPU. Everything else in the machine keeps running.
Every CPU. stop_machine() recruits all the stoppers at once and drives them through a lockstep state machine until every online CPU is sitting in a tight loop with interrupts disabled. That is total, machine-wide exclusion — the strongest guarantee the kernel can offer, and the most expensive thing it can do outside of a panic.

Those two uses share one kthread and one class, and confusing them is the single most common error about this subsystem. The stop class is both the ultimate hammer and a latency disaster, and which of those it is depends entirely on whether you called stop_one_cpu() or stop_machine().

It is worth holding this next to the other end of the exclusion spectrum. Read-write semaphores are among the lightest exclusion primitives the kernel has: an uncontended acquire is one atomic instruction on one word, and readers do not exclude each other at all. stop_machine() is the heaviest: it excludes every thread on every CPU, including interrupt handlers, and the cost scales with the number of CPUs. Both are answers to “make sure nobody else touches this,” separated by roughly six orders of magnitude in cost. The kernel’s whole synchronisation design is an effort to answer that question with the leftmost tool that works.

What This Note Owns

Notes in this vault overlap by design but must not duplicate. This note owns: the stop scheduling class as an object, the cpu_stopper / migration/N kthread, the stop_machine() family and its real users, and the mechanics of relocating a task between run queues (move_queued_task, set_task_cpu, migration_cpu_stop). It deliberately does not re-teach: the general class-cascade and the real-time priority number spaces (owned by Real-Time Scheduling SCHED_FIFO and SCHED_RR), struct rq and the enqueue/dequeue path (owned by Per-CPU Run Queues and struct rq), the mechanics of the switch itself (owned by Context Switch Mechanics switch_to and switch_mm), or the policy question of when a task should move (owned by Scheduler Load Balancing).

All code claims are pinned to v6.12 and cited to raw.githubusercontent.com/torvalds/linux/v6.12/... with line numbers from that tag. Whole-tree claims (such as the exhaustive list of stop_machine() callers below) were produced by extracting the v6.12 tarball and grepping it, not from memory. Where a fact is dated (when the stop class appeared, when the dedicated migration thread died) the date comes from the cgit history of kernel/stop_machine.c.

Mental Model

The right way to think about this is a two-tier scheme keyed on a single question: is the task I want to move currently running on a CPU, or just sitting in a run queue?

flowchart TB
  START["Want task p on CPU A<br/>to end up on CPU B"] --> Q{"Is p currently<br/>running on a CPU?"}
  Q -->|"No — queued only"| MQT["move_queued_task() under rq->lock:<br/>deactivate on A → set_task_cpu(B)<br/>→ activate on B"]
  Q -->|"Yes — running on A"| STOP["queue migration_cpu_stop<br/>to CPU A's stopper via<br/>stop_one_cpu()"]
  STOP --> PREEMPT["migration/A kthread<br/>(stop_sched_class, top priority)<br/>preempts p — p leaves the CPU"]
  PREEMPT --> RUN["migration_cpu_stop() now runs;<br/>p is no longer running<br/>→ __migrate_task() → move_queued_task()"]
  MQT --> DONE["p enqueued on CPU B"]
  RUN --> DONE

The migration decision tree. What it shows: a queued task takes the cheap direct path (move_queued_task holds the source run-queue lock the whole time); a running task takes the expensive path through the per-CPU stopper kthread, whose sole job here is to evict the victim by virtue of outranking it, after which the same move_queued_task machinery completes the move. The insight to take: the stop class exists precisely to convert a “running task” into a “not-running task” so the cheap path becomes usable — the actual relocation logic is identical in both branches.

Where the Stop Class Sits — and Why You Cannot See It

The scheduler is a stack of scheduling classes, each a const struct sched_class with a common method table. pick_next_task() walks that stack from the top and takes the first class that offers a runnable task; a class only gets asked if every class above it declined. The stop class is at the top, so it is asked first, always.

As of v6.12 there are six classes, not five — sched_ext was merged in this very release (existence-checked: kernel/sched/ext.c returns HTTP 404 at tag v6.11 and HTTP 200 at v6.12), and this machine’s Fedora kernel has CONFIG_SCHED_CLASS_EXT=y. Their order and what each one’s position means in practice:

#	Class	Policies it serves	Who is in it	What its position means in practice
1	`stop_sched_class`	none — no `SCHED_*` policy maps here	exactly one task per CPU: the `migration/N` stopper	Runs the instant it is woken. Cannot be created, joined, yielded, or re-prioritised. This is the only way the kernel can be sure something runs right now on a chosen CPU.
2	`dl_sched_class`	`SCHED_DEADLINE`	EDF-scheduled tasks with an admitted bandwidth reservation	Beats all RT and CFS work, but is itself preemptible by a stopper — so a deadline task can and does get evicted for a migration.
3	`rt_sched_class`	`SCHED_FIFO`, `SCHED_RR`	classic fixed-priority real-time tasks, priority 1–99	Even `SCHED_FIFO` priority 99 — the highest priority userspace can name — is below the stop class.
4	`fair_sched_class`	`SCHED_OTHER`, `SCHED_BATCH`, `SCHED_IDLE`	essentially every normal process	The default home; see The EEVDF Scheduler.
5	`ext_sched_class`	`SCHED_EXT`	tasks handed to a BPF scheduler	New in v6.12. Sits below fair, so a BPF scheduler cannot starve normal work.
6	`idle_sched_class`	none	the per-CPU `swapper/N` idle task	The floor: always has something to “run”.

The v6.12 class stack, top to bottom. What it shows: the stop class is not “priority 100” — it has no policy number and no priority at all, which is precisely why nothing can compete with it. The insight to take: rows 2 and 3 are the interesting ones. A SCHED_DEADLINE task with a hard bandwidth reservation, or a SCHED_FIFO 99 task in a real-time audio pipeline, will still be thrown off its CPU the moment a stopper is queued there. That is the latency cost of the stop class, and no userspace configuration can opt out of it.

The ordering is not stored in an array or a priority field that you can grep for — it is enforced by the linker. DEFINE_SCHED_CLASS(name) puts each class object into its own named section, and the linker script concatenates those sections in a hard-coded order:

/* kernel/sched/sched.h v6.12, L2499-2502 */
#define DEFINE_SCHED_CLASS(name) \
const struct sched_class name##_sched_class \
	__aligned(__alignof__(struct sched_class)) /* STRUCT_ALIGN(), vmlinux.lds.h */ \
	__section("__" #name "_sched_class")

/* include/asm-generic/vmlinux.lds.h v6.12, L129-138 */
#define SCHED_DATA				\
	STRUCT_ALIGN();				\
	__sched_class_highest = .;		\
	*(__stop_sched_class)			\
	*(__dl_sched_class)			\
	*(__rt_sched_class)			\
	*(__fair_sched_class)			\
	*(__ext_sched_class)			\
	*(__idle_sched_class)			\
	__sched_class_lowest = .;

Reading these together: every class object lands in a distinct input section, and SCHED_DATA (pulled into .rodata by RO_DATA, vmlinux.lds.h L454) emits them back-to-back in exactly the order shown, bracketed by the symbols __sched_class_highest and __sched_class_lowest. The class stack is therefore a contiguous array in memory, and “walk the classes in priority order” is a plain pointer increment — for_class_range() at sched.h L2543. The priority order of Linux’s scheduling classes is, quite literally, link order. (The class cascade itself, and the priority number spaces inside the RT class, are developed in Real-Time Scheduling SCHED_FIFO and SCHED_RR; this note only needs to establish that the stop class is above all of it.)

Demonstration: enumerating the stoppers on this box

On the machine this note was written on — Fedora 44, kernel 7.1.13-200.fc44.x86_64, 32 logical CPUs — there is exactly one stopper per CPU, and ps will show you its scheduling parameters:

$ nproc
32
$ ps -eo pid,class,rtprio,pri,psr,comm | awk 'NR==1 || /migration/'
    PID CLS RTPRIO PRI PSR COMMAND
     19 FF      99 139   0 migration/0
     25 FF      99 139   1 migration/1
     31 FF      99 139   2 migration/2
     ...
    206 FF      99 139  31 migration/31
$ ps -eo comm | grep -c '^migration/'
32

Read this carefully, because it disproves the naive expectation. There is no ps class code for the stop class. ps reports CLS=FF — SCHED_FIFO — with RTPRIO=99. /proc/19/stat agrees: field 41 (policy) is 1, which is SCHED_FIFO, and field 40 (rt_priority) is 99.

$ awk '{printf "comm=%s prio=%s rt_priority=%s policy=%s\n",$2,$18,$40,$41}' /proc/19/stat
comm=(migration/0) prio=-100 rt_priority=99 policy=1

This is deliberate, and the source says why. sched_set_stop_task() first runs the kthread through the normal sched_setscheduler path as a SCHED_FIFO task at MAX_RT_PRIO - 1 (= 99), and only then overwrites the class pointer behind the scheduler’s back:

/* kernel/sched/core.c v6.12, L3553-3571 */
void sched_set_stop_task(int cpu, struct task_struct *stop)
{
	static struct lock_class_key stop_pi_lock;
	struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
	struct task_struct *old_stop = cpu_rq(cpu)->stop;
 
	if (stop) {
		/*
		 * Make it appear like a SCHED_FIFO task, its something
		 * userspace knows about and won't get confused about.
		 *
		 * Also, it will make PI more or less work without too
		 * much confusion -- but then, stop work should not
		 * rely on PI working anyway.
		 */
		sched_setscheduler_nocheck(stop, SCHED_FIFO, &param);
 
		stop->sched_class = &stop_sched_class;

So the policy field is a deliberate lie for userspace’s benefit: p->policy stays SCHED_FIFO, p->prio stays 0 (the numerically-smallest, i.e. strongest, kernel priority — which is why /proc/PID/stat reports prio = -100, since task_prio() returns p->prio - MAX_RT_PRIO), while p->sched_class quietly points at stop_sched_class. The scheduler dispatches on sched_class; every userspace-facing interface reports policy. They disagree, on purpose.

You can confirm the prio arithmetic with a neighbouring kthread that really is an ordinary RT task — the thermal idle_inject threads run at SCHED_FIFO 50:

$ awk '{printf "comm=%s prio=%s rt_priority=%s policy=%s\n",$2,$18,$40,$41}' /proc/21/stat
comm=(idle_inject/0) prio=-51 rt_priority=50 policy=1

99 - 50 = 49, and 49 - 100 = -51. The same formula applied to the stopper gives 99 - 99 = 0, 0 - 100 = -100. The stopper is reported as being exactly as privileged as a SCHED_FIFO 99 task. It is not — it is strictly above one — but nothing in /proc will tell you that.

The practical consequence

You cannot detect the stop class from userspace, and you cannot create work in it. There is no SCHED_STOP policy to pass to sched_setscheduler(); switched_to_stop() is a BUG(). The only membership is by kernel fiat, one task per CPU. If you are trying to reason about worst-case latency on a box, the stop class is an invisible term in the equation — chrt, ps and /proc will all tell you the highest-priority thing on the system is SCHED_FIFO 99, and they will all be wrong.

On this particular host the point is doubly sharp, because RLIMIT_RTPRIO is 0 (ulimit -r → 0) and CONFIG_RT_GROUP_SCHED is not set in /boot/config-7.1.13-200.fc44.x86_64. So an unprivileged user here cannot create any real-time task at all, let alone stop-class work — every experiment below is done from the SCHED_OTHER side of the fence, observing the stoppers rather than competing with them.

Why a Running Task Needs the Stop Class

A task_struct records its current CPU (read via task_cpu(p), written via set_task_cpu()), and that CPU’s run queue holds the scheduling-class bookkeeping for it. To migrate the task you must atomically (with respect to the scheduler) take it off the source queue and put it on the destination queue. If the task is parked in a run queue but not executing, the migrating CPU simply grabs rq->lock and does the surgery: nothing else can touch the task because run-queue operations all serialize on that lock.

A running task is different. Its register state, its kernel stack pointer, and the per-CPU current pointer all reflect that it is live on its CPU right now. You cannot just rewrite task_cpu(p) to point at another CPU — the task is still physically executing here. The only safe way to move it is to first make it stop running, and the cleanest way to guarantee that is to run something on its CPU that outranks it absolutely. That something is the stop class.

The stop class’s defining property is stated in its own source file: “The stop task is the highest priority task in the system, it preempts everything and will be preempted by nothing” (stop_task.c v6.12). Recall the scheduling-class stack walked by pick_next_task(): stop → deadline → rt → fair → ext → idle (see Scheduling Classes and the sched_class Interface and Scheduler Entry Points and pick_next_task). Because stop_sched_class is first, the instant the stopper kthread becomes runnable on a CPU, the next scheduling decision on that CPU picks it — preempting even a SCHED_FIFO real-time task or a SCHED_DEADLINE task. The victim is now off the CPU, demoted from “running” to “queued (or blocked)”, and the ordinary move_queued_task() path applies.

The migration/N kthread Is the cpu_stopper

The kthread you see as migration/0, migration/1, … in ps output is not a dedicated “migration thread” in the sense of a thread whose only purpose is scheduler migration. It is the generic per-CPU CPU stopper thread, and migration is merely its most frequent customer. The naming is historical: the stopper subsystem grew out of the original migration thread.

The stopper is defined as a per-CPU structure and a per-CPU “smpboot” hotplug thread:

struct cpu_stopper {
	struct task_struct	*thread;
	raw_spinlock_t		lock;
	bool			enabled;	/* is this stopper enabled? */
	struct list_head	works;		/* list of pending works */
	struct cpu_stop_work	stop_work;	/* for stop_cpus */
	unsigned long		caller;
	cpu_stop_fn_t		fn;
};
 
static DEFINE_PER_CPU(struct cpu_stopper, cpu_stopper);
 
static struct smp_hotplug_thread cpu_stop_threads = {
	.store			= &cpu_stopper.thread,
	.thread_should_run	= cpu_stop_should_run,
	.thread_fn		= cpu_stopper_thread,
	.thread_comm		= "migration/%u",
	.create			= cpu_stop_create,
	.park			= cpu_stop_park,
	.selfparking		= true,
};

From stop_machine.c v6.12. Line by line: each CPU has a cpu_stopper holding a pointer to its kthread, a lock, an enabled flag (cleared while the CPU is parked/offline), and a works list of pending stop jobs. thread_comm = "migration/%u" is exactly why the thread shows up as migration/0 etc. — %u is filled with the CPU number. The smp_hotplug_thread registration creates one such kthread per CPU at boot and re-creates it on CPU hotplug.

The crucial wiring that makes the stopper the stop-class task is in cpu_stop_create():

static void cpu_stop_create(unsigned int cpu)
{
	sched_set_stop_task(cpu, per_cpu(cpu_stopper.thread, cpu));
}

sched_set_stop_task() installs this kthread as rq->stop for that CPU and switches it into stop_sched_class. So the identity is exact: the migration/N kthread, the cpu_stopper.thread for CPU N, and the task that stop_sched_class schedules on CPU N are all the same task_struct. They are not three separate objects.

The stop class itself is deliberately minimal. Its methods either do trivial accounting or actively forbid the operation — a stop task may never yield, may never be created by userspace, and may never have its priority changed:

static void yield_task_stop(struct rq *rq)
{
	BUG(); /* the stop task should never yield, its pointless. */
}
static void switched_to_stop(struct rq *rq, struct task_struct *p)
{
	BUG(); /* its impossible to change to this class */
}

From stop_task.c v6.12. The BUG() calls encode invariants: nothing in the kernel should ever try to demote, yield, or re-prioritize a stopper. Its pick_task_stop() simply returns rq->stop if the stopper is runnable, and select_task_rq_stop() returns the task’s current CPU unconditionally — stop tasks never migrate, which makes sense because a per-CPU stopper is meaningless on any other CPU.

The Stopper API Surface

Before tracing a migration, it is worth laying out the whole set of entry points the stopper subsystem exposes, because “queue work to one stopper” and “freeze the machine” are the same mechanism at different widths. Everything below is in kernel/stop_machine.c at v6.12, and every one of them ultimately puts a struct cpu_stop_work on some cpu_stopper.works list and wakes the corresponding migration/N kthread.

Entry point	v6.12 line	CPUs monopolised	Blocks caller?	Representative user
`stop_one_cpu(cpu, fn, arg)`	L139	1	yes — waits for completion	`migration_cpu_stop` from `set_cpus_allowed_ptr()`, `sched_exec()`, `migrate_task_to()`
`stop_one_cpu_nowait(cpu, fn, arg, work_buf)`	L384	1	no — fire and forget	active load balancing; `balance_push()` evicting tasks from a dying CPU (`core.c` L7935); `migration_cpu_stop` re-queuing itself after a race
`stop_two_cpus(cpu1, cpu2, fn, arg)`	L334	2, simultaneously	yes	`migrate_swap()` — NUMA balancing exchanging two running tasks (`core.c` L3369)
`stop_machine(fn, data, cpus)`	L622	all online CPUs	yes	MTRR updates, microcode load, clocksource switch
`stop_machine_cpuslocked(fn, data, cpus)`	L586	all online CPUs	yes	same, when the caller already holds the CPU-hotplug read lock
`stop_core_cpuslocked(cpu, fn, data)`	L635	all SMT siblings of one core	yes	Intel In-Field Scan (`drivers/platform/x86/intel/ifs/`) — added 2022-05-12, first shipped in v5.19
`stop_machine_from_inactive_cpu(fn, data, cpus)`	L677	all online CPUs	yes	x86 cache-info rendezvous from a CPU that is not yet in `cpu_online_mask`

The stopper API, widest-to-narrowest exclusion. What it shows: one subsystem serves both the cheapest per-CPU preemption and the most expensive machine-wide freeze. The insight to take: the top three rows are scheduler operations and cost a context switch or two; the bottom four are machine operations and cost every CPU in the system a stall. stop_core_cpuslocked is the interesting middle ground, added only in 2022 because Intel’s in-field-scan feature needs both SMT siblings of a core quiesced but has no reason to disturb the other 30 CPUs.

The stop_core_cpuslocked dating is an existence check, not a recollection: drivers/platform/x86/intel/ifs/runtest.c returns HTTP 404 at tag v5.18 and HTTP 200 at v5.19, and stop_core_cpuslocked appears zero times in kernel/stop_machine.c at v5.18 and twice at v5.19.

How a Running Task Actually Gets Moved

The canonical job the stopper runs for migration is migration_cpu_stop(). The high-level sequence is documented in the comment block in core.c:

we invoke migration_cpu_stop() on the target CPU using stop_one_cpu().

stopper starts to run (implicitly forcing the migrated thread off the CPU)

it checks whether the migrated task is still in the wrong runqueue.

if it’s in the wrong runqueue then the migration thread removes it and puts it into the right queue.

stopper completes and stop_one_cpu() returns and the migration is done.

(core.c v6.12). Walk it concretely. A typical caller — e.g. set_cpus_allowed_ptr() reacting to a changed affinity mask, or sched_exec() rebalancing on execve() — fills a struct migration_arg { task, dest_cpu, … } and calls:

stop_one_cpu(task_cpu(p), migration_cpu_stop, &arg);

stop_one_cpu(cpu, fn, arg) runs fn(arg) on cpu “in a process context with the highest priority preempting any task on the cpu and monopolizing it” and blocks until it completes. Its kerneldoc adds two things worth knowing: it may sleep (the caller blocks on a completion, so this is not callable from atomic context), and it returns -ENOENT if the work was never executed because the target CPU was offline — a return value callers routinely ignore and occasionally should not (stop_machine.c v6.12 kerneldoc). Internally it builds a cpu_stop_work, queues it on the target CPU’s cpu_stopper.works list, and wakes that CPU’s stopper:

static void __cpu_stop_queue_work(struct cpu_stopper *stopper,
				  struct cpu_stop_work *work,
				  struct wake_q_head *wakeq)
{
	list_add_tail(&work->list, &stopper->works);
	wake_q_add(wakeq, stopper->thread);
}

Because the woken stopper is in the top class, the destination CPU’s next __schedule() picks it, preempting whatever ran. The stopper’s main loop pulls the work and calls the function with preemption disabled (stop callbacks must not sleep):

static void cpu_stopper_thread(unsigned int cpu)
{
	/* ... pull first work off stopper->works ... */
	if (work) {
		cpu_stop_fn_t fn = work->fn;
		preempt_count_inc();
		ret = fn(arg);          /* this is migration_cpu_stop() */
		if (done) { ...; cpu_stop_signal_done(done); }
		preempt_count_dec();
		goto repeat;
	}
}

Now migration_cpu_stop() itself runs on the source CPU, which means the victim is no longer running there — the stopper displaced it. It takes p->pi_lock and the run-queue lock, re-validates that p is still on this run queue (affinity may have changed again, or the task may have blocked), and if so performs the move:

if (task_on_rq_queued(p)) {
	update_rq_clock(rq);
	rq = __migrate_task(rq, &rf, p, arg->dest_cpu);
} else {
	p->wake_cpu = arg->dest_cpu;
}

From core.c v6.12. If the task is queued, __migrate_task() (after a final is_cpu_allowed() check) calls move_queued_task(). If the task has since blocked (not on the run queue), there is nothing to dequeue — the kernel just records wake_cpu so the task wakes on the destination later (see Wakeups and try_to_wake_up).

Laid out as a sequence across the three participants, the whole dance is:

sequenceDiagram
    autonumber
    participant C as Caller<br/>(e.g. taskset on CPU 3)
    participant V as Victim task p<br/>RUNNING on CPU 7
    participant S as migration/7<br/>(stop_sched_class)
    participant D as CPU 11 run queue

    C->>C: set_cpus_allowed_ptr(p, {11})<br/>takes p->pi_lock + rq(7) lock
    Note over C: p is running and 7 is<br/>no longer in the new mask
    C->>S: stop_one_cpu(7, migration_cpu_stop, &arg)<br/>queue cpu_stop_work, wake_q_add
    C-->>C: blocks on completion
    Note over S: stopper is now runnable on CPU 7
    S-->>V: IPI / reschedule on CPU 7
    V->>V: __schedule(): pick_next_task walks<br/>classes from the top, finds rq->stop
    V-->>S: context switch — p is now merely QUEUED
    S->>S: cpu_stopper_thread(): preempt_count_inc(),<br/>then fn(arg) is migration_cpu_stop()
    S->>S: take p->pi_lock + rq(7) lock, re-check<br/>task_on_rq_queued(p) and is_cpu_allowed()
    S->>D: __migrate_task() then move_queued_task():<br/>deactivate on 7, set_task_cpu(11), activate on 11
    S->>D: wakeup_preempt() — should p preempt CPU 11's current?
    S-->>C: cpu_stop_signal_done() fires the completion
    C->>C: stop_one_cpu() returns — migration done

The full running-task migration, step by step. What it shows: five distinct things must happen in order — queue the work, wake the stopper, let the stopper preempt the victim, do the relocation, signal completion — and only step 9 onward is “the migration”; steps 3–7 exist purely to demote the victim from running to queued. The insight to take: count the context switches. CPU 7 switches into the stopper and then out of it again, and the calling CPU blocks for the whole round trip. That is the price of moving a task that is currently on-CPU, and it is why the load balancer will do almost anything to move a queued task instead (see Scheduler Load Balancing).

move_queued_task and set_task_cpu — The Shared Relocation Core

Both branches converge on move_queued_task(), which is the entire relocation primitive and the same code the cheap queued-task path uses directly:

static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf,
				   struct task_struct *p, int new_cpu)
{
	lockdep_assert_rq_held(rq);
 
	deactivate_task(rq, p, DEQUEUE_NOCLOCK);
	set_task_cpu(p, new_cpu);
	rq_unlock(rq, rf);
 
	rq = cpu_rq(new_cpu);
 
	rq_lock(rq, rf);
	WARN_ON_ONCE(task_cpu(p) != new_cpu);
	activate_task(rq, p, 0);
	wakeup_preempt(rq, p, 0);
 
	return rq;
}

From core.c v6.12. Reading it: the source rq->lock must already be held (lockdep_assert_rq_held). deactivate_task() dequeues p from the source class’s run queue and clears its on-rq state. set_task_cpu(p, new_cpu) changes the recorded CPU — and along the way invokes the class hook migrate_task_rq (for fair tasks this discounts the task’s vruntime against the new cfs_rq’s baseline), bumps p->se.nr_migrations, and fires rseq/perf/mm_cid migration notifications. The source lock is dropped, the destination lock taken, and activate_task() enqueues p on the new CPU. Finally wakeup_preempt() checks whether the newly arrived task should preempt the destination’s current task. The function returns the locked destination rq — the caller’s lock has been handed forward to a different run queue, a subtlety the caller must track.

set_task_cpu() enforces the locking discipline that makes all of this safe:

WARN_ON_ONCE(debug_locks && !(lockdep_is_held(&p->pi_lock) ||
			      lockdep_is_held(__rq_lockp(task_rq(p)))));

From core.c v6.12. The rule: to change a task’s CPU you must hold either the task’s pi_lock (used when waking a blocked task — see Wakeups and try_to_wake_up) or the run-queue lock (used when the task is runnable). This is exactly why a queued task can be migrated cheaply (you hold the rq lock) but a running task cannot be migrated by a remote CPU without first stopping it (the remote CPU does not own the live execution state).

Demonstration: Watching the Two Paths Diverge

The two-tier scheme from the Mental Model is not a teaching simplification — it is directly observable with nothing more exotic than taskset and /proc. Two counters make it visible:

/proc/PID/sched → se.nr_migrations is incremented inside set_task_cpu() every time a task’s recorded CPU changes. It counts actual relocations.
/proc/<stopper-pid>/sched → nr_switches counts how many times that migration/N kthread has been context-switched. Because the stopper sleeps unless work is queued, a rise here means the stopper genuinely ran.

First, pick two CPUs that are not SMT siblings, because placing a “before and after” experiment on two hyperthreads of one core would confuse migration with sibling scheduling:

$ cat /sys/devices/system/cpu/cpu7/topology/thread_siblings_list
7,23
$ cat /sys/devices/system/cpu/cpu11/topology/thread_siblings_list
11,27

CPU 7 and CPU 11 are on different physical cores. (On this box, CPUs 4 and 20 are siblings of one core — always read thread_siblings_list before drawing conclusions from a placement experiment.)

A running task: the stopper does the work

Start a task that is genuinely running, pin it to CPU 7, then force it to CPU 11 by changing its affinity out from under it:

$ taskset -c 7 sh -c 'while :; do :; done' & PID=$!
$ ps -o pid,psr,class,comm,pcpu -p $PID
    PID PSR CLS COMMAND         %CPU
1987881   7 TS  sh              98.5
$ awk '/nr_migrations/{print $3}' /proc/$PID/sched
1
$ taskset -p -c 11 $PID
pid 1987881's current affinity list: 7
pid 1987881's new affinity list: 11
$ ps -o pid,psr,class,comm,pcpu -p $PID
    PID PSR CLS COMMAND         %CPU
1987881  11 TS  sh              99.2
$ awk '/nr_migrations/{print $3}' /proc/$PID/sched
2

The task was burning 98.5% of a CPU — it was definitively running, not queued — and it moved from PSR 7 to PSR 11, with nr_migrations going 1 → 2. That single increment is one pass through set_task_cpu(), reached via stop_one_cpu(7, migration_cpu_stop, &arg) → the stopper preempting the shell → __migrate_task() → move_queued_task().

The controlled comparison

The interesting measurement is the contrast. Run twenty forced affinity flips between CPU 7 and CPU 11, once against a busy-looping task and once against a sleeping one, sampling the two stoppers’ nr_switches on either side:

$ # --- RUNNING (busy-loop) task: 20 forced affinity flips 7<->11 ---
task nr_migrations: 1 -> 21   (delta 20)
migration/7  nr_switches: 4059219 -> 4059233  (delta 14)
migration/11 nr_switches: 4583497 -> 4583508  (delta 11)
 
$ # --- SLEEPING task (taskset -c 7 sleep 300): the same 20 flips ---
task nr_migrations: 2 -> 2    (delta 0)
migration/7  nr_switches: 4059251 -> 4059251  (delta 0)
migration/11 nr_switches: 4583518 -> 4583518  (delta 0)

This is the whole note in six lines of output.

Against the running task, twenty affinity changes produced exactly twenty migrations, and the two stopper kthreads woke a combined 25 times across the same window — the stoppers on precisely the two CPUs involved, and nowhere near enough background noise to explain it. The kernel had to evict a running task twenty times, and each eviction is a stopper activation.

Against the sleeping task, twenty identical taskset calls produced zero migrations and zero stopper wakeups. Nothing was running, so there was nothing to preempt. set_cpus_allowed_ptr() took the task’s pi_lock, saw !task_on_rq_queued(p), updated p->cpus_mask, and — per the migration_cpu_stop() logic quoted above — simply recorded where the task should land when it next wakes. sleep(3) never woke during the experiment, so set_task_cpu() was never called and nr_migrations never moved.

flowchart LR
  subgraph run["20 flips against a RUNNING task"]
    A1["taskset changes<br/>cpus_mask 20×"] --> A2["task is on-CPU<br/>→ must be evicted"]
    A2 --> A3["stop_one_cpu() × 20"]
    A3 --> A4["migration/7 + migration/11<br/>nr_switches +25"]
    A4 --> A5["nr_migrations +20"]
  end
  subgraph slp["20 flips against a SLEEPING task"]
    B1["taskset changes<br/>cpus_mask 20×"] --> B2["task is not on any rq<br/>→ nothing to evict"]
    B2 --> B3["record p->wake_cpu only"]
    B3 --> B4["stopper nr_switches +0"]
    B4 --> B5["nr_migrations +0"]
  end

The measured divergence, drawn. What it shows: identical userspace actions (twenty taskset -p -c calls) produce completely different kernel work depending on one bit of task state — whether the task is currently on a run queue. The insight to take: “changing a task’s affinity” is not one operation with one cost. Against a sleeping task it is a mask write. Against a running task it is twenty round trips through the highest-priority scheduling class in the system. If you are writing an affinity-management daemon, this is the difference between free and expensive, and you have no direct control over which one you get.

Uncertain

Verify: that the combined +25 stopper nr_switches is entirely attributable to these twenty migrations rather than partly to unrelated background stop work on CPUs 7 and 11. Reason: with /sys/kernel/debug mode 0700 and no perf, bpftool or bpftrace installed on this host, there is no way to attribute individual stopper activations to individual cpu_stop_work items. The direction and magnitude of the effect are unambiguous (25 versus 0 across otherwise identical windows), but the exact count is not. To resolve: perf trace -e sched:sched_migrate_task or a kprobe on migration_cpu_stop, both of which need perf/bpftrace plus root.

What could not be demonstrated here

Honesty about the limits matters as much as the results:

Wanted demonstration	Blocked by
Read `/sys/kernel/debug/sched/` to see the stop class in the run-queue dump	`/sys/kernel/debug` is mode `0700`, owner `root`; no passwordless `sudo`
`trace-cmd`/`perf` record of `sched:sched_migrate_task` and `sched_switch` into `migration/N`	`perf`, `trace-cmd`, `bpftrace`, `bpftool` all not installed; `/sys/kernel/tracing` is also `0700`
Create a `SCHED_FIFO` task and show a stopper preempting it	`ulimit -r` → `0` (`RLIMIT_RTPRIO` is 0), so no RT priority can be requested at all
Trigger a real `stop_machine()` (CPU offline, MTRR write, microcode load) and time the stall	every one of these requires root
Show RT throttling interacting with the stopper	`CONFIG_RT_GROUP_SCHED` is not set in this kernel’s config — the feature does not exist here

The measurements that are above use only ps, taskset and /proc, all of which work unprivileged, which is why they are the ones reported.

Who Calls Into the Stopper for Migration

Several distinct kernel paths end in stopper work, and they differ in which helper they use and what they are trying to achieve. Conflating them is easy because they all funnel into migration_cpu_stop() or a close relative:

Caller	v6.12 location	Stopper helper	What it is for	Notes
`set_cpus_allowed_ptr()`	`core.c` L3118	`stop_one_cpu()` (via `__set_cpus_allowed_ptr_locked`, L3012)	the task’s allowed mask shrank and it is running on a now-forbidden CPU	the `sched_setaffinity()` / `taskset` path; see CPU Affinity and sched_setaffinity
`sched_exec()`	`core.c` L5443	`stop_one_cpu()`	rebalance at `execve()`, when the task’s cache and memory footprint are smallest	asks `select_task_rq(..., WF_EXEC)` first and returns early if the answer is the current CPU
`migrate_task_to()`	`core.c` L7784	`stop_one_cpu()`	NUMA balancing moving a task toward its memory	`CONFIG_NUMA_BALANCING` only — the function does not exist otherwise. Emits `trace_sched_move_numa`. See NUMA Balancing and Task Placement
`migrate_swap()`	`core.c` L3369	`stop_two_cpus()`	NUMA balancing exchanging two running tasks	needs both CPUs quiesced at once, hence `stop_two_cpus`; emits `trace_sched_swap_numa`
`balance_push()`	`core.c` L7883, queue at L7935	`stop_one_cpu_nowait()`	evict every movable task from a CPU that is going offline	runs as an `rq->balance_callback` out of `__schedule()`; the CPU-down half of hotplug
active load balancing	`kernel/sched/fair.c` L11878, `active_load_balance_cpu_stop`	`stop_one_cpu_nowait()`	pull the running task off an overloaded CPU when pulling queued tasks was not enough	the last-resort branch of periodic balancing; see Scheduler Load Balancing

Migration entry points. What it shows: five of the six are policy decisions (“this task would be better off elsewhere”) and one — balance_push — is a correctness requirement (“this CPU is dying, everything must leave”). The insight to take: migrate_task_to() and migrate_swap() are both NUMA-balancing-only and both compiled out without CONFIG_NUMA_BALANCING; if you are reading a kernel without it, the only routine producers of running-task migration are affinity changes, execve and active balancing.

Pinned and Migration-Disabled Tasks

Not every task may be moved to any CPU. Migration is constrained by:

The affinity mask (p->cpus_ptr / p->cpus_mask). A task pinned to one CPU (sched_setaffinity to a single bit) can never be migrated elsewhere — is_cpu_allowed() and __migrate_task()’s is_cpu_allowed() guard reject the move. Per-CPU kthreads (KTHREAD_IS_PER_CPU) are pinned by construction. See CPU Affinity and sched_setaffinity.
migration_disabled. Code inside a migrate_disable() / migrate_enable() section must not be moved even though preemption may be allowed. migration_cpu_stop() explicitly bails — if (is_migration_disabled(p)) goto out; — and the set_affinity_pending dance retries once migration is re-enabled. This is the modern, finer-grained replacement for blanket preempt_disable() in many PREEMPT_RT paths.
The stopper task itself never migrates (select_task_rq_stop returns the current CPU).

stop_machine — The Heavier Sibling, Not the Migration Path

It is tempting to conflate stop_machine with migration because both live in kernel/stop_machine.c and both ride the cpu_stopper infrastructure. They are different, and migration does not use stop_machine.

stop_one_cpu() (what migration uses) monopolises one CPU to run a function. Other CPUs keep running normally. Cost: two context switches on one CPU.
stop_machine() halts every online CPU simultaneously to reach global quiescence. Cost: every CPU in the machine stops dead for the duration, with interrupts disabled.

The rendezvous state machine

stop_machine_cpuslocked() builds one struct multi_stop_data and queues multi_cpu_stop to every stopper in the mask (stop_machine.c L586-620). Every participating CPU then runs the same function and advances through a shared four-state machine in lockstep, coordinated by a single atomic counter:

/* kernel/stop_machine.c v6.12, L157-196 */
enum multi_stop_state {
	MULTI_STOP_NONE,		/* Dummy starting state for thread. */
	MULTI_STOP_PREPARE,		/* Awaiting everyone to be scheduled. */
	MULTI_STOP_DISABLE_IRQ,		/* Disable interrupts. */
	MULTI_STOP_RUN,			/* Run the function */
	MULTI_STOP_EXIT,		/* Exit */
};
 
static void set_state(struct multi_stop_data *msdata,
		      enum multi_stop_state newstate)
{
	/* Reset ack counter. */
	atomic_set(&msdata->thread_ack, msdata->num_threads);
	smp_wmb();
	WRITE_ONCE(msdata->state, newstate);
}
 
/* Last one to ack a state moves to the next state. */
static void ack_state(struct multi_stop_data *msdata)
{
	if (atomic_dec_and_test(&msdata->thread_ack))
		set_state(msdata, msdata->state + 1);
}

Line by line, this is a barrier built out of one counter. set_state() arms thread_ack with the number of participating threads, issues an smp_wmb() so the counter reset is visible before the new state, and publishes the state. Every CPU that observes a new state does its work for that state and calls ack_state(), which decrements the counter; whichever CPU decrements it to zero is the one that advances everybody to the next state. No CPU can be in state N+1 while any other CPU is still in state N-1. That is exactly the “global quiescence” guarantee stop_machine() sells.

The per-CPU loop is short enough to read whole:

/* kernel/stop_machine.c v6.12, L204-256, abridged */
static int multi_cpu_stop(void *data)
{
	struct multi_stop_data *msdata = data;
	enum multi_stop_state newstate, curstate = MULTI_STOP_NONE;
	int cpu = smp_processor_id(), err = 0;
	bool is_active;
	...
	do {
		stop_machine_yield(cpumask);          /* cpu_relax(), or arch override */
		newstate = READ_ONCE(msdata->state);
		if (newstate != curstate) {
			curstate = newstate;
			switch (curstate) {
			case MULTI_STOP_DISABLE_IRQ:
				local_irq_disable();
				hard_irq_disable();
				break;
			case MULTI_STOP_RUN:
				if (is_active)
					err = msdata->fn(msdata->data);
				break;
			default:
				break;
			}
			ack_state(msdata);
		} else if (curstate > MULTI_STOP_PREPARE) {
			touch_nmi_watchdog();
		}
		rcu_momentary_eqs();
	} while (curstate != MULTI_STOP_EXIT);
 
	local_irq_restore(flags);
	return err;
}

Three details deserve attention. First, is_active — only one CPU actually runs the callback (cpumask_first(cpu_online_mask) when no explicit mask was given); every other CPU exists purely to not be executing anything. Second, touch_nmi_watchdog() is called on every spin once past MULTI_STOP_PREPARE: without it, a stop_machine() that legitimately takes a while would be reported as a hard lockup by the NMI watchdog on the 31 CPUs that are only spinning. Third, rcu_momentary_eqs() reports a momentary RCU quiescent state, because CPUs spinning with interrupts disabled would otherwise stall an RCU grace period (see Read-Copy-Update Fundamentals).

sequenceDiagram
    autonumber
    participant I as Initiator<br/>stop_machine()
    participant S0 as migration/0<br/>(is_active)
    participant S1 as migration/1
    participant SN as migration/31

    I->>I: set_state(MULTI_STOP_PREPARE)<br/>thread_ack = 32
    I->>S0: queue multi_cpu_stop
    I->>S1: queue multi_cpu_stop
    I->>SN: queue multi_cpu_stop
    Note over S0,SN: each stopper preempts whatever<br/>was running on its CPU
    S0->>S0: ack_state() — 31 left
    S1->>S1: ack_state() — 30 left
    SN->>SN: ack_state() — 0 left, advances state
    Note over I,SN: state = MULTI_STOP_DISABLE_IRQ
    S0->>S0: local_irq_disable() + hard_irq_disable()
    S1->>S1: local_irq_disable() + hard_irq_disable()
    SN->>SN: local_irq_disable() + hard_irq_disable()
    Note over I,SN: last ack advances<br/>state = MULTI_STOP_RUN
    S0->>S0: is_active — runs msdata->fn(data)
    S1-->>S1: spins in cpu_relax()<br/>+ touch_nmi_watchdog()
    SN-->>SN: spins in cpu_relax()<br/>+ touch_nmi_watchdog()
    Note over I,SN: last ack advances<br/>state = MULTI_STOP_EXIT
    S0->>S0: local_irq_restore()
    S1->>S1: local_irq_restore()
    SN->>SN: local_irq_restore()
    SN-->>I: completion signalled — machine resumes

stop_machine() bringing a 32-CPU box to a rendezvous. What it shows: the initiator does no work; it publishes a state and waits. All 32 stoppers must reach each barrier before any of them crosses it, and for the entire span between steps 10 and 20 no other code runs anywhere in the machine and no interrupt is delivered. The insight to take: the callback in step 16 might be five instructions long, and the operation still costs 32 context switches in, 32 out, three full barrier round trips, and an interrupts-off window on every CPU. stop_machine() is priced by the size of the machine, not by the size of the work.

Who actually calls stop_machine() — verified, not remembered

The received wisdom about stop_machine() users is unreliable, so the list below comes from grepping the extracted v6.12 tree rather than from recollection. Restricting to real call sites (not comments), here is what actually uses it, grouped by why:

User	v6.12 call site	Why total exclusion is required
CPU hotplug (down only)	`kernel/cpu.c` L1312, `stop_machine_cpuslocked(take_cpu_down, …, cpumask_of(cpu))`	the dying CPU must reach a point where nothing can be scheduled or IRQ-routed to it; note the mask is one CPU, not all of them
MTRR updates (x86)	`arch/x86/kernel/cpu/mtrr/mtrr.c` L191	memory-type range registers are architecturally required to be updated in a lockstep rendezvous with caches disabled; the source spells out the 15-step Intel SDM procedure above the call
Microcode load (x86)	`arch/x86/kernel/cpu/microcode/core.c` L557, `load_cpus_stopped`	a late microcode update can change instruction semantics; no CPU may be executing arbitrary code while it lands
Clocksource switch	`kernel/time/timekeeping.c` L1635, `stop_machine(change_clocksource, …)`	timekeeping must not be read while the clocksource is being swapped
x86 cache-info rendezvous	`arch/x86/kernel/cpu/cacheinfo.c` L1189/L1209	CPUID cache topology programming, including from a not-yet-online CPU via `stop_machine_from_inactive_cpu()`
arm64 alternatives & CPU features	`arch/arm64/kernel/alternative.c` L260, `cpufeature.c` L1957/L3249	arm64 patches its own text and installs KPTI nG mappings under a full freeze
arm/riscv/csky/parisc/xtensa text patching	`patch.c` / `ftrace.c` / `jump_label.c` in each arch	these architectures have no safe live-patching primitive, so they stop the machine to modify instructions
s390 STP clock sync, kprobes	`arch/s390/kernel/time.c` L699, `kprobes.c` L160/L173	architectural requirements specific to s390
Xen suspend	`drivers/xen/manage.c` L136	the whole guest must be quiesced before the hypervisor snapshots it
i915 VT-d workaround, Intel RNG init, thunderx EDAC injection	various `drivers/`	narrow hardware errata workarounds

Three corrections to the folklore

1. text_poke on x86 is the opposite of a stop_machine() user. The kerneldoc for text_poke_bp_batch() says so in as many words: “Modify multi-byte instruction by using int3 breakpoint on SMP. We completely avoid stop_machine() here, and achieve the synchronization using int3 breakpoint” (alternative.c L2233-2241). The INT3-then-patch-then-replace dance exists precisely so that x86 never has to freeze the machine to patch text. Listing “module text patching / text_poke” as a stop_machine() user inverts the actual design. It is true of arm, parisc, riscv, csky and xtensa, which lack the equivalent trick.

2. ftrace conversions do not use stop_machine() on x86 either. ftrace_run_stop_machine() exists (ftrace.c L2926-2929) but it is reached only through the __weak default arch_ftrace_update_code(), whose own comment reads “Archs can override this function if it does not need to run stop_machine() to modify code.” x86 overrides it to call ftrace_modify_all_code() directly (arch/x86/kernel/ftrace.c L256-259), routing through text_poke batching instead. So on the machine most readers are running, enabling a tracepoint does not stop the machine. On arm, riscv and csky it does.

3. CPU hotplug’s use is narrower than “hotplug uses stop_machine”. Only the offline path calls it, only for the single dying CPU’s mask, and only for the take_cpu_down step. The online path does not. Reducing hotplug’s stop_machine() dependence was an explicit multi-year project — see Srivatsa Bhat’s “stop_machine()-free CPU hotplug” series (LWN, 2012 and part 1, 2013), whose stated goal was to remove the offline path’s dependence on stop_machine() and give atomic-context code an alternative to preempt_disable() for keeping CPUs online.

The honest summary is that stop_machine() is overwhelmingly an architecture-errata and firmware-interface tool, not a general kernel-synchronisation tool. Almost every entry above is “the hardware manual says all CPUs must be quiesced.” Where a subsystem could find a cheaper protocol — x86 text patching, most of CPU hotplug — it did, and the history of this file is largely the history of that migration away.

Why it is a latency disaster

Put the cost in concrete terms on this 32-CPU box. A single stop_machine() call must: wake 32 stopper kthreads, each of which preempts whatever was running on its CPU (including any SCHED_DEADLINE or SCHED_FIFO 99 task, with no way for that task to object); pass three lockstep barriers, each requiring all 32 CPUs to arrive; hold every CPU with interrupts hard-disabled across the middle barrier; and then context-switch all 32 CPUs back. The interrupts-off window is bounded below by the slowest CPU’s time to reach the barrier — which, if one CPU is in a long interrupts-disabled region of its own, can be arbitrarily bad.

This is why stop_machine() is essentially forbidden in real-time configurations, and why the PREEMPT_RT project treated every remaining caller as a defect to be designed away (see The PREEMPT_RT Real-Time Kernel). It is also why the SMP alternatives machinery on x86 went to such lengths to patch instructions without it. A stop_one_cpu() migration costs one CPU a couple of microseconds; a stop_machine() costs the whole machine a stall that no scheduling class, no priority, and no CPU isolation setting (see CPU Isolation isolcpus and nohz_full) can protect you from.

A dated history, from the commit log

The current shape of this subsystem is the result of a series of consolidations, each of which is checkable in the cgit log for kernel/stop_machine.c:

Date	Change	Author	Significance
2005-04-16	file exists at `v2.6.12-rc2`	—	the original `stop_machine()`, separate from any migration thread
2010-05-06	`cpu_stop: implement stop_cpu[s]()` (+367 lines)	Tejun Heo	the generic per-CPU stopper is born
2010-05-06	`stop_machine: reimplement using cpu_stop`	Tejun Heo	`stop_machine()` becomes a client of the stopper, not its own mechanism
2010-05-06	`sched: replace migration_thread with cpu_stop`	Tejun Heo	the dedicated migration thread is deleted; the stopper inherits its job and its `migration/%u` name
2010-10-18	`sched: Create special class for stop/migrate work`	Peter Zijlstra	`stop_sched_class` is created — existence-checked: `kernel/sched_stoptask.c` is HTTP 404 at `v2.6.36` and HTTP 200 at `v2.6.37`
2013-02-14	`stop_machine: Use smpboot threads`	Thomas Gleixner	the stopper becomes a managed per-CPU hotplug thread (−85/+49 lines)
2013-10-09	`stop_machine: Introduce stop_two_cpus()`	Peter Zijlstra	enables `migrate_swap()` for NUMA balancing
2022-05-12	`stop_machine: Add stop_core_cpuslocked()`	Peter Zijlstra	per-core (SMT-sibling) exclusion for Intel In-Field Scan
2026-07-31	`stop_machine: Make stop_one_cpu_nowait() return void`	Yury Norov	most recent change as of this writing

The consolidation history. What it shows: the name migration/N is a fossil. The insight to take: in 2010 there really was a thread whose only job was migration, and it was removed; the generic stopper took over and kept the name so that existing tooling and documentation would not break. The scheduling class that gives it its power arrived five months later, as a separate change. So the three things a reader tends to fuse — “the migration thread”, “the stopper”, “the stop class” — have three distinct birthdays, and the modern answer is that only the last two still exist.

Failure Modes and Subtleties

The pending/retry race. Between migrate_enable()’s internal preempt_enable() and the stopper actually running, the task can be migrated by something else and is no longer current. migration_cpu_stop() handles this via struct set_affinity_pending and may re-queue itself onto the task’s new CPU with stop_one_cpu_nowait(). The comment notes a “giant hole here” on !PREEMPT kernels where this race is far more likely. This is the source of considerable historical complexity in the affinity-change path.
Hotplug interaction. A migration_cpu_stop can find its target CPU went offline mid-flight; the comment acknowledges it “might end up running on a dodgy CPU … at which point we’ll get pushed out anyway.” The CPU-down path uses balance-push stop work to evict all movable tasks before a CPU goes dead.
Stopper must not sleep. cpu_stopper_thread() runs the callback with preempt_count_inc(); a sleeping stop function would deadlock the CPU and trips a WARN_ONCE on leaked preempt count. Migration callbacks only take spinlocks, never sleeping locks.
Cost. Migrating a running task costs at least one full IPI-driven wakeup of the stopper plus two context switches (into the stopper, then into whatever runs next) plus two run-queue lock acquisitions. This is why the load balancer (see below) strongly prefers to pull queued tasks and treats moving the running task as a last resort.

How Migration Is Triggered

Migration is mechanism, not policy. The callers that decide when to move a task are:

Load balancing (Scheduler Load Balancing, Scheduling Domains and CPU Topology). Periodic, topology-aware balancing pulls queued tasks from a busy CPU’s run queue toward an idle one — and per the sched-domains documentation, it “locks both our initial CPU’s runqueue and the newly found busiest one and starts moving tasks.” This is the cheap move_queued_task path. Active balancing, when it must move the running task of a CPU, escalates to the stopper.
Affinity changes (CPU Affinity and sched_setaffinity). sched_setaffinity() may strand a running task on a now-forbidden CPU; set_cpus_allowed_ptr() fires migration_cpu_stop to relocate it.
sched_exec() — execve() is “a valuable balancing opportunity, because at this point the task has the smallest effective memory and cache footprint,” so the kernel may migrate the exec’ing task to a better CPU via stop_one_cpu().
NUMA balancing (NUMA Balancing and Task Placement) and wakeup placement (Wakeup Balancing and select_task_rq) choose CPUs at enqueue time; for blocked tasks this needs no stopper, just set_task_cpu() under pi_lock.

Common Misunderstandings

Because three distinct things share one name and one kthread, this area attracts a specific set of wrong beliefs. Each row below is contradicted by a citation elsewhere in this note:

Belief	Status	What is actually true
“Migration goes through `stop_machine()`.”	False	Migration uses `stop_one_cpu()` — one CPU, no rendezvous. `stop_machine()` and `stop_one_cpu()` share the stopper kthreads and nothing else.
“`migration/N` is a thread dedicated to scheduler migration.”	False since 2010	It is the generic `cpu_stopper` thread. Tejun Heo’s 2010-05-06 series deleted the real migration thread and kept its name.
“The stop class is `SCHED_FIFO` priority 100.”	False	It has no policy and no priority. `ps` reports `FF/99` because `sched_set_stop_task()` deliberately makes it look like `SCHED_FIFO` 99 for userspace, then overwrites `->sched_class`.
“A `SCHED_DEADLINE` task cannot be preempted.”	False	It cannot be preempted by any policy-scheduled task. A stopper preempts it, and there is no way to prevent that.
“`text_poke` / module patching stops the machine.”	False on x86	`text_poke_bp_batch()`’s own kerneldoc: “We completely avoid `stop_machine()` here.” True on arm, parisc, riscv, csky, xtensa.
“Enabling an ftrace tracepoint stops the machine.”	False on x86	x86 overrides `arch_ftrace_update_code()` to use `text_poke` batching. The `stop_machine()` path is the `__weak` fallback for architectures without a live-patching primitive.
“CPU hotplug uses `stop_machine()`.”	Half true	Only CPU-down, only the `take_cpu_down` step, and only with the single dying CPU in the mask. CPU-up does not.
“`isolcpus` / `nohz_full` protects a CPU from the stopper.”	False	Isolation keeps schedulable work away. It does not stop `stop_machine()`, which recruits every online CPU regardless. See CPU Isolation isolcpus and nohz_full.
“You can create a stop-class task if you have `CAP_SYS_NICE`.”	False	There is no `SCHED_STOP` policy, and `switched_to_stop()` is a `BUG()`. Membership is one kernel-assigned task per CPU, full stop.

Where This Sits in the Exclusion Spectrum

The stop class is the far end of a spectrum that this vault covers note by note. Ordering the kernel’s exclusion mechanisms by how much of the machine they exclude, cheapest first, puts the pieces in perspective:

Mechanism	Excludes	Sleeps?	Rough cost, uncontended	Note
RCU read-side	nobody — readers never block writers	no	near zero on non-`PREEMPT`	Read-Copy-Update Fundamentals
rwsem, read	other writers only; readers run concurrently	yes	one atomic add on one word	Read-Write Semaphores
mutex / rwsem write	one task at a time, on one object	yes	one `cmpxchg`	Kernel Mutexes, Read-Write Semaphores
spinlock	one CPU at a time, on one object	no	one `cmpxchg` + spin	Kernel Spinlocks
`local_irq_disable()`	interrupts on this CPU	no	a few cycles	Softirqs and the Softirq Vector
`stop_one_cpu()`	everything on one CPU, including RT and DL	no (callback runs with preemption off)	two context switches	this note
`stop_machine()`	everything on every CPU, interrupts included	no	2×N context switches + 3 barriers + IRQ-off window	this note

The exclusion spectrum. What it shows: the last two rows are qualitatively different from the rest — every other mechanism excludes contenders for one object, while the stop class excludes contenders for the CPU itself. The insight to take: this is why the stop class has no API for ordinary code. A lock protects data; a stopper protects an execution context. You reach for the latter only when the thing you must exclude is “arbitrary code running anywhere,” which is a hardware-level requirement, not a data-structure-level one.

vault

Explorer

Migration the Stop Class and the Migration Thread