Migration, the Stop Class and the Migration Thread

Linux keeps a separate run queue per CPU, so a runnable task lives on exactly one CPU’s queue at a time. Moving a task from one CPU to another — migration — is therefore not a free pointer swap: it must dequeue the task from the source run queue, change its recorded CPU, and enqueue it on the destination, all under the right locks. For a task that is merely queued and not currently executing, the kernel does this directly with move_queued_task() while holding the source rq->lock. But a task that is currently running on its CPU cannot be relocated out from under itself — you cannot rewrite a task’s CPU while its registers and kernel stack are live on that CPU. Linux solves this with the stop scheduling class (stop_sched_class), the highest-priority class in the system, embodied per CPU by the migration/N kthread (the cpu_stopper.thread). To move a running task, the kernel queues work to that CPU’s stopper; the stopper, having absolute priority, instantly preempts the victim, which forces it off the CPU — and now migration_cpu_stop() can safely move the no-longer-running task (kernel/sched/core.c v6.12, “This is how migration works”). This note traces that machinery as of Linux 6.12 LTS (released 2024-11-17, verified at kernel.org releases).

This note pins all code claims to 6.12 LTS. The migration/stopper machinery is long-stable, but symbol names (pcpu_hot.current_task, the DEFINE_SCHED_CLASS macro, the sched_class method set) are stated as they appear in 6.12 and may differ in other releases.

Mental Model

The right way to think about this is a two-tier scheme keyed on a single question: is the task I want to move currently running on a CPU, or just sitting in a run queue?

flowchart TB
  START["Want task p on CPU A<br/>to end up on CPU B"] --> Q{"Is p currently<br/>running on a CPU?"}
  Q -->|"No — queued only"| MQT["move_queued_task() under rq->lock:<br/>deactivate on A → set_task_cpu(B)<br/>→ activate on B"]
  Q -->|"Yes — running on A"| STOP["queue migration_cpu_stop<br/>to CPU A's stopper via<br/>stop_one_cpu()"]
  STOP --> PREEMPT["migration/A kthread<br/>(stop_sched_class, top priority)<br/>preempts p — p leaves the CPU"]
  PREEMPT --> RUN["migration_cpu_stop() now runs;<br/>p is no longer running<br/>→ __migrate_task() → move_queued_task()"]
  MQT --> DONE["p enqueued on CPU B"]
  RUN --> DONE

The migration decision tree. What it shows: a queued task takes the cheap direct path (move_queued_task holds the source run-queue lock the whole time); a running task takes the expensive path through the per-CPU stopper kthread, whose sole job here is to evict the victim by virtue of outranking it, after which the same move_queued_task machinery completes the move. The insight to take: the stop class exists precisely to convert a “running task” into a “not-running task” so the cheap path becomes usable — the actual relocation logic is identical in both branches.

Why a Running Task Needs the Stop Class

A task_struct records its current CPU (read via task_cpu(p), written via set_task_cpu()), and that CPU’s run queue holds the scheduling-class bookkeeping for it. To migrate the task you must atomically (with respect to the scheduler) take it off the source queue and put it on the destination queue. If the task is parked in a run queue but not executing, the migrating CPU simply grabs rq->lock and does the surgery: nothing else can touch the task because run-queue operations all serialize on that lock.

A running task is different. Its register state, its kernel stack pointer, and the per-CPU current pointer all reflect that it is live on its CPU right now. You cannot just rewrite task_cpu(p) to point at another CPU — the task is still physically executing here. The only safe way to move it is to first make it stop running, and the cleanest way to guarantee that is to run something on its CPU that outranks it absolutely. That something is the stop class.

The stop class’s defining property is stated in its own source file: “The stop task is the highest priority task in the system, it preempts everything and will be preempted by nothing” (stop_task.c v6.12). Recall the scheduling-class stack walked by pick_next_task(): stop → deadline → rt → fair → ext → idle (see Scheduling Classes and the sched_class Interface and Scheduler Entry Points and pick_next_task). Because stop_sched_class is first, the instant the stopper kthread becomes runnable on a CPU, the next scheduling decision on that CPU picks it — preempting even a SCHED_FIFO real-time task or a SCHED_DEADLINE task. The victim is now off the CPU, demoted from “running” to “queued (or blocked)”, and the ordinary move_queued_task() path applies.

The migration/N kthread Is the cpu_stopper

The kthread you see as migration/0, migration/1, … in ps output is not a dedicated “migration thread” in the sense of a thread whose only purpose is scheduler migration. It is the generic per-CPU CPU stopper thread, and migration is merely its most frequent customer. The naming is historical: the stopper subsystem grew out of the original migration thread.

The stopper is defined as a per-CPU structure and a per-CPU “smpboot” hotplug thread:

struct cpu_stopper {
	struct task_struct	*thread;
	raw_spinlock_t		lock;
	bool			enabled;	/* is this stopper enabled? */
	struct list_head	works;		/* list of pending works */
	struct cpu_stop_work	stop_work;	/* for stop_cpus */
	unsigned long		caller;
	cpu_stop_fn_t		fn;
};
 
static DEFINE_PER_CPU(struct cpu_stopper, cpu_stopper);
 
static struct smp_hotplug_thread cpu_stop_threads = {
	.store			= &cpu_stopper.thread,
	.thread_should_run	= cpu_stop_should_run,
	.thread_fn		= cpu_stopper_thread,
	.thread_comm		= "migration/%u",
	.create			= cpu_stop_create,
	.park			= cpu_stop_park,
	.selfparking		= true,
};

From stop_machine.c v6.12. Line by line: each CPU has a cpu_stopper holding a pointer to its kthread, a lock, an enabled flag (cleared while the CPU is parked/offline), and a works list of pending stop jobs. thread_comm = "migration/%u" is exactly why the thread shows up as migration/0 etc. — %u is filled with the CPU number. The smp_hotplug_thread registration creates one such kthread per CPU at boot and re-creates it on CPU hotplug.

The crucial wiring that makes the stopper the stop-class task is in cpu_stop_create():

static void cpu_stop_create(unsigned int cpu)
{
	sched_set_stop_task(cpu, per_cpu(cpu_stopper.thread, cpu));
}

sched_set_stop_task() installs this kthread as rq->stop for that CPU and switches it into stop_sched_class. So the identity is exact: the migration/N kthread, the cpu_stopper.thread for CPU N, and the task that stop_sched_class schedules on CPU N are all the same task_struct. They are not three separate objects.

The stop class itself is deliberately minimal. Its methods either do trivial accounting or actively forbid the operation — a stop task may never yield, may never be created by userspace, and may never have its priority changed:

static void yield_task_stop(struct rq *rq)
{
	BUG(); /* the stop task should never yield, its pointless. */
}
static void switched_to_stop(struct rq *rq, struct task_struct *p)
{
	BUG(); /* its impossible to change to this class */
}

From stop_task.c v6.12. The BUG() calls encode invariants: nothing in the kernel should ever try to demote, yield, or re-prioritize a stopper. Its pick_task_stop() simply returns rq->stop if the stopper is runnable, and select_task_rq_stop() returns the task’s current CPU unconditionally — stop tasks never migrate, which makes sense because a per-CPU stopper is meaningless on any other CPU.

How a Running Task Actually Gets Moved

The canonical job the stopper runs for migration is migration_cpu_stop(). The high-level sequence is documented in the comment block in core.c:

we invoke migration_cpu_stop() on the target CPU using stop_one_cpu().

stopper starts to run (implicitly forcing the migrated thread off the CPU)

it checks whether the migrated task is still in the wrong runqueue.

if it’s in the wrong runqueue then the migration thread removes it and puts it into the right queue.

stopper completes and stop_one_cpu() returns and the migration is done.

(core.c v6.12). Walk it concretely. A typical caller — e.g. set_cpus_allowed_ptr() reacting to a changed affinity mask, or sched_exec() rebalancing on execve() — fills a struct migration_arg { task, dest_cpu, … } and calls:

stop_one_cpu(task_cpu(p), migration_cpu_stop, &arg);

stop_one_cpu(cpu, fn, arg) runs fn(arg) on cpu “in a process context with the highest priority preempting any task on the cpu and monopolizing it” and blocks until it completes (stop_machine.c v6.12 kerneldoc). Internally it builds a cpu_stop_work, queues it on the target CPU’s cpu_stopper.works list, and wakes that CPU’s stopper:

static void __cpu_stop_queue_work(struct cpu_stopper *stopper,
				  struct cpu_stop_work *work,
				  struct wake_q_head *wakeq)
{
	list_add_tail(&work->list, &stopper->works);
	wake_q_add(wakeq, stopper->thread);
}

Because the woken stopper is in the top class, the destination CPU’s next __schedule() picks it, preempting whatever ran. The stopper’s main loop pulls the work and calls the function with preemption disabled (stop callbacks must not sleep):

static void cpu_stopper_thread(unsigned int cpu)
{
	/* ... pull first work off stopper->works ... */
	if (work) {
		cpu_stop_fn_t fn = work->fn;
		preempt_count_inc();
		ret = fn(arg);          /* this is migration_cpu_stop() */
		if (done) { ...; cpu_stop_signal_done(done); }
		preempt_count_dec();
		goto repeat;
	}
}

Now migration_cpu_stop() itself runs on the source CPU, which means the victim is no longer running there — the stopper displaced it. It takes p->pi_lock and the run-queue lock, re-validates that p is still on this run queue (affinity may have changed again, or the task may have blocked), and if so performs the move:

if (task_on_rq_queued(p)) {
	update_rq_clock(rq);
	rq = __migrate_task(rq, &rf, p, arg->dest_cpu);
} else {
	p->wake_cpu = arg->dest_cpu;
}

From core.c v6.12. If the task is queued, __migrate_task() (after a final is_cpu_allowed() check) calls move_queued_task(). If the task has since blocked (not on the run queue), there is nothing to dequeue — the kernel just records wake_cpu so the task wakes on the destination later (see Wakeups and try_to_wake_up).

move_queued_task and set_task_cpu — The Shared Relocation Core

Both branches converge on move_queued_task(), which is the entire relocation primitive and the same code the cheap queued-task path uses directly:

static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf,
				   struct task_struct *p, int new_cpu)
{
	lockdep_assert_rq_held(rq);
 
	deactivate_task(rq, p, DEQUEUE_NOCLOCK);
	set_task_cpu(p, new_cpu);
	rq_unlock(rq, rf);
 
	rq = cpu_rq(new_cpu);
 
	rq_lock(rq, rf);
	WARN_ON_ONCE(task_cpu(p) != new_cpu);
	activate_task(rq, p, 0);
	wakeup_preempt(rq, p, 0);
 
	return rq;
}

From core.c v6.12. Reading it: the source rq->lock must already be held (lockdep_assert_rq_held). deactivate_task() dequeues p from the source class’s run queue and clears its on-rq state. set_task_cpu(p, new_cpu) changes the recorded CPU — and along the way invokes the class hook migrate_task_rq (for fair tasks this discounts the task’s vruntime against the new cfs_rq’s baseline), bumps p->se.nr_migrations, and fires rseq/perf/mm_cid migration notifications. The source lock is dropped, the destination lock taken, and activate_task() enqueues p on the new CPU. Finally wakeup_preempt() checks whether the newly arrived task should preempt the destination’s current task. The function returns the locked destination rq — the caller’s lock has been handed forward to a different run queue, a subtlety the caller must track.

set_task_cpu() enforces the locking discipline that makes all of this safe:

WARN_ON_ONCE(debug_locks && !(lockdep_is_held(&p->pi_lock) ||
			      lockdep_is_held(__rq_lockp(task_rq(p)))));

From core.c v6.12. The rule: to change a task’s CPU you must hold either the task’s pi_lock (used when waking a blocked task — see Wakeups and try_to_wake_up) or the run-queue lock (used when the task is runnable). This is exactly why a queued task can be migrated cheaply (you hold the rq lock) but a running task cannot be migrated by a remote CPU without first stopping it (the remote CPU does not own the live execution state).

Pinned and Migration-Disabled Tasks

Not every task may be moved to any CPU. Migration is constrained by:

The affinity mask (p->cpus_ptr / p->cpus_mask). A task pinned to one CPU (sched_setaffinity to a single bit) can never be migrated elsewhere — is_cpu_allowed() and __migrate_task()’s is_cpu_allowed() guard reject the move. Per-CPU kthreads (KTHREAD_IS_PER_CPU) are pinned by construction. See CPU Affinity and sched_setaffinity.
migration_disabled. Code inside a migrate_disable() / migrate_enable() section must not be moved even though preemption may be allowed. migration_cpu_stop() explicitly bails — if (is_migration_disabled(p)) goto out; — and the set_affinity_pending dance retries once migration is re-enabled. This is the modern, finer-grained replacement for blanket preempt_disable() in many PREEMPT_RT paths.
The stopper task itself never migrates (select_task_rq_stop returns the current CPU).

stop_machine — The Heavier Sibling, Not the Migration Path

It is tempting to conflate stop_machine with migration because both live in kernel/stop_machine.c and both ride the cpu_stopper infrastructure. They are different, and migration does not use stop_machine.

stop_one_cpu() (what migration uses) monopolizes one CPU to run a function. Other CPUs keep running normally.
stop_machine() halts every online CPU simultaneously to reach global quiescence — it queues multi_cpu_stop to all stoppers and runs them through a synchronized state machine (MULTI_STOP_PREPARE → MULTI_STOP_DISABLE_IRQ → MULTI_STOP_RUN → MULTI_STOP_EXIT), with all but one CPU spinning with interrupts disabled while the chosen CPU runs the callback. It is used for genuinely global, can’t-tolerate-concurrency operations: MTRR updates, some text/code patching, certain CPU hotplug steps.

static int multi_cpu_stop(void *data)
{
	/* ... synchronized state machine ... */
	switch (curstate) {
	case MULTI_STOP_DISABLE_IRQ:
		local_irq_disable();
		hard_irq_disable();
		break;
	case MULTI_STOP_RUN:
		if (is_active)
			err = msdata->fn(msdata->data);
		break;
	...
	}
}

From stop_machine.c v6.12. The takeaway: stop_machine is a system-wide freeze and is enormously expensive; migration is a single-CPU preemption. They share the stopper kthreads but are otherwise unrelated mechanisms. Saying “migration goes through stop_machine” is wrong.

Failure Modes and Subtleties

The pending/retry race. Between migrate_enable()’s internal preempt_enable() and the stopper actually running, the task can be migrated by something else and is no longer current. migration_cpu_stop() handles this via struct set_affinity_pending and may re-queue itself onto the task’s new CPU with stop_one_cpu_nowait(). The comment notes a “giant hole here” on !PREEMPT kernels where this race is far more likely. This is the source of considerable historical complexity in the affinity-change path.
Hotplug interaction. A migration_cpu_stop can find its target CPU went offline mid-flight; the comment acknowledges it “might end up running on a dodgy CPU … at which point we’ll get pushed out anyway.” The CPU-down path uses balance-push stop work to evict all movable tasks before a CPU goes dead.
Stopper must not sleep. cpu_stopper_thread() runs the callback with preempt_count_inc(); a sleeping stop function would deadlock the CPU and trips a WARN_ONCE on leaked preempt count. Migration callbacks only take spinlocks, never sleeping locks.
Cost. Migrating a running task costs at least one full IPI-driven wakeup of the stopper plus two context switches (into the stopper, then into whatever runs next) plus two run-queue lock acquisitions. This is why the load balancer (see below) strongly prefers to pull queued tasks and treats moving the running task as a last resort.

How Migration Is Triggered

Migration is mechanism, not policy. The callers that decide when to move a task are:

Load balancing (Scheduler Load Balancing, Scheduling Domains and CPU Topology). Periodic, topology-aware balancing pulls queued tasks from a busy CPU’s run queue toward an idle one — and per the sched-domains documentation, it “locks both our initial CPU’s runqueue and the newly found busiest one and starts moving tasks.” This is the cheap move_queued_task path. Active balancing, when it must move the running task of a CPU, escalates to the stopper.
Affinity changes (CPU Affinity and sched_setaffinity). sched_setaffinity() may strand a running task on a now-forbidden CPU; set_cpus_allowed_ptr() fires migration_cpu_stop to relocate it.
sched_exec() — execve() is “a valuable balancing opportunity, because at this point the task has the smallest effective memory and cache footprint,” so the kernel may migrate the exec’ing task to a better CPU via stop_one_cpu().
NUMA balancing (NUMA Balancing and Task Placement) and wakeup placement (Wakeup Balancing and select_task_rq) choose CPUs at enqueue time; for blocked tasks this needs no stopper, just set_task_cpu() under pi_lock.

vault

Explorer

Migration the Stop Class and the Migration Thread