vault

Wakeups and try_to_wake_up

13 min read

Wakeups and try_to_wake_up

When a blocked task’s wait condition becomes true — data arrives on a socket, a mutex is released, a timer fires — something must turn that sleeping task_struct back into a runnable one and get it onto a CPU’s run queue. That something is try_to_wake_up() (universally abbreviated TTWU), the single chokepoint for all wakeups in kernel/sched/core.c (core.c v6.12). It atomically checks whether the task is in a wake-able state, flips its state to TASK_RUNNING, decides which CPU it should run on via select_task_rq(), places it on that run queue (often on a remote CPU by sending an inter-processor interrupt rather than touching the remote run queue directly), and finally checks whether the freshly-woken task should preempt whatever is currently running there. Every convenience wrapper — wake_up_process(), wake_up_state(), wake_up_q(), and the wait-queue wake_up() family — bottoms out in try_to_wake_up(). This note is pinned to Linux 6.12 LTS (the v6.12 tag, released 2024-11-17).

The function is, in the words of its own kernel comment, code that “races really badly with just about everything” and is full of memory barriers; the goal here is to make its structure legible without glossing the concurrency that justifies it.

Mental Model

A wakeup is a hand-off between two parties that may be on different CPUs and running concurrently: the waker (whoever calls try_to_wake_up, e.g. the thread releasing a lock) and the wakee (the blocked task p). The wakee may be in any of several states: still actually running on a CPU (about to sleep but not finished), already off the run queue and fully asleep, asleep-but-still-on-the-tree (sched_delayed, see Runnable State Enqueue and Dequeue), or even waking itself. TTWU must handle all of these without ever taking two run-queue locks at once if it can avoid it — taking two rq->locks is the expensive, deadlock-prone case the whole design dances around.

The serialization key is p->pi_lock (the task’s “process integrity” lock), not rq->lock. By taking pi_lock, TTWU stabilizes the fields it needs to make a placement decision — p->sched_class, p->cpus_ptr (the affinity mask), p->sched_task_group — and serializes against other concurrent wakeups of the same task. It then takes the target rq->lock only briefly, at the enqueue.

flowchart TD
  START["try_to_wake_up(p, state, flags)"] --> SELF{"p == current?"}
  SELF -->|yes| FAST["ttwu_state_match + ttwu_do_wakeup<br/>(no locks)"]
  SELF -->|no| PI["LOCK p->pi_lock<br/>smp_mb__after_spinlock()"]
  PI --> MATCH{"state &amp;<br/>p->__state ?"}
  MATCH -->|no match| OUT["return 0 (no wakeup)"]
  MATCH -->|match| ONRQ{"p->on_rq ?<br/>(ttwu_runnable)"}
  ONRQ -->|"on_rq (incl.<br/>sched_delayed)"| RUNNABLE["re-mark RUNNING<br/>maybe re-enqueue delayed<br/>wakeup_preempt(); done"]
  ONRQ -->|"off-queue"| WAKING["state = TASK_WAKING"]
  WAKING --> WL{"on_cpu &amp;&amp;<br/>ttwu_queue_cond?"}
  WL -->|yes| IPI["__ttwu_queue_wakelist:<br/>push to remote wake_list,<br/>send IPI"]
  WL -->|no| SEL["wait on_cpu clear;<br/>cpu = select_task_rq(p)"]
  SEL --> Q["ttwu_queue(p, cpu):<br/>LOCK rq->lock<br/>ttwu_do_activate()"]
  IPI -.->|"remote CPU runs"| PEND["sched_ttwu_pending():<br/>ttwu_do_activate() locally"]
  Q --> ACT["activate_task → enqueue_task<br/>+ wakeup_preempt()"]
  PEND --> ACT

The decision tree of try_to_wake_up. What it shows: the three fast exits (waking self with no locks; the task never being in a wakeable state; the task still on the run queue so only its state needs flipping) and the slow path that picks a CPU and enqueues — either locally under rq->lock or remotely by queuing onto the target’s wake_list and sending an IPI. The insight: the entire structure is organized to avoid taking the wakee’s run-queue lock; only ttwu_queue() (local enqueue) and ttwu_runnable() (already-queued case) ever touch an rq->lock, and the IPI path offloads even that to the target CPU.

Mechanical Walk-through

Entry and the convenience wrappers

try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) takes the target task, a mask of task states that may be woken, and modifier flags (WF_*). The state mask is why a wake_up_interruptible() will not wake a task in TASK_UNINTERRUPTIBLE: the caller passes TASK_INTERRUPTIBLE, and if p’s state is not in the mask, TTWU returns 0 without doing anything (core.c v6.12, ~line 4127).

The wrappers fix the arguments (core.c v6.12, ~line 4412):

int wake_up_process(struct task_struct *p)
{
	return try_to_wake_up(p, TASK_NORMAL, 0);
}
int wake_up_state(struct task_struct *p, unsigned int state)
{
	return try_to_wake_up(p, state, 0);
}

TASK_NORMAL is TASK_INTERRUPTIBLE | TASK_UNINTERRUPTIBLE, so wake_up_process() wakes a task regardless of whether its sleep was interruptible. The wait-queue machinery (see Wait Queues and Task Blocking) calls default_wake_function, which calls try_to_wake_up(curr->private, mode, wake_flags) with the mode the sleeper registered.

The waking-self fast path

The first branch handles p == current — a task waking itself, which happens with self-targeted signals and some lock paths. Because the task is, by definition, already running on this CPU and on its run queue, no locks are needed: it just matches the state and calls ttwu_do_wakeup() (which writes TASK_RUNNING) and returns. The code even SCHED_WARN_ONs p->se.sched_delayed, because a running task cannot also be a delayed-dequeue sleeper.

State check under pi_lock

For the normal case TTWU takes p->pi_lock with IRQs saved, then issues smp_mb__after_spinlock() — a full memory barrier ensuring the caller’s CONDITION = 1 store (e.g. “the buffer now has data”) is ordered before the load of p->__state. This is the partner of the smp_store_mb() inside set_current_state() that the sleeper executed; together they guarantee the classic sleep/wake race cannot drop a wakeup. It then calls ttwu_state_match(p, state, &success) (core.c v6.12, ~line 3976), which tests state & p->__state. If there is no match — the task is already running, or in a state the caller did not ask to wake — TTWU breaks out and returns 0. ttwu_state_match also handles PREEMPT_RT’s saved-state trick: a task blocked on an RT mutex carries its “real” state in p->saved_state, and the matcher restores it rather than spuriously waking the task.

Already on the run queue: ttwu_runnable()

After confirming a state match, TTWU loads p->on_rq (with an smp_rmb() ordering it after the state load) and, if set, calls ttwu_runnable() (core.c v6.12, ~line 3731). This covers two situations. First, the textbook race: the sleeper executed set_current_state(TASK_UNINTERRUPTIBLE) but has not yet reached schedule(), so it is still queued; all TTWU must do is set __state back to TASK_RUNNING. Second, the sched_delayed case: the task slept but EEVDF left it on the tree (see Runnable State Enqueue and Dequeue), so ttwu_runnable re-activates it in place:

	if (task_on_rq_queued(p)) {
		update_rq_clock(rq);
		if (p->se.sched_delayed)
			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
		if (!task_on_cpu(rq, p))
			wakeup_preempt(rq, p, wake_flags);
		ttwu_do_wakeup(p);
		ret = 1;
	}

This is the only path in TTWU that takes the wakee’s existing rq->lock (via __task_rq_lock) — and the kernel comment marks it “unavoidable.” It re-enqueues a delayed task cheaply, runs wakeup_preempt() if the task is not currently on a CPU, flips it to TASK_RUNNING, and is done. No CPU selection, no migration.

TASK_WAKING and the migration window

If the task is genuinely off-queue (on_rq == 0), it slept fully and may be moved to a better CPU. TTWU first issues smp_acquire__after_ctrl_dep() — a barrier paired with the smp_store_release(&p->on_rq, 0) in __block_task() — to be sure the previous CPU has finished descheduling the task and the task “no longer cares about its own p->state.” Then it sets p->__state = TASK_WAKING. This transient state exists precisely so TTWU can drop pi_lock before the enqueue while still telling any other concurrent waker “this task is being woken, hands off.” See Process States and the Task State Machine for where TASK_WAKING sits.

Before selecting a CPU, TTWU must wait until the task has fully left its old CPU. The wakee might still be on_cpu == 1 (the old CPU is mid-schedule(), having dequeued the task but not yet finished the context switch). TTWU does smp_cond_load_acquire(&p->on_cpu, !VAL) — spin until on_cpu clears — unless it can offload the work via the wakelist (next section).

CPU selection: select_task_rq()

With the task quiesced, TTWU calls cpu = select_task_rq(p, p->wake_cpu, &wake_flags). This is the placement brain — it consults scheduling domains, idle-CPU search, and energy/capacity heuristics to pick where the task should run. That logic is large enough to be its own topic; see Wakeup Balancing and select_task_rq. If the chosen CPU differs from task_cpu(p), TTWU sets WF_MIGRATED, updates PSI/iowait accounting, and calls set_task_cpu(p, cpu) to move the task’s run-queue association.

Local vs. remote enqueue: ttwu_queue() and the IPI

Finally ttwu_queue(p, cpu, wake_flags) performs the enqueue (core.c v6.12, ~line 3943):

static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
{
	struct rq *rq = cpu_rq(cpu);
	struct rq_flags rf;
 
	if (ttwu_queue_wakelist(p, cpu, wake_flags))
		return;
 
	rq_lock(rq, &rf);
	update_rq_clock(rq);
	ttwu_do_activate(rq, p, wake_flags, &rf);
	rq_unlock(rq, &rf);
}

There are two ways to enqueue. The direct way (the rq_lock/ttwu_do_activate/rq_unlock block) takes the target run queue’s lock and enqueues there — fine when the target is the local CPU or shares cache. The remote-wakelist way (ttwu_queue_wakelist) avoids touching the remote run queue’s data at all: it pushes p onto the target CPU’s wake_list (a lockless llist) and sends an IPI so the target CPU does the enqueue itself. The decision is made by ttwu_queue_cond() (core.c v6.12, ~line 3874), gated by the TTWU_QUEUE scheduler feature (default true on SMP, features.h v6.12). The wakelist is chosen when:

  • the task is not a sched_ext task (SCX may need select_task_rq to actually run, so the optimization is skipped — sched_ext and BPF-Defined Schedulers);
  • the target CPU is active (not in hotplug transition) and still in p’s affinity mask;
  • the target CPU does not share cache with the waker (!cpus_share_cache) — queuing remotely avoids dragging the remote run queue’s cache lines onto the waker’s CPU; or
  • the target shares cache but is idle (!cpu_rq(cpu)->nr_running) — offload the activation to the idle CPU so the busy waker is not slowed, and avoid stacking tasks.

When the wakelist is used, __ttwu_queue_wakelist() sets rq->ttwu_pending = 1, records p->sched_remote_wakeup, and calls __smp_call_single_queue(cpu, &p->wake_entry.llist) to enqueue the IPI. On the receiving CPU, sched_ttwu_pending() runs (core.c v6.12, ~line 3762): it locks its own run queue, drains the llist, and calls ttwu_do_activate() for each pending task — so the cost of the enqueue (and the cache traffic) is paid by the CPU the task will actually run on, not by the waker. There is an optimization for an idle target: if the idle task is in its polling loop, call_function_single_prep_ipi() can flip its need_resched poll flag and skip the physical IPI entirely (trace_sched_wake_idle_without_ipi).

ttwu_do_activate() and the preemption check

Whether reached directly or from sched_ttwu_pending, ttwu_do_activate() is where the task becomes runnable (core.c v6.12, ~line 3656):

	int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
	if (p->sched_contributes_to_load)
		rq->nr_uninterruptible--;
	... /* WF_MIGRATED / iowait accounting */
	activate_task(rq, p, en_flags);
	wakeup_preempt(rq, p, wake_flags);
	ttwu_do_wakeup(p);

It decrements nr_uninterruptible (undoing the increment __block_task did when the task slept in D state — this is how load average accounting balances). It calls activate_task() with ENQUEUE_WAKEUP, which routes to enqueue_task_fair() and inserts the entity into the EEVDF tree (see Runnable State Enqueue and Dequeue). Then wakeup_preempt(rq, p, wake_flags) asks the target run queue’s scheduling class: does this newly-runnable task deserve to preempt the current task? For the fair class this is check_preempt_wakeup_fair, which compares EEVDF eligibility/virtual deadlines and may set the need_resched flag (see The need_resched Flag and Preemption Points and Wakeup Preemption and check_preempt_wakeup). Setting need_resched does not switch immediately — it marks that a reschedule is due at the next safe preemption point. Finally ttwu_do_wakeup() writes p->__state = TASK_RUNNING and fires the sched_wakeup tracepoint.

Deferred batches: wake_up_q()

Some code paths need to wake several tasks while holding a lock they cannot drop mid-loop (e.g. a futex releasing many waiters). The wake_q mechanism lets them stage wakeups and fire them later. wake_q_add(head, task) links the task onto a wake_q_head and takes a reference; wake_up_q(head) then walks the list and calls wake_up_process() on each (core.c v6.12, ~line 1051):

void wake_up_q(struct wake_q_head *head)
{
	struct wake_q_node *node = head->first;
	while (node != WAKE_Q_TAIL) {
		struct task_struct *task = container_of(node, struct task_struct, wake_q);
		node = node->next;
		task->wake_q.next = NULL;
		wake_up_process(task);
		put_task_struct(task);
	}
}

The point is latency and lock-hold-time: by deferring the actual try_to_wake_up calls until after the critical section, the waker keeps its lock for the minimum time and the wakeups (with their potential IPIs and preemptions) happen outside it. The comment warns that the wakeup “can come instantly” — wake_q_add must be used as if it were wake_up_process(), i.e. the task must already be ready to run.

Failure Modes and Common Misunderstandings

The lost-wakeup race. The canonical bug TTWU is built to prevent: waker sets CONDITION = 1 and calls wake; sleeper checks CONDITION (still sees 0 due to reordering), sets its state to sleeping, and schedule()s — sleeping forever. The fix is the barrier pairing: set_current_state() (sleeper) issues smp_store_mb, and try_to_wake_up issues smp_mb__after_spinlock before reading state. Code that hand-rolls a wait loop with a bare p->state = TASK_INTERRUPTIBLE assignment instead of set_current_state() reintroduces this race. Always use set_current_state() / the wait-queue helpers (see Wait Queues and Task Blocking).

Assuming a wakeup means an immediate switch. try_to_wake_up returning 1 means the task is now runnable and may have set need_resched — it does not mean the woken task is running, nor that the waker yielded. The actual switch waits for a preemption point. A wake_up_process() followed by code that assumes the wakee already ran is wrong.

Return value of 0 is not failure. try_to_wake_up returns 0 when the task’s state did not match the wake mask — typically because it was already running or already woken by someone else. This is normal and not an error; callers that treat 0 as “wakeup failed, retry” can spin.

TASK_WAKING confusion. A task observed in TASK_WAKING is not blocked and not yet enqueued — it is mid-flight inside TTWU between dropping pi_lock and the enqueue. Tooling that walks task states must treat it as transient.

TTWU is the kernel-internal primitive; userspace reaches it indirectly. A futex(FUTEX_WAKE) ultimately calls wake_up_q/wake_up_process on the kernel side (contrast with Go’s runtime, which parks/unparks goroutines in userspace and only touches a futex when an OS thread must block — see Futex and OS Synchronization Primitives and GMP Scheduler Model). Within the kernel, wake_up() / wake_up_interruptible() on a wait queue, complete() on a completion, and up() on a semaphore are all thin layers over try_to_wake_up. The choice between them is about what you are waiting on, not about a different wakeup mechanism — they converge on TTWU.

Production Notes

The remote-wakelist/IPI path (TTWU_QUEUE) and its ttwu_queue_cond heuristics are performance-critical and have been tuned repeatedly; the cache-sharing and idle-CPU conditions exist because earlier versions either bounced run-queue cache lines across NUMA nodes (hurting throughput) or stacked too many wakees onto one CPU during a burst. Operators can observe wakeup behavior through scheduler statistics (/proc/schedstat, nr_wakeups, nr_wakeups_remote, nr_wakeups_migrate recorded in ttwu_stat) and can toggle NO_TTWU_QUEUE via /sys/kernel/debug/sched/features to A/B-test the remote-enqueue optimization when diagnosing wakeup latency. The sched_wakeup and sched_waking tracepoints fired along this path are the backbone of latency tools like perf sched and runqlat.

The fair-class callback named here, check_preempt_wakeup_fair, is verified as the .wakeup_preempt method in the fair class’s sched_class table (fair.c v6.12, line 13577).

See Also

Graph View

Backlinks