vault

kthreadd and Kernel Threads

14 min read

kthreadd and Kernel Threads

A kernel thread is a schedulable task that lives entirely inside the kernel: it runs only in kernel mode, never returns to userspace, and — crucially — has no userspace address space of its own (current->mm == NULL). The kernel needs many such threads to do background work asynchronously: flushing dirty pages, reclaiming memory, servicing deferred interrupt work, draining workqueues. All of them are manufactured by a single special task, kthreadd, which always holds PID 2 and is the parent of every kernel thread on the system. kthreadd exists so that thread creation always happens from a clean, known kernel context — never inherited from whatever userspace process happened to trigger the creation (a modprobe, a CPU hotplug event). This note walks the factory: how kthread_create()/kthread_run() queue a request, how kthreadd services it by forking, and the lifecycle primitives (kthread_should_stop, kthread_stop, parking, CPU binding). It is pinned to Linux 6.12 LTS (kernel/kthread.c, init/main.c).

Mental Model — A Single-Threaded Factory Behind a Work Queue

Think of kthreadd as a lone factory worker sitting beside an inbox. Anyone in the kernel who wants a new kernel thread does not create it directly. Instead they fill out an order form (a struct kthread_create_info), drop it in the inbox (kthread_create_list), ring a bell (wake_up_process(kthreadd_task)), and then block waiting for the finished product. kthreadd wakes, picks up each form, performs the actual fork-equivalent that brings a new task into existence, and hands the result back through a completion. The requester unblocks with a freshly minted (but still stopped) kernel thread.

The reason for this indirection is context hygiene. Kernel threads are created from wildly varying contexts — a userspace process invoking modprobe to load a driver, the CPU-hotplug machinery bringing a core online, a filesystem mounting. If a new kernel thread simply forked from that caller, it would inherit the caller’s signal handlers, its CPU affinity, its cgroup, its memory policy, possibly its open files and userspace mapping. By routing every creation through kthreadd, every kernel thread instead inherits kthreadd’s deliberately scrubbed environment: signals ignored, default NUMA policy, the housekeeping CPU mask, the root cgroup. The source file’s header comment states this directly: “Creation is done via kthreadd, so that we get a clean environment even if we’re invoked from userspace (think modprobe, hotplug cpu, etc.).” (kernel/kthread.c).

sequenceDiagram
    participant Caller as Any kernel code<br/>(e.g. kswapd setup,<br/>workqueue, modprobe)
    participant List as kthread_create_list<br/>(+ kthread_create_lock)
    participant Kd as kthreadd (PID 2)
    participant New as New kernel thread

    Caller->>List: list_add_tail(create)
    Caller->>Kd: wake_up_process(kthreadd_task)
    Caller->>Caller: wait_for_completion_killable(&done)
    Kd->>Kd: wake, drain list under lock
    Kd->>New: create_kthread() -> kernel_thread(kthread, ...)
    New->>New: kthread(): reset sched/affinity,<br/>set TASK_UNINTERRUPTIBLE
    New->>Caller: complete(done) -> requester unblocks
    Note over New: thread is created but STOPPED;<br/>requester (or kthread_run) calls<br/>wake_up_process() to start it
    New->>New: __kthread_parkme(); ret = threadfn(data)

Figure: the creation handshake. What it shows: the requester never forks; it enqueues a request and blocks, kthreadd does the actual fork, and the new thread signals completion before it ever runs threadfn. The insight: there are two separate “go” signals — kthreadd brings the thread into existence (stopped), and a second wake_up_process() actually starts it running threadfn. kthread_create() gives you the stopped thread; kthread_run() does both steps for you.

What a Kernel Thread Actually Is

A normal userspace process is a struct task_struct whose ->mm field points to a struct mm_struct — the descriptor of its virtual address space (page tables, VMAs, the heap, the stack, mapped files). A kernel thread is a task_struct whose ->mm is NULL. It has no userspace page tables because it never executes userspace code; it only ever runs kernel functions. This is the defining property, and tools rely on it: htop and ps identify kernel threads partly by this kernel-only nature, conventionally displaying their names in brackets like [kswapd0] (htop issue #1001).

But a CPU always needs some page table loaded in its memory-management unit, even while running a thread that has no address space of its own. The kernel solves this with the active_mm field. When the scheduler switches to a kernel thread, it does not tear down and reload page tables; it leaves the previous task’s address space installed and records it in the kernel thread’s active_mm (this is called lazy TLB mode). The kernel thread thus borrows an address space it never touches. The borrowed mm is pinned with a lazy-TLB reference count so it cannot be freed out from under the borrower. This is why you will see current->mm == NULL but current->active_mm != NULL inside a kernel thread.

Occasionally a kernel thread genuinely needs to touch userspace memory — for example the user-mode helper machinery, or io_uring/vhost worker threads acting on behalf of a userspace process. For that, the kernel provides kthread_use_mm() / kthread_unuse_mm(), which temporarily adopt a real address space. The mechanism (from kernel/kthread.c) takes a reference with mmgrab(mm), then under task_lock and with interrupts disabled sets tsk->active_mm = mm; tsk->mm = mm; and calls switch_mm_irqs_off() to install the real page tables; kthread_unuse_mm() reverses it, clearing tsk->mm = NULL and returning to lazy-TLB mode. The two functions guard against using them on a non-kthread with WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)). Every kernel thread carries the PF_KTHREAD flag in task->flags, which is how the rest of the kernel cheaply asks “is this a kernel thread?”.

Familiar examples, each a kernel thread parented by kthreadd:

  • ksoftirqd/N — one per CPU, processes softirqs when the softirq load is too high to handle inline (see ksoftirqd and Softirq Load).
  • kworker/... — the workqueue worker pool threads that execute deferred work_struct callbacks (see Workqueue Internals kworker and Worker Pools).
  • kswapd0 (one per NUMA node, kswapdN) — background page reclaim under memory pressure (see kswapd and Background Reclaim).
  • migration/N — the per-CPU stop-machine / task-migration thread, a high-priority “stopper” used to move tasks between CPUs and run code with a CPU held exclusively. It is a per-CPU, CPU-bound, parkable kernel thread, the archetype of the binding/parking machinery described below.

Why PID 2 and Not PID 1 — the Genealogy of Tasks

Three special tasks bootstrap the system, and their PIDs are not arbitrary. The very first task, PID 0, is the idle task (the “swapper”) — it is what a CPU runs when it has nothing else to do; it is not created by fork but hand-built during early boot. From PID 0, the function rest_init spawns two children in a deliberate order, visible in init/main.c:

/* PID 1: the userspace init */
pid = user_mode_thread(kernel_init, NULL, CLONE_FS);
...
/* PID 2: the kernel thread daemon */
pid = kernel_thread(kthreadd, NULL, NULL, CLONE_FS | CLONE_FILES);
rcu_read_lock();
kthreadd_task = find_task_by_pid_ns(pid, &init_pid_ns);
rcu_read_unlock();

kernel_init (PID 1) is spawned first so that it claims PID 1, then kthreadd (PID 2) immediately after. The comment in rest_init explains the ordering: “We need to spawn init first so that it obtains pid 1, however the init task will end up wanting to create kthreads, which, if we schedule it before we create kthreadd, will OOPS.” The new kernel_init is in fact made to wait for kthreadd to finish setting up, via a kthreadd_done completion, before it does any real work.

The deep reason all kernel threads descend from PID 2, never PID 1, is reparenting. When any process dies leaving still-running children, those orphaned children are reparented — adopted by a new parent so they still have somewhere to be reaped. The default reparenting target is PID 1, the userspace init (see Reaping Orphans and the subreaper). If kernel threads were children of PID 1, then a userspace init (systemd, say) would suddenly find itself the parent of [kswapd0] and [migration/3]. That is conceptually wrong and operationally dangerous: userspace init has no business reaping or signalling kernel threads, and the bookkeeping for kernel threads must never leak into the userspace process tree. By rooting the entire kernel-thread subtree at PID 2, the kernel keeps its own threads cleanly partitioned from userspace’s process tree — orphaned kernel threads reparent to kthreadd, not to userspace init. PID 2 is therefore guaranteed by construction: it is created right after PID 1, before PID 1 is scheduled, and every kernel thread is forked from it (Linux Foundation forum discussion).

Mechanical Walk-through — From Request to Running Thread

1. The request (kthread_create / kthread_create_on_node). A caller invokes the macro kthread_create(threadfn, data, namefmt, ...), which expands to kthread_create_on_node(... NUMA_NO_NODE ...) (include/linux/kthread.h). Internally __kthread_create_on_node() kmallocs a struct kthread_create_info, stashing the function pointer threadfn, the data argument, the NUMA node, an on-stack completion done, and a full_name built with kvasprintf. It then enqueues the request and rings the bell:

spin_lock(&kthread_create_lock);
list_add_tail(&create->list, &kthread_create_list);
spin_unlock(&kthread_create_lock);
 
wake_up_process(kthreadd_task);
...
wait_for_completion_killable(&done);

The wait is killable for a specific reason spelled out in the comment: while kthreadd is allocating memory for the new thread, the requesting task might itself be chosen by the OOM killer. A killable wait lets the requester die rather than deadlock the OOM killer.

2. kthreadd’s service loop. kthreadd() (kernel/kthread.c) first scrubs its own context — set_task_comm(tsk, "kthreadd"), ignore_signals(tsk), restricts itself to the housekeeping CPU mask, sets PF_NOFREEZE, initialises its cgroup — then spins forever:

for (;;) {
    set_current_state(TASK_INTERRUPTIBLE);
    if (list_empty(&kthread_create_list))
        schedule();              /* sleep until woken */
    __set_current_state(TASK_RUNNING);
 
    spin_lock(&kthread_create_lock);
    while (!list_empty(&kthread_create_list)) {
        struct kthread_create_info *create;
        create = list_entry(kthread_create_list.next, ...);
        list_del_init(&create->list);
        spin_unlock(&kthread_create_lock);
        create_kthread(create);   /* the actual fork */
        spin_lock(&kthread_create_lock);
    }
    spin_unlock(&kthread_create_lock);
}

Note the careful state dance: it sets TASK_INTERRUPTIBLE before checking the list, so a wakeup racing with the emptiness check cannot be lost.

3. The fork (create_kthread → kernel_thread). create_kthread() calls:

pid = kernel_thread(kthread, create, create->full_name,
                    CLONE_FS | CLONE_FILES | SIGCHLD);

kernel_thread() is the in-kernel fork. The new task’s entry point is the static function kthread() (note: lowercase, distinct from kthreadd), and create is passed as its argument. CLONE_FS | CLONE_FILES share the filesystem-context and open-file table with kthreadd (a clean slate); SIGCHLD makes kthreadd the parent for child-reaping purposes.

4. The new thread’s preamble (kthread()). The new task starts in kthread(). It copies threadfn/data out of the create struct (which lives on the creator’s stack, so it must be read promptly), resets scheduler policy to SCHED_NORMAL and CPU affinity to the housekeeping mask (in case it inherited something), then announces itself and stops:

__set_current_state(TASK_UNINTERRUPTIBLE);
create->result = current;
preempt_disable();
complete(done);                 /* tell creator: "I exist" */
schedule_preempt_disabled();    /* ...then immediately sleep */
preempt_enable();
 
ret = -EINTR;
if (!test_bit(KTHREAD_SHOULD_STOP, &self->flags)) {
    cgroup_kthread_ready();
    __kthread_parkme(self);
    ret = threadfn(data);       /* the real work, once woken */
}
kthread_exit(ret);

This is the crux of the two-phase start. After complete(done), the requester’s kthread_create() returns a fully formed but stopped task_struct. The thread will not run threadfn until someone calls wake_up_process() on it. If the caller used kthread_run() instead, the macro does that wakeup automatically:

#define kthread_run(threadfn, data, namefmt, ...)                  \
({                                                                 \
    struct task_struct *__k                                        \
        = kthread_create(threadfn, data, namefmt, ## __VA_ARGS__); \
    if (!IS_ERR(__k))                                              \
        wake_up_process(__k);                                      \
    __k;                                                           \
})

The two-phase design (kthread_create then wake_up_process) exists so the caller can bind the thread to a CPU (or otherwise configure it) before it ever runs — see binding, below.

Lifecycle Primitives — Stop, Park, Bind

Stopping (kthread_should_stop / kthread_stop). A long-running kernel thread typically loops while (!kthread_should_stop()) { ... }. kthread_should_stop() simply tests a per-thread flag: test_bit(KTHREAD_SHOULD_STOP, &to_kthread(current)->flags). The owner shuts it down with kthread_stop(k), which sets that bit, unparks the thread, posts a TIF_NOTIFY_SIGNAL so a sleeping thread wakes, wake_up_process(k), then blocks on the thread’s exited completion and returns the thread’s threadfn return value:

int kthread_stop(struct task_struct *k)
{
    ...
    set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
    kthread_unpark(k);
    set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
    wake_up_process(k);
    wait_for_completion(&kthread->exited);
    ret = kthread->result;
    ...
    return ret;
}

The contract: the thread must cooperate by periodically checking kthread_should_stop() and returning. There is no forced kill — kthread_stop is a polite request the thread must honour.

Parking (kthread_park / kthread_unpark / kthread_should_park). Parking is “stop, but keep the thread alive so it can be restarted.” When a CPU is taken offline, its per-CPU kernel threads (like migration/N) cannot keep running there but should not be destroyed either — they will be needed again when the CPU comes back. kthread_park() sets KTHREAD_SHOULD_PARK, wakes the thread, and waits for it to reach the parked state. The thread, in __kthread_parkme(), transitions to the special TASK_PARKED state and sleeps. kthread_unpark() clears the bit, re-binds the thread to its CPU if it is per-CPU, and wakes it; threadfn is then re-entered. The distinction from stopping: a stopped thread is gone forever (its threadfn has returned); a parked thread is merely suspended and threadfn runs again on unpark.

CPU binding (kthread_bind / kthread_create_on_cpu). Many kernel threads must run on a specific CPU (per-CPU ksoftirqd, migration, workqueue-bound workers). kthread_bind(p, cpu) pins a just-created, still-stopped thread to one CPU — it must be stopped, because binding sets PF_NO_SETAFFINITY (forbidding later affinity changes) and the implementation wait_task_inactive()s on the thread, which is only safe while it is not running. This is precisely why creation is two-phase: kthread_create_on_cpu() creates the stopped thread, calls kthread_bind(p, cpu), records to_kthread(p)->cpu = cpu, and only then is the thread woken. Binding before the first run guarantees threadfn executes on the right CPU from its very first instruction. The KTHREAD_IS_PER_CPU flag (set via kthread_set_per_cpu) marks threads that must be re-bound on unpark after CPU hotplug.

Failure Modes and Common Misunderstandings

“A kernel thread is just a thread of the kernel process.” No. There is no “kernel process.” Each kernel thread is an independent task_struct with its own PID, scheduled exactly like any other task. What makes it special is only mm == NULL and PF_KTHREAD.

Dereferencing userspace pointers in a kernel thread. Because a kernel thread has no mm, a raw userspace virtual address is meaningless inside it. Code that needs userspace memory must first kthread_use_mm() to adopt the target address space; forgetting this leads to faults or, worse, reading whatever happened to be in the borrowed active_mm. The WARN_ON_ONCE(tsk->mm) guards in kthread_use_mm catch nesting bugs.

Expecting kthread_stop to be instantaneous or forceful. It blocks until the thread’s threadfn actually returns. A kernel thread stuck in an uninterruptible sleep, or one that never checks kthread_should_stop(), will hang kthread_stop() indefinitely — a classic cause of a wedged module unload (rmmod hangs in D state). The fix is in the thread’s loop, which must check the stop condition and be wakeable.

The OOM / fork-failure path. create_kthread() can fail if kernel_thread() returns a negative PID (out of memory, PID exhaustion). It then propagates ERR_PTR(pid) back through the completion, so kthread_create() returns an IS_ERR() pointer. Callers that skip the IS_ERR() check and dereference the result will crash. Both kthread_create and kthread_run can return ERR_PTR(-ENOMEM).

Confusing kthreadd (the daemon, PID 2) with kthread (the per-thread entry function). The lowercase kthread() is the static function every new kernel thread starts in; kthreadd() is the daemon that forks them. Same file, one letter apart, very different roles.

Production Notes

In top/htop/ps, kernel threads appear with bracketed names ([kworker/0:1], [ksoftirqd/0], [kswapd0]) and a parent PID (PPID) of 2. A quick way to enumerate every kernel thread on a live system is to list tasks whose parent is kthreadd: e.g. ps -e --ppid 2 shows the direct children of PID 2, and (because some kernel threads spawn further kernel threads, also via kthreadd) the broader set is everything with mm == NULL. The bracketed-name convention plus PPID 2 is the operator’s everyday signal that a task is kernel-internal. Per-CPU kernel threads scale with core count, so on a 128-core box you will see 128 ksoftirqd and migration threads — normal, and mostly idle.

A frequent real-world interaction is CPU isolation (isolcpus= / the housekeeping mask): note that kthread() and kthreadd() both call set_cpus_allowed_ptr(..., housekeeping_cpumask(HK_TYPE_KTHREAD)). This is the kernel actively keeping freely-floating kernel threads off isolated CPUs so that latency-sensitive workloads pinned there are not disturbed by background kernel work. Per-CPU bound threads (which must run on the isolated CPU) are the exception. This interplay is why understanding kthreadd’s affinity defaults matters for real-time and HPC tuning.

See Also

Graph View