A workqueue is the kernel’s general-purpose mechanism for running deferred work in a context where it may sleep. You describe a job as a work_struct — a struct that holds a function pointer — initialize it with INIT_WORK(), and hand it to the kernel with queue_work() (or the shorthand schedule_work()). Sometime later, a kernel worker thread named kworker/... picks the work item off its queue and runs your callback in ordinary process context, so the callback may take mutexes, allocate memory with GFP_KERNEL, and block on I/O — all the things a softirq or hard-interrupt handler must never do (per the workqueue documentation, Linux v6.12). Since 2010 this has been implemented by the Concurrency-Managed Workqueue (CMWQ), whose central idea is to decouple the work items a driver queues from the worker threads that actually run them: instead of one thread per workqueue, all workqueues share a small pool of per-CPU worker threads, and the pool grows or shrinks just enough to keep the CPU busy without starving. This is the bottom half to reach for whenever the deferred work can sleep but is not bound to a specific interrupt line (when it is, prefer a threaded IRQ).
What this note owns, and what it does not. This note owns the concurrency-managed idea the title names — why one runnable worker per pool suffices, the nr_running accounting and the two scheduler hooks that maintain it, the WQ_CPU_INTENSIVE opt-out and its auto-detector — together with the front-end API (work_struct, queue_work, the WQ_* flags, max_active, flush and cancel), the predefined system workqueues, and how all of it is read off a live machine. The deeper backend plumbing — worker creation, the pool manager, idle reaping, pool_workqueue internals and the rescuer’s mayday protocol — belongs to Workqueue Internals kworker and Worker Pools. The design-time choice between workqueues and the other deferral mechanisms belongs to Choosing a Deferral Mechanism; the comparison of all five mechanisms as observed on a running system lives in Softirqs and the Softirq Vector. Link, do not duplicate.
Overlapping note — flagged, not merged
Workqueue Internals kworker and Worker Pools also carries sections titled “Concurrency Management — One Runnable Worker” and “The kworker Threads and Reading Their Names”, and its own scope statement claims concurrency management for the backend note. The two notes now genuinely overlap on that topic: this note treats it as the defining model the title refers to (premise, scheduler hooks, opt-out, worked timing example), while the sibling treats it as one step in the backend pipeline. Neither has been edited to remove the other’s material, because silently deleting verified prose is worse than a documented overlap. A later editorial pass should decide which note keeps the mechanism and reduce the other to a cross-link. duplicate-candidateuncertain
Mental Model
A workqueue separates what to do (the work item) from who does it (the worker thread) and when (later, in process context). The work item is a passive data structure — “a simple struct that holds a pointer to the function that is to be executed asynchronously” (workqueue docs, v6.12). The workqueue is a named queue you submit to. The actual execution context is a kworker thread drawn from a shared pool. This three-way decoupling is the whole design: you never create or manage a thread yourself, you just submit a function and the CMWQ machinery finds (or spawns) a worker to run it.
flowchart LR
subgraph Driver
W1["work_struct A<br/>(func ptr)"]
W2["work_struct B"]
end
W1 -->|"queue_work(wq, &A)"| Q["workqueue<br/>(system_wq / custom)"]
W2 -->|"queue_work(wq, &B)"| Q
Q --> POOL["per-CPU worker pool<br/>(shared by ALL workqueues)"]
POOL --> K1["kworker thread 1"]
POOL --> K2["kworker thread 2<br/>(spawned when 1 sleeps)"]
K1 -->|"runs A().func"| RUNA["A's callback<br/>process ctx — CAN sleep"]
K2 -->|"runs B().func"| RUNB["B's callback<br/>process ctx — CAN sleep"]
The CMWQ decoupling. What it shows: drivers queue passive work_struct items onto named workqueues, but those queues all feed into a small set of shared per-CPU worker pools; a pool runs work on kworker threads and spawns another worker only when the running one goes to sleep. The insight to take: the number of threads is not the number of workqueues — that was the old design’s fatal flaw. CMWQ keeps thread count low by sharing pools and managing concurrency dynamically, so you can create as many logically-distinct workqueues as you like without spawning a thread for each.
Why CMWQ Exists — The Old Design’s Failure
Understanding CMWQ requires knowing what it replaced, because the concurrency-managed part is a direct answer to a real scaling disaster. In the original workqueue implementation (workqueue docs, v6.12):
“In the original wq implementation, a multi threaded (MT) wq had one worker thread per CPU and a single threaded (ST) wq had one worker thread system-wide.”
This had several compounding problems, all quoted from the kernel docs:
Thread proliferation. “A single MT wq needed to keep around the same number of workers as the number of CPUs,” and across many workqueues this “saturated the default 32k PID space just booting up” on large machines. Every workqueue cost N threads whether or not it had work.
Rigid, separate pools. “Each wq maintained its own separate worker pool. An MT wq could provide only one execution context per CPU while an ST wq one for the whole system.” Concurrency was fixed at creation time, not on demand.
Deadlock-prone. “Work items had to compete for those very limited execution contexts leading to various problems including proneness to deadlocks around the single execution context.” If a work item blocked, everything behind it on that single context was stuck.
Forced bad trade-offs. Subsystems like libata ended up “choosing to use ST wq for polling PIOs and accepting an unnecessary limitation that no two polling PIOs can progress at the same time,” just to conserve threads.
CMWQ (merged in 2.6.36, 2010) fixes all of this with two design goals, again quoted:
“Use per-CPU unified worker pools shared by all wq to provide flexible level of concurrency on demand without wasting a lot of resource.”
“Automatically regulate worker pool and level of concurrency so that the API users don’t need to worry about such details.”
The mechanism for “automatically regulate” is a hook into the scheduler. CMWQ maintains, per CPU, two worker pools — one for normal-priority work and one for high-priority work — and these pools are shared by every workqueue targeting that CPU. The pool is told whenever one of its workers wakes up or goes to sleep (workqueue docs, v6.12):
“The worker-pool is notified whenever an active worker wakes up or sleeps and keeps track of the number of the currently runnable workers.”
And the regulation rule itself:
“As long as there are one or more runnable workers on the CPU, the worker-pool doesn’t start execution of a new work, but, when the last running worker goes to sleep, it immediately schedules a new worker so that the CPU doesn’t sit idle while there are pending work items.”
That is the heart of “concurrency-managed”: keep exactly one worker running per pool per CPU when possible (no pointless context-switching), but the instant the running worker blocks (e.g. your callback called mutex_lock() and slept), wake another worker so pending work keeps flowing and the CPU never idles with work waiting. The pool/kworker internals — how workers are created, parked, and destroyed — are deferred to Workqueue Internals kworker and Worker Pools; this note covers the API and model.
Concurrency Management — the mechanism the name refers to
This is the section the note is named for, and it is the part that summaries of workqueues almost always skip. “Concurrency-managed” is not a slogan; it names a specific, small, and rather elegant piece of machinery that hooks directly into the CPU scheduler. Everything else in CMWQ — pools, flags, max_active — is scaffolding around this one idea.
The premise, stated precisely
One running worker per pool per CPU is enough, as long as a worker that stops running is immediately replaced.
The reasoning behind that premise is worth walking through, because it is not obvious. Suppose you have a hundred pending work items on one CPU. How many threads should run them? The naive answers are both wrong. One hundred wastes an enormous amount of memory on kernel stacks and makes the scheduler thrash between them — and it buys nothing, because a single CPU can only execute one thing at a time anyway. One is nearly right, and would be exactly right if work items never blocked: the CPU is saturated, the items complete in order, no context switch is wasted. But work items do block — that is the entire reason a workqueue exists rather than a softirq — and the moment the single worker calls mutex_lock() and sleeps, the CPU goes idle with ninety-nine items still pending. That is the failure the old one-thread-per-workqueue design suffered from in a different form.
CMWQ’s answer is to take the “one worker” model and patch exactly the hole in it: keep one worker running, and the instant it stops running, start another. The documentation states the rule in a single sentence (workqueue.rst, v6.12):
“As long as there are one or more runnable workers on the CPU, the worker-pool doesn’t start execution of a new work, but, when the last running worker goes to sleep, it immediately schedules a new worker so that the CPU doesn’t sit idle while there are pending work items.”
The thread count therefore tracks the number of blocked work items, not the number of queued ones and not the number of workqueues. A pool where nothing ever blocks converges to a single worker no matter how much work you throw at it. A pool where every item blocks on I/O grows a worker per outstanding I/O. Neither case required the programmer to think about it.
The accounting: nr_running and WORKER_NOT_RUNNING
The implementation is one integer per pool and two scheduler callbacks. The integer is worker_pool::nr_running (workqueue.c:200); the whole of concurrency management is four small predicates that read it:
Read them as four questions the pool asks itself. Is there work with nobody running? → wake someone (need_more_worker). Do I have an idle worker to wake, or must I create one from scratch? → may_start_working / need_to_create_worker. I am a running worker that just finished an item — should I take another? → keep_working, which says yes only while nr_running <= 1, i.e. only while I am the only runner. The <= 1 rather than == 1 is not sloppiness: an unbound pool never increments nr_running at all, so the test must also be true at zero. The comment on need_more_worker makes that explicit — “because unbound workers never contribute to nr_running, this function will always return %true for unbound pools as long as the worklist isn’t empty,” which is the one-line statement of why unbound pools are not concurrency-managed.
nr_running is maintained from two places. The first is worker_set_flags()/worker_clr_flags(), which adjust it whenever a worker enters or leaves the WORKER_NOT_RUNNING state — a composite mask of WORKER_PREP | WORKER_CPU_INTENSIVE | WORKER_UNBOUND | WORKER_REBOUND (workqueue.c:96). A worker in any of those states is, for accounting purposes, not running, even though it is on the CPU burning cycles. That distinction is the hinge on which WQ_CPU_INTENSIVE turns.
The second is the scheduler itself.
The scheduler hooks — wq_worker_sleeping() and wq_worker_running()
This is the part that makes CMWQ different from every userspace thread pool: it does not poll for blocked workers, it is told. The core scheduler calls into the workqueue code on the exact transitions:
/* kernel/workqueue.c:1413-1454, v6.12 — called from schedule() */void wq_worker_sleeping(struct task_struct *task){ struct worker *worker = kthread_data(task); ... if (worker->flags & WORKER_NOT_RUNNING) return; /* already not counted; nothing to do */ ... WRITE_ONCE(worker->sleeping, 1); raw_spin_lock_irq(&pool->lock); ... pool->nr_running--; if (kick_pool(pool)) /* <-- wake/create a replacement NOW */ worker->current_pwq->stats[PWQ_STAT_CM_WAKEUP]++; raw_spin_unlock_irq(&pool->lock);}/* called when a worker returns from schedule() */void wq_worker_running(struct task_struct *task){ struct worker *worker = kthread_data(task); if (!READ_ONCE(worker->sleeping)) return; preempt_disable(); if (!(worker->flags & WORKER_NOT_RUNNING)) worker->pool->nr_running++; preempt_enable(); worker->current_at = worker->task->se.sum_exec_runtime; /* reset CPU-intensive clock */ WRITE_ONCE(worker->sleeping, 0);}
The load-bearing line is kick_pool(pool) inside wq_worker_sleeping(). It runs from inside schedule(), on the blocking worker’s own stack, before the CPU has been handed to anyone else. The replacement worker is therefore made runnable in the same scheduling operation that takes the sleeper off the CPU — which is why the documentation can say “immediately” without hedging. There is no timer, no watchdog, no periodic check. The statistic it bumps, PWQ_STAT_CM_WAKEUP, is literally “concurrency-management wakeups,” and is exported per-workqueue in the debugfs dump.
Two details in that code repay attention. WRITE_ONCE(worker->sleeping, 1) is a guard against double-decrementing: if the worker is preempted between setting the flag and reaching wq_worker_running(), the early if (READ_ONCE(worker->sleeping)) return in a re-entered wq_worker_sleeping() prevents a second decrement. And the WORKER_NOT_RUNNING re-check after taking pool->lock exists because unbind_workers() (CPU hot-unplug) can preempt in between and reset nr_running — decrementing after that would corrupt the count on the newly-unbound pool. Both comments are in the source, and both are the kind of race that only shows up on a 128-CPU machine under hotplug.
stateDiagram-v2
direction LR
state "IDLE — on pool->idle_list — nr_running += 0" as Idle
state "RUNNING — executing work->func() — nr_running += 1" as Running
state "BLOCKED — inside schedule() — nr_running += 0" as Blocked
state "CPU_INTENSIVE — still on CPU, but nr_running += 0" as CpuInt
state "DYING — idle 5 min and too_many_workers()" as Dying
[*] --> Idle: create_worker()
Idle --> Running: kick_pool() wakes it, nr_running++
Running --> Blocked: work blocks, wq_worker_sleeping(), nr_running--, kick_pool()
Blocked --> Running: wakeup, wq_worker_running(), nr_running++, current_at reset
Running --> CpuInt: WQ_CPU_INTENSIVE, or wq_worker_tick() over threshold, nr_running--
CpuInt --> Running: work item finishes, nr_running++
Running --> Idle: worklist empty or not keep_working(), nr_running--
Idle --> Dying: idle_timer after IDLE_WORKER_TIMEOUT
Dying --> [*]
The concurrency-management state machine for one worker, annotated with its effect on pool->nr_running. What it shows: the pool does not count threads, it counts runnable threads — and three different transitions decrement that count for three different reasons. Blocking decrements it because the CPU is genuinely free. Going idle decrements it because there is no work. Becoming CPU_INTENSIVE decrements it while the worker is still executing on the CPU — a deliberate lie to the accounting, which is exactly what the flag is for. The insight to take: every transition that reaches nr_running == 0 with a non-empty worklist triggers kick_pool(), and that single invariant is the whole of concurrency management.
WQ_CPU_INTENSIVE — the opt-out, and why it is needed
The premise breaks for one class of work item: the one that never sleeps. A callback that computes for fifty milliseconds without blocking keeps nr_running at 1 for the entire fifty milliseconds. keep_working() stays true for it and need_more_worker() stays false for everyone else, so every other item in that pool on that CPU is stalled behind it. The pool is behaving correctly by its own rule — there is a runnable worker — but the rule assumed work items are short.
WQ_CPU_INTENSIVE is the escape: it makes process_one_work() call worker_set_flags(worker, WORKER_CPU_INTENSIVE) before dispatching (workqueue.c:3179), which decrements nr_running even though the worker is about to run flat out. The pool then sees nr_running == 0, kicks another worker, and the two compete for the CPU under ordinary scheduler rules instead of one blocking the other. The docs put it as: “runnable CPU intensive work items will not prevent other work items in the same worker-pool from starting execution.” The flag is meaningless for unbound workqueues, because unbound pools never counted nr_running in the first place.
The kernel’s own worked example makes the effect concrete. With w0 burning 5 ms, sleeping 10 ms, burning 5 ms; w1 and w2 each burning 5 ms then sleeping 10 ms; all on one CPU:
Time (ms)
Original workqueue (one context)
CMWQ, max_active ≥ 3
CMWQ, w1/w2 on a WQ_CPU_INTENSIVE queue
0
w0 starts, burns
w0 starts, burns
w0 starts, burns
5
w0 sleeps (CPU idle)
w0 sleeps → w1 starts
w0 sleeps → w1 and w2 both start
10
(idle)
w1 sleeps → w2 starts
w1 sleeps
15
w0 wakes, burns
w2 sleeps; w0 wakes, burns
w2 sleeps; w0 wakes, burns
20
w0 finishes; w1 starts
w0 finishes; w1 finishes
w0 finishes; w1 finishes
25
w1 sleeps
w2 finishes — all done at 25 ms
w2 finishes — all done at 25 ms
35
w1 finishes; w2 starts
—
—
50
w2 finishes — all done at 50 ms
—
—
Same three work items, same CPU, same total CPU demand: 50 ms wall-clock under the old design, 25 ms under CMWQ. The gain is entirely from filling the 10 ms sleep windows, which is precisely what kick_pool() on wq_worker_sleeping() exists to do. (Table transcribed from the “Example Execution Scenarios” section of workqueue.rst, v6.12.)
Automatic CPU-intensive detection — and a default-value trap
Relying on driver authors to remember WQ_CPU_INTENSIVE did not work, so since the CMWQ rework the kernel detects the condition itself. wq_worker_tick() is called from sched_tick() — i.e. from the timer interrupt, on every tick during which a kworker is the current task (workqueue.c:1459):
worker->current_at is a snapshot of the task’s sum_exec_runtime taken when the item was dispatched and re-taken on every wakeup, so the comparison measures CPU time consumed without sleeping — not wall-clock, which would misfire on a preempted worker. Exceed the threshold and the worker is demoted to CPU_INTENSIVE exactly as if the flag had been set, and a replacement is kicked.
A default-value trap worth remembering
Reading the declaration alone gives the wrong default. In kernel/workqueue.c:
static unsigned long wq_cpu_intensive_thresh_us = ULONG_MAX; /* line 418 */module_param_named(cpu_intensive_thresh_us, wq_cpu_intensive_thresh_us, ulong, 0644);
ULONG_MAX is a sentinel, not a value — the comment above it says so: “ULONG_MAX indicates that the user hasn’t overridden it with a boot parameter. The actual value is initialized in wq_cpu_intensive_thresh_init().” That function sets 10,000 µs (10 ms), and then scales it up on slow machines: bogo = max(loops_per_jiffy / 500000 * HZ, 1), and if (bogo < 4000) thresh = min(thresh * 4000 / bogo, USEC_PER_SEC) — up to a 1-second ceiling. The stated rationale is that 10 ms “is just below what most consider human-perceivable,” but “the kernel also runs on a lot slower CPUs including microcontrollers where the threshold is way too low.”
Verified against the running machine rather than assumed. Its bogomips is 5988.64, comfortably above 4000, so no scaling should apply and the value should be exactly 10,000:
The prediction and the measurement agree. On a 400-BogoMIPS embedded board the same code would yield 100,000 µs instead.
One companion knob only exists if the kernel was built for it: wq_cpu_intensive_warning_thresh (default 4) controls how often wq_cpu_intensive_report() prints a warning naming the offending work function, and it is compiled in only under CONFIG_WQ_CPU_INTENSIVE_REPORT. On the measurement box that option is not set, so these warnings will never appear in dmesg there — worth knowing before concluding from a silent log that no work item is hogging a worker.
Mechanical Walk-through — The API
With the concurrency model established, the API itself is small. Three types, four verbs, and one allocation function.
The work item. A job is a struct work_struct (from workqueue_types.h, v6.12):
typedef void (*work_func_t)(struct work_struct *work);struct work_struct { atomic_long_t data; /* packed: pending bit + pool/wq pointer */ struct list_head entry; /* links into the pool's worklist */ work_func_t func; /* your callback */#ifdef CONFIG_LOCKDEP struct lockdep_map lockdep_map;#endif};
The data field is a clever packed word: it holds both a pending bit (set while the item is queued, so re-queuing the same item is a no-op) and a pointer to the pool/workqueue it belongs to. entry links it into a worklist; func is the callback. Note the callback receives the work_struct * itself, not arbitrary context — the idiom is to embed work_struct inside your own struct and recover the outer struct with container_of().
Initializing and queuing. You initialize a work item with INIT_WORK(&my_work, my_func) (or DECLARE_WORK(name, func) for a static one), then submit it. queue_work() puts it on a specific workqueue; schedule_work() is the shorthand that targets the default system_wq (workqueue.h, v6.12):
The boolean return is important and routinely ignored: it is true if the item was queued, and false if the work was already queued (its pending bit was set). This is the documented behavior — “Returns false if work was already queued, true otherwise.” So queuing the same work_struct twice before it runs coalesces into a single execution, not two. WORK_CPU_UNBOUND means “let the workqueue pick the CPU” (for a normal per-CPU workqueue this means the queuing CPU).
Delayed work. To run something after a delay, use struct delayed_work, which is just a work_struct plus a timer (workqueue.h, v6.12):
You init it with INIT_DELAYED_WORK(&dw, func) and queue it with queue_delayed_work(wq, &dw, delay) or schedule_delayed_work(&dw, delay), where delay is in jiffies. Mechanically, the timer fires after delay, and its expiry is what actually queues the embedded work_struct onto the workqueue — so the delay is a timer wait, then the work runs in a worker as usual. This is the standard way to implement “retry in 100 ms” or a periodic poll (by re-queuing from the callback).
The default system workqueues. Most code never creates its own workqueue; it uses one of the predefined ones declared in workqueue.h (v6.12), each with a documented purpose:
system_wq — “one used by schedule[_delayed]_work[_on](). Multi-CPU multi-threaded.” The general default.
system_highpri_wq — “similar to system_wq but for work items which require WQ_HIGHPRI.” Runs on the high-priority (elevated-nice) pool.
system_long_wq — “similar to system_wq but may host long running works.” Keeps slow jobs from clogging the general queue.
system_unbound_wq — “unbound workqueue. Workers are not bound to any specific CPU.” For CPU-intensive or latency-fluctuating work the scheduler should be free to place.
system_freezable_wq — “equivalent to system_wq except that it’s freezable” (drains during system suspend).
system_power_efficient_wq and system_freezable_power_efficient_wq — become unbound when the workqueue.power_efficient kernel parameter is set, otherwise behave like their non-efficient counterparts.
Creating a custom workqueue. When you need isolation, ordering, or specific behavior, allocate your own with alloc_workqueue() (v6.12):
__printf(1, 4) struct workqueue_struct *alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...);
fmt is a printf-style name (shows up in kworker thread names and /sys), flags is an OR of WQ_* flags (below), and max_active bounds concurrency. max_active == 0 requests the default. The active-limit constants (v6.12) are:
max_active is, per the docs, “the maximum number of execution contexts per CPU which can be assigned to the work items of a wq” — “This is always a per-CPU attribute, even for unbound workqueues.” The default of 256 (with a hard ceiling of 512) is deliberately high so it is “not the limiting factor while providing protection in runaway cases.” You rarely need to tune it; the common reasons to call alloc_workqueue() at all are the flags or to get an ordered (max_active == 1, unbound) queue.
The full path of one work item, end to end
sequenceDiagram
autonumber
participant DRV as Driver (any context)
participant WQ as workqueue<br/>(front end, no threads)
participant PWQ as pool_workqueue<br/>(per-CPU binding)
participant POOL as worker_pool<br/>(worklist + nr_running)
participant W as kworker thread
participant CB as your work.func()
DRV->>WQ: queue_work(wq, work)
WQ->>WQ: test_and_set WORK_STRUCT_PENDING on work.data
alt bit was already set
WQ-->>DRV: return false — coalesced, one execution
else bit was clear
WQ->>PWQ: pick pwq for this CPU (or unbound pod)
PWQ->>POOL: list_add_tail(work.entry, pool.worklist)
POOL->>POOL: kick_pool() — need_more_worker() ?
POOL->>W: wake_up_process(first_idle_worker)
WQ-->>DRV: return true
end
W->>POOL: worker_thread(): worker_clr_flags(WORKER_IDLE), nr_running++
W->>POOL: process_one_work(): current_at = sum_exec_runtime
POOL->>W: strscpy(worker.desc, pwq.wq.name) [the ps suffix]
W->>W: set_work_pool_and_clear_pending() — PENDING cleared BEFORE the call
Note over W: work may now be re-queued while still running
W->>CB: work.func(work)
CB->>CB: mutex_lock / kmalloc(GFP_KERNEL) / block on I/O
CB->>POOL: schedule() calls wq_worker_sleeping(): nr_running--, kick_pool()
POOL->>POOL: a SECOND worker starts the next item
CB-->>W: callback returns
W->>POOL: cond_resched(), worker_clr_flags(WORKER_CPU_INTENSIVE)
W->>POOL: pwq_dec_nr_in_flight() — may release a flush colour
W->>POOL: keep_working() ? next item : go idle, nr_running--
A single work item from queue_work() to completion. What it shows: three separate hand-offs — driver to workqueue, workqueue to pool via a per-CPU pool_workqueue, and pool to a concrete thread — and the two moments that surprise people. Step 2: the pending bit lives in work->data itself, so queuing an already-queued item returns false and coalesces rather than running twice. Step 13: the pending bit is cleared before your callback runs, which is deliberate — it lets a callback re-queue itself — but means “pending” and “running” are genuinely different states — and a callback that re-queues itself is already pending again by the time flush_work() returns, which is why the teardown primitive you need is cancel_work_sync(), not flush_work(). The insight to take: the second worker at step 18 is the entire value proposition. Without it the CPU would idle from step 17 until the I/O completed.
Worker Pools — the backend, and how to see it
A workqueue is a front end: a name, a set of flags, a max_active budget and a flush domain. It owns no threads. The threads live in worker pools, and the mapping from workqueue to pool is what determines where your callback actually runs. The v6.12 kernel has three kinds of pool, and NR_STD_WORKER_POOLS is 2 — meaning “two standard pools per CPU”, normal and highpri — which is a constant people routinely misread as “two pools per CPU, total.”
std_nice[] = { 0, HIGHPRI_NICE_LEVEL } → 0 and −20
Yes
Yes — this is where nr_running lives
kworker/<cpu>:<id> and kworker/<cpu>:<id>H
Unbound pools
Dynamic — one per distinct workqueue_attrs
created on demand by get_unbound_pool(); “the number of these backing pools is dynamic”
Whatever the attrs say (0 or −20 for the two default sets)
Yes
No — unbound workers never contribute to nr_running
kworker/u<poolid>:<id>
Rescuers
One per WQ_MEM_RECLAIM workqueue
worker->rescue_wq set; not a pool member in the normal sense
RESCUER_NICE_LEVEL = −20
Yes
n/a — summoned by mayday, not by concurrency management
kworker/R-<wqname>
flowchart TB
subgraph FE["Front end — workqueues (no threads of their own)"]
direction LR
SW["system_wq<br/>name 'events'<br/>flags 0"]
SH["system_highpri_wq<br/>name 'events_highpri'<br/>WQ_HIGHPRI"]
SU["system_unbound_wq<br/>name 'events_unbound'<br/>WQ_UNBOUND"]
SB["system_bh_wq<br/>name 'events_bh'<br/>WQ_BH"]
MY["your alloc_workqueue()<br/>e.g. 'kblockd', WQ_MEM_RECLAIM | WQ_HIGHPRI"]
end
subgraph BE0["Backend — CPU 0"]
P0N["cpu_worker_pools[0]<br/>nice 0<br/>concurrency-managed"]
P0H["cpu_worker_pools[1]<br/>nice -20<br/>concurrency-managed"]
B0N["bh_worker_pools[0]<br/>softirq TASKLET"]
B0H["bh_worker_pools[1]<br/>softirq HI"]
end
subgraph BE1["Backend — CPU 1 ... CPU 31 (identical)"]
P1N["cpu_worker_pools[0]"]
P1H["cpu_worker_pools[1]"]
end
subgraph BEU["Backend — unbound pools (dynamic, pool ids 128+)"]
U0["unbound pool id 128<br/>cpumask = one cache-shard pod<br/>NOT concurrency-managed"]
U1["unbound pool id 129 ..."]
end
SW -->|"per-CPU pwq"| P0N
SW -->|"per-CPU pwq"| P1N
SH --> P0H
SH --> P1H
MY --> P0H
MY --> P1H
SB --> B0N
SU --> U0
SU --> U1
MY -.->|"WQ_MEM_RECLAIM adds"| R["kworker/R-kblockd<br/>rescuer, nice -20"]
P0N --> W1["kworker/0:0<br/>kworker/0:1 ..."]
P0H --> W2["kworker/0:0H<br/>kworker/0:1H ..."]
U0 --> W3["kworker/u128:0<br/>kworker/u128:1 ..."]
Bound versus unbound pool topology, with real names from the measurement box. What it shows: many workqueues, few pools. Every non-unbound workqueue gets a pool_workqueue (“pwq”) per CPU pointing at one of that CPU’s two standard pools — so system_wq and a driver’s private queue with the same flags land on the same threads. Unbound workqueues instead get pools created to match their attributes, shared with any other unbound queue asking for identical attributes. The insight to take: creating a workqueue is nearly free — it allocates a name, a flush domain and some per-CPU pwq structs, not threads. That is what makes it reasonable to create a private workqueue for isolation, which under the pre-2010 design would have cost one thread per CPU.
The predefined system workqueues
Most code never calls alloc_workqueue(). It uses one of nine queues created during workqueue_init_early() (workqueue.c:7790–7809). The “kernel name” column matters more than it looks: that string, not the C variable name, is what appears in the kworker thread suffix and in /proc, so it is how you recognise them on a live system.
C symbol
Kernel name (seen in ps)
Allocated as
Use it for
system_wq
events
alloc_workqueue("events", 0, 0)
The default. schedule_work() targets this. Docs warn: “There are users which expect relatively short queue flush time. Don’t queue works which can run for too long.”
system_highpri_wq
events_highpri
WQ_HIGHPRI, 0
Work that must not queue behind bulk events work; runs on the nice −20 pool
system_long_wq
events_long
0, 0
Long-running work. Flags are identical to system_wq — the separation is purely a flush domain, so slow jobs do not lengthen system_wq’s flush
system_unbound_wq
events_unbound
WQ_UNBOUND, WQ_MAX_ACTIVE
Work the scheduler should place freely. Note the explicit max_active of 512, not the 256 default
system_freezable_wq
events_freezable
WQ_FREEZABLE, 0
Work that must be drained across suspend
system_power_efficient_wq
events_power_efficient
WQ_POWER_EFFICIENT, 0
Identical to system_wq unless workqueue.power_efficient=1
system_freezable_power_efficient_wq
events_freezable_pwr_efficient
WQ_FREEZABLE | WQ_POWER_EFFICIENT, 0
Both of the above
system_bh_wq
events_bh
WQ_BH, 0
Atomic, softirq-context work — the tasklet replacement. v6.9+
system_bh_highpri_wq
events_bh_highpri
WQ_BH | WQ_HIGHPRI, 0
As above, out of the HI softirq rather than TASKLET. v6.9+
That system_long_wq and system_wq have byte-identical flags is the clearest demonstration of what a workqueue is in CMWQ: not an execution resource, but “a domain for forward progress guarantee, flush and work item attributes,” as the documentation puts it. The two queues share every worker thread on the machine. What they do not share is flush_workqueue() scope — and that is the entire reason both exist.
Uncertain — these names are being renamed upstream
Verify before copying any system_*_wq name into new code written against a post-v6.12 kernel. Reason: measured by existence-check on 2026-09-12, v7.1’s include/linux/workqueue.h carries extern struct workqueue_struct *system_wq; /* use system_percpu_wq, this will be removed */ alongside new system_percpu_wq, system_dfl_wq and system_dfl_long_wq symbols, and schedule_work() there expands to queue_work(system_percpu_wq, work). The v6.12 names above are correct for v6.12 and still compile on v7.1, but system_wq is explicitly marked for removal. To resolve: re-read include/linux/workqueue.h at the target release before writing new code. uncertain
The Workqueue Flags
alloc_workqueue()’s flags argument selects behavior. The complete public enum at v6.12 (workqueue.h:369–412):
enum wq_flags { WQ_BH = 1 << 0, /* execute in bottom half (softirq) context */ WQ_UNBOUND = 1 << 1, /* not bound to any cpu */ WQ_FREEZABLE = 1 << 2, /* freeze during suspend */ WQ_MEM_RECLAIM = 1 << 3, /* may be used for memory reclaim */ WQ_HIGHPRI = 1 << 4, /* high priority */ WQ_CPU_INTENSIVE = 1 << 5, /* cpu intensive workqueue */ WQ_SYSFS = 1 << 6, /* visible in sysfs, see workqueue_sysfs_register() */ WQ_POWER_EFFICIENT = 1 << 7, __WQ_DESTROYING = 1 << 15, /* internal: workqueue is destroying */ __WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */ __WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */ __WQ_LEGACY = 1 << 18, /* internal: create*_workqueue() */ /* BH wq only allows the following flags */ __WQ_BH_ALLOWS = WQ_BH | WQ_HIGHPRI,};
Bits 15–18 are double-underscore-prefixed and are not for callers — but they are worth knowing about, because __WQ_ORDERED in particular is why one long-standing idiom stopped working (see below), and __WQ_BH_ALLOWS encodes a hard constraint the compiler will not catch for you.
Flag
Which pool it targets
What it actually changes
Reach for it when
Gotcha
WQ_BH
Per-CPU BH pool (one pseudo-worker per CPU)
Work runs in softirq context on the queueing CPU, in queueing order, and cannot sleep
Replacing a tasklet; you need tasklet-class latency without the tasklet API’s design errors
Only WQ_HIGHPRI may be combined with it (__WQ_BH_ALLOWS), and max_activemust be 0. Added in v6.9
WQ_UNBOUND
A dynamically created unbound pool matching the queue’s attributes
Workers are not pinned; the scheduler places and migrates them. Concurrency management is off — nr_running is never counted
Concurrency demand fluctuates wildly, or the work is long and CPU-bound
You lose cache locality and gain scheduler overhead. max_active semantics differ (see the contradiction noted below)
WQ_FREEZABLE
unchanged
The queue participates in the suspend freeze: pending work is drained and nothing new starts until thaw
Work that must not run across a suspend/resume boundary
A freezable queue that a resume path waits on is a hang
WQ_MEM_RECLAIM
unchanged, plus a dedicated rescuer thread
Guarantees at least one execution context “regardless of memory pressure” — a kworker/R-<name> thread is created up front
Mandatory if the work can ever run on a memory-reclaim path
Omitting it is a latent deadlock that only fires under memory pressure. 83 rescuers exist on the measurement box
WQ_HIGHPRI
Per-CPU highpri pool
Workers run at nice −20 (HIGHPRI_NICE_LEVEL = MIN_NICE). Normal and highpri pools “don’t interact with each other” — separate workers, separate concurrency management
Latency-sensitive work that must not queue behind bulk work
It is a nice level, not a real-time priority. A SCHED_FIFO task still preempts it
WQ_CPU_INTENSIVE
unchanged
Excludes items from nr_running accounting so they cannot stall the pool
Bound work that burns CPU without sleeping
Meaningless for unbound queues. Also now largely redundant: auto-detection fires at 10 ms
WQ_SYSFS
unchanged
Exposes the queue under /sys/bus/workqueue/devices/<name>/ for runtime tuning of nice, cpumask, max_active, affinity_scope
Any queue that may consume significant CPU — the docs recommend it explicitly
Creates a stable ABI you then have to keep
WQ_POWER_EFFICIENT
Per-CPU normally; unbound if workqueue.power_efficient=1
Lets a distro trade a little throughput for idle-core preservation
A queue identified as a significant power consumer
A no-op unless the boot parameter or CONFIG_WQ_POWER_EFFICIENT_DEFAULT is set. Both are off on the measurement box (power_efficient = N)
The WQ_POWER_EFFICIENT header comment is the clearest explanation in the tree of why per-CPU is not free: per-CPU queues “exclude the scheduler from choosing the CPU to execute the worker threads, which has an unfortunate side effect of increasing power consumption… a per-cpu work item scheduled from an interrupt handler on an idle CPU will force the scheduler to execute the work item on that CPU breaking the idleness.” Locality and power are in direct tension, and WQ_UNBOUND versus per-CPU is exactly that trade.
Two of these deserve more than a table row.
WQ_MEM_RECLAIM and the rescuer. The deadlock it prevents is circular and easy to miss: a work item on the reclaim path needs a worker to run it; the pool has no idle worker, so it must create one; creating a kthread allocates memory; allocating memory under pressure triggers reclaim; reclaim is waiting on the work item. WQ_MEM_RECLAIM breaks the cycle by allocating a rescuer thread at alloc_workqueue() time, when memory is plentiful. The rescuer sits idle until a pool fails to make progress, at which point the mayday mechanism summons it — MAYDAY_INITIAL_TIMEOUT is HZ/100 (10 ms at CONFIG_HZ=1000) and then MAYDAY_INTERVAL is HZ/10 (100 ms), with CREATE_COOLDOWN of one full second between failed worker-creation attempts (workqueue.c:111–125). Rescuers run at RESCUER_NICE_LEVEL = MIN_NICE (−20) and are the kworker/R-<wqname> threads visible in ps.
WQ_HIGHPRI is nice −20, not real-time — and this matters. Both HIGHPRI_NICE_LEVEL and RESCUER_NICE_LEVEL are MIN_NICE, i.e. −20, which is the most favourable Completely Fair Scheduler weight, not a different scheduling class. On the measurement box every one of the 258 kworker threads reports scheduling class TS (SCHED_OTHER); not one is FF or RR. A threaded IRQ handler at SCHED_FIFO 50 preempts all of them unconditionally. If you need a workqueue callback to beat a real-time task, a workqueue is the wrong mechanism — see Real-Time Scheduling SCHED_FIFO and SCHED_RR.
max_active, and a documentation contradiction inside v6.12 itself
max_active bounds how many of this workqueue’s items may be in flight at once; 0 requests the default of 256, and the hard ceiling is 512. These are deliberately high — “not the limiting factor while providing protection in runaway cases” — and the documentation’s own advice is that “unless there is a specific need for throttling the number of active work items, specifying ‘0’ is recommended.”
The v6.12 tree contradicts itself here — and the .rst is the stale half
Documentation/core-api/workqueue.rst states that max_active “determines the maximum number of execution contexts per CPU… This is always a per-CPU attribute, even for unbound workqueues.” The alloc_workqueue() kerneldoc in include/linux/workqueue.h:478–498 says the opposite for unbound queues: “For unbound workqueues, @max_active limits the number of in-flight work items for the whole system. e.g. @max_active of 16 indicates that that there can be at most 16 work items executing for the workqueue in the whole system.”
The code settles it in favour of the header. wq_update_node_max_active() (workqueue.c:1573) exists solely to split an unbound workqueue’s max_active across NUMA nodes in proportion to each node’s share of cpumask_weight_and(effective, cpu_online_mask), writing the result into wq->node_nr_active[node]->max; admission is then gated by tryinc_node_nr_active(), which compares against that per-node cap. A per-CPU attribute would need no such machinery. So for unbound workqueues the limit is system-wide, enforced per NUMA node; for per-CPU workqueues it remains genuinely per-CPU.
The header also documents the hazard this creates and the mitigation: an unlucky CPU distribution could give a node a cap lower than the number of interdependent work items and deadlock, so every node is guaranteed at least min_active = min(max_active, WQ_DFL_MIN_ACTIVE) — that is, at least 8. Read the .rst paragraph as historical.
Ordered workqueues: the idiom changed, and the old one is now wrong. The long-standing recipe for “execute strictly one at a time, in submission order” was alloc_workqueue(name, WQ_UNBOUND, 1). At v6.12 that no longer does the job, and the documentation says so outright: “While the combination of @max_active of 1 and WQ_UNBOUND used to achieve this behavior, this is no longer the case. Use alloc_ordered_workqueue() instead.” The reason is visible in the macro — it sets an internal flag the caller cannot reach:
Without __WQ_ORDERED the workqueue core does not apply the ordering guarantee, regardless of max_active. Only WQ_FREEZABLE and WQ_MEM_RECLAIM are meaningful in the flags argument of an ordered queue. The three legacy constructors are now thin wrappers over this with __WQ_LEGACY set — create_singlethread_workqueue(name) is alloc_ordered_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM, name) — and the docs say the create_*workqueue() family is “deprecated and scheduled for removal.”
Affinity Scopes, WQ_BH, and Dating the Recent Rework
CMWQ is often described as a 2010 design, and the core concurrency mechanism genuinely is. But the unbound half has been substantially reworked in the last two years, and the two biggest changes both post-date most of the tutorial material about workqueues. Each claim below was established by existence-checking the same file at successive release tags — fetching it and recording whether the symbol is present — rather than from memory, and then cross-checked against dated commits on git.kernel.org.
Feature
First release containing it
Evidence
Concurrency-Managed Workqueue
2.6.36 (2010)
Documentation/core-api/workqueue.rst is dated “September, 2010”, authored by Tejun Heo; contemporaneous coverage in LWN, “Working on workqueues”
grep WQ_AFFN returns 0 hits in both include/linux/workqueue.h and kernel/workqueue.c at v6.4 and v6.5, and 12 and 25 hits respectively at v6.6
WQ_BH / system_bh_wq / system_bh_highpri_wq
v6.9
grep WQ_BH in include/linux/workqueue.h: 0 hits at v6.8, 2 at v6.9. Corroborated by the LWN article’s own index entry (“Releases/6.9”) and by Tejun Heo’s follow-up fix “workqueue: Fix queue_work_on() with BH workqueues”, dated 2024-02-14 on git.kernel.org
WQ_AFFN_CACHE_SHARD (sub-LLC shards)
v7.1 — after the v6.12 pin
grep cache_shard in kernel/workqueue.c: 0 hits at v6.12 through v7.0 inclusive, 14 hits at v7.1. cgit dates the series to 2026-04-01 (Breno Leitao), merged in “Merge tag wq-for-7.1” on 2026-04-15
The in-tree documentation misdates its own feature
Documentation/core-api/workqueue.rst states, in the “Conclusion and Recommendations” subsection: “Affinity scopes are introduced in Linux v6.5. To emulate the previous behavior, use strict ‘numa’ affinity scope.” The existence check above contradicts the first sentence: at tag v6.5 neither include/linux/workqueue.h nor kernel/workqueue.c contains the string WQ_AFFN at all, and both do at v6.6. The likely explanation is that the documentation was written against the development tree during the v6.5 cycle and the series slipped a release. The advice in the second sentence is unaffected. Cite v6.6, and treat this as a reminder that “introduced in vX” statements inside the kernel’s own docs are hypotheses to be checked, not facts.
Why affinity scopes were added. Before v6.6, unbound pools were partitioned by NUMA node, which on a modern chiplet CPU is far too coarse: an AMD Ryzen 9 3900X presents one NUMA node but four separate L3 caches, so an unbound worker could be woken on a core whose cache shares nothing with the issuer. The affn_scope attribute lets a workqueue choose the granularity of its “pod” — cpu, smt, cache (last-level cache), numa, or system — with cache as the default (wq_affn_dfl = WQ_AFFN_CACHE, workqueue.c:435). There is also a strict variant: non-strict lets the scheduler migrate a worker out of its pod, strict does not.
The documentation backs the default with real numbers — a dm-crypt/fio benchmark on exactly that 12-core/24-thread, four-L3 Ryzen, /dev/dm-0 on a Samsung 990 PRO, boost disabled, five runs each. The trade-off it exposes is between locality and work conservation, and it depends entirely on how many issuers there are:
Issuers (fio --numjobs)
system scope
cache scope
cache (strict)
24 — work spread across the machine
1159.40 ±1.34 MiBps
1166.40 ±0.89
1166.00 ±0.71
8 — enough work to saturate
1155.40 ±0.89
1154.40 ±1.14
1112.00 ±4.64 (−3.7%)
4 — not enough to saturate
993.60 ±1.82
973.40 ±1.52 (−2%)
828.20 ±4.49 (−20%)
With plenty of issuers, cache affinity is a free 0.6% win from better locality. With few issuers, strict cache affinity costs 20% of bandwidth, because workers pile up in one pod while other pods sit idle — the scheduler is no longer permitted to fix the imbalance. Non-strict cache is the compromise that was chosen as default: it keeps most of the locality benefit and loses only 2% in the worst measured case. The documentation is candid that this is a workaround: “the loss of work-conservation in non-strict affinity scopes is likely originating from the scheduler. There is no theoretical reason why the kernel wouldn’t be able to do the right thing… it is possible that future scheduler improvements may make most of these tunables unnecessary.”
It also records a conclusion worth remembering: “An unbound workqueue with strict ‘cpu’ affinity scope behaves the same as WQ_CPU_INTENSIVE per-cpu workqueue. There is no real advan[t]age to the latter and an unbound workqueue provides a lot more flexibility.” In other words, the v6.6+ position is that WQ_CPU_INTENSIVE is largely obsolete as a design choice — it survives mainly as the internal mechanism the auto-detector uses.
Point-in-time check against the running machine. The box these measurements came from runs 7.1.13, and its live parameters show the next generation of this same idea already in place:
$ for f in /sys/module/workqueue/parameters/*; do echo "$(basename $f) = $(cat $f)"; donecache_shard_size = 8cpu_intensive_thresh_us = 10000debug_force_rr_cpu = Ndefault_affinity_scope = cache_shardpower_efficient = N
cache_shard is not a v6.12 value — the v6.12 enum offers only default, cpu, smt, cache, numa, system. WQ_AFFN_CACHE_SHARD was added in v7.1 and made the default in the same series, splitting a large last-level cache into synthetic sub-shards of cache_shard_size CPUs (8 here). Anyone reading default_affinity_scope on a current distro kernel and looking it up in v6.12 documentation will not find it. This is the practical reason the vault pins a release when citing kernel internals: the workqueue’s unbound half is under active development and its defaults have changed twice in three years.
Reading kworker Thread Names on a Live System
Everything above is directly observable. This section works through the naming convention, because a kworker name encodes which pool the thread belongs to, what priority it runs at, and which workqueue it last served — and almost nobody can read it. All output below is from a 32-CPU AMD workstation running Fedora 44, kernel 7.1.13-200.fc44; the code is cited at v6.12 and the differences are called out where they exist.
And the -/+ suffix comes from a second place, wq_worker_comm() (workqueue.c:6420–6450), whose comment is the definition: “->desc tracks information (wq name or set_worker_desc()) for the latest execution. If current, prepend '+', otherwise '-'.”
Component
Example
Meaning
Source
kworker/<cpu>:<id>
kworker/11:3
Bound worker. <cpu> is the CPU it is pinned to; <id> is a per-pool IDA index, not a CPU or a PID
pool->cpu, worker->id
trailing H
kworker/11:3H
The worker’s pool has attrs->nice < 0 — i.e. it is the highpri pool (nice −20)
pool->attrs->nice < 0 ? "H" : ""
kworker/u<N>:<id>
kworker/u130:2
Unbound worker. <N> is the worker-pool ID — not a CPU, not a CPU count, not a NUMA node
pool->id
kworker/R-<name>
kworker/R-writeback
Rescuer for the WQ_MEM_RECLAIM workqueue named <name>
worker->rescue_wq->name
-<desc> suffix
kworker/0:1H-kblockd
The worker is idle; <desc> is the workqueue it last served
worker->current_work == NULL
+<desc> suffix
kworker/u130:2+events_unbound
The worker is currently executing a work item from <desc>
worker->current_work != NULL
<desc> itself
kblockd, btrfs-endio
Defaults to pwq->wq->name, set in process_one_work(); a driver can override it per-item with set_worker_desc()
strscpy(worker->desc, pwq->wq->name, …)
The u<N> field is the one that is almost universally misread, including in otherwise good write-ups: it is a pool ID, and pool IDs are allocated by idr_alloc() in worker_pool_assign_id() starting from 0. Bound pools are assigned first, during workqueue_init_early(), in a loop over for_each_possible_cpu() that initialises the two BH pools then the two standard pools per CPU (workqueue.c:7755–7769). So on a machine with P possible CPUs, IDs 0 … 4P−1 are consumed before any unbound pool exists, and the first unbound pool gets ID 4P.
That is a falsifiable prediction, so here it is tested. The measurement box has 32 possible CPUs, which predicts the first unbound pool ID should be 4 × 32 = 128:
The lowest unbound pool ID is exactly 128, and there are six unbound pools in total. Anyone who has read u128 as “128 CPUs” or “128 workers” now has a concrete counter-example: this machine has 32 CPUs and 258 kworker threads.
Four things fall straight out of that census, and each confirms a claim made earlier from source.
64 bound-highpri workers on 32 CPUs is exactly two per CPU, all at nice −20. That is HIGHPRI_NICE_LEVEL = MIN_NICE as declared, and it is the CMWQ steady state: a pool that is never forced to grow keeps the one or two workers it needs.
83 rescuers, all at nice −20 (RESCUER_NICE_LEVEL). Since alloc_workqueue() creates exactly one rescuer per WQ_MEM_RECLAIM workqueue, this is a direct count of the reclaim-relevant workqueues alive on the box. This is the concrete cost of the flag: 83 kernel threads permanently resident so that memory reclaim can never deadlock on worker creation. It is also a good sense of scale for how many kernel subsystems declare themselves reclaim-relevant.
58 unbound workers across six pools, split 16 at nice −20 and 42 at nice 0 — the two default attribute sets (unbound_std_wq_attrs[NR_STD_WORKER_POOLS], normal and highpri) instantiated per affinity pod.
Every single one is scheduling class TS. Not one kworker on this machine is SCHED_FIFO or SCHED_RR, while ten threaded IRQ handlers are (FF, priority 50). WQ_HIGHPRI buys a CFS weight, not a scheduling class.
A sample of names showing the suffixes in action:
$ ps -eo pid,class,ni,comm | grep kworker | head -4 3763 TS -20 kworker/0:1H-kblockd <- bound CPU 0, highpri pool, idle, last ran kblockd1942064 TS 0 kworker/u128:0-comp_1.2.0 <- unbound pool 128, idle, last ran comp_1.2.01980583 TS 0 kworker/u128:1-dm_vblank_control_workqueue ... kworker/u130:3+events_unbound <- unbound pool 130, RUNNING an events_unbound item
Note the last name is 28 characters. TASK_COMM_LEN is 16 (include/linux/sched.h:306), so this string cannot be coming from task->comm. It is synthesised on read: proc_task_name() in fs/proc/array.c tests p->flags & PF_WQ_WORKERbefore testing PF_KTHREAD and calls wq_worker_comm() into a 64-byte buffer (array.c:100–118). Two practical consequences: the descriptive suffix is only visible through /proc, so a tool reading comm from a tracepoint or a BPF program sees the truncated 15-character form; and the suffix is a live read of worker->desc under pool->lock, so it changes as the thread picks up different work.
What the worker population tells you
The distribution above is the healthy steady state — pools at their minimum, mostly idle, - suffixes everywhere. Two abnormal patterns are worth recognising:
A pool that has grown large (say kworker/7:0 through kworker/7:40) means many work items on CPU 7 blocked simultaneously. That is CMWQ working as designed, but it points at a subsystem doing a lot of concurrent blocking I/O. Workers are reaped after IDLE_WORKER_TIMEOUT = 300 * HZ — five minutes — and only while too_many_workers() holds, which requires nr_idle > 2 && (nr_idle - 2) * 4 >= nr_busy (workqueue.c:956–962). So a burst leaves a long, harmless tail.
A + suffix that never changes on a bound worker means one work item has held that CPU’s pool slot for a long time. Below 10 ms the auto-detector has not fired yet; above it the worker has been demoted to CPU_INTENSIVE and the pool has kicked a replacement, so the symptom is self-limiting but the underlying item still wants either WQ_UNBOUND or WQ_CPU_INTENSIVE.
Uncertain — the authoritative view was not readable
Verify: per-pool nr_running, nr_idle, per-workqueue PWQ_STAT_CM_WAKEUP / PWQ_STAT_CPU_INTENSIVE counters, and the actual workqueue-to-pool mapping on this machine. Reason: the kernel exports all of it under /sys/kernel/debug/workqueue/, and the in-tree tools tools/workqueue/wq_dump.py and tools/workqueue/wq_monitor.py read it — but on this Fedora host /sys/kernel/debug is mode 0700 (drwx------ root root) and there is no passwordless sudo, so every reading above had to come from ps and /sys/module/workqueue/parameters/ instead. CONFIG_WQ_WATCHDOG is also unset in this kernel, so the “pool stalls” warning that would otherwise appear in dmesg is compiled out. To resolve: as root, cat /sys/kernel/debug/workqueue/*, or run sudo tools/workqueue/wq_dump.py from a kernel source tree matching the running release. A less invasive partial alternative is ls /sys/bus/workqueue/devices/ for queues created with WQ_SYSFS. uncertain
Synchronization — Flushing and Cancelling
Because work runs later, in another thread, you must synchronize before tearing down whatever the work touches. Three helpers (v6.12):
flush_work(work) — blocks until the given work item has finished its current execution (if running/queued) and returns. It does not prevent re-queuing; it just waits for the in-flight instance.
cancel_work_sync(work) — “cancel a work and wait for it to finish.” It dequeues the item if still pending and, if it was already running, waits for it to complete. After it returns, the work is guaranteed not to be running — this is what you call in a driver’s remove()/cleanup path before freeing the structure the work dereferences.
cancel_delayed_work_sync(dwork) — the same, for delayed_work: it also cancels the pending timer.
The canonical bug is freeing the object a work item points into while the work is still queued or running. cancel_work_sync() exists precisely to close that race; using flush_work() where you meant cancel_work_sync() (so the work re-queues itself after you flush) is a classic mistake. Note also that flush_scheduled_work() and flushing the wholesystem_wq are discouraged — the header wraps flush_scheduled_work() with a compile-time warning and flush_workqueue() warns at runtime if you flush a system-wide queue, because flushing a shared queue waits for unrelated work and invites deadlocks.
Code Example — A Typical Driver Pattern
struct mydev { struct work_struct irq_work; /* embedded work item */ struct delayed_work poll_work; /* periodic poll */ struct mutex lock; /* ... */};/* the deferred callback — process context, MAY sleep */static void mydev_do_work(struct work_struct *w){ struct mydev *d = container_of(w, struct mydev, irq_work); mutex_lock(&d->lock); /* legal: we can sleep here */ process_pending(d); /* may block on slow I/O */ mutex_unlock(&d->lock);}/* hard-IRQ handler (atomic) just defers the heavy work */static irqreturn_t mydev_isr(int irq, void *dev_id){ struct mydev *d = dev_id; schedule_work(&d->irq_work); /* false if already queued — that's fine */ return IRQ_HANDLED;}static int mydev_probe(struct mydev *d){ INIT_WORK(&d->irq_work, mydev_do_work); INIT_DELAYED_WORK(&d->poll_work, mydev_poll); schedule_delayed_work(&d->poll_work, msecs_to_jiffies(500)); return 0;}static void mydev_remove(struct mydev *d){ cancel_work_sync(&d->irq_work); /* MUST: no work in flight */ cancel_delayed_work_sync(&d->poll_work); /* MUST: cancel timer + work */ /* now safe to free d */}
The load-bearing details: the hard-IRQ handler does the minimum and schedule_work()s the rest — the callback then runs in a kworker where mutex_lock() and blocking I/O are legal. container_of() recovers the device struct from the embedded work_struct. The remove() path mustcancel_work_sync() and cancel_delayed_work_sync() before freeing, or a worker could dereference freed memory. If mydev_poll re-queues itself with schedule_delayed_work() it becomes a periodic poller — and then onlycancel_delayed_work_sync() reliably stops the cycle.
Failure Modes and Common Misunderstandings
Use-after-free on teardown. Freeing the structure a work item points into without cancel_work_sync() first. The fix is always to cancel-and-wait in the cleanup path.
Flushing a shared system workqueue. Calling flush_scheduled_work() or flushing system_wq waits for every unrelated work item on that queue, which is slow and deadlock-prone (your work waits on a queue that something else is blocking). The kernel now warns about this; flush a specific work_struct or a private workqueue instead.
A CPU-bound work item starving the pool. A callback that loops without sleeping holds the pool’s running slot, blocking everything behind it on that CPU. Either mark the queue WQ_CPU_INTENSIVE (so it’s excluded from concurrency accounting) or WQ_UNBOUND (so the scheduler can balance it). This is the single most common CMWQ performance pathology.
Forgetting WQ_MEM_RECLAIM. A latent deadlock that only manifests under memory pressure — exactly when you can least afford it. Any work that can run during reclaim must set this flag to get a rescuer.
Expecting ordering or per-CPU guarantees you didn’t ask for. A plain workqueue does not guarantee FIFO global ordering across CPUs; for strict ordering you need an ordered workqueue — and at v6.12 that means alloc_ordered_workqueue(name, 0), not the old alloc_workqueue(name, WQ_UNBOUND, 1) recipe, which no longer sets the required internal __WQ_ORDERED flag. See the max_active discussion above.
Alternatives and When to Choose Them
Across the four deferral mechanisms, a workqueue is the choice when the deferred work can sleep and is general — not tied to re-enabling one specific interrupt line. If the sleepable work is tightly bound to one interrupt (you want oneshot line-masking and synchronize_irq() to cover it), a threaded IRQ is the better fit — it is essentially a workqueue specialized to one line, with IRQ-subsystem integration. If the work must stay atomic (cannot sleep) and needs the lowest latency, the kernel core uses a softirq; if you historically wanted simple serialized atomic deferral you’d reach for a tasklet, but tasklets are being retired — the modern replacement is a WQ_BH workqueue, which gives softirq-context execution through this same API. So: sleepable + general → workqueue; sleepable + one-line → threaded IRQ; atomic → softirq or WQ_BH workqueue.
Production Notes
CMWQ is everywhere in the kernel — block-layer completions, filesystem writeback, USB, the driver core, and countless drivers all defer through workqueues. The design is widely regarded as one of the cleaner subsystem rewrites in the kernel precisely because it solved the thread-explosion and deadlock problems of the old implementation without changing the INIT_WORK / schedule_work API drivers already used: old callers kept working while gaining shared pools underneath. The ongoing tasklet phase-out toward WQ_BH (see Tasklets and Their Deprecation) means new atomic deferral increasingly flows through this same API too, which makes CMWQ the single most important deferral subsystem to understand.
Reading a busy machine. When kworker threads dominate top, the useful first question is which workqueue, and the answer is in the thread name — the -/+ suffix is the workqueue that thread last ran or is running. kworker/12:2+btrfs-endio is a filesystem completion, kworker/0:1H-kblockd is block-layer work on the highpri pool. Aggregating those suffixes across all 258 threads with ps -eo comm | grep kworker | sed 's/.*[-+]//' | sort | uniq -c | sort -rn gives a subsystem-level attribution in one line, with no tooling and no privileges.
The one flag you must not forget is WQ_MEM_RECLAIM. Everything else in this note is a performance consideration; that one is a correctness requirement, and getting it wrong produces a deadlock that only manifests under memory pressure, i.e. during an incident. The rule from the documentation is absolute: “All wq which might be used in the memory reclaim paths MUST have this flag set.” If you are unsure whether your work can run during reclaim, set it — the cost is one resident kernel thread.
Do not flush shared queues.flush_scheduled_work() and flush_workqueue(system_wq) wait for every unrelated work item queued by every other subsystem. The header now wraps flush_scheduled_work() in a compile-time warning and the core warns at runtime when a system-wide queue is flushed, precisely because this pattern produced real deadlocks. Flush a specific work_struct, or create a private workqueue and flush that.
Tune through WQ_SYSFS, not through code. A workqueue created with WQ_SYSFS appears under /sys/bus/workqueue/devices/<name>/ with writable nice, cpumask, max_active and (for unbound queues) affinity_scope and affinity_strict attributes. The documentation recommends it explicitly for “workqueue usages that may consume a significant amount of CPU,” because the right affinity scope is genuinely workload- and topology-dependent — as the dm-crypt numbers above show, the best choice swings by 20% depending only on how many issuers there are.
Diagnostics, and what a hardened distro takes away./sys/kernel/debug/workqueue/ exposes per-pool and per-workqueue state, and the in-tree tools/workqueue/wq_dump.py and wq_monitor.py render it. CONFIG_WQ_WATCHDOG adds a stall detector that prints “BUG: workqueue lockup” when a pool makes no progress, and CONFIG_WQ_CPU_INTENSIVE_REPORT names work functions that exceed the CPU-intensive threshold. All three are unavailable on the Fedora 44 machine used here: debugfs is mode 0700, and both Kconfig options are unset. Before concluding from a clean dmesg that no workqueue is stalling, check zgrep WQ_WATCHDOG /boot/config-$(uname -r) — absence of the warning may mean absence of the detector.
Version awareness is not optional for this subsystem. Unlike softirqs, whose vector list has not changed since 2009, the workqueue’s unbound half has changed defaults twice in three years: affinity scopes with a cache default in v6.6, and cache_shard superseding it as default in v7.1. WQ_BH arrived in v6.9. The system_wq symbol itself is marked for removal in v7.1 in favour of system_percpu_wq. When you cite workqueue behaviour, name the release you read.
See Also
Threaded Interrupt Handlers — the line-bound sleepable deferral; a threaded IRQ is conceptually a workqueue specialized to one interrupt