vault

Workqueues and the Concurrency-Managed Workqueue

17 min read

Workqueues and the Concurrency-Managed Workqueue

A workqueue is the kernel’s general-purpose mechanism for running deferred work in a context where it may sleep. You describe a job as a work_struct — a struct that holds a function pointer — initialize it with INIT_WORK(), and hand it to the kernel with queue_work() (or the shorthand schedule_work()). Sometime later, a kernel worker thread named kworker/... picks the work item off its queue and runs your callback in ordinary process context, so the callback may take mutexes, allocate memory with GFP_KERNEL, and block on I/O — all the things a softirq or hard-interrupt handler must never do (per the workqueue documentation, Linux v6.12). Since 2010 this has been implemented by the Concurrency-Managed Workqueue (CMWQ), whose central idea is to decouple the work items a driver queues from the worker threads that actually run them: instead of one thread per workqueue, all workqueues share a small pool of per-CPU worker threads, and the pool grows or shrinks just enough to keep the CPU busy without starving. This is the bottom half to reach for whenever the deferred work can sleep but is not bound to a specific interrupt line (when it is, prefer a threaded IRQ).

Mental Model

A workqueue separates what to do (the work item) from who does it (the worker thread) and when (later, in process context). The work item is a passive data structure — “a simple struct that holds a pointer to the function that is to be executed asynchronously” (workqueue docs, v6.12). The workqueue is a named queue you submit to. The actual execution context is a kworker thread drawn from a shared pool. This three-way decoupling is the whole design: you never create or manage a thread yourself, you just submit a function and the CMWQ machinery finds (or spawns) a worker to run it.

flowchart LR
    subgraph Driver
      W1["work_struct A<br/>(func ptr)"]
      W2["work_struct B"]
    end
    W1 -->|"queue_work(wq, &A)"| Q["workqueue<br/>(system_wq / custom)"]
    W2 -->|"queue_work(wq, &B)"| Q
    Q --> POOL["per-CPU worker pool<br/>(shared by ALL workqueues)"]
    POOL --> K1["kworker thread 1"]
    POOL --> K2["kworker thread 2<br/>(spawned when 1 sleeps)"]
    K1 -->|"runs A().func"| RUNA["A's callback<br/>process ctx — CAN sleep"]
    K2 -->|"runs B().func"| RUNB["B's callback<br/>process ctx — CAN sleep"]

The CMWQ decoupling. What it shows: drivers queue passive work_struct items onto named workqueues, but those queues all feed into a small set of shared per-CPU worker pools; a pool runs work on kworker threads and spawns another worker only when the running one goes to sleep. The insight to take: the number of threads is not the number of workqueues — that was the old design’s fatal flaw. CMWQ keeps thread count low by sharing pools and managing concurrency dynamically, so you can create as many logically-distinct workqueues as you like without spawning a thread for each.

Why CMWQ Exists — The Old Design’s Failure

Understanding CMWQ requires knowing what it replaced, because the concurrency-managed part is a direct answer to a real scaling disaster. In the original workqueue implementation (workqueue docs, v6.12):

“In the original wq implementation, a multi threaded (MT) wq had one worker thread per CPU and a single threaded (ST) wq had one worker thread system-wide.”

This had several compounding problems, all quoted from the kernel docs:

  • Thread proliferation. “A single MT wq needed to keep around the same number of workers as the number of CPUs,” and across many workqueues this “saturated the default 32k PID space just booting up” on large machines. Every workqueue cost N threads whether or not it had work.
  • Rigid, separate pools. “Each wq maintained its own separate worker pool. An MT wq could provide only one execution context per CPU while an ST wq one for the whole system.” Concurrency was fixed at creation time, not on demand.
  • Deadlock-prone. “Work items had to compete for those very limited execution contexts leading to various problems including proneness to deadlocks around the single execution context.” If a work item blocked, everything behind it on that single context was stuck.
  • Forced bad trade-offs. Subsystems like libata ended up “choosing to use ST wq for polling PIOs and accepting an unnecessary limitation that no two polling PIOs can progress at the same time,” just to conserve threads.

CMWQ (merged in 2.6.36, 2010) fixes all of this with two design goals, again quoted:

“Use per-CPU unified worker pools shared by all wq to provide flexible level of concurrency on demand without wasting a lot of resource.”

“Automatically regulate worker pool and level of concurrency so that the API users don’t need to worry about such details.”

The mechanism for “automatically regulate” is a hook into the scheduler. CMWQ maintains, per CPU, two worker pools — one for normal-priority work and one for high-priority work — and these pools are shared by every workqueue targeting that CPU. The pool is told whenever one of its workers wakes up or goes to sleep (workqueue docs, v6.12):

“The worker-pool is notified whenever an active worker wakes up or sleeps and keeps track of the number of the currently runnable workers.”

And the regulation rule itself:

“As long as there are one or more runnable workers on the CPU, the worker-pool doesn’t start execution of a new work, but, when the last running worker goes to sleep, it immediately schedules a new worker so that the CPU doesn’t sit idle while there are pending work items.”

That is the heart of “concurrency-managed”: keep exactly one worker running per pool per CPU when possible (no pointless context-switching), but the instant the running worker blocks (e.g. your callback called mutex_lock() and slept), wake another worker so pending work keeps flowing and the CPU never idles with work waiting. The pool/kworker internals — how workers are created, parked, and destroyed — are deferred to Workqueue Internals kworker and Worker Pools; this note covers the API and model.

Mechanical Walk-through — The API

The work item. A job is a struct work_struct (from workqueue_types.h, v6.12):

typedef void (*work_func_t)(struct work_struct *work);
 
struct work_struct {
	atomic_long_t data;        /* packed: pending bit + pool/wq pointer */
	struct list_head entry;    /* links into the pool's worklist */
	work_func_t func;          /* your callback */
#ifdef CONFIG_LOCKDEP
	struct lockdep_map lockdep_map;
#endif
};

The data field is a clever packed word: it holds both a pending bit (set while the item is queued, so re-queuing the same item is a no-op) and a pointer to the pool/workqueue it belongs to. entry links it into a worklist; func is the callback. Note the callback receives the work_struct * itself, not arbitrary context — the idiom is to embed work_struct inside your own struct and recover the outer struct with container_of().

Initializing and queuing. You initialize a work item with INIT_WORK(&my_work, my_func) (or DECLARE_WORK(name, func) for a static one), then submit it. queue_work() puts it on a specific workqueue; schedule_work() is the shorthand that targets the default system_wq (workqueue.h, v6.12):

static inline bool queue_work(struct workqueue_struct *wq, struct work_struct *work)
{
	return queue_work_on(WORK_CPU_UNBOUND, wq, work);
}
 
static inline bool schedule_work(struct work_struct *work)
{
	return queue_work(system_wq, work);
}

The boolean return is important and routinely ignored: it is true if the item was queued, and false if the work was already queued (its pending bit was set). This is the documented behavior — “Returns false if work was already queued, true otherwise.” So queuing the same work_struct twice before it runs coalesces into a single execution, not two. WORK_CPU_UNBOUND means “let the workqueue pick the CPU” (for a normal per-CPU workqueue this means the queuing CPU).

Delayed work. To run something after a delay, use struct delayed_work, which is just a work_struct plus a timer (workqueue.h, v6.12):

struct delayed_work {
	struct work_struct work;
	struct timer_list timer;
	struct workqueue_struct *wq;
	int cpu;
};

You init it with INIT_DELAYED_WORK(&dw, func) and queue it with queue_delayed_work(wq, &dw, delay) or schedule_delayed_work(&dw, delay), where delay is in jiffies. Mechanically, the timer fires after delay, and its expiry is what actually queues the embedded work_struct onto the workqueue — so the delay is a timer wait, then the work runs in a worker as usual. This is the standard way to implement “retry in 100 ms” or a periodic poll (by re-queuing from the callback).

The default system workqueues. Most code never creates its own workqueue; it uses one of the predefined ones declared in workqueue.h (v6.12), each with a documented purpose:

  • system_wq — “one used by schedule[_delayed]_work[_on](). Multi-CPU multi-threaded.” The general default.
  • system_highpri_wq — “similar to system_wq but for work items which require WQ_HIGHPRI.” Runs on the high-priority (elevated-nice) pool.
  • system_long_wq — “similar to system_wq but may host long running works.” Keeps slow jobs from clogging the general queue.
  • system_unbound_wq — “unbound workqueue. Workers are not bound to any specific CPU.” For CPU-intensive or latency-fluctuating work the scheduler should be free to place.
  • system_freezable_wq — “equivalent to system_wq except that it’s freezable” (drains during system suspend).
  • system_power_efficient_wq and system_freezable_power_efficient_wq — become unbound when the workqueue.power_efficient kernel parameter is set, otherwise behave like their non-efficient counterparts.

Creating a custom workqueue. When you need isolation, ordering, or specific behavior, allocate your own with alloc_workqueue() (v6.12):

__printf(1, 4) struct workqueue_struct *
alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...);

fmt is a printf-style name (shows up in kworker thread names and /sys), flags is an OR of WQ_* flags (below), and max_active bounds concurrency. max_active == 0 requests the default. The active-limit constants (v6.12) are:

enum wq_consts {
	WQ_MAX_ACTIVE		= 512,
	WQ_UNBOUND_MAX_ACTIVE	= WQ_MAX_ACTIVE,
	WQ_DFL_ACTIVE		= WQ_MAX_ACTIVE / 2,   /* = 256 */
	WQ_DFL_MIN_ACTIVE	= 8,
};

max_active is, per the docs, “the maximum number of execution contexts per CPU which can be assigned to the work items of a wq” — “This is always a per-CPU attribute, even for unbound workqueues.” The default of 256 (with a hard ceiling of 512) is deliberately high so it is “not the limiting factor while providing protection in runaway cases.” You rarely need to tune it; the common reasons to call alloc_workqueue() at all are the flags or to get an ordered (max_active == 1, unbound) queue.

The Workqueue Flags

alloc_workqueue()’s flags argument selects behavior. The enum and its inline comments (v6.12):

enum wq_flags {
	WQ_BH			= 1 << 0, /* execute in bottom half (softirq) context */
	WQ_UNBOUND		= 1 << 1, /* not bound to any cpu */
	WQ_FREEZABLE		= 1 << 2, /* freeze during suspend */
	WQ_MEM_RECLAIM		= 1 << 3, /* may be used for memory reclaim */
	WQ_HIGHPRI		= 1 << 4, /* high priority */
	WQ_CPU_INTENSIVE	= 1 << 5, /* cpu intensive workqueue */
	WQ_SYSFS		= 1 << 6, /* visible in sysfs */
	WQ_POWER_EFFICIENT	= 1 << 7,
};

The four that matter most in practice, with their documented semantics:

  • WQ_UNBOUND — “Work items queued to an unbound wq are served by the special worker-pools which host workers which are not bound to any specific CPU.” Use it when concurrency requirements fluctuate widely, or for “long running CPU intensive workloads which can be better managed by the system scheduler” — because an unbound worker can be migrated and load-balanced like any task, whereas a bound worker is pinned to the queuing CPU.
  • WQ_MEM_RECLAIM — “All wq which might be used in the memory reclaim paths MUST have this flag set.” This is the most safety-critical flag. The danger: a work item on the reclaim path needs a worker thread to run, but creating a worker thread needs memory, which under memory pressure needs reclaim, which needs the work item to run — a deadlock. WQ_MEM_RECLAIM breaks the cycle by reserving a dedicated rescue worker (a “rescuer”) for the workqueue, guaranteeing at least one execution context exists no matter how starved memory is. If your work can ever run while the system is reclaiming memory and you forget this flag, you have a latent deadlock.
  • WQ_HIGHPRI — “Work items of a highpri wq are queued to the highpri worker-pool of the target cpu. Highpri worker-pools are served by worker threads with elevated nice level.” Importantly, “normal and highpri worker-pools don’t interact with each other” — each is a separate pool with its own concurrency management, so high-priority work cannot be blocked behind normal work.
  • WQ_CPU_INTENSIVE — “Work items of a CPU intensive wq do not contribute to the concurrency level.” Since CMWQ’s whole regulation is based on “is a worker runnable,” a CPU-bound work item that never sleeps would otherwise hold the pool’s single running slot indefinitely and starve everything behind it. Marking it CPU-intensive removes it from the concurrency accounting, so “runnable CPU intensive work items will not prevent other work items in the same worker-pool from starting execution” — the scheduler time-slices them instead.

WQ_FREEZABLE makes the queue participate in suspend (work is “drained and no new work item starts execution until thawed”). WQ_BH is a newer addition — “BH workqueues can be considered a convenience interface to softirq… always per-CPU and all BH work items are executed in the queueing CPU’s softirq context.” WQ_BH is the modern, recommended replacement for tasklets: it gives you softirq-context (atomic, non-sleeping) execution through the workqueue API, which is why the kernel is migrating tasklet users onto it.

Synchronization — Flushing and Cancelling

Because work runs later, in another thread, you must synchronize before tearing down whatever the work touches. Three helpers (v6.12):

  • flush_work(work) — blocks until the given work item has finished its current execution (if running/queued) and returns. It does not prevent re-queuing; it just waits for the in-flight instance.
  • cancel_work_sync(work) — “cancel a work and wait for it to finish.” It dequeues the item if still pending and, if it was already running, waits for it to complete. After it returns, the work is guaranteed not to be running — this is what you call in a driver’s remove()/cleanup path before freeing the structure the work dereferences.
  • cancel_delayed_work_sync(dwork) — the same, for delayed_work: it also cancels the pending timer.

The canonical bug is freeing the object a work item points into while the work is still queued or running. cancel_work_sync() exists precisely to close that race; using flush_work() where you meant cancel_work_sync() (so the work re-queues itself after you flush) is a classic mistake. Note also that flush_scheduled_work() and flushing the whole system_wq are discouraged — the header wraps flush_scheduled_work() with a compile-time warning and flush_workqueue() warns at runtime if you flush a system-wide queue, because flushing a shared queue waits for unrelated work and invites deadlocks.

Code Example — A Typical Driver Pattern

struct mydev {
	struct work_struct  irq_work;   /* embedded work item */
	struct delayed_work poll_work;  /* periodic poll */
	struct mutex        lock;
	/* ... */
};
 
/* the deferred callback — process context, MAY sleep */
static void mydev_do_work(struct work_struct *w)
{
	struct mydev *d = container_of(w, struct mydev, irq_work);
 
	mutex_lock(&d->lock);           /* legal: we can sleep here */
	process_pending(d);             /* may block on slow I/O */
	mutex_unlock(&d->lock);
}
 
/* hard-IRQ handler (atomic) just defers the heavy work */
static irqreturn_t mydev_isr(int irq, void *dev_id)
{
	struct mydev *d = dev_id;
	schedule_work(&d->irq_work);    /* false if already queued — that's fine */
	return IRQ_HANDLED;
}
 
static int mydev_probe(struct mydev *d)
{
	INIT_WORK(&d->irq_work, mydev_do_work);
	INIT_DELAYED_WORK(&d->poll_work, mydev_poll);
	schedule_delayed_work(&d->poll_work, msecs_to_jiffies(500));
	return 0;
}
 
static void mydev_remove(struct mydev *d)
{
	cancel_work_sync(&d->irq_work);           /* MUST: no work in flight */
	cancel_delayed_work_sync(&d->poll_work);  /* MUST: cancel timer + work */
	/* now safe to free d */
}

The load-bearing details: the hard-IRQ handler does the minimum and schedule_work()s the rest — the callback then runs in a kworker where mutex_lock() and blocking I/O are legal. container_of() recovers the device struct from the embedded work_struct. The remove() path must cancel_work_sync() and cancel_delayed_work_sync() before freeing, or a worker could dereference freed memory. If mydev_poll re-queues itself with schedule_delayed_work() it becomes a periodic poller — and then only cancel_delayed_work_sync() reliably stops the cycle.

Failure Modes and Common Misunderstandings

Use-after-free on teardown. Freeing the structure a work item points into without cancel_work_sync() first. The fix is always to cancel-and-wait in the cleanup path.

Flushing a shared system workqueue. Calling flush_scheduled_work() or flushing system_wq waits for every unrelated work item on that queue, which is slow and deadlock-prone (your work waits on a queue that something else is blocking). The kernel now warns about this; flush a specific work_struct or a private workqueue instead.

A CPU-bound work item starving the pool. A callback that loops without sleeping holds the pool’s running slot, blocking everything behind it on that CPU. Either mark the queue WQ_CPU_INTENSIVE (so it’s excluded from concurrency accounting) or WQ_UNBOUND (so the scheduler can balance it). This is the single most common CMWQ performance pathology.

Forgetting WQ_MEM_RECLAIM. A latent deadlock that only manifests under memory pressure — exactly when you can least afford it. Any work that can run during reclaim must set this flag to get a rescuer.

Expecting ordering or per-CPU guarantees you didn’t ask for. A plain workqueue does not guarantee FIFO global ordering across CPUs; for strict ordering you need an ordered workqueue (alloc_workqueue(name, WQ_UNBOUND, 1) — max_active == 1).

Alternatives and When to Choose Them

Across the four deferral mechanisms, a workqueue is the choice when the deferred work can sleep and is general — not tied to re-enabling one specific interrupt line. If the sleepable work is tightly bound to one interrupt (you want oneshot line-masking and synchronize_irq() to cover it), a threaded IRQ is the better fit — it is essentially a workqueue specialized to one line, with IRQ-subsystem integration. If the work must stay atomic (cannot sleep) and needs the lowest latency, the kernel core uses a softirq; if you historically wanted simple serialized atomic deferral you’d reach for a tasklet, but tasklets are being retired — the modern replacement is a WQ_BH workqueue, which gives softirq-context execution through this same API. So: sleepable + general → workqueue; sleepable + one-line → threaded IRQ; atomic → softirq or WQ_BH workqueue.

Production Notes

CMWQ is everywhere in the kernel — block-layer completions, filesystem writeback, USB, the driver core, and countless drivers all defer through workqueues. The kworker/<cpu>:<id> and kworker/u<n>:<id> threads you see at the top of a busy top are CMWQ’s shared pool workers (the u prefix marks unbound pools). The CMWQ design is widely regarded as one of the cleaner subsystem rewrites in the kernel precisely because it solved the thread-explosion and deadlock problems of the old design without changing the simple INIT_WORK / schedule_work API drivers already used — old callers kept working while gaining shared pools underneath. For diagnosing stalls, /sys/kernel/debug/workqueue/ and the wq_monitor.py tracing tool expose per-pool backlog and the “pool stalled” warnings the kernel emits when a work item hogs a worker for too long. The ongoing tasklet phase-out toward WQ_BH (see Tasklets and Their Deprecation) means new atomic-deferral code increasingly flows through the workqueue API too, making CMWQ the single most important deferral subsystem to understand.

See Also

Graph View

Backlinks