Softirqs and the Softirq Vector

A softirq (short for “software interrupt”) is the lowest-level, highest-frequency mechanism Linux uses to defer work out of a hardware interrupt handler. When a device’s top-half handler has done the bare minimum, it raises a softirq — sets a per-CPU pending bit — and returns; the kernel then runs the pending softirqs slightly later, on the return path from the hardware interrupt (and, under sustained load, in a dedicated per-CPU kernel thread called ksoftirqd). Unlike the four bottom-half mechanisms layered above it, softirqs are a statically fixed, compile-time set: a small enum of exactly ten kinds (HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, RCU), and only the kernel core may add one — there is no dynamic registration for drivers (interrupt.h, v6.12). The defining property that makes softirqs both powerful and dangerous is their reentrancy model: the same softirq type can run concurrently on different CPUs at the same time, so any state a softirq touches must be per-CPU or explicitly Symmetric MultiProcessing (SMP)-locked — a sharp contrast with tasklets, which the kernel serializes so a given tasklet never runs on two CPUs at once. To stop softirq processing from starving everything else, the kernel bounds each processing pass with MAX_SOFTIRQ_RESTART (10) restarts plus a 2-millisecond time budget before punting the remainder to ksoftirqd (softirq.c, v6.12).

What this note owns. The generic softirq mechanism: the vector enum and its ordering, raise_softirq/open_softirq, the handle_softirqs() drain loop and its budget, the reentrancy model, and how all of it is observed from userspace. It deliberately does not re-explain its neighbours: The Network Receive Path owns NAPI and the NET_RX workload, ksoftirqd and Softirq Load owns the overflow thread in depth, Choosing a Deferral Mechanism owns the design-time decision procedure, The PREEMPT_RT Real-Time Kernel owns the real-time conversion, and Workqueues and the Concurrency-Managed Workqueue owns the sleepable alternative. Links, not duplication.

Mental Model

Think of the softirq subsystem as a fixed bank of ten labelled mailboxes, one bank per CPU. The labels are nailed on at kernel build time — NET_RX, TIMER, and so on — and you cannot add a mailbox at runtime. To “raise” a softirq is to flip the flag on one mailbox on your CPU, saying “this kind of work is pending here.” Periodically — most importantly when a hardware interrupt finishes — a clerk (__do_softirq/handle_softirqs) sweeps the bank from the lowest-numbered mailbox up, and for each flag that is set, runs that softirq’s one registered handler function. The clerk works fast but is on a leash: it will not sweep forever (at most ten passes, or two milliseconds), after which it hands the rest to a background worker thread so the CPU can return to userspace.

flowchart TB
  HW["Hardware IRQ handler<br/>(top half)"] -->|"raise_softirq(NET_RX)"| BIT["Set pending bit<br/>per-CPU __softirq_pending"]
  BIT --> RET["Hard IRQ returns:<br/>irq_exit() checks pending"]
  RET --> DS["__do_softirq() / handle_softirqs()"]
  DS --> LOOP{"restart loop<br/>(snapshot pending,<br/>clear it, run set bits)"}
  LOOP --> VEC["For each set bit, lowest first:<br/>softirq_vec[nr].action()"]
  VEC --> H0["HI"]
  VEC --> H1["TIMER"]
  VEC --> H3["NET_RX"]
  VEC --> H9["RCU (last)"]
  VEC --> CHECK{"more pending AND<br/>time_before(end) AND<br/>!need_resched AND<br/>--max_restart?"}
  CHECK -->|yes| LOOP
  CHECK -->|no| KSOFT["wakeup_softirqd():<br/>defer rest to ksoftirqd"]

One pass through the softirq machinery. What it shows: a top half raises a softirq by setting a per-CPU pending bit; on return from the hard IRQ, irq_exit() triggers __do_softirq(), which snapshots and clears the pending word, then runs the handler for each set bit in fixed numeric priority order (HI=0 first, RCU=9 last). After the sweep it re-checks pending and loops — but only while it has restarts left, is under its 2 ms budget, and nobody needs the CPU; otherwise it wakes ksoftirqd to finish the rest. The insight to take: softirqs trade latency for throughput — they run almost immediately and in atomic context, but the kernel must actively bound them or they would livelock the machine under a packet flood.

The Softirq Vector — a fixed, ordered set

The complete softirq vector is this enum in include/linux/interrupt.h:

enum
{
    HI_SOFTIRQ=0,
    TIMER_SOFTIRQ,
    NET_TX_SOFTIRQ,
    NET_RX_SOFTIRQ,
    BLOCK_SOFTIRQ,
    IRQ_POLL_SOFTIRQ,
    TASKLET_SOFTIRQ,
    SCHED_SOFTIRQ,
    HRTIMER_SOFTIRQ,
    RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */
 
    NR_SOFTIRQS
};

Two facts are encoded in this list, and both matter. First, the numeric value is the priority: __do_softirq scans pending bits from bit 0 upward, so HI_SOFTIRQ (0) runs before TIMER_SOFTIRQ (1) before … before RCU_SOFTIRQ (9). HI_SOFTIRQ is the “high-priority tasklet” softirq; TASKLET_SOFTIRQ (6) is the normal-priority one — this is the entire reason a “high-priority tasklet” is faster than an ordinary one. The trailing comment “RCU should always be the last softirq” reflects a deliberate ordering choice: Read-Copy-Update (RCU) callback processing is placed last so grace-period work happens after the latency-sensitive network and timer softirqs. NR_SOFTIRQS is the sentinel count (10), used to size arrays.

Second — and this is the fact that makes softirqs feel archaic — the list has not grown since 2009. BLOCK_IOPOLL_SOFTIRQ (later renamed IRQ_POLL_SOFTIRQ when it was generalised out of the block layer for 4.5) was the last vector added, in 2.6.32 (LWN, February 2019). Re-checking that claim against the v6.12 tree — fifteen years after 2.6.32 shipped in December 2009 — gives the same answer: still ten, still the same ten, still in the same order. The comment sitting directly above the enum in interrupt.h explains why, and is worth quoting in full because it is the kernel’s own policy statement on the matter:

/* PLEASE, avoid to allocate new softirqs, if you need not _really_ high
   frequency threaded job scheduling. For almost all the purposes
   tasklets are more than enough. F.e. all serial device BHs et
   al. should be converted to tasklets, not to softirqs.
 */

That comment is itself a fossil — it recommends tasklets, which the kernel has since spent years trying to remove (LWN, “The end of tasklets”) — but its first clause still holds exactly. Adding a softirq vector would, in Thomas Gleixner’s summary of the mechanism, add to “a conglomerate of mostly unrelated jobs, which run in the context of a randomly chosen victim w/o the ability to put any control on them.”

A third, less-known constant lives just below the enum and encodes which vectors survive CPU hot-unplug:

#define SOFTIRQ_HOTPLUG_SAFE_MASK (BIT(TIMER_SOFTIRQ) | BIT(IRQ_POLL_SOFTIRQ) |\
				   BIT(HRTIMER_SOFTIRQ) | BIT(RCU_SOFTIRQ))

Its comment names the reason for each of the four: RCU has rcutree_migrate_callbacks() and rcutree_report_cpu_dead(), IRQ_POLL has irq_poll_cpu_dead(), and the two timer vectors have (hr)timers_dead_cpu() — each of which migrates the dying CPU’s queue somewhere else. Those four can therefore be ignored if they are still pending after ksoftirqd has been parked on an offlining CPU, because someone else will pick the work up. The other six cannot, and a pending bit among them at that point is a bug. This is a good illustration of how much per-vector special-casing hides behind what looks like a uniform array.

Each entry’s meaning:

The ten vectors, with their owners

Every slot is claimed by exactly one core subsystem, which calls open_softirq() once during its own initialisation. The table below was built by grepping the v6.12 tree for open_softirq( and recording the file and line of each of the ten call sites, so the “registered by” column is the actual code, not a recollection. The /proc/softirqs label column is the string from softirq_to_name[] (softirq.c:64–67), which is what you will match against when reading the counters on a live machine.

#	Enum constant	`/proc/softirqs` label	Handler, registered by (v6.12)	What it does
0	`HI_SOFTIRQ`	`HI`	`tasklet_hi_action` — softirq.c:911	Runs high-priority tasklets. Being vector 0 is the entire mechanism by which a “high-priority tasklet” beats a normal one.
1	`TIMER_SOFTIRQ`	`TIMER`	`run_timer_softirq` — kernel/time/timer.c:2758	Expires the low-resolution timer wheel (`struct timer_list`). See The Timer Wheel and timer_list.
2	`NET_TX_SOFTIRQ`	`NET_TX`	`net_tx_action` — net/core/dev.c:12154	Frees completed transmit buffers and drains the per-CPU `output_queue` of qdiscs needing a re-run.
3	`NET_RX_SOFTIRQ`	`NET_RX`	`net_rx_action` — net/core/dev.c:12155	The NAPI poll loop. Owned by The Network Receive Path; see NET_RX and NET_TX Softirqs.
4	`BLOCK_SOFTIRQ`	`BLOCK`	`blk_done_softirq` — block/blk-mq.c:4954	Completes block-I/O requests that were redirected to another CPU, not all of them — see the measurement below.
5	`IRQ_POLL_SOFTIRQ`	`IRQ_POLL`	`irq_poll_softirq` — lib/irq_poll.c:214	A NAPI-style polled-completion library generalised out of the block layer, used by storage HBAs (mpt3sas, be2iscsi, ib_srp).
6	`TASKLET_SOFTIRQ`	`TASKLET`	`tasklet_action` — softirq.c:910	Runs normal-priority tasklets, and (since v6.9) also the `WQ_BH` workqueue pseudo-workers.
7	`SCHED_SOFTIRQ`	`SCHED`	`sched_balance_softirq` — kernel/sched/fair.c:13674	Runs the fair-class load balancer out of line from the scheduler tick. See Per-CPU Run Queues and struct rq.
8	`HRTIMER_SOFTIRQ`	`HRTIMER`	`hrtimer_run_softirq` — kernel/time/hrtimer.c:2243	Expires soft high-resolution timers only — those armed with `HRTIMER_MODE_SOFT`. Hard hrtimers never touch this vector; see High-Resolution Timers and hrtimers.
9	`RCU_SOFTIRQ`	`RCU`	`rcu_core_si` — kernel/rcu/tree.c:5712	Invokes queued Read-Copy-Update callbacks and advances grace periods. Registered conditionally — see below.

Three entries in that table deserve a correction to the folklore.

HRTIMER_SOFTIRQ is not vestigial, but it is not what its name suggests either. High-resolution timer expiry moved out of softirq context into the hard-IRQ hrtimer_interrupt() path for the 4.2 release (LWN, “Per-vector software-interrupt masking”). What re-populated the slot was HRTIMER_MODE_SOFT: a caller can ask for a timer whose callback must not run in hard-IRQ context, and those are collected on a separate set of clock bases. In v6.12 the vector is raised from exactly two places — hrtimer_interrupt() and hrtimer_run_queues() — and both guard it with if (!ktime_before(now, cpu_base->softirq_expires_next)), i.e. “only if a soft timer is actually due.” Because soft-mode hrtimers are rare, the HRTIMER column in /proc/softirqs is normally almost flat; on the 32-CPU Fedora 44 box used for the measurements below it stood at 36,999 lifetime against 54.9 million for TIMER, and did not move at all across several load tests. This resolves a gap that an earlier revision of this note flagged as uncertain.

RCU_SOFTIRQ is the one conditional registration — the slot can be left permanently empty. Alone among the ten, its open_softirq() call sits inside an if:

static bool use_softirq = !IS_ENABLED(CONFIG_PREEMPT_RT);   /* tree.c:106 */
module_param(use_softirq, bool, 0444);
...
	if (use_softirq)                                        /* tree.c:5711 */
		open_softirq(RCU_SOFTIRQ, rcu_core_si);

When use_softirq is false, invoke_rcu_core() calls invoke_rcu_core_kthread() instead of raise_softirq(RCU_SOFTIRQ) (tree.c:2873) and a per-CPU rcuc/N thread is spawned to do the work (tree.c:2945). Two things flip it: building with CONFIG_PREEMPT_RT, which makes it default to false, or passing rcutree.use_softirq=0 on the kernel command line, which works on any kernel because the parameter is exported read-only at 0444. This is the single softirq vector that ordinary administrators can switch off, and it is a standard low-latency tuning knob — an rcuc kthread is a schedulable, prioritisable, SCHED_FIFO-capable entity, whereas RCU_SOFTIRQ is not. On the Fedora 44 box measured here, /sys/module/rcutree/parameters/use_softirq reads Y, and RCU is correspondingly the single busiest column in /proc/softirqs. The trailing comment on the enum — “Preferable RCU should always be the last softirq” — is a scheduling-order statement, not a registration one: grace-period work is deliberately placed behind the latency-sensitive timer and network vectors.

BLOCK_SOFTIRQ fires far less often than people expect. blk_mq_complete_request_remote() returns false — meaning “complete inline, right here in hard-IRQ context” — whenever the request’s hardware context maps a single software context on this very CPU, or the request is polled (blk-mq.c:1201–1204). Only when completion must be redirected does a softirq appear, and even then the usual route is an inter-processor interrupt (IPI) whose handler, __blk_mq_complete_request_remote(), raises BLOCK_SOFTIRQ on the target CPU. The direct blk_mq_raise_softirq() path is reserved for single-hardware-queue devices (rq->q->nr_hw_queues == 1). On a modern multi-queue NVMe device with per-CPU queue pairs, almost every completion takes the inline path and the BLOCK counter barely moves — which is exactly what the measurement section below shows.

The critical structural point: this list is the whole universe. A driver cannot call something like register_softirq() — there is no such API. The closest thing, open_softirq(), only installs the handler function for a slot the core already reserved:

void open_softirq(int nr, void (*action)(void))
{
    softirq_vec[nr].action = action;
}

softirq_vec[] is the per-handler table, static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp;, where struct softirq_action is just { void (*action)(void); }. So open_softirq(NET_RX_SOFTIRQ, net_rx_action) wires the networking receive function into slot 3 — and only the networking core, timer core, RCU core, etc., call this, once, at boot. Drivers that want deferred atomic work use a tasklet (built on the TASKLET/HI slots) or — increasingly preferred in the 6.12 era — a workqueue; they never get their own softirq.

Raising and Running — the Mechanism

Raising: flip a per-CPU pending bit

A top half schedules a softirq with raise_softirq() (or the irq-off variant if interrupts are already disabled):

void raise_softirq(unsigned int nr)
{
    unsigned long flags;
    local_irq_save(flags);
    raise_softirq_irqoff(nr);
    local_irq_restore(flags);
}
 
inline void raise_softirq_irqoff(unsigned int nr)
{
    __raise_softirq_irqoff(nr);
    if (!in_interrupt() && should_wake_ksoftirqd())
        wakeup_softirqd();
}
 
void __raise_softirq_irqoff(unsigned int nr)
{
    lockdep_assert_irqs_disabled();
    trace_softirq_raise(nr);
    or_softirq_pending(1UL << nr);
}

The substance is the last function: or_softirq_pending(1UL << nr) ORs bit nr into the per-CPU pending bitmask (__softirq_pending, accessed via local_softirq_pending()). That is all “raising” is — no list, no allocation, just a set bit on the current CPU. The wrapping is about correctness: raise_softirq brackets the bit-set with local_irq_save/restore because the pending word is read-modify-written and must not race with a hard IRQ on the same CPU. The raise_softirq_irqoff variant additionally notices: if we are not already inside an interrupt (!in_interrupt()) — i.e. some process-context code raised a softirq — and ksoftirqd should run, it wakes ksoftirqd immediately, because there is no imminent return-from-IRQ to drain the softirq otherwise.

Running: `__do_softirq` on return from the hard IRQ

The pending softirqs are drained primarily on the return path from a hardware interrupt: irq_exit() checks local_softirq_pending() and, if anything is set (and we are not nested), calls into __do_softirq(). In v6.12 the entry point delegates to handle_softirqs():

asmlinkage __visible void __softirq_entry __do_softirq(void)
{
    handle_softirqs(false);
}

The heart of handle_softirqs() is the bounded restart loop:

unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
int max_restart = MAX_SOFTIRQ_RESTART;
pending = local_softirq_pending();
...
restart:
    set_softirq_pending(0);          /* atomically grab+clear the pending word */
    local_irq_enable();
    h = softirq_vec;
    while ((softirq_bit = ffs(pending))) {
        ...
        trace_softirq_entry(vec_nr);
        h->action();                 /* run this softirq's handler */
        trace_softirq_exit(vec_nr);
        h++;
        pending >>= softirq_bit;
    }
    local_irq_disable();
    pending = local_softirq_pending();
    if (pending) {
        if (time_before(jiffies, end) && !need_resched() &&
            --max_restart)
            goto restart;
        wakeup_softirqd();
    }

Walking this carefully:

Snapshot and clear. set_softirq_pending(0) zeroes the per-CPU pending word and the loop works from a local copy (pending). This is the key to the “run on return from IRQ” model: any softirq raised while this pass is running sets a fresh bit that the snapshot missed, so it will be caught by the next local_softirq_pending() read after the sweep — not by this one. This makes raising idempotent and avoids unbounded in-pass growth.
Re-enable hard IRQs during handler execution. local_irq_enable() — softirq handlers run with hardware interrupts enabled (so a NIC can still interrupt while we process packets). They run with bottom halves disabled (softirqs masked on this CPU), which is what gives a softirq local non-reentrancy: a softirq cannot interrupt another softirq on the same CPU. It can, however, run on another CPU — see reentrancy below.
Dispatch in priority order. ffs(pending) (find-first-set) returns the lowest set bit, h->action() calls that slot’s handler, and pending >>= softirq_bit shifts it out so the next ffs finds the next one. Starting from softirq_vec and advancing h keeps the handler pointer in step with the bit. Because ffs always returns the lowest bit, handlers run in the enum order — fixed priority, HI first, RCU last.
Bounded restart. After the sweep, with IRQs disabled, it re-reads pending. If new softirqs arrived, it will goto restart — but only if all three conditions hold: time_before(jiffies, end) (still under the 2 ms budget), !need_resched() (no higher-priority task is waiting), and --max_restart is still nonzero (fewer than 10 passes so far). If any fails, it calls wakeup_softirqd() and returns, leaving the rest for the ksoftirqd thread.

The bound: `MAX_SOFTIRQ_RESTART` and the time budget

The two limits are #define MAX_SOFTIRQ_TIME msecs_to_jiffies(2) and #define MAX_SOFTIRQ_RESTART 10 (softirq.c:475–476). Either alone has a failure mode — the time check is the normal governor, but if jiffies is frozen (inside stop_machine(), say) it never trips, so the hard count of 10 guarantees termination regardless. This bound is the entire reason a sustained packet flood cannot livelock the machine purely in __do_softirq: after 2 ms or 10 restarts the work moves to a schedulable thread that competes fairly for the CPU, restoring responsiveness (LWN, “Software interrupts and realtime”). The constants, the kernel’s own justification for having two of them, and what they evaluate to at different CONFIG_HZ values are treated in full below, under The budget constants, and what they are on a real machine.

Where Softirqs Actually Run — five entry points, not one

“Softirqs run on return from a hardware interrupt” is true but incomplete, and the incompleteness is where most confusion about softirq latency comes from. In v6.12 there are five distinct places from which a pending softirq can be drained, and they have wildly different latency and priority characteristics. Knowing which one fired is the difference between “this packet was processed in a microsecond” and “this packet waited behind a nice 0 thread on a loaded CPU.”

#	Entry point	Called from	Context it runs in	Preemptible?	Typical latency
1	`invoke_softirq()` → `__do_softirq()`	`irq_exit_rcu()`, on the return path of every hard IRQ (softirq.c:627)	Hard-IRQ stack (or `do_softirq_own_stack()`), bottom halves disabled	No	Sub-microsecond; the common case
2	`__local_bh_enable_ip()` → `do_softirq()`	Any `local_bh_enable()` / `spin_unlock_bh()` / `rcu_read_unlock_bh()` that drops the last BH-disable	Process context of whoever released the lock	No, BH still off	Immediate, but charged to an unrelated task
3	`run_ksoftirqd()` → `handle_softirqs(true)`	The per-CPU `ksoftirqd/N` smpboot thread (softirq.c:919)	Ordinary kernel thread, `SCHED_OTHER`, nice 0	Yes — it is a normal task	Whenever the scheduler gets round to it
4	`wakeup_softirqd()` from `raise_softirq_irqoff()`	Process-context code raising a softirq outside any interrupt (softirq.c:670)	As #3	Yes	As #3
5	`do_softirq_post_smp_call_flush()`	After flushing the SMP call-function queue; a `PREEMPT_RT`-only safety valve (softirq.c:290)	RT: ksoftirqd wakeup	—	RT kernels only

Entry point 3 is the one that surprises people, because it changes the scheduling class of the work. ksoftirqd/N is registered as a struct smp_hotplug_thread with .thread_comm = "ksoftirqd/%u" and nothing else — no priority is set anywhere in spawn_ksoftirqd() (softirq.c:974), so the threads inherit the default SCHED_OTHER at nice 0. Confirmed directly on the measurement box, where all 32 of them report scheduling class TS and nice 0:

$ ps -eo pid,class,rtprio,ni,comm | grep ksoftirqd | tail -4
    189 TS       -   0 ksoftirqd/28
    195 TS       -   0 ksoftirqd/29
    201 TS       -   0 ksoftirqd/30
    207 TS       -   0 ksoftirqd/31

This produces the duality Jonathan Corbet identified: softirq work is “either the highest priority or the lowest priority work on the system,” with nothing in between, depending purely on whether it drained inline or overflowed into ksoftirqd (LWN, “Software interrupts and realtime”). A machine can look perfectly responsive right up to the moment the budget is exceeded, and then abruptly start dropping packets because the receive path is now competing with make -j32 on equal terms.

There is also a boot-time override worth knowing about. invoke_softirq() begins if (!force_irqthreads() || !__this_cpu_read(ksoftirqd)) — so when forced IRQ threading is on, entry point 1 is disabled entirely and every softirq is punted to ksoftirqd. force_irqthreads() is a static key enabled by the threadirqs early parameter (kernel/irq/manage.c:35) and hardcoded to true under CONFIG_PREEMPT_RT (interrupt.h:518). Notably, threadirqs is not documented in Documentation/admin-guide/kernel-parameters.txt at v6.12 — a grep of that file returns nothing — so the early_param() call is the only authoritative record that it exists.

The handoff, with both thresholds marked

flowchart TD
    HW["Device raises hardware IRQ"] --> TH["Top half runs<br/>(hard-IRQ context)"]
    TH --> RS["raise_softirq_irqoff(nr)<br/>or_softirq_pending(1UL &lt;&lt; nr)"]
    RS --> EXIT["irq_exit_rcu()"]
    EXIT --> Q1{"local_softirq_pending()<br/>non-zero AND not nested?"}
    Q1 -->|no| RET["return to interrupted code"]
    Q1 -->|yes| Q2{"force_irqthreads()<br/>i.e. threadirqs / PREEMPT_RT?"}
    Q2 -->|yes| WAKE
    Q2 -->|no| HS["handle_softirqs(false)<br/>end = jiffies + MAX_SOFTIRQ_TIME<br/>max_restart = MAX_SOFTIRQ_RESTART"]
    HS --> SWEEP["set_softirq_pending(0); local_irq_enable()<br/>ffs() loop: run softirq_vec[n].action()<br/>lowest vector number first"]
    SWEEP --> Q3{"local_softirq_pending()<br/>non-zero again?"}
    Q3 -->|no| DONE["done — return to interrupted code"]
    Q3 -->|yes| Q4{"time_before(jiffies, end)<br/>AND !need_resched()<br/>AND --max_restart != 0 ?"}
    Q4 -->|"all three hold"| SWEEP
    Q4 -->|"any one fails"| WAKE["wakeup_softirqd()"]
    WAKE --> KS["ksoftirqd/N wakes<br/>SCHED_OTHER, nice 0"]
    KS --> KSR["run_ksoftirqd()<br/>handle_softirqs(true)<br/>then cond_resched()"]
    KSR --> Q3

The full raise-to-drain path with the two budget checks made explicit. What it shows: every hardware interrupt ends in irq_exit_rcu(), which decides between draining inline (fast, atomic, unpreemptible) and waking ksoftirqd (slow, schedulable, fair). The goto restart edge is the inline loop; the wakeup_softirqd() edge is the overflow valve. Note that the same three-condition test guards both the inline restart and ksoftirqd’s own loop — ksoftirqd calls handle_softirqs(true), the identical function, so it too gives up after 2 ms and calls cond_resched() rather than monopolising the CPU. The insight to take: the entire softirq design is one bounded loop with a single escape hatch, and every softirq latency problem is a question of which side of that hatch the work landed on.

The budget constants, and what they are on a real machine

/* kernel/softirq.c:475-476, v6.12 */
#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
#define MAX_SOFTIRQ_RESTART 10

MAX_SOFTIRQ_TIME is expressed in jiffies, so its concrete value depends on CONFIG_HZ — which makes it one of those constants you cannot state without naming the kernel build. The table below gives the arithmetic for the common configurations, and for the box these measurements were taken on.

Constant	Source value	On the measurement box (`CONFIG_HZ=1000`)	At `CONFIG_HZ=250` (Debian/Ubuntu default)	At `CONFIG_HZ=100`	What happens when it is hit
`MAX_SOFTIRQ_TIME`	`msecs_to_jiffies(2)`	2 jiffies = 2 ms	1 jiffy (rounded up) = 4 ms	1 jiffy = 10 ms	`time_before(jiffies, end)` fails → `wakeup_softirqd()`
`MAX_SOFTIRQ_RESTART`	`10`	10 sweeps	10 sweeps	10 sweeps	`--max_restart` reaches 0 → `wakeup_softirqd()`
`need_resched()`	(not a constant)	checked every sweep	same	same	A higher-priority task is runnable → `wakeup_softirqd()` immediately
`net.core.netdev_budget`	sysctl, default 300	300 packets	—	—	NAPI’s own inner budget; `net_rx_action` returns, softirq loop continues
`net.core.netdev_budget_usecs`	sysctl, default 2000	2000 µs = 2 ms	—	—	NAPI’s inner time budget, deliberately equal to `MAX_SOFTIRQ_TIME`

Two observations follow from this table that are easy to miss.

First, at CONFIG_HZ=100 the time budget is five times looser than the code comment implies. msecs_to_jiffies(2) cannot round down to zero — it rounds up to one jiffy — so a kernel ticking at 100 Hz gives __do_softirq up to 10 ms per pass rather than 2 ms. The “2 ms” in the comment is the intent, not a guarantee, and the guarantee degrades exactly on the slow, low-tick embedded systems where it matters most. The restart counter is what actually bounds those machines.

Second, the NAPI budget nests inside the softirq budget, and the two time limits were set equal on purpose. netdev_budget_usecs defaults to 2000 µs — the same 2 ms as MAX_SOFTIRQ_TIME (Documentation/admin-guide/sysctl/net.rst). So under a receive flood the inner NAPI loop and the outer softirq loop expire at roughly the same moment, and raising netdev_budget alone without also understanding MAX_SOFTIRQ_TIME buys much less than operators expect: the outer loop hands off to ksoftirqd regardless. Live values on the measurement box:

$ sysctl net.core.netdev_budget net.core.netdev_budget_usecs
net.core.netdev_budget = 300
net.core.netdev_budget_usecs = 2000

The kernel’s own comment explains why there must be two backstops rather than one: “The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in certain cases, such as stop_machine(), jiffies may cease to increment and so we need the MAX_SOFTIRQ_RESTART limit as well to make sure we eventually return from this method. These limits have been established via experimentation. The two things to balance is latency against fairness — we want to handle softirqs as soon as possible, but they should not be able to lock up the box.” (softirq.c:462–474). If jiffies is frozen, the time check never trips; the hard count of 10 guarantees termination anyway.

One further subtlety inside handle_softirqs() that is invisible from the outside: the function clears PF_MEMALLOC on entry and restores it on exit. The comment explains that the current task’s context is borrowed for softirq execution, so a softirq handler must not inherit the interrupted task’s emergency-memory-reserve privileges — network receive would otherwise be able to dip into reserves it has no claim on. It is a small thing, but it is the clearest statement in the code of what a softirq actually is: not a thread, but a hijacking of whatever thread happened to be running.

The Reentrancy Model — the defining property

This is the single most consequential thing to understand about softirqs, and the most common source of bugs. The same softirq type can execute on multiple CPUs at the same time. If two NICs (or two queues of one NIC) on CPU 0 and CPU 1 both raise NET_RX_SOFTIRQ, then net_rx_action runs on both CPUs concurrently. The kernel does not serialize a softirq against itself across CPUs — only against itself on the same CPU (because bottom halves are disabled during a pass). The source comment in softirq.c is blunt about this design: it notes the model gives “something sort of weak cpu binding” and that, e.g., NET RX is “multithreaded” with “no global serialization.”

The direct consequence: any data a softirq handler reads or writes must be either per-CPU or protected by an SMP-safe lock. A global counter incremented in a softirq without locking is a data race; a global list manipulated in a softirq needs a spinlock. The idiomatic pattern is per-CPU data — which is why the kernel’s per-CPU local_t atomic operations explicitly document that they are usable “from any context (process, irq, softirq, nmi, …)” and that “Variables touched by local ops must be per cpu variables” (local_ops, v6.12). Per-CPU storage sidesteps the cross-CPU race entirely: each CPU’s softirq instance touches only its own copy.

This is precisely where tasklets differ and why they were invented. A tasklet is layered on the TASKLET/HI softirqs but the tasklet core guarantees a given tasklet runs on only one CPU at a time (it is serialized against itself), so tasklet code can be written without SMP locking of its own state — at the cost of that serialization being a scalability bottleneck. Choosing between “raw softirq” (max parallelism, you do the locking) and “tasklet” (serialized, simpler) is the classic trade-off — though in 6.12-era kernels tasklets are discouraged in favor of workqueues; see Choosing a Deferral Mechanism.

What a raise-and-drain cycle looks like in time

sequenceDiagram
    autonumber
    participant NIC as NIC (hardware)
    participant TH as Top half<br/>(hard IRQ, CPU 3)
    participant BM as __softirq_pending<br/>(per-CPU bitmask, CPU 3)
    participant SW as handle_softirqs()<br/>(CPU 3, BH disabled)
    participant KS as ksoftirqd/3<br/>(SCHED_OTHER, nice 0)

    NIC->>TH: MSI-X vector fires
    TH->>BM: set bit 3 in pending word [NET_RX]
    Note over BM: bitmask = 0b0000001000
    TH->>TH: napi_schedule, return IRQ_HANDLED
    TH->>SW: irq_exit_rcu sees pending, calls in
    SW->>BM: set_softirq_pending(0) — snapshot then clear
    Note over BM: live bitmask = 0<br/>local copy = 0b0000001000
    SW->>SW: local_irq_enable — hard IRQs back ON
    SW->>SW: ffs finds bit 3, runs net_rx_action
    NIC->>TH: another packet arrives MID-SWEEP
    TH->>BM: set bit 3 in pending word [NET_RX]
    Note over BM: bitmask = 0b0000001000 again<br/>(missed by this sweep's snapshot)
    SW->>SW: sweep ends, local_irq_disable
    SW->>BM: re-read local_softirq_pending, non-zero
    alt under 2 ms, no need_resched, restarts left
        SW->>SW: goto restart, sweep again inline
    else budget exhausted
        SW->>KS: wakeup_softirqd
        KS->>KS: handle_softirqs(true), then cond_resched
    end

One raise-and-drain cycle with a concurrent re-raise. What it shows: the pending word is a bitmask, not a queue — raising NET_RX twice sets the same bit twice and produces one handler call, so softirqs coalesce for free. The snapshot-and-clear at step 6 is what makes a mid-sweep raise safe: the new bit lands in the now-zeroed live bitmask and is caught by the re-read at step 12 rather than extending the current sweep indefinitely. The insight to take: hardware interrupts are enabled during the sweep (step 7) but bottom halves are disabled, which is the exact combination that gives a softirq local non-reentrancy while still allowing the device to keep interrupting — and it is why the bounded restart loop, not the sweep itself, is where livelock is prevented.

Softirqs Under PREEMPT_RT

Roughly a third of kernel/softirq.c at v6.12 is a second, parallel implementation guarded by #ifdef CONFIG_PREEMPT_RT, and it changes what a softirq is deeply enough that reading only the non-RT half will mislead you about the code you are looking at. The ownership of the real-time conversion itself belongs to The PREEMPT_RT Real-Time Kernel; what follows is only the softirq-shaped part of it.

On a non-RT kernel, “bottom halves disabled” is a count in the per-task preempt_count, and a section with BH disabled is therefore also non-preemptible. Under PREEMPT_RT that is unacceptable — a task must be able to be preempted inside a BH-disabled section — so the accounting is split in two (softirq.c:105–125):

struct softirq_ctrl {
	local_lock_t	lock;
	int		cnt;
};
static DEFINE_PER_CPU(struct softirq_ctrl, softirq_ctrl) = {
	.lock = INIT_LOCAL_LOCK(softirq_ctrl.lock),
};

A per-task counter (task::softirqs_disabled_cnt) backs softirq_count(), in_softirq() and in_serving_softirqs(), while a per-CPU counter plus a local_lock_t provides the actual mutual exclusion. The kernel’s own comment gives the reason for each: the per-task count is what the in_* predicates must consult “because these counts are only valid when the task holding softirq_ctrl::lock is running,” and the per-CPU count “prevents pointless wakeups of ksoftirqd in case that the task which is in a softirq disabled section is preempted or blocks.” Disabling bottom halves on an RT kernel is thus acquiring a lock, not incrementing a counter — which means it can block, which means it is a scheduling event, which is the whole point.

Three concrete behavioural differences follow, all visible in the v6.12 source:

	Non-RT (`!CONFIG_PREEMPT_RT`)	`CONFIG_PREEMPT_RT`
`invoke_softirq()` on IRQ exit	Runs `__do_softirq()` inline on the IRQ stack unless `threadirqs` is set (softirq.c:419)	Never runs inline — always `wakeup_softirqd()` (softirq.c:277)
`should_wake_ksoftirqd()`	Unconditionally `true`	`!this_cpu_read(softirq_ctrl.cnt)` — suppressed while someone holds the BH lock
BH-disabled section	Non-preemptible (`preempt_count`)	Preemptible; backed by a `local_lock_t` and an `rcu_read_lock()`
`RCU_SOFTIRQ`	Registered (`use_softirq` defaults true)	Not registered — `use_softirq` defaults to `!IS_ENABLED(CONFIG_PREEMPT_RT)`; work goes to `rcuc/N` threads
`force_irqthreads()`	Static key, off unless `threadirqs`	Compile-time `true` (interrupt.h:518)

The net effect is that on an RT kernel essentially all softirq work becomes schedulable thread work, which is exactly the property real-time users need and exactly the throughput cost everyone else declines to pay. There is also a small RT-only safety valve, do_softirq_post_smp_call_flush(), which WARN_ON_ONCEs if flushing the SMP call-function queue raised a softirq that was not pending beforehand — the comment notes that the only known block-layer code that did this is disabled on RT, and the warning exists to catch any new offender.

Point-in-time note: config PREEMPT_RT has existed as a Kconfig symbol since v5.3, but no architecture set ARCH_SUPPORTS_RT until v6.12 — so v6.12 is the first mainline release where an ordinary x86_64 make menuconfig will let you turn it on. The measurement box in this note is not an RT kernel: CONFIG_PREEMPT_RT is unset, CONFIG_PREEMPT_LAZY=y and CONFIG_PREEMPT_DYNAMIC=y, which is Fedora’s standard desktop configuration.

Observing Softirqs on a Live System

Everything above is observable without root. This section reproduces the measurements on a 32-CPU AMD workstation running Fedora 44, kernel 7.1.13-200.fc44 — a stable, not longterm, release (kernel.org/releases.json, 2026-09-12: mainline 7.3-rc2, stable 7.2.5, longterm branches 6.18.51 / 6.12.109 / 6.6.156 / 6.1.187 / 5.15.220). The code cited throughout this note is pinned to v6.12, which is the vault’s reference longterm branch; the live readings corroborate the shapes described here, not the v6.12 line numbers. Where the running kernel and v6.12 are known to differ, it is called out.

`/proc/softirqs` — one row per vector, one column per CPU

The file’s row order is the enum order, and the row labels are literally softirq_to_name[]. Summing across CPUs on an idle-ish desktop after several weeks of uptime:

$ awk 'NR==1{next}{s=0;for(i=2;i<=NF;i++)s+=$i; printf "%-10s %15d\n",$1,s}' /proc/softirqs
HI:                8535467
TIMER:            54899060
NET_TX:                997
NET_RX:           54886926
BLOCK:              123755
IRQ_POLL:                0
TASKLET:         471959363
SCHED:           404927977
HRTIMER:            36999
RCU:             490167662

Four things in that snapshot are worth stopping on, because each contradicts a plausible guess.

IRQ_POLL is exactly zero. No driver on this machine uses the irq_poll library — it is the preserve of enterprise storage HBAs. A zero here is normal, not broken.

NET_TX is 997 against NET_RX’s 54.9 million — a ratio of roughly 1 : 55,000 on a machine that has certainly transmitted about as many packets as it received. The reason is that modern NAPI drivers reclaim transmit descriptors inside their NET_RX poll function rather than raising a separate NET_TX softirq; net_tx_action is now mostly a qdisc re-run and deferred-skb-free path. Reading NET_TX as “transmit volume” is simply wrong.

RCU is the busiest vector on the machine, ahead of TASKLET and SCHED. Because rcutree.use_softirq is Y here, every callback batch invocation lands in vector 9.

HRTIMER is four orders of magnitude below TIMER — 36,999 versus 54.9 million — which is the empirical form of the “soft hrtimers are rare” claim made earlier.

Before-and-after: which vectors actually move under load

The counters are monotonic, so differencing two snapshots isolates one workload. Five seconds of idle, as a control:

--- IDLE 5s baseline (deltas) ---
HI:             +0      NET_RX:      +633     TASKLET:  +13300
TIMER:       +2427      BLOCK:         +2     SCHED:    +17152
NET_TX:         +0      IRQ_POLL:      +0     HRTIMER:      +0    RCU: +20232

Now 512 MB pushed through a loopback TCP socket (a Python sender and sink on 127.0.0.1, ~8,000 × 64 KiB writes), which takes a couple of seconds:

--- 512 MB over loopback (deltas) ---
HI:              +0
TIMER:          +60
NET_TX:          +0
NET_RX:      +16284
BLOCK:           +0
IRQ_POLL:        +0
TASKLET:        +52
SCHED:         +373
HRTIMER:         +0
RCU:           +341

This is as clean a signal as /proc/softirqs ever gives: NET_RX moves by 16,284 while every other vector stays at or below its idle rate, and several are pinned at zero. It also demonstrates the NET_TX point concretely — this workload is entirely transmit on one socket and receive on another, yet NET_TX does not move at all, because the loopback device delivers by handing the skb to the receive path on the same CPU rather than running a transmit-completion softirq.

Contrast that with a network workload involving real hardware — sixty concurrent HTTPS fetches over the physical NIC:

--- 60 concurrent HTTPS fetches (deltas) ---
TIMER:        +576      NET_RX:      +471      SCHED:  +3548
TASKLET:     +2198      RCU:        +4209      others: +0

NET_RX moves far less than the loopback test, because the bytes are fewer, but TIMER and SCHED move a lot more: sixty processes being created, scheduled across 32 CPUs and armed with socket timeouts. Comparing these two runs is a good way to internalise that /proc/softirqs measures kernel deferred-work events, not bytes.

And a block-I/O workload — 400 MB written with oflag=direct followed by sync:

--- 400 MB O_DIRECT write + sync (deltas) ---
BLOCK:           +0
TIMER:          +55
SCHED:         +445
RCU:           +528

BLOCK does not move at all. This is the blk_mq_complete_request_remote() behaviour described earlier: the NVMe device has a hardware queue per CPU, so every completion interrupt arrives on the CPU that submitted the request, the redirect test fails, and the request is completed inline in hard-IRQ context without ever raising BLOCK_SOFTIRQ. A reader who expected BLOCK to track disk traffic would conclude their instrumentation was broken; in fact the vector has been largely bypassed by blk-mq’s per-CPU queue design. See The Multi-Queue Block Layer blk-mq.

Uncertain

Verify: that the zero BLOCK delta is caused specifically by the rq->mq_hctx->nr_ctx == 1 && rq->mq_ctx->cpu == raw_smp_processor_id() early-return rather than by the REQ_POLLED branch or by the writes being absorbed before reaching the device. Reason: the code path is read and cited at v6.12, but the running kernel is 7.1.13 and /sys/kernel/debug is mode 0700 on this Fedora host, so blk-mq debugfs could not be inspected to confirm the queue mapping. To resolve: as root, read /sys/kernel/debug/block/<dev>/hctx*/ to confirm nr_ctx per hardware context, or compare /proc/interrupts NVMe rows before and after to confirm completions arrived on the submitting CPU. uncertain

Per-CPU distribution and the ksoftirqd threads

/proc/softirqs is per-CPU, which is what makes it useful for diagnosing an affinity problem. On this box the NET_RX column is heavily skewed:

$ grep NET_RX /proc/softirqs | tr -s ' ' '\n' | tail -n +2 | sort -rn | head -5
8030932
2721389
2603662
2427763
2385435

One CPU carries 8.0 million receive softirqs while the runners-up carry 2.4–2.7 million each — a 3× imbalance, the classic signature of a NIC whose interrupts are not being spread, or of a flow-hash that concentrates. That diagnosis, and the irqbalance / Receive Packet Steering remedies for it, belong to IRQ Affinity and irqbalance and The Network Receive Path.

The threads that drain the overflow are one per CPU, spawned by smpboot_register_percpu_thread():

$ ps -eo comm | grep -c ksoftirqd    # 32 CPUs, 32 threads
32

A useful sanity check when reasoning about latency on this machine: of 66 numeric IRQ lines in /proc/interrupts, only 10 have threaded handlers, and every one of those runs SCHED_FIFO at real-time priority 50 —

$ ps -eo pid,class,rtprio,comm | grep 'irq/'
    227 FF      50 irq/9-acpi
    239 FF      50 irq/26-AMD-Vi
    246 FF      50 irq/33-pciehp
    247 FF      50 irq/34-pciehp
    382 FF      50 irq/27-ACPI:Event
    ... (10 total)

So on this box the interrupt half of ten lines is real-time-prioritised, while all softirq work that overflows is SCHED_OTHER nice 0. That asymmetry is the practical meaning of the “highest or lowest priority” duality. Note also that CONFIG_RT_GROUP_SCHED is not set in this Fedora kernel — v6.12 is the release that made that gating explicit — so there is no real-time bandwidth throttling to rescue the system if one of those FF 50 threads spins; and RLIMIT_RTPRIO is 0 for the unprivileged account, so the comparison cannot be extended by creating real-time threads of one’s own. Details belong to Real-Time Scheduling SCHED_FIFO and SCHED_RR and Threaded Interrupt Handlers.

What could not be measured here

Uncertain

Verify: the per-vector time cost (as opposed to count) of softirq processing, and the actual frequency with which the 2 ms / 10-restart budget is exceeded on this machine. Reason: /proc/softirqs counts invocations only. The tools that would answer it — perf, bpftrace, bpftool, and the softirq_entry/softirq_exit tracepoints under /sys/kernel/tracing — are either not installed on this host or live under /sys/kernel/debug, which is mode 0700 with no passwordless sudo. To resolve: sudo mount -o remount,mode=755 /sys/kernel/debug (or run as root) and then cat /sys/kernel/tracing/events/irq/softirq_exit/enable, or install bpftrace and histogram softirq_entry→softirq_exit deltas per vec. A cruder unprivileged proxy is the softirq column of /proc/stat per CPU, which gives aggregate time but not a per-vector breakdown. uncertain

Failure Modes and Common Misunderstandings

“I’ll add my own softirq for my driver.” You cannot — open_softirq only fills a core-reserved slot, and the enum is closed. The fix is a tasklet or workqueue. Attempting to reuse an existing slot (say TASKLET) for unrelated work is wrong and will collide with tasklet machinery.
Sleeping in a softirq. Softirq handlers run in atomic context (bottom halves disabled). Calling anything that may sleep — mutex_lock, kmalloc(GFP_KERNEL), copy_to_user — is a bug that trips BUG: scheduling while atomic or sleeping-in-atomic warnings. Defer such work to a workqueue.
Unlocked shared state. The most insidious bug: a softirq updates a global structure without a lock, works fine on a uniprocessor or under light load, and corrupts data only when the softirq fires on two CPUs at once. Per-CPU data or spin_lock_bh() (which also blocks the softirq on the local CPU) is the cure.
Softirq starvation / ksoftirqd at 100%. Under a packet flood, __do_softirq repeatedly hits its 10-restart / 2 ms ceiling and dumps work on ksoftirqd, which then runs hot. Seeing ksoftirqd/N pinning a CPU in top is the signature of softirq overload — usually network receive. This is by design (it preserves fairness) but signals the box is interrupt-bound; mitigations are NAPI tuning and spreading IRQs across CPUs.
Assuming softirqs run “immediately.” They run on the next return-from-IRQ or local_bh_enable(), or in ksoftirqd — not synchronously at raise_softirq(). Code that needs the deferred work done before proceeding has a logic error.

Alternatives and When to Choose Them

Softirqs sit at the bottom of the deferral stack (Top Halves and Bottom Halves). Choose a raw softirq essentially never as a driver author — they exist for the kernel core’s hottest paths (networking, timers, RCU, block) where the overhead of any higher abstraction would matter and the core can do its own per-CPU locking. Choose a tasklet if you historically wanted simple, serialized, atomic deferral without writing SMP locks — but note tasklets are being phased out. Choose a workqueue (or a WQ_BH workqueue) or a threaded IRQ when the deferred work can sleep — takes mutexes, allocates with GFP_KERNEL, or touches userspace — because those run in schedulable kernel threads. The softirq’s distinguishing advantages are lowest latency (runs on return-from-IRQ, no scheduler round-trip) and full cross-CPU parallelism; its costs are the atomic-context restrictions and the per-CPU/locking burden it pushes onto the handler.

The decision procedure for picking among these is owned by Choosing a Deferral Mechanism, which compares them along five design axes — this note does not repeat that. What follows is the complementary view that note does not carry: the same five mechanisms compared by what they look like from outside the kernel, on a running machine. Every value in the “observed on the measurement box” column was read from ps -eo pid,class,rtprio,ni,comm or /proc on the Fedora 44 / 7.1.13 host described above, not inferred.

Property	Softirq	Tasklet	`WQ_BH` workqueue	Workqueue (threaded)	Threaded IRQ	Plain kthread
Execution context	Softirq (atomic)	Softirq (atomic)	Softirq (atomic)	Process	Process	Process
May it sleep?	No	No	No	Yes	Yes	Yes
Has a visible task?	Only on overflow: `ksoftirqd/N`	No — runs inside `TASKLET`/`HI`	No — pseudo-worker inside `TASKLET`/`HI`	Yes: `kworker/<cpu>:<id>`	Yes: `irq/<n>-<name>`	Yes: whatever you named it
Scheduling class observed	`TS` (`SCHED_OTHER`) nice 0	n/a (borrows current task)	n/a (borrows current task)	`TS` nice 0 or −20	`FF` (`SCHED_FIFO`) prio 50	`TS` nice 0 by default
Priority tunable at runtime?	No — `ksoftirqd` nice is not set by the kernel and renicing it is discouraged	No	No	Only by pool: `WQ_HIGHPRI` → nice −20	Yes — `chrt` on the `irq/N-*` PID	Yes
Per-CPU affinity	Always — pending bitmask is per-CPU	Always — per-CPU `tasklet_vec`	Always — per-CPU BH pool	Bound pool: yes. `WQ_UNBOUND`: no	Follows the IRQ’s affinity mask	Wherever you pin it
Same instance concurrent on 2 CPUs?	Yes — you must lock	No — serialized against itself	No — one BH context per CPU	Yes, across CPUs	No — one thread per action	No
Who may create one	Kernel core only, compile-time	Any module	Any code, `alloc_workqueue(.., WQ_BH, 0)`	Any code	Any driver, `request_threaded_irq()`	Any code, `kthread_run()`
Where to count it	`/proc/softirqs` row	`TASKLET` / `HI` rows	`TASKLET` / `HI` rows	`/proc/PID/stat` per `kworker`	`/proc/interrupts` + the thread’s CPU time	Normal process accounting
Observed on the measurement box	32 × `ksoftirqd/N`, all `TS 0`	folded into `TASKLET` (472 M)	folded into `TASKLET`	258 `kworker` threads	10 of 66 IRQ lines, all `FF 50`	e.g. `kswapd0`, `khugepaged` (`TS`)

Two rows in that table carry most of the practical weight.

“Priority tunable at runtime?” is the reason SCHED_FIFO threaded IRQ handlers and softirqs behave so differently under load. A threaded handler is a first-class task: an administrator can chrt it up or down, pin it, and see it in top. Overflowed softirq work has none of that — it is SCHED_OTHER nice 0 and there is no supported knob to change that, because raising ksoftirqd’s priority makes it preempt the very tasks that softirqs exist to serve, and lowering it starves the network stack. This is precisely the complaint that drives the PREEMPT_RT work, and the reason rcutree.use_softirq=0 exists as an escape hatch for the one vector that has one.

“Same instance concurrent on 2 CPUs?” is the axis on which softirqs are genuinely unusual, and it is the source of most softirq bugs. Everything else in the table is either per-CPU-serialized or one-thread-per-instance; the raw softirq is the only mechanism where the same handler function can be executing on thirty-two CPUs simultaneously with no framework-provided mutual exclusion whatsoever.

A note on the composite case, because it is easy to get wrong: a WQ_BH workqueue is not an eleventh softirq vector, and not a fourth kind of thing. In v6.12 its work items are executed inside the two tasklet softirqs, and the code says so with no ambiguity:

/* kernel/softirq.c:807-817, v6.12 */
static __latent_entropy void tasklet_action(void)
{
	workqueue_softirq_action(false);          /* run BH-workqueue items FIRST */
	tasklet_action_common(this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
}
 
static __latent_entropy void tasklet_hi_action(void)
{
	workqueue_softirq_action(true);
	tasklet_action_common(this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
}

Three consequences. First, WQ_BH activity is counted in the TASKLET and HI rows of /proc/softirqs, not in a new row — there is no way to separate the two from that file. Second, WQ_BH items run ahead of the tasklets on the same vector within a single sweep, because the call comes first. Third, the arrangement is explicitly temporary: the comment above workqueue_softirq_action() in kernel/workqueue.c states that “Once conversion from tasklet is complete, the need_more_worker() test can be dropped… After full conversion, we’ll add worker->softirq_action, directly use the softirq action and obtain the worker pointer from the softirq_action pointer” (workqueue.c:3586–3598). Tejun Heo chose to route through tasklets when the feature was merged in v6.9 specifically to avoid priority inversion between WQ_BH items and the 500-odd tasklet users still in the tree (LWN, “The end of tasklets”). See Workqueues and the Concurrency-Managed Workqueue for the API side and Tasklets and Their Deprecation for the migration.

Production Notes

The softirq/ksoftirqd design is a textbook case of bounded work in interrupt context. The 2 ms + 10-restart governor was tuned to keep softirq processing from monopolizing a CPU while still draining bursts inline for low latency; the moment it overflows, work moves to ksoftirqd, which since kernel 2.6.23 runs at normal user priority — making softirq work “either the highest priority or the lowest priority work on the system” depending on whether it runs inline or in ksoftirqd (LWN 520076). That duality is why a CPU servicing a flood can simultaneously feel snappy (inline drains keep up under moderate load) and then suddenly show ksoftirqd burning a core (overflow under heavy load).

The operational playbook that follows from everything above is short, and mostly consists of not reaching for the obvious knob.

Do not renice ksoftirqd. It is the first thing people try when ksoftirqd/N shows up at the top of top, and it is wrong in both directions. Raising its priority lets deferred network processing preempt the applications that consume the packets, which converts a throughput problem into a livelock. Lowering it starves the receive path and produces drops. The kernel sets no priority for these threads at all — spawn_ksoftirqd() registers a bare smp_hotplug_thread with only a thread_comm — and that neutrality is deliberate.

ksoftirqd at 100% is a symptom, not the disease. It means the inline budget is being exhausted repeatedly, which on almost every real system means network receive. The real fixes live elsewhere: spreading the NIC’s interrupts across CPUs (IRQ Affinity and irqbalance), enabling Receive Packet Steering or Receive Flow Steering, increasing interrupt coalescing on the NIC so fewer, larger batches arrive, or raising net.core.netdev_budget together with an understanding that the outer 2 ms softirq budget still caps the pass. See Interrupt Storms and Livelock for the failure mode this is all defending against.

The one vector you can actually turn off is RCU. rcutree.use_softirq=0 on the kernel command line moves RCU callback invocation into per-CPU rcuc/N threads, which are prioritisable and pinnable. On a machine where /proc/softirqs shows RCU as the dominant column — as on the box measured above, at 490 million — this is a genuine lever for latency-sensitive workloads, and it is the mechanism PREEMPT_RT uses by default. It costs throughput, because a thread wakeup is more expensive than a softirq raise.

Read /proc/softirqs per-CPU, not summed. The summed view tells you which subsystem is busy; only the per-CPU columns tell you whether the busyness is balanced, and imbalance is the far more common and more actionable pathology. A single CPU carrying 3× the NET_RX of its neighbours, as measured above, is a configuration problem, not a capacity problem.

Version awareness. Softirq internals are unusually stable — the vector list is unchanged since 2009 and the 2 ms / 10 restart constants have been in place for over a decade — but the surrounding mechanisms are not. WQ_BH (v6.9) changed what the TASKLET and HI rows count; ARCH_SUPPORTS_RT landing in v6.12 changed who can realistically build the RT variant. When citing softirq behaviour, name the release.

vault

Explorer

Softirqs and the Softirq Vector

Softirqs and the Softirq Vector

Mental Model

The Softirq Vector — a fixed, ordered set

The ten vectors, with their owners

Raising and Running — the Mechanism

Raising: flip a per-CPU pending bit

Running: `__do_softirq` on return from the hard IRQ

The bound: `MAX_SOFTIRQ_RESTART` and the time budget

Where Softirqs Actually Run — five entry points, not one

The handoff, with both thresholds marked

The budget constants, and what they are on a real machine

The Reentrancy Model — the defining property

What a raise-and-drain cycle looks like in time

Softirqs Under PREEMPT_RT

Observing Softirqs on a Live System

`/proc/softirqs` — one row per vector, one column per CPU

Before-and-after: which vectors actually move under load

Per-CPU distribution and the ksoftirqd threads

What could not be measured here

Failure Modes and Common Misunderstandings

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents

Backlinks

vault

Explorer

Softirqs and the Softirq Vector

Softirqs and the Softirq Vector

Mental Model

The Softirq Vector — a fixed, ordered set

The ten vectors, with their owners

Raising and Running — the Mechanism

Raising: flip a per-CPU pending bit

Running: __do_softirq on return from the hard IRQ

The bound: MAX_SOFTIRQ_RESTART and the time budget

Where Softirqs Actually Run — five entry points, not one

The handoff, with both thresholds marked

The budget constants, and what they are on a real machine

The Reentrancy Model — the defining property

What a raise-and-drain cycle looks like in time

Softirqs Under PREEMPT_RT

Observing Softirqs on a Live System

/proc/softirqs — one row per vector, one column per CPU

Before-and-after: which vectors actually move under load

Per-CPU distribution and the ksoftirqd threads

What could not be measured here

Failure Modes and Common Misunderstandings

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents

Backlinks

Running: `__do_softirq` on return from the hard IRQ

The bound: `MAX_SOFTIRQ_RESTART` and the time budget

`/proc/softirqs` — one row per vector, one column per CPU