vault

Softirqs and the Softirq Vector

15 min read

Softirqs and the Softirq Vector

A softirq (short for “software interrupt”) is the lowest-level, highest-frequency mechanism Linux uses to defer work out of a hardware interrupt handler. When a device’s top-half handler has done the bare minimum, it raises a softirq — sets a per-CPU pending bit — and returns; the kernel then runs the pending softirqs slightly later, on the return path from the hardware interrupt (and, under sustained load, in a dedicated per-CPU kernel thread called ksoftirqd). Unlike the four bottom-half mechanisms layered above it, softirqs are a statically fixed, compile-time set: a small enum of exactly ten kinds (HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, RCU), and only the kernel core may add one — there is no dynamic registration for drivers (interrupt.h, v6.12). The defining property that makes softirqs both powerful and dangerous is their reentrancy model: the same softirq type can run concurrently on different CPUs at the same time, so any state a softirq touches must be per-CPU or explicitly Symmetric MultiProcessing (SMP)-locked — a sharp contrast with tasklets, which the kernel serializes so a given tasklet never runs on two CPUs at once. To stop softirq processing from starving everything else, the kernel bounds each processing pass with MAX_SOFTIRQ_RESTART (10) restarts plus a 2-millisecond time budget before punting the remainder to ksoftirqd (softirq.c, v6.12).

Mental Model

Think of the softirq subsystem as a fixed bank of ten labelled mailboxes, one bank per CPU. The labels are nailed on at kernel build time — NET_RX, TIMER, and so on — and you cannot add a mailbox at runtime. To “raise” a softirq is to flip the flag on one mailbox on your CPU, saying “this kind of work is pending here.” Periodically — most importantly when a hardware interrupt finishes — a clerk (__do_softirq/handle_softirqs) sweeps the bank from the lowest-numbered mailbox up, and for each flag that is set, runs that softirq’s one registered handler function. The clerk works fast but is on a leash: it will not sweep forever (at most ten passes, or two milliseconds), after which it hands the rest to a background worker thread so the CPU can return to userspace.

flowchart TB
  HW["Hardware IRQ handler<br/>(top half)"] -->|"raise_softirq(NET_RX)"| BIT["Set pending bit<br/>per-CPU __softirq_pending"]
  BIT --> RET["Hard IRQ returns:<br/>irq_exit() checks pending"]
  RET --> DS["__do_softirq() / handle_softirqs()"]
  DS --> LOOP{"restart loop<br/>(snapshot pending,<br/>clear it, run set bits)"}
  LOOP --> VEC["For each set bit, lowest first:<br/>softirq_vec[nr].action()"]
  VEC --> H0["HI"]
  VEC --> H1["TIMER"]
  VEC --> H3["NET_RX"]
  VEC --> H9["RCU (last)"]
  VEC --> CHECK{"more pending AND<br/>time_before(end) AND<br/>!need_resched AND<br/>--max_restart?"}
  CHECK -->|yes| LOOP
  CHECK -->|no| KSOFT["wakeup_softirqd():<br/>defer rest to ksoftirqd"]

One pass through the softirq machinery. What it shows: a top half raises a softirq by setting a per-CPU pending bit; on return from the hard IRQ, irq_exit() triggers __do_softirq(), which snapshots and clears the pending word, then runs the handler for each set bit in fixed numeric priority order (HI=0 first, RCU=9 last). After the sweep it re-checks pending and loops — but only while it has restarts left, is under its 2 ms budget, and nobody needs the CPU; otherwise it wakes ksoftirqd to finish the rest. The insight to take: softirqs trade latency for throughput — they run almost immediately and in atomic context, but the kernel must actively bound them or they would livelock the machine under a packet flood.

The Softirq Vector — a fixed, ordered set

The complete softirq vector is this enum in include/linux/interrupt.h:

enum
{
    HI_SOFTIRQ=0,
    TIMER_SOFTIRQ,
    NET_TX_SOFTIRQ,
    NET_RX_SOFTIRQ,
    BLOCK_SOFTIRQ,
    IRQ_POLL_SOFTIRQ,
    TASKLET_SOFTIRQ,
    SCHED_SOFTIRQ,
    HRTIMER_SOFTIRQ,
    RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */
 
    NR_SOFTIRQS
};

Two facts are encoded in this list, and both matter. First, the numeric value is the priority: __do_softirq scans pending bits from bit 0 upward, so HI_SOFTIRQ (0) runs before TIMER_SOFTIRQ (1) before … before RCU_SOFTIRQ (9). HI_SOFTIRQ is the “high-priority tasklet” softirq; TASKLET_SOFTIRQ (6) is the normal-priority one — this is the entire reason a “high-priority tasklet” is faster than an ordinary one. The trailing comment “RCU should always be the last softirq” reflects a deliberate ordering choice: Read-Copy-Update (RCU) callback processing is placed last so grace-period work happens after the latency-sensitive network and timer softirqs. NR_SOFTIRQS is the sentinel count (10), used to size arrays. Each entry’s meaning:

  • HI / TASKLET — the two tasklet softirqs (high and normal priority). Tasklets are built on top of softirqs, which is why they appear here.
  • TIMER — expiry of low-resolution kernel timers (timer_list).
  • NET_TX / NET_RX — network transmit-completion and receive processing; the NAPI poll loop runs in NET_RX. These are owned by the networking stack — see NET_RX and NET_TX Softirqs.
  • BLOCK — block-I/O completion (finishing disk/NVMe requests).
  • IRQ_POLL — a generic interrupt-polling library (irq_poll), used by some block and storage drivers to batch completions NAPI-style.
  • SCHED — scheduler load-balancing and related deferred scheduler work.
  • HRTIMER — historically high-resolution timer expiry (the softirq still exists; much hrtimer work moved to hard-IRQ context, but the vector slot remains).
  • RCU — invoking queued RCU callbacks.

The critical structural point: this list is the whole universe. A driver cannot call something like register_softirq() — there is no such API. The closest thing, open_softirq(), only installs the handler function for a slot the core already reserved:

void open_softirq(int nr, void (*action)(void))
{
    softirq_vec[nr].action = action;
}

softirq_vec[] is the per-handler table, static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp;, where struct softirq_action is just { void (*action)(void); }. So open_softirq(NET_RX_SOFTIRQ, net_rx_action) wires the networking receive function into slot 3 — and only the networking core, timer core, RCU core, etc., call this, once, at boot. Drivers that want deferred atomic work use a tasklet (built on the TASKLET/HI slots) or — increasingly preferred in the 6.12 era — a workqueue; they never get their own softirq.

Raising and Running — the Mechanism

Raising: flip a per-CPU pending bit

A top half schedules a softirq with raise_softirq() (or the irq-off variant if interrupts are already disabled):

void raise_softirq(unsigned int nr)
{
    unsigned long flags;
    local_irq_save(flags);
    raise_softirq_irqoff(nr);
    local_irq_restore(flags);
}
 
inline void raise_softirq_irqoff(unsigned int nr)
{
    __raise_softirq_irqoff(nr);
    if (!in_interrupt() && should_wake_ksoftirqd())
        wakeup_softirqd();
}
 
void __raise_softirq_irqoff(unsigned int nr)
{
    lockdep_assert_irqs_disabled();
    trace_softirq_raise(nr);
    or_softirq_pending(1UL << nr);
}

The substance is the last function: or_softirq_pending(1UL << nr) ORs bit nr into the per-CPU pending bitmask (__softirq_pending, accessed via local_softirq_pending()). That is all “raising” is — no list, no allocation, just a set bit on the current CPU. The wrapping is about correctness: raise_softirq brackets the bit-set with local_irq_save/restore because the pending word is read-modify-written and must not race with a hard IRQ on the same CPU. The raise_softirq_irqoff variant additionally notices: if we are not already inside an interrupt (!in_interrupt()) — i.e. some process-context code raised a softirq — and ksoftirqd should run, it wakes ksoftirqd immediately, because there is no imminent return-from-IRQ to drain the softirq otherwise.

Running: __do_softirq on return from the hard IRQ

The pending softirqs are drained primarily on the return path from a hardware interrupt: irq_exit() checks local_softirq_pending() and, if anything is set (and we are not nested), calls into __do_softirq(). In v6.12 the entry point delegates to handle_softirqs():

asmlinkage __visible void __softirq_entry __do_softirq(void)
{
    handle_softirqs(false);
}

The heart of handle_softirqs() is the bounded restart loop:

unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
int max_restart = MAX_SOFTIRQ_RESTART;
pending = local_softirq_pending();
...
restart:
    set_softirq_pending(0);          /* atomically grab+clear the pending word */
    local_irq_enable();
    h = softirq_vec;
    while ((softirq_bit = ffs(pending))) {
        ...
        trace_softirq_entry(vec_nr);
        h->action();                 /* run this softirq's handler */
        trace_softirq_exit(vec_nr);
        h++;
        pending >>= softirq_bit;
    }
    local_irq_disable();
    pending = local_softirq_pending();
    if (pending) {
        if (time_before(jiffies, end) && !need_resched() &&
            --max_restart)
            goto restart;
        wakeup_softirqd();
    }

Walking this carefully:

  1. Snapshot and clear. set_softirq_pending(0) zeroes the per-CPU pending word and the loop works from a local copy (pending). This is the key to the “run on return from IRQ” model: any softirq raised while this pass is running sets a fresh bit that the snapshot missed, so it will be caught by the next local_softirq_pending() read after the sweep — not by this one. This makes raising idempotent and avoids unbounded in-pass growth.

  2. Re-enable hard IRQs during handler execution. local_irq_enable() — softirq handlers run with hardware interrupts enabled (so a NIC can still interrupt while we process packets). They run with bottom halves disabled (softirqs masked on this CPU), which is what gives a softirq local non-reentrancy: a softirq cannot interrupt another softirq on the same CPU. It can, however, run on another CPU — see reentrancy below.

  3. Dispatch in priority order. ffs(pending) (find-first-set) returns the lowest set bit, h->action() calls that slot’s handler, and pending >>= softirq_bit shifts it out so the next ffs finds the next one. Starting from softirq_vec and advancing h keeps the handler pointer in step with the bit. Because ffs always returns the lowest bit, handlers run in the enum order — fixed priority, HI first, RCU last.

  4. Bounded restart. After the sweep, with IRQs disabled, it re-reads pending. If new softirqs arrived, it will goto restart — but only if all three conditions hold: time_before(jiffies, end) (still under the 2 ms budget), !need_resched() (no higher-priority task is waiting), and --max_restart is still nonzero (fewer than 10 passes so far). If any fails, it calls wakeup_softirqd() and returns, leaving the rest for the ksoftirqd thread.

The bound: MAX_SOFTIRQ_RESTART and the time budget

The two limits are defined with an explanatory comment in softirq.c:

#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
#define MAX_SOFTIRQ_RESTART 10

The comment: “We restart softirq processing for at most MAX_SOFTIRQ_RESTART times, but break the loop if need_resched() is set or after 2 ms. The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in certain cases, such as stop_machine(), jiffies may cease to increment and so we need the MAX_SOFTIRQ_RESTART limit as well to make sure we eventually return.” Two backstops, because either alone has a failure mode: the time check is the normal governor, but if jiffies is frozen (e.g. inside stop_machine()) it never trips, so the hard count of 10 guarantees termination regardless. This bound is the entire reason a sustained packet flood cannot livelock the machine purely in __do_softirq — after 2 ms or 10 restarts the work moves to a schedulable thread (ksoftirqd) that competes fairly for the CPU, restoring responsiveness (LWN, “Software interrupts and realtime”).

The Reentrancy Model — the defining property

This is the single most consequential thing to understand about softirqs, and the most common source of bugs. The same softirq type can execute on multiple CPUs at the same time. If two NICs (or two queues of one NIC) on CPU 0 and CPU 1 both raise NET_RX_SOFTIRQ, then net_rx_action runs on both CPUs concurrently. The kernel does not serialize a softirq against itself across CPUs — only against itself on the same CPU (because bottom halves are disabled during a pass). The source comment in softirq.c is blunt about this design: it notes the model gives “something sort of weak cpu binding” and that, e.g., NET RX is “multithreaded” with “no global serialization.”

The direct consequence: any data a softirq handler reads or writes must be either per-CPU or protected by an SMP-safe lock. A global counter incremented in a softirq without locking is a data race; a global list manipulated in a softirq needs a spinlock. The idiomatic pattern is per-CPU data — which is why the kernel’s per-CPU local_t atomic operations explicitly document that they are usable “from any context (process, irq, softirq, nmi, …)” and that “Variables touched by local ops must be per cpu variables” (local_ops, v6.12). Per-CPU storage sidesteps the cross-CPU race entirely: each CPU’s softirq instance touches only its own copy.

This is precisely where tasklets differ and why they were invented. A tasklet is layered on the TASKLET/HI softirqs but the tasklet core guarantees a given tasklet runs on only one CPU at a time (it is serialized against itself), so tasklet code can be written without SMP locking of its own state — at the cost of that serialization being a scalability bottleneck. Choosing between “raw softirq” (max parallelism, you do the locking) and “tasklet” (serialized, simpler) is the classic trade-off — though in 6.12-era kernels tasklets are discouraged in favor of workqueues; see Choosing a Deferral Mechanism.

Failure Modes and Common Misunderstandings

  • “I’ll add my own softirq for my driver.” You cannot — open_softirq only fills a core-reserved slot, and the enum is closed. The fix is a tasklet or workqueue. Attempting to reuse an existing slot (say TASKLET) for unrelated work is wrong and will collide with tasklet machinery.
  • Sleeping in a softirq. Softirq handlers run in atomic context (bottom halves disabled). Calling anything that may sleep — mutex_lock, kmalloc(GFP_KERNEL), copy_to_user — is a bug that trips BUG: scheduling while atomic or sleeping-in-atomic warnings. Defer such work to a workqueue.
  • Unlocked shared state. The most insidious bug: a softirq updates a global structure without a lock, works fine on a uniprocessor or under light load, and corrupts data only when the softirq fires on two CPUs at once. Per-CPU data or spin_lock_bh() (which also blocks the softirq on the local CPU) is the cure.
  • Softirq starvation / ksoftirqd at 100%. Under a packet flood, __do_softirq repeatedly hits its 10-restart / 2 ms ceiling and dumps work on ksoftirqd, which then runs hot. Seeing ksoftirqd/N pinning a CPU in top is the signature of softirq overload — usually network receive. This is by design (it preserves fairness) but signals the box is interrupt-bound; mitigations are NAPI tuning and spreading IRQs across CPUs.
  • Assuming softirqs run “immediately.” They run on the next return-from-IRQ or local_bh_enable(), or in ksoftirqd — not synchronously at raise_softirq(). Code that needs the deferred work done before proceeding has a logic error.

Alternatives and When to Choose Them

Softirqs sit at the bottom of the four-mechanism deferral stack (Top Halves and Bottom Halves). Choose a raw softirq essentially never as a driver author — they exist for the kernel core’s hottest paths (networking, timers, RCU, block) where the overhead of any higher abstraction would matter and the core can do its own per-CPU locking. Choose a tasklet if you historically wanted simple, serialized, atomic deferral without writing SMP locks — but note tasklets are being phased out. Choose a workqueue (or a BH workqueue) or a threaded IRQ when the deferred work can sleep — takes mutexes, allocates with GFP_KERNEL, or touches userspace — because those run in schedulable kernel threads. The softirq’s distinguishing advantages are lowest latency (runs on return-from-IRQ, no scheduler round-trip) and full cross-CPU parallelism; its costs are the atomic-context restrictions and the per-CPU/locking burden it pushes onto the handler.

Production Notes

The softirq/ksoftirqd design is a textbook case of bounded work in interrupt context. The 2 ms + 10-restart governor was tuned to keep softirq processing from monopolizing a CPU while still draining bursts inline for low latency; the moment it overflows, work moves to ksoftirqd, which since kernel 2.6.23 runs at normal user priority — making softirq work “either the highest priority or the lowest priority work on the system” depending on whether it runs inline or in ksoftirqd (LWN 520076). That duality is why a CPU servicing a flood can simultaneously feel snappy (inline drains keep up under moderate load) and then suddenly show ksoftirqd burning a core (overflow under heavy load). On the real-time front, PREEMPT_RT historically pushed softirq processing into per-softirq threads to make it preemptible; tracking that is out of scope here. For observability, /proc/softirqs shows per-CPU counts for each of the ten softirq types — a fast way to see which softirq is hot (a soaring NET_RX column under load points squarely at network receive).

Uncertain

Verify: that the HRTIMER_SOFTIRQ slot’s current role in v6.12 is limited (much hrtimer expiry having moved to hard-IRQ context, leaving the softirq for softirq-marked timers). Reason: the slot is confirmed present in the v6.12 enum, but its exact usage split between hard-IRQ and softirq paths was not traced to kernel/time/hrtimer.c during this research. To resolve: read hrtimer_interrupt() / __hrtimer_run_queues() and the HRTIMER_SOFTIRQ raise sites in kernel/time/hrtimer.c at v6.12. uncertain

See Also

Graph View

Backlinks