vault

Interrupt Storms and Livelock

16 min read

Interrupt Storms and Livelock

When interrupts arrive faster than the system can fully process them, an interrupt-driven kernel does not merely slow down — it can collapse. Because hardware interrupts have absolute priority over all software, a device interrupting at line rate can pin a CPU at ~100% in hard-IRQ and softirq context, leaving no cycles for the very code that consumes the work the interrupts announced. The pathology is receive livelock: as offered load rises past a saturation point, delivered throughput does not plateau — it falls, heading toward zero, while the machine is maximally busy doing nothing useful. The classic statement is Mogul & Ramakrishnan, “Eliminating Receive Livelock in an Interrupt-Driven Kernel” (USENIX 1996), who define the condition precisely: “the system spends all its time processing interrupts, to the exclusion of other necessary tasks” so that “no packets are delivered to the user application or the output of the system” (Mogul & Ramakrishnan 1996). Linux’s defenses are layered: NAPI (switch a flooded device from interrupts to polling), hardware interrupt coalescing, a softirq time budget that hands off to ksoftirqd, spurious-interrupt disabling for a wedged line, and threaded IRQs for priority control.

Pinned to Linux 6.12 LTS (released 2024-11-17). This note explains the failure and its mitigations; the mechanisms themselves are detailed in their own leaves — NAPI and Polled Receive, ksoftirqd and Softirq Load, Softirqs and the Softirq Vector, Shared Interrupt Lines — and cross-linked rather than duplicated.

Mental Model

The intuition is a priority inversion built into the hardware. A hardware interrupt is not a normal task the scheduler can deprioritize; it preempts whatever is running, including the kernel’s own protocol stack and the application. So if you draw a graph of delivered throughput (work that actually reaches its consumer) against offered load (interrupts per second), a naive interrupt-driven system rises, peaks at the Maximum Loss Free Receive Rate (MLFRR) — the highest input rate it can sustain with zero loss — and then, instead of staying flat, declines, because each additional interrupt steals cycles from completing the work already accepted. Mogul & Ramakrishnan put it exactly: “Any purely interrupt-driven system using fixed interrupt priorities will suffer from receive livelock under input overload conditions. Once the input rate exceeds the reciprocal of the CPU cost of processing one input event, any task scheduled at a lower priority will not get a chance to run” (Mogul & Ramakrishnan 1996).

flowchart LR
  subgraph G["Delivered throughput vs offered load"]
    direction TB
    A["Low load:<br/>interrupts cheap,<br/>throughput tracks load"] --> B["Knee = MLFRR<br/>(max loss-free rate)"]
    B --> C["Ideal/robust system:<br/>throughput stays flat,<br/>excess dropped EARLY"]
    B --> D["Livelock-prone system:<br/>throughput FALLS toward 0<br/>as load rises further"]
  end

Throughput as offered load increases. What it shows: every realizable system has a knee at the MLFRR; above it some input must be dropped. The fork in the road is how the system behaves past the knee — a robust system holds throughput flat by dropping excess early (before investing work), while a livelock-prone system lets interrupt overhead consume the CPU so that throughput decreases with rising load, reaching zero at the livelock point. The insight to take: the goal of every mitigation below is to bend the curve from the falling line back to the flat line — i.e., to make the system shed load gracefully instead of catastrophically.

Why Throughput Collapses: the Mechanism

Trace a single received packet through a 1990s-style (and, structurally, still-relevant) interrupt-driven path. The NIC raises an interrupt; the CPU drops into the device’s hard-IRQ handler at high Interrupt Priority Level (IPL), which copies the packet off the card and queues it, then raises a software interrupt for protocol processing at a lower IPL; that softirq runs IP/TCP processing and queues the packet to a socket; finally the application reads it. Mogul & Ramakrishnan describe this 4.2BSD structure and note the fatal property: “Tasks performed at interrupt level, by definition, have absolute priority over all other tasks. If the event rate is high enough to cause the system to spend all of its time responding to interrupts, then nothing else will happen, and the system throughput will drop to zero” (Mogul & Ramakrishnan 1996).

The collapse is self-reinforcing. Each hard-IRQ preempts the softirq that would consume earlier packets; the softirq queue and socket queues fill; the queues are finite, so packets get dropped — but they are dropped late, after the hard-IRQ handler has already spent cycles pulling them off the card. The paper’s sharpest observation: “A livelocked system wastes all of the effort it puts into partially processing received packets, since they are all discarded” (Mogul & Ramakrishnan 1996). Every drop is paid for twice — once to receive, once to throw away — and at saturation all the CPU goes to receive-and-discard, leaving nothing for the protocol stack or application. That is livelock: “a state of the system where no useful progress is being made, because some necessary resource is entirely consumed with processing receiver interrupts.” Crucially it is not a deadlock — “When the input load drops sufficiently, the system leaves this state, and is again able to make forward progress” — but while the flood lasts, the machine is wedged.

The paper is careful about one tempting non-fix: batching (processing several packets per interrupt) raises the MLFRR but does not cure livelock — “Batching can shift the livelock point but cannot, by itself, prevent livelock” (Mogul & Ramakrishnan 1996). It moves the knee right; it does not change the shape of the falling curve past the knee.

The paper’s prescription is the blueprint Linux later adopted: a hybrid of interrupts and polling. “Since a purely interrupt-driven system leads to livelock, and a purely polling system adds unnecessary latency, we employ a hybrid design, in which the system polls only when triggered by an interrupt, and interrupts happen only while polling is suspended. During low loads, packet arrivals are unpredictable and we use interrupts to avoid latency. During high loads, we know that packets are arriving at or near the system’s saturation rate, so we use polling to ensure progress and fairness, and only re-enable interrupts when no more work is pending” (Mogul & Ramakrishnan 1996). And the rule about when to drop: “Once the system has invested enough work in an incoming packet to the point where it is about to be queued, it makes more sense to process that packet to completion than to drop it” — drop early (at the card), not late.

Linux Mitigation 1: NAPI — the Canonical Fix

NAPI (originally “New API,” now just the name) is Linux’s direct implementation of Mogul & Ramakrishnan’s hybrid. Under light load a NIC interrupts per packet (or per small batch) for low latency. The moment an interrupt fires, the driver does not process the packet in the hard-IRQ handler; it calls napi_schedule() to schedule a NAPI poll and — the key step — masks the device’s receive interrupt. The kernel documentation is explicit: “Drivers should keep the interrupts masked after scheduling the NAPI instance” (NAPI docs, v6.12). With interrupts masked, the device cannot interrupt again; instead the kernel polls it — repeatedly calling the driver’s poll() method (in NET_RX softirq context) to pull packets in batches.

The poll() method takes a budget: “drivers can process completions for any number of Tx packets but should only process up to budget number of Rx packets” and “returns the amount of work done. If the driver still has outstanding work to do (e.g. budget was exhausted) the poll method should return exactly budget” (NAPI docs, v6.12). The budget caps how much one poll can do, guaranteeing the CPU returns to the scheduler. Only when traffic drains below budget — the device is no longer flooded — does the driver call napi_complete_done(), which re-arms the hardware interrupt. So under flood the device is effectively polled (zero interrupts, bounded work per round); under light load it is interrupt-driven (low latency). This is exactly “interrupts only to initiate polling… re-enable interrupts when no more work is pending.” NAPI is the reason a modern Linux box does not livelock at 10/40/100 GbE. See NAPI and Polled Receive for the full state machine.

Linux Mitigation 2: Hardware Interrupt Coalescing / Moderation

Below NAPI, the NIC hardware itself can coalesce (a.k.a. moderate) interrupts: instead of one interrupt per frame, the card waits until either N frames have arrived (rx-frames) or a T-microsecond timer expires (rx-usecs), then raises a single interrupt for the batch. Tunable via ethtool -C <iface> rx-usecs N rx-frames M (and adaptive variants adaptive-rx on). This is the “batching” of the paper — it raises the MLFRR and cuts per-interrupt overhead, and it composes with NAPI (fewer interrupts to start polls). On its own it only shifts the knee; combined with NAPI it is genuinely effective because NAPI converts the saved interrupts into bounded polling rather than letting more interrupts in.

Uncertain

Verify: that rx-usecs/rx-frames/adaptive-rx remain the exact ethtool -C parameter names and behave identically on current drivers as of v6.12. Reason: ethtool coalescing parameters are driver-dependent and not all NICs implement every knob; the names are stable and well-established but were not re-verified against the v6.12 ethtool man page or a specific driver in this task. To resolve: check ethtool -C output on a target NIC and the driver’s ethtool_ops.set_coalesce. uncertain

Linux Mitigation 3: the Softirq Time Budget and ksoftirqd

Even with NAPI, softirq processing (NET_RX runs the polls) could in principle monopolize a CPU. Linux caps it. After a hard interrupt, pending softirqs run on the way out; the loop that runs them, handle_softirqs() in kernel/softirq.c, is bounded by both a time limit and a restart-count limit (v6.12):

#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
#define MAX_SOFTIRQ_RESTART 10

with the rationale spelled out in the source: “We restart softirq processing for at most MAX_SOFTIRQ_RESTART times, but break the loop if need_resched() is set or after 2 ms. The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in certain cases, such as stop_machine(), jiffies may cease to increment and so we need the MAX_SOFTIRQ_RESTART limit as well to make sure we eventually return from this method. These limits have been established via experimentation. The two things to balance is latency against fairness — we want to handle softirqs as soon as possible, but they should not be able to lock up the box” (kernel/softirq.c, v6.12).

The enforcing tail of handle_softirqs() is:

pending = local_softirq_pending();
if (pending) {
	if (time_before(jiffies, end) && !need_resched() &&
	    --max_restart)
		goto restart;
 
	wakeup_softirqd();
}

Walking it: after one pass over the pending softirqs, if more are still pending, the loop restarts only if (a) less than 2 ms has elapsed (time_before(jiffies, end), where end = jiffies + MAX_SOFTIRQ_TIME was set on entry), (b) no reschedule is pending (!need_resched()), and (c) the restart counter --max_restart (initialized to MAX_SOFTIRQ_RESTART = 10) has not hit zero. If any condition fails, it stops re-looping in interrupt context and calls wakeup_softirqd() — handing the remaining softirq work off to the per-CPU ksoftirqd kernel thread. That is the pivotal move: ksoftirqd is a schedulable, normal-priority task, so once softirq work is pushed there, it competes fairly with other tasks under the scheduler instead of running at unbounded interrupt-time priority. A CPU pegged in ksoftirqd (visible in top) is the modern symptom of an overloaded softirq path — but because it is a thread, the application still gets cycles, and the machine does not livelock. See ksoftirqd and Softirq Load and Softirqs and the Softirq Vector.

(Implementation note for accuracy: in v6.12 the function long known as __do_softirq is named handle_softirqs(bool ksirqd); the budget logic above is unchanged from the historical __do_softirq.)

Linux Mitigation 4: Spurious-Interrupt Disabling (a Wedged Line)

A different storm is a stuck line: a device (or a buggy shared-line peer) asserts an interrupt that no registered handler claims, over and over — every handler returns IRQ_NONE. Left alone this is a hard hang. The generic IRQ layer’s note_interrupt() (kernel/irq/spurious.c) watches for it. It counts: desc->irq_count increments every interrupt and desc->irqs_unhandled increments whenever the handler chain returns IRQ_NONE. When irq_count reaches 100,000, it checks the unhandled ratio, and if essentially all of them were unhandled it shuts the line down:

if (unlikely(desc->irqs_unhandled > 99900)) {
	/*
	 * The interrupt is stuck
	 */
	__report_bad_irq(desc, action_ret);
	/*
	 * Now kill the IRQ
	 */
	printk(KERN_EMERG "Disabling IRQ #%d\n", irq);
	desc->istate |= IRQS_SPURIOUS_DISABLED;
	desc->depth++;
	irq_disable(desc);

So if 99,900 of the last 100,000 interrupts on a line went unhandled, the kernel declares it stuck, prints the famous Disabling IRQ #N message to the log, sets IRQS_SPURIOUS_DISABLED, and masks the line with irq_disable(). A recovery poll timer is then started to periodically re-test whether the device has settled. This trades the affected device’s functionality for the survival of the whole machine — a deliberate “amputate the limb” policy. The Disabling IRQ #N line in dmesg is the canonical fingerprint of a screaming/shared-line interrupt problem; see Shared Interrupt Lines for how a misbehaving device on a shared line triggers it.

Uncertain

Verify: the exact 99900/100000 constants and the Disabling IRQ #%d string in v6.12 kernel/irq/spurious.c, plus the precise recovery-poll-timer behaviour. Reason: the code block and message were obtained via a fetch summary of the v6.12 blob and not every surrounding line (e.g. the irq_count == 100000 reset and try_misrouted_irq path) was quoted verbatim. To resolve: read note_interrupt() and __report_bad_irq() at tag v6.12 directly. uncertain

Linux Mitigation 5: Threaded IRQs for Priority Control

The deepest structural fix mirrors the paper’s “do almost nothing at high IPL” approach: move handler work out of hard-IRQ context into a kernel thread, where the scheduler — not the interrupt hardware — decides priority. With request_threaded_irq(), the hard handler does the bare minimum (acknowledge the device, mask it, return IRQ_WAKE_THREAD) and the real work runs in a dedicated kthread. The kernel-doc explains: “@handler is still called in hard interrupt context and has to check whether the interrupt originates from the device. If yes it needs to disable the interrupt on the device and return IRQ_WAKE_THREAD which will wake up the handler thread and run @thread_fn” (kernel/irq/manage.c, v6.12). The thread runs at FIFO priority (sched_set_fifo(current) in irq_thread()), so it can be prioritized against other work — a latency-critical IRQ thread can be pinned and boosted, a noisy one deprioritized — which a hard-IRQ handler can never be. IRQF_ONESHOT keeps the line masked until the thread finishes (“Run thread_fn with interrupt line masked”), preventing a re-fire storm while the thread runs.

This is the design PREEMPT_RT generalizes: with forced threading every handler that can be threaded is, via irq_forced_thread_fn (“if (force_irqthreads() && test_bit(IRQTF_FORCED_THREAD, …)) handler_fn = irq_forced_thread_fn”), turning interrupt processing into ordinary, preemptible, schedulable work. PREEMPT_RT was mainlined in 6.12, making threaded-by-default a first-class configuration. See Threaded Interrupt Handlers. The trade-off is latency: a scheduling round-trip per interrupt is slower than running inline, so threaded IRQs buy priority control and livelock immunity at the cost of a wakeup.

Failure Modes, Symptoms, and Diagnosis

The signature of an interrupt storm is a CPU at or near 100% system (not user) time, concentrated in %irq and %softirq columns of mpstat -P ALL 1 (or %si/%hi in top), while application throughput is flat or falling. cat /proc/interrupts shows one line’s count climbing at an extreme rate. If it is a wedged line, dmesg carries Disabling IRQ #N; if it is a packet flood, the climbing count is the NIC’s IRQ and ksoftirqd/<cpu> is at the top of top. /proc/softirqs shows NET_RX dominating. A useful confirmation that NAPI is the right lever: dropping rx-usecs/rx-frames to 0 makes it worse (more interrupts), raising coalescing makes it better — the opposite of a normal latency problem.

Mogul & Ramakrishnan’s own measurements showed the dramatic difference: their unmodified system’s forwarding throughput peaked and then fell to zero as input rate climbed, while the polling/quota-modified kernel held a flat plateau well past the saturation point (Mogul & Ramakrishnan 1996). That flat-versus-falling distinction is exactly what a healthy NAPI-driven Linux box exhibits today: offered load can vastly exceed capacity, and goodput simply plateaus (excess dropped at the NIC ring) instead of collapsing.

A subtle modern failure is softirq starvation of one CPU when all NIC interrupts land on a single core (no affinity spreading): that core livelocks-in-miniature (its ksoftirqd saturates) while others idle. The fix is not more budget but spreading — Receive-Side Scaling (multiple hardware queues, each with its own NAPI instance and IRQ) plus affinity. This is why high-throughput tuning pairs NAPI with multi-queue NICs and careful IRQ affinity; see IRQ Affinity and irqbalance.

Alternatives and When to Choose Them

  • Pure polling (busy-poll / SO_BUSY_POLL, DPDK, XDP) — for extreme rates, abandon interrupts entirely and spin a dedicated core polling the NIC. This is the “almost everything at high level, no interrupts” extreme; it gives the lowest latency and highest throughput but burns a whole core even when idle — the latency cost the paper warned of, paid as power. Right for dedicated packet-processing appliances, wrong for general-purpose servers. XDP/AF_XDP and DPDK live here.
  • Interrupt coalescing alone — cheap, no code; raises the MLFRR but does not change the falling-curve shape, so insufficient by itself for true overload. Use as a complement to NAPI, not a substitute.
  • Threaded IRQs / PREEMPT_RT — when bounded latency for other work matters more than raw throughput (audio, industrial control, RT systems). Trades per-interrupt latency for schedulability.
  • NAPI — the default and correct choice for essentially all network drivers; it is the production answer.

The honest summary: NAPI plus multi-queue affinity plus modest coalescing is what keeps real servers alive under flood; the other mechanisms are either the safety net (softirq budget, spurious disabling) or the specialist’s tool (pure polling, RT threading).

See Also

Graph View

Backlinks