vault

Non-Maskable Interrupts and the NMI Watchdog

16 min read

Non-Maskable Interrupts and the NMI Watchdog

A Non-Maskable Interrupt (NMI) is an interrupt that ordinary interrupt-disabling cannot block. local_irq_disable() / the x86 CLI instruction clears the interrupt-enable flag and stops maskable device interrupts — but the NMI ignores that flag entirely. That property is exactly what makes the NMI useful: it can reach a CPU that is spinning with interrupts disabled, which is precisely the situation an ordinary interrupt-driven watchdog could never observe. Linux uses NMIs for hardware-error reporting (machine checks), the NMI hardlockup watchdog, perf PMU profiling (so a profiler can sample even IRQ-disabled kernel code), Magic SysRq over NMI, kgdb, and the crash/panic IPI that one CPU sends to freeze the others. The price of being unmaskable is a vicious corner case — the nested-NMI problem — and an extremely restricted execution context: an NMI handler cannot sleep, cannot take normal locks, and must avoid anything that could fault in unbounded ways. This note covers all of it against the v6.12 source.

Mental Model

A regular maskable interrupt is a polite signal: the CPU honours a “do not disturb” sign (IF, the interrupt-enable flag), so a critical section can run local_irq_disable() and be sure no device handler will preempt it. The NMI is the fire alarm: it does not check the do-not-disturb sign, because the events it carries — a memory machine-check, a wedged CPU, a profiler tick that must not be biased by IRQ-disabled regions — are too important to be silenceable by ordinary code. The catch is that because nothing in software can mask it, the NMI handler runs in a context with almost no guarantees: it can land mid-way through any code, including code that holds locks the NMI handler would want, or code that was itself mid-NMI.

flowchart TB
  CRIT["CPU in critical section<br/>local_irq_disable() / CLI set"] -->|"maskable IRQ blocked"| BLK["Device IRQ: pending,<br/>NOT delivered"]
  CRIT -->|"NMI ignores IF"| NMI["NMI delivered anyway"]
  NMI --> ENTRY["exc_nmi() on the NMI IST stack"]
  ENTRY --> LATCH{"software latch:<br/>nmi_state == NOT_RUNNING?"}
  LATCH -->|"no (already running)"| MARK["set state = LATCHED,<br/>return; rerun later"]
  LATCH -->|"yes"| RUN["state = EXECUTING<br/>default_do_nmi()"]
  RUN --> DISP["dispatch handler chain<br/>NMI_LOCAL: perf, watchdog,<br/>then SERR / IO_CHECK / unknown"]

How an NMI reaches a CPU that has masked ordinary interrupts, and the software latch guarding re-entry. What it shows: while local_irq_disable() blocks the device IRQ, the NMI is delivered regardless; it enters exc_nmi() and consults a per-CPU software state machine before running, because the hardware’s own NMI masking is unreliable across the handler’s lifetime. The insight to take: the NMI’s super-power (unmaskability) is also its curse — the kernel has to rebuild, in software, the re-entrancy protection that masking normally provides for free.

What “Non-Maskable” Actually Means

On x86, the RFLAGS.IF bit gates maskable interrupts: CLI clears it (interrupts off), STI sets it (interrupts on). The kernel wrappers local_irq_disable()/local_irq_save() manipulate IF. The NMI is delivered on a separate signal path and is not gated by IF at all — see Disabling and Masking Interrupts for the maskable side. The hardware does provide a limited form of NMI masking: once an NMI is taken, the CPU blocks further NMIs until the next IRET instruction. This auto-masking is the root of the nested-NMI problem below, because IRET is also how the kernel returns from other exceptions, so an NMI handler that takes a page fault or breakpoint inadvertently re-opens the NMI window when that nested exception’s IRET executes.

Because of unmaskability, the NMI is the kernel’s tool of last resort for reaching a CPU that has stopped responding to normal interrupts. If a CPU is spinning forever in a kernel critical section with IF cleared, no timer interrupt, no IPI on a normal vector, nothing maskable will ever run there. An NMI still will. That is the whole basis of the hardlockup watchdog.

Uses of the NMI

The kernel registers NMI handlers on a small set of sources defined in arch/x86/include/asm/nmi.h:

enum {
	NMI_LOCAL=0,
	NMI_UNKNOWN,
	NMI_SERR,
	NMI_IO_CHECK,
	NMI_MAX
};

NMI_LOCAL is the CPU-local source (perf counters, the watchdog, KGDB, backtrace requests); NMI_UNKNOWN is the catch-all for NMIs nobody claimed; NMI_SERR and NMI_IO_CHECK are the legacy chipset NMI reasons (system error / I/O-channel check) read from port 0x61 (nmi.h, v6.12). The real-world uses:

  • Hardware error / machine check. Uncorrectable memory or bus errors are signalled to the CPU; the chipset/firmware can raise an NMI (the SERR path) that the kernel surfaces as a fatal report. (The CPU’s dedicated #MC machine-check exception is a separate vector with its own IST stack — see IRQ Stacks and Per-CPU Interrupt Handling — but external hardware errors can also arrive as NMIs.)
  • The NMI hardlockup watchdog (the main subject below): a periodic NMI generated from a PMU counter detects a CPU stuck with interrupts disabled.
  • perf PMU profiling. The performance-monitoring unit (PMU) is programmed to overflow a hardware counter after N cycles/instructions; the overflow is delivered as an NMI. Sampling via NMI is deliberate: it lets perf record profile code that runs with interrupts disabled (interrupt handlers, spinlock critical sections, the scheduler’s innards) — a maskable sampling interrupt would be blind to exactly those hot regions and would bias the profile. This is registered on NMI_LOCAL.
  • Magic SysRq and crash backtraces. A SysRq like “dump all CPU backtraces” sends an NMI to every CPU so each prints its stack even if wedged. On x86 this uses __apic_send_IPI_mask(mask, NMI_VECTOR) and a handler registered on NMI_LOCAL named "arch_bt" (hw_nmi.c, v6.12).
  • kgdb. The kernel debugger uses NMIs to round up CPUs into the debug stub.
  • The crash / panic IPI-NMI. When one CPU panics, it must stop the others before dumping (kdump). It sends an NMI IPI; default_do_nmi() even special-cases this — while spinning to acquire nmi_reason_lock, it calls run_crash_ipi_callback(regs) so a panicking CPU’s request is honoured promptly. See Inter-Processor Interrupts for the IPI delivery mechanism.

Handlers are registered with register_nmi_handler(type, fn, flags, name), which builds an nmiaction and links it onto the per-source chain via __register_nmi_handler (nmi.c, v6.12). Because NMIs are edge-triggered and only one can be latched at a time, nmi_handle() walks the entire handler list for a source on each NMI, summing how many claimed it — so that if two events coalesced into one delivered NMI, both handlers still run:

list_for_each_entry_rcu(a, &desc->head, list) {
	...
	thishandled = a->handler(type, regs);
	handled += thishandled;
	...
	nmi_check_duration(a, delta);
}

nmi_check_duration() warns ("NMI handler ... took too long to run") if a handler exceeds a threshold — a hard requirement, because a slow NMI handler stalls everything on that CPU with no way to be preempted (nmi.c, v6.12).

The Nested-NMI Problem and the Software Latch

The hardware masks NMIs only until IRET. But an NMI handler can legitimately trigger events that themselves execute IRET: a page fault (if it touches a not-present page), a breakpoint (#BP/#DB, e.g. if kprobes or the debugger has instrumented something the handler calls), or, in SEV-ES guests, a #VC. When that nested exception returns via IRET, the CPU re-enables NMIs — even though the outer NMI is still running. A second NMI can now preempt the first, and since both share the same per-CPU NMI IST stack, the second would clobber the first’s frame. The v6.12 comment states it plainly:

 * NMIs can page fault or hit breakpoints which will cause it to lose
 * its NMI context with the CPU when the breakpoint or page fault does an IRET.
 *
 * As a result, NMIs can nest if NMIs get unmasked due an IRET during
 * NMI processing.

Linux solves this with a per-CPU software state machine — a software latch that simulates the masking the hardware dropped:

enum nmi_states {
	NMI_NOT_RUNNING = 0,
	NMI_EXECUTING,
	NMI_LATCHED,
};
static DEFINE_PER_CPU(enum nmi_states, nmi_state);

The logic in exc_nmi() (nmi.c, v6.12):

DEFINE_IDTENTRY_RAW(exc_nmi)
{
	...
	if (this_cpu_read(nmi_state) != NMI_NOT_RUNNING) {
		this_cpu_write(nmi_state, NMI_LATCHED);
		return;
	}
	this_cpu_write(nmi_state, NMI_EXECUTING);
	this_cpu_write(nmi_cr2, read_cr2());
 
nmi_restart:
	...
	default_do_nmi(regs);
	...
	if (this_cpu_dec_return(nmi_state))
		goto nmi_restart;
}

Walk the states:

  1. NOT_RUNNING → EXECUTING. The first NMI sees nmi_state == NMI_NOT_RUNNING, flips it to NMI_EXECUTING, and proceeds to default_do_nmi().
  2. A nested NMI arrives. If a second NMI preempts while state is EXECUTING (or already LATCHED), the very first check this_cpu_read(nmi_state) != NMI_NOT_RUNNING is true, so it sets the state to NMI_LATCHED and returns immediately without running any handler. The nested NMI is recorded, not processed inline — which is what protects the outer NMI’s stack frame. The comment notes “the latch is binary”: multiple nested NMIs collapse into a single re-run.
  3. The outer NMI finishes and re-runs if latched. At the end, this_cpu_dec_return(nmi_state) decrements the state. If it was NMI_EXECUTING (value 1) it becomes NMI_NOT_RUNNING (0) and the handler exits. If a nested NMI had bumped it to NMI_LATCHED (value 2), the decrement yields NMI_EXECUTING (1, non-zero), so goto nmi_restart runs default_do_nmi() again to service the NMI that was deferred.

There is a second subtlety the code handles: the saved fault address register. read_cr2() is stashed in nmi_cr2 on entry and restored on exit — because if the NMI took a page fault, the nested fault would have overwritten CR2, corrupting the value the interrupted (pre-NMI) page-fault handler was about to read. The save/restore must straddle the latch-clearing to avoid a race with yet another nested NMI (nmi.c, v6.12). The lesson: an NMI’s IST stack (from IRQ Stacks and Per-CPU Interrupt Handling) gives it a valid stack, but not protection against a second NMI overwriting that stack — the software latch is what closes that gap.

The NMI Watchdog — Hardlockup vs Softlockup

Linux’s lockup detector is two cooperating detectors. They catch different failures, by different mechanisms, on different timescales.

Hardlockup detector — the NMI half

A hard lockup is a CPU stuck in kernel mode with interrupts disabled — it is not even taking timer interrupts, so nothing maskable can notice. Only an NMI can. The perf-based hardlockup detector (kernel/watchdog_perf.c) creates a pinned per-CPU perf_event that counts CPU cycles and overflows roughly once per watchdog_thresh seconds:

static struct perf_event_attr wd_hw_attr = {
	.type		= PERF_TYPE_HARDWARE,
	.config		= PERF_COUNT_HW_CPU_CYCLES,
	.size		= sizeof(struct perf_event_attr),
	.pinned		= 1,
	.disabled	= 1,
};

The sample period is hw_nmi_get_sample_period(watchdog_thresh) = cpu_khz * 1000 * watchdog_thresh cycles — i.e. about watchdog_thresh seconds of CPU cycles (hw_nmi.c, v6.12). When the counter overflows, the PMU raises an NMI, whose handler is watchdog_overflow_callback() → watchdog_hardlockup_check(). That function does not directly time anything; it checks whether the CPU’s hrtimer interrupt counter advanced since the last NMI:

static bool is_hardlockup(unsigned int cpu)
{
	int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu));
	if (per_cpu(hrtimer_interrupts_saved, cpu) == hrint)
		return true;          /* counter didn't move -> stuck */
	per_cpu(hrtimer_interrupts_saved, cpu) = hrint;
	return false;
}

The watchdog hrtimer bumps hrtimer_interrupts every time it fires (watchdog_hardlockup_kick()). If, by the time the NMI samples, that counter has not moved, the CPU has not been processing its periodic timer — it is hard-locked. The handler then prints "Watchdog detected hard LOCKUP on cpu %d" and, if hardlockup_panic is set, calls nmi_panic() (watchdog.c, v6.12). The official v6.12 documentation states it the same way: “An NMI perf event is generated every ‘watchdog_thresh’ seconds… If any CPU in the system does not receive any hrtimer interrupt during that time the ‘hardlockup detector’… will generate a kernel warning or call panic” (lockup-watchdogs, v6.12). Enabling it logs "Enabled. Permanently consumes one hw-PMU counter." — a real cost, since that PMU counter is then unavailable to perf.

There is also a buddy hardlockup detector (watchdog_buddy_check_hardlockup) for systems without a usable PMU NMI: each CPU checks its neighbour’s hrtimer counter from its own timer interrupt, trading the NMI for cross-CPU checking. It detects hard lockups except the rare case where the checking CPU itself is also locked.

Softlockup detector — the hrtimer half

A soft lockup is a CPU stuck in kernel mode without disabling interrupts — it still takes interrupts, but a task monopolizes the CPU so completely that nothing else (including the scheduler’s normal work) makes progress for too long. Because interrupts still arrive, you do not need an NMI; a high-resolution timer suffices. Each CPU arms a per-CPU hrtimer (watchdog_timer_fn, started HRTIMER_MODE_REL_HARD) that fires every sample_period. On each tick it kicks a short job onto the CPU via the CPU stopper (stop_one_cpu_nowait(... softlockup_fn ...)); softlockup_fn() just updates a per-CPU timestamp (update_touch_ts()). The detector fires when that timestamp has not advanced for too long:

static int get_softlockup_thresh(void)
{
	return watchdog_thresh * 2;
}
...
if (time_after(now, period_ts + get_softlockup_thresh()))
	return now - touch_ts;   /* it's a soft lockup */

So the soft-lockup threshold is 2 * watchdog_thresh seconds (watchdog.c, v6.12). When tripped, watchdog_timer_fn() prints "BUG: soft lockup - CPU#%d stuck for %us!" and, if softlockup_panic is set, panics. The sample_period is get_softlockup_thresh() * (NSEC_PER_SEC / NUM_SAMPLE_PERIODS) with NUM_SAMPLE_PERIODS == 5, i.e. the timer fires five times within the soft-lockup window (the code comment notes “4 seconds by default” for the default 10 s threshold: 20 s / 5 = 4 s).

Uncertain

Verify: this note describes the v6.12 softlockup detector as using the CPU stopper (stop_one_cpu_nowait + cpu_stop_work + softlockup_fn) kicked by an hrtimer, not a persistent per-CPU watchdog/N smpboot kthread. The watchdog/N kernel thread was the older implementation; in v6.12 source kernel/watchdog.c there is no smpboot_register_percpu_thread/watchdog/ thread name — it is the stop-machine job described above. Reason: the v6.12 source was read directly and shows the stopper mechanism, but the exact release in which the watchdog/N thread was removed in favour of the stopper was not pinned to a specific commit/version in this task. To resolve: git log --oneline -- kernel/watchdog.c for the smpboot-thread → cpu-stopper conversion and note the version. The mechanism as described matches v6.12. uncertain

Configuration

The knobs (all sysctl under kernel.):

watchdog_thresh   = 10          # seconds; hardlockup ~ 10s, softlockup = 2*thresh = 20s
nmi_watchdog      = 1           # enable/disable the NMI hardlockup detector
softlockup_panic  = 0/1         # panic instead of just warning on soft lockup
hardlockup_panic  = 0/1         # panic instead of just warning on hard lockup
watchdog_cpumask              # which CPUs run the detector

watchdog_thresh = 0 disables the lockup detector entirely. Setting watchdog_thresh via /proc/sys/kernel/watchdog_thresh re-derives sample_period and reprograms both detectors (watchdog.c, v6.12). The default watchdog_thresh of 10 is set in the source: int __read_mostly watchdog_thresh = 10;.

Failure Modes and Diagnosis

  • BUG: soft lockup - CPU#N stuck for 22s! — a task spun in the kernel (or a too-long busy loop) for over 2*watchdog_thresh. The accompanying stack trace points at the looping code. Common causes: a non-terminating loop holding no lock, a too-large bounded loop, or a livelock. The CPU is not dead (interrupts still work), it is just not yielding.
  • Watchdog detected hard LOCKUP on cpu N — the CPU stopped taking timer interrupts, almost always a deadlock or infinite loop with interrupts disabled (e.g. a spinlock taken twice, or a spin_lock without _irqsave that then self-deadlocks on a re-entrant interrupt). Only the NMI could see it. If it escalates to panic, you get a kdump for post-mortem.
  • NMI handler (...) took too long to run — some registered NMI handler is too slow. Because NMIs cannot be preempted and can stall the CPU, this is a correctness smell, not just a perf nit.
  • Uhhuh. NMI received for unknown reason ... — an NMI that no handler claimed (unknown_nmi_error). Often spurious hardware/firmware NMIs; with unknown_nmi_panic set it becomes fatal, which is sometimes deliberately enabled in production to force a crash dump on otherwise-silent hardware faults.
  • PMU contention. Because the hardlockup detector permanently consumes one hardware PMU counter, profiling tools have one fewer counter. On counter-starved CPUs this can make perf stat fall back to multiplexing. Disabling nmi_watchdog frees the counter — a real trade-off in performance-engineering setups.

Restricted Context — What an NMI Handler May Not Do

An NMI handler runs in the most restricted context in the kernel. It cannot sleep (there is no task context to block — the same fundamental reason covered in Interrupt Context and Why It Cannot Sleep, but stricter). It must avoid normal locks: a spinlock it tries to take might already be held by the very code the NMI interrupted, on the same CPU, deadlocking instantly — so NMI code uses raw_spin_trylock patterns (as default_do_nmi does for nmi_reason_lock) or lock-free per-CPU state. It must avoid anything that can fault unboundedly. It cannot rely on RCU in the usual way (special irqentry_nmi_enter/exit handling is needed). And it must be fast — the nmi_check_duration() watchdog-on-the-watchdog exists because a slow NMI handler stalls the CPU with no recourse. These constraints are why NMI handlers are deliberately tiny: sample a counter, set a flag, print a line, request a backtrace — never real work.

Alternatives and When to Choose Them

For detecting stuck CPUs, the NMI hardlockup detector is the only thing that works against IRQ-disabled lockups; there is no maskable-interrupt alternative that can see them. The buddy detector is the fallback when no PMU NMI is available (some virtualized or constrained platforms) — strictly weaker (it cannot catch a lockup where the checker is also locked) but needs no PMU. For userspace hangs and watchdog-timer-style “is the box alive” monitoring, the kernel’s separate hardware watchdog subsystem (/dev/watchdog, watchdogd) is the right tool — it resets the machine if userspace stops petting it, a different layer from the per-CPU lockup detectors here. For profiling, NMI-based PMU sampling is preferred over timer-based sampling precisely because it is unmaskable and so unbiased across IRQ-disabled regions; timer-based perf sampling is the fallback when PMU NMIs are unavailable, at the cost of blindness to the hottest atomic paths.

Production Notes

In latency-sensitive deployments the NMI watchdog is frequently disabled (nmi_watchdog=0 on the kernel command line, or kernel.nmi_watchdog=0 sysctl), for two reasons: it permanently holds a PMU counter that the team wants for profiling, and the periodic NMI itself is a small, hard-to-mask source of jitter on isolated CPUs. This pairs with the CPU-isolation work in CPU isolation — a nohz_full/isolcpus core is configured to be as interruption-free as possible, and an unmaskable periodic NMI fights that goal. The trade-off is real: disabling the watchdog means a genuine hard lockup on those cores goes silently undetected until something else notices.

Conversely, in fleet reliability settings the watchdog is often configured to panic (hardlockup_panic=1, softlockup_panic=1) so that a wedged machine produces a kdump crash image and reboots, rather than sitting dark and unresponsive. The hard-lockup-via-NMI path is the only way to get a useful crash dump out of a CPU that has disabled interrupts and stopped responding — which is exactly the unmaskability property that started this note.

See Also

Graph View

Backlinks