IRQ Stacks and Per-CPU Interrupt Handling

When a hardware interrupt fires on x86-64 Linux, the handler does not run on the interrupted task’s kernel stack. The kernel switches to a dedicated, per-CPU interrupt stack (the hardirq stack) before invoking the device handler, then switches back on return. The reason is brutal arithmetic: a thread’s kernel stack is only 16 KiB (THREAD_SIZE, four 4 KiB pages on x86-64), and an interrupt can land on top of an already-deep syscall path. Without a separate stack, a deep call chain plus a nested interrupt could silently run off the bottom of the 16 KiB and corrupt whatever lies below — a class of bug with no clean failure signature. A separate hardirq stack bounds interrupt-induced depth to its own 16 KiB region. On top of that, x86-64 maintains a small set of always-valid special stacks — the Interrupt Stack Table (IST) — that the CPU switches to in hardware for events like the Non-Maskable Interrupt (NMI), double-fault (#DF), machine-check (#MC), and debug (#DB), so those handlers work even when the regular stack pointer is garbage. This note traces all of it against the v6.12 source.

Mental Model

Think of each CPU as owning a small fleet of stacks, not one. The “normal” stack is whichever task is currently running on the CPU — its 16 KiB kernel stack, used for syscalls, page faults from the task, and ordinary kernel work. Layered beside it are interrupt-only stacks that exist independently of any task: one hardirq stack for device-interrupt handlers, and a handful of IST stacks that the hardware itself selects for a fixed set of catastrophic or re-entrancy-sensitive events. The key property of all of these is that they are per-CPU: every logical CPU gets its own private copy, because two CPUs can be servicing interrupts simultaneously and they must never share a stack.

flowchart TB
  subgraph CPU0["Per-CPU stacks for CPU 0"]
    TASK["Task kernel stack<br/>16 KiB (THREAD_SIZE)<br/>syscalls, faults from task"]
    HIRQ["Hardirq stack<br/>16 KiB (IRQ_STACK_SIZE)<br/>device IRQ + softirq handlers"]
    IST_NMI["IST: NMI stack (8 KiB)"]
    IST_DF["IST: #DF double-fault (8 KiB)"]
    IST_MCE["IST: #MC machine-check (8 KiB)"]
    IST_DB["IST: #DB debug (8 KiB)"]
  end
  SYS["int 0x80 / syscall / page fault<br/>from the running task"] --> TASK
  DEV["Device IRQ vector<br/>(NIC, disk, timer)"] -->|"software switch:<br/>save rsp, set rsp=hardirq TOS"| HIRQ
  NMI["NMI"] -->|"hardware switches via IDT IST=1"| IST_NMI
  DF["#DF"] -->|"hardware, IST=0"| IST_DF

The per-CPU stack fleet on x86-64. What it shows: ordinary task work uses the task’s own 16 KiB kernel stack; a device interrupt triggers a software stack switch onto the per-CPU hardirq stack; and a small set of special events cause the hardware to switch to a fixed IST stack chosen by an index in the Interrupt Descriptor Table (IDT) entry. The insight to take: “which stack am I on?” depends on how the kernel was entered. Device IRQs and the catastrophic exceptions deliberately leave the task stack untouched, which is what makes deep-fault and corrupt-stack recovery possible.

Why a Separate Stack at All — the 16 KiB Budget

On x86-64 the kernel stack size is fixed by THREAD_SIZE, defined in arch/x86/include/asm/page_64_types.h:

#define THREAD_SIZE_ORDER	(2 + KASAN_STACK_ORDER)
#define THREAD_SIZE  (PAGE_SIZE << THREAD_SIZE_ORDER)

With the page size of 4 KiB and KASAN_STACK_ORDER equal to 0 in a normal (non-KASAN) build, THREAD_SIZE_ORDER is 2, so THREAD_SIZE = 4096 << 2 = 16384 bytes — 16 KiB (page_64_types.h, v6.12). Note the KASAN_STACK_ORDER: when the Kernel Address Sanitizer (KASAN) is compiled in, this order becomes 1 and the stack doubles to 32 KiB, because instrumented code consumes more stack. The default kernel ships 16 KiB.

That 16 KiB is the entire budget for everything the kernel does on behalf of a task: the syscall entry frame, every function in a deep call chain (think a write going through the VFS layer, a filesystem, the block layer, a device driver), and any local variables along the way. The kernel has no automatic stack growth — there is no guard-fault-and-extend mechanism the way userspace stacks grow. If you run past the bottom of the 16 KiB, you start scribbling on adjacent memory.

Now consider what an interrupt adds. Interrupts are asynchronous: a network card can raise its IRQ at literally any instruction boundary, including deep inside that VFS→filesystem→block call chain. If the interrupt handler ran on the same stack, its frames would stack on top of the already-consumed depth. A moderately deep syscall plus a moderately deep interrupt handler — and interrupts can themselves nest if a higher-priority vector arrives — could together exceed 16 KiB. The failure is insidious: no fault, just silent corruption of the thread_info/scheduling data that historically lived at the bottom of the stack, or of a neighbouring page.

The fix is to give interrupts their own stack. When a device interrupt is delivered while the CPU was running task code, the kernel switches to the per-CPU hardirq stack, so the interrupt’s depth is charged against a separate 16 KiB region rather than eating into the task’s remaining budget. The task’s stack depth and the interrupt’s stack depth are now independent; neither can overflow the other.

The Hardirq Stack — `DEFINE_PER_CPU` and `hardirq_stack_ptr`

The hardirq stack’s storage is declared in arch/x86/kernel/irq_64.c:

DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible;
DECLARE_INIT_PER_CPU(irq_stack_backing_store);

DEFINE_PER_CPU_PAGE_ALIGNED allocates one struct irq_stack per CPU, page-aligned — this is the actual backing memory (irq_64.c, v6.12). Its size is IRQ_STACK_SIZE, defined right next to THREAD_SIZE:

#define IRQ_STACK_ORDER (2 + KASAN_STACK_ORDER)
#define IRQ_STACK_SIZE (PAGE_SIZE << IRQ_STACK_ORDER)

So the hardirq stack is also 16 KiB — same order as the task stack. Each CPU has exactly one, which is fine because interrupts on a single CPU cannot truly run in parallel: the CPU is either running task code or one interrupt, and nested interrupts unwind LIFO on that same per-CPU stack.

The kernel does not chase the backing store directly in the hot path; it caches a pointer to the top of stack (TOS). On x86-64 the stack grows downward, so the usable top is the highest address. irq_init_percpu_irqstack() sets up the per-CPU pointer:

/* Store actual TOS to avoid adjustment in the hotpath */
per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;

pcpu_hot.hardirq_stack_ptr is the per-CPU TOS pointer, living in the pcpu_hot “hot data” structure so it is one cache-line fetch away during interrupt entry (irq_64.c, v6.12). The - 8 reserves a slot at the very top — that slot is where the entry macro stores the old stack pointer so the kernel can switch back and so stack unwinders can walk from the IRQ stack back to the interrupted stack.

The actual stack switch

The switch itself is a small inline-assembly macro, call_on_stack, in arch/x86/include/asm/irq_stack.h. Its four documented steps are exactly what you would draw on a whiteboard:

#define call_on_stack(stack, func, asm_call, argconstr...)		\
{									\
	register void *tos asm("r11");					\
	tos = ((void *)(stack));					\
	asm_inline volatile(						\
	"movq	rsp				\n"		\
		asm_call						\
	"popq	%%rsp					\n"		\
	...

Line by line: movq %rsp, (%tos) stores the current stack pointer into the reserved top slot of the IRQ stack (so unwinders can link back, and so the old rsp is recoverable). movq %tos, %rsp switches the live stack pointer to the top of the IRQ stack. Then asm_call invokes the C handler — which now runs entirely on the IRQ stack. Finally popq %rsp reloads the original stack pointer from that top slot, returning to the interrupted stack exactly where it left off (irq_stack.h, v6.12).

Switch only when needed — the `_cond` wrapper

Crucially, the kernel does not always switch. The call_on_irqstack_cond macro guards the switch:

if (user_mode(regs) || __this_cpu_read(pcpu_hot.hardirq_stack_inuse)) {
	irq_enter_rcu();
	func(c_args);
	irq_exit_rcu();
} else {
	__this_cpu_write(pcpu_hot.hardirq_stack_inuse, true);
	call_on_irqstack(func, asm_call, constr);
	__this_cpu_write(pcpu_hot.hardirq_stack_inuse, false);
}

There are two cases where it runs the handler without switching. First, if the interrupt came from user mode (user_mode(regs)): the CPU already switched from the tiny user-entry context onto the (empty) task kernel stack, so there is plenty of room and no reason to switch again. Second, if the hardirq stack is already in use (hardirq_stack_inuse is true) — i.e. a nested interrupt arrived while we were already on the IRQ stack — it just keeps running on the IRQ stack rather than re-entering it, since it is already the right stack. The per-CPU hardirq_stack_inuse flag is the bookkeeping that makes this safe; it is set before the switch and cleared after, with interrupts disabled across both writes (irq_stack.h, v6.12).

The Softirq Stack — Reused, Not Separate, on x86-64

A common misconception is that x86-64 has a distinct softirq stack. It does not. The kernel does have the concept of running softirqs on their own stack — gated by CONFIG_SOFTIRQ_ON_OWN_STACK, which x86 selects via HAVE_SOFTIRQ_ON_OWN_STACK in arch/x86/Kconfig — but on x86-64 the implementation reuses the same per-CPU hardirq stack:

#ifdef CONFIG_SOFTIRQ_ON_OWN_STACK
#define do_softirq_own_stack()						\
{									\
	__this_cpu_write(pcpu_hot.hardirq_stack_inuse, true);		\
	call_on_irqstack(__do_softirq, ASM_CALL_ARG0);			\
	__this_cpu_write(pcpu_hot.hardirq_stack_inuse, false);		\
}
#endif

do_softirq_own_stack() calls __do_softirq via call_on_irqstack — the same pcpu_hot.hardirq_stack_ptr (irq_stack.h, v6.12). And arch/x86/include/asm/softirq_stack.h for 64-bit simply includes irq_stack.h. This is safe because the comment in the code spells out the invariant: do_softirq_own_stack() is only called from task context when bottom halves are about to be re-enabled and softirqs are pending — “The interrupt stack cannot be in use here” — so reusing it cannot collide with an in-flight hardirq. The hardirq_stack_inuse flag is set anyway so that if an interrupt arrives during softirq processing, the _cond logic above sees the stack as busy and does not try to re-switch onto it. The takeaway: on x86-64, “the softirq stack” and “the hardirq stack” are the same 16 KiB region, used at non-overlapping times. (32-bit x86 historically had genuinely separate hardirq and softirq stacks; that is the asm-generic path the header falls back to for non-x86-64.) For the deferred-work side of softirqs see Softirqs and the Softirq Vector.

The Interrupt Stack Table (IST) — Hardware-Selected Always-Valid Stacks

The hardirq stack handles device interrupts, but those are delivered while the kernel is in a sane state — rsp points somewhere valid. A harder problem is an event that can fire when the stack pointer itself is corrupt, unmapped, or mid-switch. If such an event tried to push its entry frame onto whatever rsp happens to point at, it would fault while trying to handle a fault — an unrecoverable cascade. x86-64 solves this in hardware with the Interrupt Stack Table (IST): each entry in the Interrupt Descriptor Table (IDT) can name one of up to 7 IST slots, and when that vector is delivered the CPU unconditionally loads rsp from that slot before pushing anything, regardless of the current stack. The handler therefore always starts on a known-good stack.

Linux assigns IST indices in page_64_types.h:

/* The index for the tss.ist[] array. The hardware limit is 7 entries. */
#define	IST_INDEX_DF		0
#define	IST_INDEX_NMI		1
#define	IST_INDEX_DB		2
#define	IST_INDEX_MCE		3
#define	IST_INDEX_VC		4

So Linux uses dedicated IST stacks for (page_64_types.h, v6.12):

DF, double-fault (index 0) — raised when a fault occurs while delivering another fault. The textbook case is a kernel stack overflow: pushing the entry frame faults because the next page is unmapped (a guard page), and that fault-during-fault becomes a DF. The DF handler runs on its own IST stack precisely so it can report the overflow instead of triple-faulting (which would reset the machine).
NMI (index 1) — the non-maskable interrupt, which can fire at any instruction including ones where the stack is mid-manipulation.
DB, debug (index 2) — hardware breakpoints / single-step, which must work even while debugging the entry code itself.
MC, machine-check (index 3) — a hardware-detected error (bad RAM, bus error) that can occur at any moment and must be servable even from a wedged state.
VC, VMM Communication (index 4) — used by AMD SEV-ES encrypted guests; present when CONFIG_AMD_MEM_ENCRYPT is set.

The IST stacks live in the per-CPU CPU entry area (cea), laid out by ESTACKS_MEMBERS in arch/x86/include/asm/cpu_entry_area.h:

#define ESTACKS_MEMBERS(guardsize, optional_stack_size)		\
	char	DF_stack_guard[guardsize];			\
	char	DF_stack[EXCEPTION_STKSZ];			\
	char	NMI_stack_guard[guardsize];			\
	char	NMI_stack[EXCEPTION_STKSZ];			\
	char	DB_stack_guard[guardsize];			\
	char	DB_stack[EXCEPTION_STKSZ];			\
	char	MCE_stack_guard[guardsize];			\
	char	MCE_stack[EXCEPTION_STKSZ];			\
	...
struct cea_exception_stacks {
	ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ)
};

Each IST stack is EXCEPTION_STKSZ, which is PAGE_SIZE << (1 + KASAN_STACK_ORDER) = 8 KiB in a normal build — half the size of the hardirq stack, because IST handlers must be short. Critically, in the effective mapping (cea_exception_stacks) each stack is preceded by a PAGE_SIZE guard page (PAGE_SIZE is passed as guardsize) (cpu_entry_area.h, v6.12). If an IST handler overflows its 8 KiB, it hits the unmapped guard page and faults immediately and detectably, rather than corrupting the adjacent stack. The physical backing store (struct exception_stacks, ESTACKS_MEMBERS(0, …)) has no guard pages — the guard pages exist only in the virtually-mapped entry-area alias.

Uncertain

Verify: the precise hardware semantics that IST switching happens unconditionally on rsp regardless of the current privilege/stack, and the “up to 7 IST entries” limit, are properties of the Intel/AMD architecture (the IDT IST field), not of Linux. The Linux source comment states “The hardware limit is 7 entries,” which corroborates it, but this note did not fetch the Intel SDM / AMD APM directly. Reason: primary CPU-vendor manuals not consulted in this task. To resolve: cross-check Intel SDM Vol. 3, §6.14.5 (“Interrupt Stack Table”) and the IDT gate-descriptor format. uncertain

Stack-Overflow Detection — Guard Pages and `CONFIG_VMAP_STACK`

The IST guard pages above protect the special stacks. The ordinary task and hardirq stacks are protected by CONFIG_VMAP_STACK, which maps the stack out of vmalloc space with a guard page below it. For the hardirq stack, irq_64.c does this explicitly:

#ifdef CONFIG_VMAP_STACK
static int map_irq_stack(unsigned int cpu)
{
	char *stack = (char *)per_cpu_ptr(&irq_stack_backing_store, cpu);
	struct page *pages[IRQ_STACK_SIZE / PAGE_SIZE];
	...
	va = vmap(pages, IRQ_STACK_SIZE / PAGE_SIZE, VM_MAP, PAGE_KERNEL);
	if (!va)
		return -ENOMEM;
	per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
	return 0;
}
#else
static int map_irq_stack(unsigned int cpu)
{
	void *va = per_cpu_ptr(&irq_stack_backing_store, cpu);
	per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
	return 0;
}
#endif

With CONFIG_VMAP_STACK on, the backing pages are re-mapped into vmalloc address space via vmap(), where the surrounding vmalloc region naturally leaves an unmapped guard page — overrun the stack and you take a page fault on the guard page rather than corrupting the next allocation (irq_64.c, v6.12). The comment in the #else branch explains the exception: VMAP stacks are disabled “due to KASAN,” in which case the plain per-CPU backing store is used directly with no guard pages. The same vmap-with-guard mechanism backs task kernel stacks — see vmalloc and Virtually Contiguous Memory and The task_struct Process Descriptor for the task-stack side. The net effect: a stack overflow on a VMAP_STACK kernel produces a clean, diagnosable page fault (which on x86-64, if it overflows into the guard page during entry, escalates to the DF handler on its own IST stack) instead of silent memory corruption.

Failure Modes and How to Diagnose Them

Silent stack overflow without VMAP_STACK. On older or specially-configured kernels without CONFIG_VMAP_STACK, overrunning the 16 KiB stack corrupts adjacent memory with no immediate signal — you see a delayed, seemingly unrelated crash. The diagnosis is the smoking gun of a huge on-stack local (a multi-kilobyte array on the stack) or unbounded recursion in kernel code. The fix is structural: never put large buffers on the kernel stack; the compiler flag -Wframe-larger-than= and the kernel’s frame-size warnings exist to catch this.
Double-fault on overflow with VMAP_STACK. With guard pages, an overflow shows up as a #DF (double-fault) report — the entry-frame push hit the guard page. The oops names exc_double_fault and shows a stack trace bottoming out near the overflowing function. This is the good outcome: the bug is caught precisely.
“Which stack am I on?” confusion in unwinder output. Because device IRQs switch stacks, a stack trace taken inside an interrupt handler shows a transition from the IRQ stack back to the interrupted task’s stack (the unwinder follows the saved rsp in the top slot). If that link were broken — e.g. a hand-rolled asm path that forgot to store the old rsp — the trace would truncate at the IRQ-stack boundary. The call_on_stack macro’s first instruction exists precisely to keep this link intact.
IST re-entrancy hazards. IST stacks are not automatically nesting-safe: if the same IST vector fires twice before the first returns, the second would reuse the same stack and clobber the first’s frame. This is exactly why the NMI path needs the elaborate software re-entrancy handling described in Non-Maskable Interrupts and the NMI Watchdog — the IST gives it a valid stack, but not protection against a second NMI overwriting it.

Alternatives and Architecture Differences

The “separate IRQ stack” design is x86-64-specific in its details but the idea is common. On 32-bit x86, the kernel historically maintained genuinely separate per-CPU hardirq and softirq stacks (the asm-generic/softirq_stack.h path), because the smaller address space and stack budget made sharing riskier. On ARM64, interrupts also run on a per-CPU IRQ stack (irq_stack_ptr) for the same overflow-bounding reason, and the equivalent of guard-page overflow detection exists via VMAP_STACK there too. The IST mechanism specifically is an x86 architectural feature — there is no IST on ARM64; ARM64 instead uses separate exception levels and the SP_EL1/SP_EL0 split plus per-CPU stacks for its analogue of “always-valid” handler stacks.

A naive alternative — no separate interrupt stack, just run handlers on the current stack — is what very early/simple kernels do and what Linux did long ago. It is simpler but fragile: it couples interrupt depth to task depth and makes stack-size tuning a global guessing game. The per-CPU interrupt stack decouples the two and lets each be sized for its own worst case.

Production Notes

The 16 KiB kernel stack is a real, frequently-hit constraint in driver and filesystem development. Deeply layered I/O paths — for example, a stacked block setup (a filesystem on dm-crypt on LVM on a RAID on the device) — can approach the limit, which is one historical motivation for VMAP_STACK (so the overflow is at least caught) and for the kernel’s aggressive -Wframe-larger-than discipline. The interrupt stack is what keeps an interrupt arriving mid-way down such a chain from being the straw that breaks it.

Per-CPU-ness matters for scaling: because every CPU has its own hardirq and IST stacks, interrupt handling on many cores in parallel involves zero stack contention — no locking, no shared cache lines for the stack memory itself. This is a quiet but important part of why interrupt handling scales linearly with core count. The pcpu_hot placement of hardirq_stack_ptr is a micro-optimization in the same spirit: the one pointer the entry path must read is kept in a hot per-CPU cache line.

Finally, the IST stacks are why catastrophic-event handlers can be trusted. When you read a machine-check or NMI oops in dmesg, the reason it was able to be printed at all is that the handler ran on a pre-reserved, always-mapped IST stack rather than on whatever broken stack the failure interrupted.

vault

Explorer

IRQ Stacks and Per-CPU Interrupt Handling

IRQ Stacks and Per-CPU Interrupt Handling

Mental Model

Why a Separate Stack at All — the 16 KiB Budget

The Hardirq Stack — `DEFINE_PER_CPU` and `hardirq_stack_ptr`

The actual stack switch

Switch only when needed — the `_cond` wrapper

The Softirq Stack — Reused, Not Separate, on x86-64

The Interrupt Stack Table (IST) — Hardware-Selected Always-Valid Stacks

Stack-Overflow Detection — Guard Pages and `CONFIG_VMAP_STACK`

Failure Modes and How to Diagnose Them

Alternatives and Architecture Differences

Production Notes

See Also

Graph View

Table of Contents

Backlinks

vault

Explorer

IRQ Stacks and Per-CPU Interrupt Handling

IRQ Stacks and Per-CPU Interrupt Handling

Mental Model

Why a Separate Stack at All — the 16 KiB Budget

The Hardirq Stack — DEFINE_PER_CPU and hardirq_stack_ptr

The actual stack switch

Switch only when needed — the _cond wrapper

The Softirq Stack — Reused, Not Separate, on x86-64

The Interrupt Stack Table (IST) — Hardware-Selected Always-Valid Stacks

Stack-Overflow Detection — Guard Pages and CONFIG_VMAP_STACK

Failure Modes and How to Diagnose Them

Alternatives and Architecture Differences

Production Notes

See Also

Graph View

Table of Contents

Backlinks

The Hardirq Stack — `DEFINE_PER_CPU` and `hardirq_stack_ptr`

Switch only when needed — the `_cond` wrapper

Stack-Overflow Detection — Guard Pages and `CONFIG_VMAP_STACK`