vault

Interrupt Injection and the Virtual APIC

14 min read

Interrupt Injection and the Virtual APIC

When a guest is supposed to receive an interrupt, the CPU is not in guest mode at a convenient instruction boundary — so KVM cannot simply “deliver” it. Instead, on the next VM-entry, KVM tells the hardware “enter the guest and immediately take this interrupt vector,” using a field in the hardware control structure called the VM-entry interruption-information field (Intel) or event_inj (AMD). Getting there requires KVM to emulate the guest’s interrupt controllers in software: the per-vCPU local APIC (LAPIC) with its pending-interrupt bitmaps, the chipset-level I/O APIC and legacy 8259 PIC that route device interrupt lines, and the priority logic that decides which of several pending interrupts wins and whether the guest can take one at all. This is the software injection path — the one used when hardware AVIC is unavailable or disabled, and the substrate the in-kernel LAPIC always provides. This note traces it in the Linux 6.12 long-term-support (LTS) kernel (arch/x86/kvm/lapic.c).

The APIC (Advanced Programmable Interrupt Controller) is x86’s interrupt controller. Each CPU (or vCPU) has a local APIC that receives interrupts and presents the highest-priority one to the core; a chipset I/O APIC turns device interrupt pins into messages addressed to local APICs; the older PIC (Programmable Interrupt Controller, the 8259 pair) is the legacy equivalent. A GSI (Global System Interrupt) is KVM’s abstract interrupt-line number that an IRQ routing table maps onto a specific controller pin or an MSI message. This note assumes you know that an interrupt arrived (e.g. from an irqfd asserting a GSI) and explains what KVM does next.

Mental Model: A Software APIC Backing a Hardware Entry Field

Think of KVM’s in-kernel LAPIC as a faithful software model of a real local APIC’s registers, living in a 4 KiB page (apic->regs), plus a thin layer that, at each VM-entry, copies the winning interrupt into a hardware field the CPU consumes. The two registers that matter most are the IRR (Interrupt Request Register) — a 256-bit bitmap of interrupts that have arrived but not yet been delivered — and the ISR (In-Service Register) — a 256-bit bitmap of interrupts currently being serviced (delivered to the core, not yet acknowledged with an EOI). Injection is the act of moving the highest-priority eligible bit from IRR to ISR and telling the hardware to vector into the guest. Everything else — priority comparison, masking, the interrupt-window dance — is gating around that single move.

flowchart TB
  GSI["GSI asserted<br/>(irqfd / IOAPIC / MSI / timer)"]
  ACC["__apic_accept_irq()<br/>delivery_mode = FIXED"]
  DLV["deliver_interrupt hook"]
  IRR["set bit in IRR<br/>(kvm_lapic_set_irr)"]
  REQ["KVM_REQ_EVENT + kick vCPU"]
  CHK["kvm_check_and_inject_events()<br/>priority: exception > NMI > IRQ"]
  ALLOW{"interrupt_allowed?<br/>IF=1, no STI/MOV-SS shadow"}
  WIN["enable_irq_window:<br/>set INTR_WINDOW_EXITING"]
  PICK["kvm_cpu_get_interrupt()<br/>highest IRR vector; IRR->ISR"]
  INJ["write VM-entry interruption-info<br/>(VMCS) / event_inj (VMCB)"]
  GSI --> ACC --> DLV --> IRR --> REQ --> CHK --> ALLOW
  ALLOW -->|no| WIN
  ALLOW -->|yes| PICK --> INJ

The software interrupt-injection pipeline. What it shows: an asserted GSI lands a bit in the LAPIC’s IRR; KVM raises KVM_REQ_EVENT and kicks the vCPU out of (or before) guest mode; at injection time KVM checks priority and whether the guest can architecturally accept an interrupt; if not, it arms an interrupt-window exit to retry the moment the guest unmasks; if so, it pops the highest IRR vector into ISR and writes the hardware entry field so the next VM-entry vectors the guest. The insight to take: injection is never “push”; it is “mark pending, then on the next safe boundary, if allowed, pop-and-vector — otherwise arm a trap to retry.”

The In-Kernel irqchip and the Split irqchip

Before any injection, the VMM chooses where the interrupt controllers live. Three configurations exist on x86:

  1. Fully userspace irqchip — KVM models none of it; the VMM emulates LAPIC, IOAPIC, and PIC and uses ioctls to inject. Slow and rare; mainly historical.
  2. Full in-kernel irqchip (KVM_CREATE_IRQCHIP) — KVM emulates all three: the LAPIC (per-vCPU), the IOAPIC, and the PIC, in-kernel. The ioctl’s handler in x86.c is explicit: kvm_pic_init(kvm), then kvm_ioapic_init(kvm), then kvm_setup_default_irq_routing(kvm), finally setting kvm->arch.irqchip_mode = KVM_IRQCHIP_KERNEL (arch/x86/kvm/x86.c). It must be done before any vCPU exists (kvm->created_vcpus check).
  3. Split irqchip (KVM_CAP_SPLIT_IRQCHIP) — the LAPIC stays in-kernel (it is on the hot path and must be fast) but the IOAPIC and PIC are emulated in userspace. Enabling the capability sets kvm->arch.irqchip_mode = KVM_IRQCHIP_SPLIT and records nr_reserved_ioapic_pins, the number of GSIs reserved for the userspace IOAPIC (x86.c).

The split model is what modern fast VMMs use (it is the default in QEMU’s kernel-irqchip=split and the norm for microVMs). The rationale is sharp: the LAPIC is touched on every interrupt and EOI and must be in-kernel for APICv and irqfd to work at all, but the IOAPIC/PIC are touched rarely (mostly at boot, during routing changes) and keeping them in userspace shrinks the kernel attack surface and gives the VMM full control over routing and MSI translation. Note the constraint from arch/x86/kvm/irq.c: a resampling irqfd requires the full kernel irqchip, while a basic irqfd works with any in-kernel irqchip including split. The code distinguishes these with helpers irqchip_in_kernel, irqchip_kernel (full), and irqchip_split. In split mode, KVM’s external-interrupt query consults pending_external_vector (a vector the userspace IOAPIC handed KVM) rather than an in-kernel PIC’s output:

if (irqchip_split(v->kvm))
	return pending_userspace_extint(v);   /* split */
else
	return v->kvm->arch.vpic->output;     /* full in-kernel PIC */

The IRR and ISR Bitmaps

The LAPIC registers live in a page, addressed by the architectural offsets (APIC_IRR, APIC_ISR, APIC_TMR, etc.). Each is logically a 256-bit bitmap (one bit per vector, 0–255), but stored sparsely: the real APIC lays each 32-bit sub-register 16 bytes apart, so KVM’s accessors compute the position with two macros (lapic.h):

#define VEC_POS(v) ((v) & (32 - 1))        /* bit within a 32-bit word */
#define REG_POS(v) (((v) >> 5) << 4)       /* byte offset of that word (stride 16) */

VEC_POS(v) is v mod 32 — the bit inside one 32-bit register. REG_POS(v) is (v / 32) * 16 — the byte offset of the register holding vector v, with the 16-byte stride that mirrors hardware. So setting an IRR bit is set_bit(VEC_POS(vec), apic->regs + APIC_IRR + REG_POS(vec)), wrapped as kvm_lapic_set_irr. Setting the IRR bit is the actual act of “the interrupt arrived at this vCPU.”

Two queries drive priority. apic_find_highest_irr scans the IRR for the highest set vector (using find_highest_vector, which walks the registers high-to-low and uses __fls — find-last-set — on the first non-zero word). apic_find_highest_isr does the same for the ISR, but is optimized with a highest_isr_cache because the highest in-service vector is known the moment it is set. A subtlety worth internalizing: apic->irr_pending is “just a hint” — when not in APICv mode, KVM keeps it as a fast “is anything pending?” flag, recomputed precisely (apic_search_irr) when needed; with virtual interrupt delivery it is always treated as true because hardware can set IRR bits behind KVM’s back.

Priority: Why Interrupts Wait Behind Exceptions

A vCPU can have several events pending at once — a page fault to inject (an exception), a queued NMI, and several device interrupts. The x86 architecture imposes a strict priority, and KVM enforces it in kvm_check_and_inject_events (x86.c). The order is: re-injected events first, then pending exceptions, then SMI, then NMI, then maskable external interrupts. The code threads a single boolean can_inject through:

can_inject = !kvm_event_needs_reinjection(vcpu);
 
if (vcpu->arch.exception.pending) {
	...
	kvm_inject_exception(vcpu);
	vcpu->arch.exception.injected = true;
	can_inject = false;
}
...
if (vcpu->arch.nmi_pending) {
	r = can_inject ? kvm_x86_call(nmi_allowed)(vcpu, true) : -EBUSY;
	...
}
if (kvm_cpu_has_injectable_intr(vcpu)) {
	r = can_inject ? kvm_x86_call(interrupt_allowed)(vcpu, true) : -EBUSY;
	...
}

The reasoning is architectural: an exception is a synchronous fault of the instruction the guest just tried to run, so it must be delivered before any asynchronous interrupt, and only one event can be vectored per VM-entry (can_inject becomes false after the first). Within the LAPIC itself there is a second priority layer — the PPR (Processor Priority Register) — which compares the highest in-service vector and the guest-programmed TPR (Task Priority Register) to decide whether a given pending IRR vector outranks what is already being serviced. apic_has_interrupt_for_ppr returns the highest IRR vector only if its priority class (vector & 0xF0) exceeds the current PPR; otherwise the interrupt waits. This is how a guest masks low-priority interrupts while servicing a high-priority one without any of it leaving the LAPIC model.

The Injection Mechanism: VM-Entry Interruption-Information

Once KVM has decided an interrupt is allowed and has chosen the vector, it pops it from IRR into ISR and writes the hardware entry field. The vector selection and acknowledgement live in kvm_cpu_get_interrupt → kvm_apic_ack_interrupt, which does apic_clear_irr(vector) (the interrupt is no longer merely requested) and apic_set_isr(vector) (it is now in service), then recomputes PPR. The actual hardware write is vendor-specific.

On Intel VMX (vmx.c), vmx_inject_irq builds a 32-bit value and writes it to the VMCS:

intr = irq | INTR_INFO_VALID_MASK;     /* bit 31 = valid; bits 7:0 = vector */
...
intr |= INTR_TYPE_EXT_INTR;            /* bits 10:8 = type = external interrupt */
vmcs_write32(VM_ENTRY_INTR_INFO_FIELD, intr);

On the next VMLAUNCH/VMRESUME, the CPU sees the valid bit and vectors the guest through its IDT to the given vector exactly as if a real external interrupt had fired — no exit, the guest just “takes the interrupt” at entry. An NMI is the same field with INTR_TYPE_NMI_INTR and vector 2.

On AMD SVM (svm.c), the equivalent is the VMCB event_inj field:

svm->vmcb->control.event_inj = vcpu->arch.interrupt.nr |
			       SVM_EVTINJ_VALID | type;   /* type = SVM_EVTINJ_TYPE_INTR */

Note the software-LAPIC delivery hooks themselves: vmx_deliver_interrupt / svm_deliver_interrupt are what __apic_accept_irq calls when a FIXED-mode interrupt is accepted. In the non-APICv case svm_deliver_interrupt does exactly kvm_lapic_set_irr(vector, apic) then signals KVM_REQ_EVENT and kicks the vCPU — i.e. it just marks the IRR bit and forces the vCPU back to the injection check. The VMX path first tries a posted interrupt (vmx_deliver_posted_interrupt) and only falls back to kvm_lapic_set_irr + KVM_REQ_EVENT + kick when APICv is inactive. That fallback is precisely the software path this note describes.

Interrupt-Window Exiting: When the Guest Has Interrupts Masked

The hard case is an interrupt arriving while the guest cannot take one — it has cleared the interrupt flag (IF=0 in RFLAGS), or it is inside the one-instruction shadow after STI/MOV SS. KVM detects this in the vendor interrupt_allowed hook. On VMX:

bool __vmx_interrupt_blocked(struct kvm_vcpu *vcpu)
{
	return !(vmx_get_rflags(vcpu) & X86_EFLAGS_IF) ||
	       (vmcs_read32(GUEST_INTERRUPTIBILITY_INFO) &
		(GUEST_INTR_STATE_STI | GUEST_INTR_STATE_MOV_SS));
}

If interrupts are blocked, KVM must not inject (the architecture forbids it) but also must not drop the interrupt. The solution is interrupt-window exiting: KVM asks the CPU to force a VM exit the instant the guest becomes able to accept an interrupt. vmx_enable_irq_window simply sets a VMCS execution control:

void vmx_enable_irq_window(struct kvm_vcpu *vcpu)
{
	exec_controls_setbit(to_vmx(vcpu), CPU_BASED_INTR_WINDOW_EXITING);
}

With that bit set, the CPU exits with reason “interrupt window” the moment the guest’s instruction stream reaches a point where IF=1 and no shadow is active. KVM’s handler then clears the bit and re-raises the event so injection is retried:

static int handle_interrupt_window(struct kvm_vcpu *vcpu)
{
	exec_controls_clearbit(to_vmx(vcpu), CPU_BASED_INTR_WINDOW_EXITING);
	kvm_make_request(KVM_REQ_EVENT, vcpu);
	++vcpu->stat.irq_window_exits;
	return 1;
}

This loop — “can’t inject now, arm a window, exit when unmasked, retry” — is visible in kvm_check_and_inject_events, where after a failed interrupt_allowed it calls kvm_x86_call(enable_irq_window). The irq_window_exits statistic is a useful diagnostic: a high count means the guest spends a lot of time with interrupts masked while interrupts pile up. On AMD, the equivalent uses the V_IRQ virtual-interrupt mechanism (svm_set_vintr), and svm_enable_irq_window additionally has to cope with the global-interrupt-flag (GIF) state and may temporarily inhibit AVIC for an ExtINT that AVIC cannot deliver.

Failure Modes and Common Misunderstandings

The most common conceptual error is thinking injection “pushes” an interrupt into a running guest. It does not: with the software path the vCPU must not be in guest mode at the moment of the VMCS write, so __apic_accept_irq ends by kicking the target vCPU (kvm_vcpu_kick) to force a VM exit if it is currently running, so that the next entry picks up the new IRR bit. Forgetting this kick would let an interrupt sit in IRR indefinitely while the vCPU spins in the guest — which is exactly the bug the memory barriers in svm_deliver_interrupt guard against (the comment notes the ordering “guarantees that either VMRUN will see and process the new vIRR entry, or that svm_complete_interrupt_delivery will signal the doorbell”).

A second subtlety is EOI and the ISR. An interrupt stays in ISR until the guest writes EOI; the LAPIC then clears the highest ISR bit (apic_clear_isr) and lowers PPR so the next-priority pending interrupt can be delivered. If the guest is a level-triggered IOAPIC source, the EOI must also propagate back to deassert the line — KVM uses an EOI exit bitmap (ioapic_handled_vectors, recomputed by vcpu_scan_ioapic) so the CPU traps EOIs for exactly the level-triggered vectors that need it, and for split-irqchip routes the EOI notification out to the userspace IOAPIC. A misconfigured EOI exit bitmap manifests as a level interrupt that fires once and never again (line never deasserted) or storms (line never seen as handled).

Uncertain

Verify: the exact path by which a split-irqchip EOI for a level-triggered IOAPIC pin reaches the userspace IOAPIC model (the KVM_EXIT_IOAPIC_EOI userspace exit and its interaction with KVM_CAP_SPLIT_IRQCHIP). Reason: this note verified the in-kernel IRR/ISR/PPR mechanics and the irqchip-mode setup against v6.12 source, but did not fetch the IOAPIC EOI-exit emulation glue (ioapic.c / the KVM_EXIT_IOAPIC_EOI handler). To resolve: read arch/x86/kvm/ioapic.c and the KVM_EXIT_IOAPIC_EOI documentation at v6.12. uncertain

Alternatives and When to Choose Them

The software injection path described here is the fallback and the always-available baseline. Its cost is one VM exit per interrupt at minimum (to kick the vCPU and re-enter with the entry field set), plus potential interrupt-window exits when the guest masks interrupts. The alternative is hardware AVIC (AMD): the guest reads and writes its own APIC registers (TPR, EOI, ICR) without trapping, and the hardware maintains the virtual IRR/ISR and even delivers interrupts to a running vCPU with no exit at all via posted interrupts. When APICv is active the IRR is managed partly by hardware (the PIR, posted-interrupt request, syncs into IRR), which is why so much of lapic.c has if (apic->apicv_active) branches — the software model and the hardware model must stay coherent. The decision is not the VMM’s to make per-interrupt: KVM uses APICv whenever the CPU supports it and nothing inhibits it (nested guests, certain guest configurations, an ExtINT through the PIC), falling back to this software path otherwise. So this note is not a “legacy alternative” — it is the code that runs whenever APICv is inhibited, which happens routinely.

Production Notes

For diagnosis, kvm_stat / the per-vCPU statistics expose irq_injections, irq_window_exits, and nmi_injections; a VM with high irq_window_exits relative to irq_injections is fighting a guest that masks interrupts heavily (or a too-aggressive interrupt source). Tracepoints kvm_inj_virq, kvm_apic_accept_irq, and kvm_apicv_accept_irq let you watch each acceptance and distinguish the software path (kvm_inj_virq fires on the VMCS write) from the APICv path (kvm_apicv_accept_irq fires when delivery was handled without injection). The split-irqchip model is the production default for QEMU microVM-style configurations and for Firecracker/Cloud Hypervisor, because it keeps the hot LAPIC in-kernel while letting the lean userspace VMM own IOAPIC/MSI routing; the most common operational mistake is leaving the irqchip fully in userspace (or off), which silently disables APICv, irqfd, and posted interrupts and makes every interrupt expensive.

See Also

Graph View