vault

VM Exit Reasons and Handling

16 min read

VM Exit Reasons and Handling

A VM exit is the hardware event that yanks control away from a running guest and hands it back to the hypervisor. While a guest runs in VMX non-root mode (Intel) or SVM guest mode (AMD), the CPU executes guest instructions directly on the silicon — until it hits something the hypervisor configured to intercept (a sensitive instruction, a device access, a nested-page-table miss, a pending host interrupt). At that instant the processor atomically saves guest state, loads host state from the VMCS (Intel Virtual Machine Control Structure) or VMCB (AMD Virtual Machine Control Block), and resumes the host inside the kernel — recording why it exited in a small integer, the basic exit reason (Intel) or exitcode (AMD). KVM (the Kernel-based Virtual Machine) reads that number, dispatches it through a per-vendor handler table (vmx_handle_exit → kvm_vmx_exit_handlers[], svm_handle_exit → svm_exit_handlers[]), and makes the single most important decision in the whole subsystem: can I resolve this in-kernel and re-enter the guest, or must I return to the userspace VMM? Because every exit costs thousands of cycles of state save/restore plus the round-trip, the entire performance story of virtualization reduces to how many exits happen and how cheaply each is handled. (All code and constants below are pinned to Linux 6.12 LTS, x86, released 2024-11-17.)

This note is the conceptual keystone of the Linux Virtualization MOC: once you understand that a VM exit is the universal cost unit, the rest of the subsystem reads as “how do we avoid this exit?” — virtio batches I/O exits, vhost handles them in-kernel, VFIO passthrough removes device exits, and APICv/posted interrupts remove interrupt exits.

Mental Model

Think of a VM exit as a synchronous trap into the hypervisor, structurally similar to how a system call traps a user process into the kernel — except the privilege axis is orthogonal to the ring 0–3 axis. The guest has its own complete ring 0 (its own kernel runs privileged inside the VM), so the trap is not “ring 3 → ring 0”; it is “non-root mode → root mode,” a transition the silicon implements with dedicated microcode that swaps the entire CPU state through the VMCS/VMCB. The reason for the trap is encoded as a number, and KVM’s job is to be a giant switch statement on that number.

The crucial mental refinement is that not all exits are equal in cost, and KVM is built as a cost ladder. The cheapest outcome is the fast path: a handful of exits (a timer-MSR write, a HLT that immediately has pending work) are resolved in a few instructions with interrupts still disabled, before KVM even unwinds the entry sequence, and the guest is re-entered. One rung up is the in-kernel slow path: KVM runs the full handler (emulate CPUID, walk the nested page tables for an EPT violation, read a virtual MSR), then returns “re-enter the guest” to its own run loop — never touching userspace. The most expensive rung is the exit to userspace: KVM gives up, writes kvm_run->exit_reason to one of the KVM_EXIT_* codes, and returns from the ioctl(KVM_RUN) so the userspace VMM can emulate a device, deliver an event, or tear the VM down.

flowchart TB
  G["Guest runs in non-root / guest mode"]
  HW["CPU traps: VM EXIT<br/>save guest state, load host state<br/>record basic exit reason / exitcode"]
  FP{"fastpath<br/>exit?"}
  DISP["vmx_handle_exit / svm_handle_exit<br/>index handler table by reason"]
  H{"handler return<br/>value"}
  REENTER["re-enter guest<br/>(in-kernel resolved)"]
  USER["fill kvm_run->exit_reason<br/>return from ioctl(KVM_RUN)"]
  VMM["userspace VMM<br/>emulate I/O / inject / destroy"]
  G --> HW --> FP
  FP -->|"yes (HLT, MSR-write)"| REENTER
  FP -->|"no"| DISP --> H
  H -->|"return 1"| REENTER
  H -->|"return 0"| USER --> VMM
  REENTER --> G
  VMM -->|"ioctl(KVM_RUN)"| G

The VM-exit decision tree. What it shows: every exit is classified first by the architecture-specific fast path, then dispatched through the per-vendor handler table; the handler’s integer return value is the fork — 1 means “I handled it, re-enter the guest,” 0 means “I cannot, return to userspace via kvm_run.” The insight to take: the two cheap outcomes (fastpath, in-kernel return 1) keep control inside the kernel; only return 0 pays the full round-trip to userspace, so a “fast” guest is one whose exits almost all resolve to return 1 or never happen at all.

The Exit Reason Codes

When the CPU performs a VM exit it writes a numeric reason into the control structure. Intel and AMD encode this completely differently, and KVM has separate tables for each — but x86-generic emulation handlers (kvm_emulate_cpuid, kvm_emulate_halt) are shared between them.

Intel: basic exit reasons

On Intel VT-x, the VMCS field VM-exit reason contains a 16-bit basic exit reason in its low bits (plus modifier flags such as “VM-entry failure” in the high bits). KVM defines these as small consecutive integers in arch/x86/include/uapi/asm/vmx.h. The lowest-numbered ones are the hottest. A representative slice, with the exact values from v6.12 (vmx.h, v6.12):

#define EXIT_REASON_EXCEPTION_NMI       0   /* guest hit an exception or NMI */
#define EXIT_REASON_EXTERNAL_INTERRUPT  1   /* a host interrupt fired while guest ran */
#define EXIT_REASON_TRIPLE_FAULT        2   /* guest faulted while handling a fault */
#define EXIT_REASON_INTERRUPT_WINDOW    7   /* guest is now ready to take an interrupt */
#define EXIT_REASON_CPUID               10
#define EXIT_REASON_HLT                 12
#define EXIT_REASON_CR_ACCESS           28  /* mov to/from CR0/CR3/CR4/CR8 */
#define EXIT_REASON_IO_INSTRUCTION      30  /* IN/OUT port I/O */
#define EXIT_REASON_MSR_READ            31  /* RDMSR */
#define EXIT_REASON_MSR_WRITE           32  /* WRMSR */
#define EXIT_REASON_EPT_VIOLATION       48  /* nested-page-table fault (the MMIO/demand-fault path) */
#define EXIT_REASON_EPT_MISCONFIG       49
#define EXIT_REASON_PREEMPTION_TIMER    52  /* the VMX preemption timer fired */
#define EXIT_REASON_APIC_ACCESS         44
#define EXIT_REASON_BUS_LOCK            74
#define EXIT_REASON_NOTIFY              75

Each exit also leaves a qualification (the exit qualification field) describing the specifics — for an IO_INSTRUCTION exit it encodes the port number, the operand size, and direction; for an EPT_VIOLATION it encodes the faulting guest-physical address and the access type. KVM reads these via helpers like vmx_get_exit_qual().

AMD: SVM exitcodes

On AMD-V (SVM), the VMCB control area holds an 8-byte exitcode at control.exit_code, plus two exit-info words (exit_info_1, exit_info_2) that play the role of Intel’s qualification. AMD’s numbering is a structured opcode map rather than a dense sequence — control-register accesses, debug-register accesses, and exceptions occupy fixed numeric ranges, defined in arch/x86/include/uapi/asm/svm.h (svm.h, v6.12):

#define SVM_EXIT_READ_CR0      0x000   /* CR reads occupy 0x000..0x00f */
#define SVM_EXIT_WRITE_CR0     0x010   /* CR writes occupy 0x010..0x01f */
#define SVM_EXIT_EXCP_BASE     0x040   /* exceptions: SVM_EXIT_EXCP_BASE + vector */
#define SVM_EXIT_INTR          0x060   /* physical INTR while guest ran */
#define SVM_EXIT_NMI           0x061
#define SVM_EXIT_VINTR         0x064   /* virtual interrupt window */
#define SVM_EXIT_CPUID         0x072
#define SVM_EXIT_HLT           0x078
#define SVM_EXIT_IOIO          0x07b   /* port I/O */
#define SVM_EXIT_MSR           0x07c   /* RDMSR/WRMSR */
#define SVM_EXIT_VMMCALL       0x081   /* the AMD hypercall instruction */
#define SVM_EXIT_NPF           0x400   /* Nested Page Fault — AMD's EPT_VIOLATION analogue */
#define SVM_EXIT_ERR           -1      /* invalid guest state at VMRUN */

The conceptual mapping is worth memorizing because it recurs everywhere: Intel EPT_VIOLATION (48) ≡ AMD NPF (0x400) — both are nested-page-table faults and both drive demand-paging and MMIO detection; Intel IO_INSTRUCTION (30) ≡ AMD IOIO (0x07b); Intel MSR_READ/MSR_WRITE (31/32) ≡ AMD MSR (0x07c) with a direction bit; Intel VMCALL ≡ AMD VMMCALL. The vendor difference is real at the bit level but mostly evaporates once KVM dispatches into shared emulation code.

Uncertain

Verify: the precise modifier-bit layout of the Intel VMCS VM-exit reason field (the high bits beyond the 16-bit basic reason — e.g. bit 31 “VM-entry failure,” the enclave/SMM bits). KVM reads it as a union vmx_exit_reason with .basic and .failed_vmentry members in vmx.c, but the full bitfield semantics live in the Intel SDM Vol. 3, “Basic VM-Exit Information,” which was not fetched for this note. Reason: only the kernel’s union vmx_exit_reason usage was inspected, not the architectural spec. To resolve: cross-check against the Intel SDM Appendix C exit-reason table. uncertain

Mechanical Walk-through

From hardware exit to the dispatcher

After the CPU performs the VM exit, the architecture-specific entry/exit routine (vmx_vcpu_run on Intel, svm_vcpu_run on AMD) returns into KVM’s vcpu_enter_guest(). Before doing anything heavyweight, KVM gives the vendor code a chance at the fast path. On Intel this is vmx_exit_handlers_fastpath(), which switches on the basic reason and handles only three cases entirely with interrupts off (vmx.c, v6.12):

static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu,
                                             bool force_immediate_exit)
{
        if (is_guest_mode(vcpu) &&
            to_vmx(vcpu)->exit_reason.basic != EXIT_REASON_PREEMPTION_TIMER)
                return EXIT_FASTPATH_NONE;       /* nested L2: only the timer is fastpathed */
 
        switch (to_vmx(vcpu)->exit_reason.basic) {
        case EXIT_REASON_MSR_WRITE:
                return handle_fastpath_set_msr_irqoff(vcpu);   /* e.g. TSC-deadline / x2APIC ICR */
        case EXIT_REASON_PREEMPTION_TIMER:
                return handle_fastpath_preemption_timer(vcpu, force_immediate_exit);
        case EXIT_REASON_HLT:
                return handle_fastpath_hlt(vcpu);
        default:
                return EXIT_FASTPATH_NONE;        /* fall through to the full slow path */
        }
}

A return of EXIT_FASTPATH_REENTER_GUEST lets vcpu_enter_guest() loop straight back into the guest without ever calling the heavyweight handle_exit. The set is deliberately tiny — these are exits where the entire response is “update one timer/MSR register and resume” — because the value of the fast path is that it never re-enables interrupts or unwinds the FPU/MSR save state. EXIT_REASON_MSR_WRITE is fastpathed because guests hammer the x2APIC ICR (inter-processor-interrupt) and TSC-deadline-timer MSRs constantly; HLT is fastpathed because a vCPU that halts but immediately has pending events should not pay a full round-trip.

The dispatch tables

If the fast path declines, vcpu_enter_guest() calls the vendor’s handle_exit hook with the previously-computed exit_fastpath. On Intel that is vmx_handle_exit → __vmx_handle_exit, which after some nested-virtualization and failed-entry checks indexes the handler table (vmx.c, v6.12):

exit_handler_index = array_index_nospec((u16)exit_reason.basic,
                                        kvm_vmx_max_exit_handlers);
if (!kvm_vmx_exit_handlers[exit_handler_index])
        goto unexpected_vmexit;
 
return kvm_vmx_exit_handlers[exit_handler_index](vcpu);

array_index_nospec() is a Spectre-v1 mitigation: it clamps the attacker-influenced index inside bounds so speculative execution cannot read past the table. The table itself maps each basic reason to a handler function (kvm_vmx_exit_handlers[]):

static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
        [EXIT_REASON_EXCEPTION_NMI]       = handle_exception_nmi,
        [EXIT_REASON_EXTERNAL_INTERRUPT]  = handle_external_interrupt,
        [EXIT_REASON_IO_INSTRUCTION]      = handle_io,
        [EXIT_REASON_CPUID]               = kvm_emulate_cpuid,    /* shared with SVM */
        [EXIT_REASON_MSR_READ]            = kvm_emulate_rdmsr,    /* shared with SVM */
        [EXIT_REASON_MSR_WRITE]           = kvm_emulate_wrmsr,    /* shared with SVM */
        [EXIT_REASON_HLT]                 = kvm_emulate_halt,     /* shared with SVM */
        [EXIT_REASON_EPT_VIOLATION]       = handle_ept_violation,
        [EXIT_REASON_EPT_MISCONFIG]       = handle_ept_misconfig,
        /* ... ~45 entries total ... */
};

AMD’s svm_handle_exit → svm_invoke_exit_handler does the same against svm_exit_handlers[], which is indexed by the much larger exitcode space (entries at 0x000, 0x010, 0x040+vector, 0x400, …). Both vendors special-case the hottest handlers before the indirect table call under CONFIG_MITIGATION_RETPOLINE, calling them directly to dodge the retpoline cost of an indirect branch:

#ifdef CONFIG_MITIGATION_RETPOLINE
        if (exit_reason.basic == EXIT_REASON_MSR_WRITE)
                return kvm_emulate_wrmsr(vcpu);
        else if (exit_reason.basic == EXIT_REASON_PREEMPTION_TIMER)
                return handle_preemption_timer(vcpu);
        else if (exit_reason.basic == EXIT_REASON_INTERRUPT_WINDOW)
                return handle_interrupt_window(vcpu);
        else if (exit_reason.basic == EXIT_REASON_EXTERNAL_INTERRUPT)
                return handle_external_interrupt(vcpu);
        else if (exit_reason.basic == EXIT_REASON_HLT)
                return kvm_emulate_halt(vcpu);
        else if (exit_reason.basic == EXIT_REASON_EPT_MISCONFIG)
                return handle_ept_misconfig(vcpu);
#endif

That this list of “hot, worth-avoiding-an-indirect-branch” exits is exactly the MSR/timer/interrupt-window/external-interrupt/HLT/EPT-misconfig set is itself a documentary record of which exits dominate real workloads.

The fork: return 1 versus return 0

Every exit handler returns an int, and that integer is the most important value in KVM. The convention, throughout arch/x86/kvm/, is:

  • return 1 — “I handled this exit; re-enter the guest.” vcpu_enter_guest() returns this up to vcpu_run(), which loops straight back into another VM entry without returning from the ioctl.
  • return 0 — “I cannot handle this in-kernel; exit to userspace.” The handler has already written kvm_run->exit_reason; vcpu_run() propagates the 0 and the ioctl(KVM_RUN) returns to the VMM.
  • return <0 — a real error (-EINTR, -EIO); the ioctl returns that errno.

The canonical illustration is __kvm_emulate_halt, the shared HLT handler (x86.c, v6.12):

static int __kvm_emulate_halt(struct kvm_vcpu *vcpu, int state, int reason)
{
        ++vcpu->stat.halt_exits;
        if (lapic_in_kernel(vcpu)) {               /* in-kernel APIC: KVM owns wakeups */
                if (kvm_vcpu_has_events(vcpu))
                        vcpu->arch.pv.pv_unhalted = false;
                else
                        vcpu->arch.mp_state = state;   /* mark halted; run loop will block */
                return 1;                              /* stay in the kernel */
        } else {
                vcpu->run->exit_reason = reason;       /* reason == KVM_EXIT_HLT */
                return 0;                              /* hand the halt to userspace */
        }
}

This single function captures the whole design: the same guest instruction (HLT) resolves in-kernel (return 1) when KVM emulates the local APIC, or bounces to userspace (return 0, KVM_EXIT_HLT) when the VMM does. The decision is a policy of how the VM was configured, not a property of the instruction.

What “exit to userspace” looks like on the wire

The KVM_EXIT_* codes that land in kvm_run->exit_reason are the userspace-visible exit reasons — a much smaller, architecture-neutral set than the vendor codes, defined in include/uapi/linux/kvm.h (kvm.h, v6.12). The ones a VMM author actually handles:

KVM_EXIT_*valuemeaning / VMM action
KVM_EXIT_IO2guest did IN/OUT KVM can’t satisfy; emulate the port device
KVM_EXIT_HLT5guest halted (userspace-APIC VMs); block until an event
KVM_EXIT_MMIO6guest touched unbacked guest-physical memory; emulate the device register
KVM_EXIT_IRQ_WINDOW_OPEN7guest is now ready to take an injected interrupt
KVM_EXIT_SHUTDOWN8triple fault / reset; usually destroy or reboot the VM
KVM_EXIT_FAIL_ENTRY9hardware refused the VM entry (bad guest state)
KVM_EXIT_INTR10a host signal interrupted KVM_RUN; loop and re-enter
KVM_EXIT_INTERNAL_ERROR17KVM hit an unhandlable condition; suberror says which
KVM_EXIT_SYSTEM_EVENT24guest requested shutdown/reset/crash/suspend
KVM_EXIT_X86_RDMSR / WRMSR29 / 30a filtered MSR access userspace asked to see
KVM_EXIT_NOTIFY37guest stalled without an event window (a DoS guard)

The detailed shape of each payload lives in The kvm_run Shared Structure; here the point is the mapping: a vendor exit code like Intel IO_INSTRUCTION is reduced — via handle_io → kvm_fast_pio → (for unhandled ports) KVM_EXIT_IO — into the architecture-neutral userspace code. For port I/O specifically, handle_io extracts port/size/direction from the exit qualification and calls kvm_fast_pio(); only ports that KVM cannot satisfy itself produce a KVM_EXIT_IO (vmx.c handle_io, v6.12).

Counting exits

KVM increments per-vCPU statistics on every exit, exposed via debugfs (/sys/kernel/debug/kvm/<pid>-<vm>/vcpu<N>/). vcpu->stat.exits counts all slow-path exits, with sub-counters such as io_exits, mmio_exits, halt_exits, irq_window_exits, and nmi_window_exits. These are the first diagnostic when a VM is slow — a guest spending all its time in io_exits or mmio_exits is doing register-poke device emulation that virtio would batch away. The kvm_exit tracepoint (visible via perf kvm stat and trace-cmd) records the raw reason per exit, which is how you discover, e.g., that an idle guest is generating millions of HLT exits because halt-polling is misconfigured.

Failure Modes and Diagnostics

KVM_EXIT_INTERNAL_ERROR with suberror = KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON. This is the unexpected_vmexit path: the CPU produced a basic reason with no handler entry (or out of range). KVM dumps the VMCS (if kvm_intel.dump_invalid_vmcs=1) and bails to userspace with the raw reason in internal.data[0]. It almost always means a CPU feature was exposed to the guest that KVM does not actually emulate.

KVM_INTERNAL_ERROR_DELIVERY_EV. Seen when an event (interrupt/exception) was being delivered into the guest at the moment of the exit, and the exit type makes re-delivery unsafe — KVM detects the “vectoring info valid” condition for an exit reason it cannot safely retry and refuses to spin in an infinite fault loop (the comment in __vmx_handle_exit specifically calls out the EPT_MISCONFIG-during-delivery trap that would otherwise loop forever).

KVM_EXIT_FAIL_ENTRY. The hardware rejected the VM entry itself — guest state failed the architectural consistency checks. fail_entry.hardware_entry_failure_reason carries the VMCS VM_INSTRUCTION_ERROR (or the raw exit reason with the failed-entry bit). This is the symptom of a corrupted VMCS, an unsupported guest CR/EFER combination, or a bug in nested state restore.

KVM_EXIT_NOTIFY with KVM_NOTIFY_CONTEXT_INVALID. With KVM_CAP_X86_NOTIFY_VMEXIT enabled, the CPU forces an exit if the guest runs too long without an “event window” — a guard against a guest wedging a physical CPU (a denial-of-service primitive). If flags & KVM_NOTIFY_CONTEXT_INVALID, the VMCS is corrupted and resuming is unsafe.

Silent exit storms. The nastiest failure mode is not a crash but a correctness-preserving but catastrophically slow exit storm: a guest device driver that pokes an emulated register in a tight loop, or an unaccelerated APIC generating an exit per timer tick. Nothing breaks; the guest is just 10–100× slower than it should be. The only way to see it is to count exits (perf kvm stat live) and find the dominant reason.

Alternatives and Why the Subsystem Avoids Exits

Every other performance feature in the Linux Virtualization MOC exists to eliminate a category of exit:

  • IO_INSTRUCTION / MMIO exits from register-by-register device emulation → replaced by the virtio shared-memory ring, which trades thousands of per-register exits for a single batched notification (a “virtqueue kick”).
  • virtio notification exits → handled in-kernel by vhost (the device backend runs in a kernel thread, so the kick never reaches userspace) or removed entirely by ioeventfd (a doorbell write becomes an eventfd signal, not an exit-to-userspace).
  • Device interrupts and all I/O exits → removed by VFIO/SR-IOV passthrough: the guest owns real hardware, so its MMIO never traps.
  • Interrupt-injection exits (EXTERNAL_INTERRUPT, INTERRUPT_WINDOW, APIC accesses) → removed by AVIC and posted interrupts, which let the guest read/write its virtual APIC and receive interrupts with zero exits.

The contrast with full software emulation (no hardware virtualization at all, e.g. QEMU’s TCG) is instructive: TCG never takes a VM exit because there is no guest mode — it interprets/JITs every guest instruction, which is slow for a different reason (no native execution). Hardware-assisted KVM is fast precisely because it runs guest code natively between exits; the exit is the toll booth on an otherwise free highway.

Production Notes

In practice the exit profile is the single most useful KVM observability signal. AWS, Google, and the upstream KVM maintainers all reason about guest performance in terms of “exits per second per vCPU.” A well-tuned Linux guest with virtio, APICv, and halt-polling can run for milliseconds between heavy (userspace) exits; a poorly-configured guest with emulated devices and a software APIC can exit tens of thousands of times per second. The kvm_exit/kvm_entry tracepoint pair (timestamp delta = “time in guest”) is the standard way to measure this, and perf kvm stat report aggregates the reasons into a histogram — the first thing to run on a slow VM. The fast-path set (MSR_WRITE, HLT, PREEMPTION_TIMER) was added and grown over many releases precisely because profiling showed those three reasons dominating the exit count on interrupt- and timer-heavy workloads, and shaving the in-kernel handling cost of the most frequent reason is worth more than optimizing any rare one.

See Also

Graph View

Backlinks