vault

VM Entry and VM Exit Mechanics

15 min read

VM Entry and VM Exit Mechanics

A VM entry is the single hardware instruction that hands the physical CPU from the hypervisor to a guest; a VM exit is the hardware event that snatches it back. On Intel VMX the entry instruction is VMLAUNCH (the first time a given control structure is used) or VMRESUME (every time thereafter); on AMD SVM it is the one instruction VMRUN. Each performs a world switch: the CPU atomically saves the hypervisor’s architectural state, loads the guest’s from the VMCB, and begins executing guest code directly on the silicon (Intel SDM Vol 3C; AMD SVM 33047 §2.2). A VM exit reverses the swap and stamps a reason code so the hypervisor knows what privileged thing the guest tried. The whole performance story of virtualization is the frequency of these transitions times their cost — measured at roughly 300 cycles for an entry/exit pair on modern hardware (VMXbench). (Code and constants are pinned to Linux 6.12 LTS, x86, released 2024-11-17; architectural facts from Intel SDM Vol 3C and the AMD SVM/APM manuals.)

This note is the hardware verb. The data it operates on is VMCS and VMCB (Virtual Machine Control Structure); the software loop that issues it is the KVM vCPU Run Loop; the dispatch of an exit once back in the kernel is VM Exit Reasons and Handling. The modes it transitions between are Root Mode and Non-Root Mode.

Mental Model

Picture the CPU as having two complete “personalities” and a single instruction that flips between them. On VM entry the CPU freezes the hypervisor’s personality into the control structure, thaws the guest’s, and runs the guest at full native speed — no interpreter, no binary translation, real instructions on real hardware. The guest keeps running until it does something the hypervisor asked to be told about (a sensitive instruction, a fault, an external interrupt arriving, a missing page in the second-level page tables). At that instant the CPU freezes the guest, thaws the hypervisor exactly where it left off (right after the entry instruction), and records why. The hypervisor handles the cause and re-enters. The cost is the freeze/thaw — hundreds of cycles each way, because the CPU must flush pipelines, swap a large architectural state, and re-establish address-translation and interrupt context.

sequenceDiagram
  participant HV as Hypervisor (VMX root / host)
  participant CPU as Physical CPU
  participant G as Guest (VMX non-root)
  HV->>CPU: VMLAUNCH / VMRESUME (Intel) or VMRUN (AMD)
  Note over CPU: VM ENTRY — save host state,<br/>load guest state from VMCS/VMCB,<br/>consistency-check guest state
  CPU->>G: execute guest natively
  Note over G: runs until a sensitive event:<br/>CPUID, HLT, CR/MSR access,<br/>EPT/NPT fault, external interrupt
  G-->>CPU: trap (sensitive event)
  Note over CPU: VM EXIT — save guest state,<br/>load host state, write exit reason
  CPU->>HV: resume at HOST_RIP (right after entry insn)
  Note over HV: read exit reason, handle, loop

One round trip of the world switch. What it shows: a single instruction (VMLAUNCH/VMRESUME/VMRUN) enters the guest; the guest runs natively until a configured event forces a VM exit; the CPU returns control to the hypervisor at the address stored in the control structure’s host area, with the exit reason filled in. The insight to take: the guest runs real instructions at near-native speed between transitions — the only overhead is the transitions themselves, so virtualization performance is entirely “how few times must we round-trip, and how cheap is each trip.”

VMLAUNCH vs VMRESUME — the launch-state machine

On Intel, the two entry instructions exist because of a launch-state stored in the current VMCS. VMLAUNCH “requires the VMCS launch state to be clear and sets it to launched upon success”; VMRESUME “requires the VMCS launch state to already be launched” (felixcloutier VMLAUNCH/VMRESUME). In other words: the very first entry into a freshly-set-up VMCS uses VMLAUNCH; every subsequent re-entry after a VM exit uses VMRESUME. Getting this wrong is a hard fail — VMLAUNCH on an already-launched VMCS raises VMXERR_VMLAUNCH_NONCLEAR_VMCS (error 4), and VMRESUME on a never-launched VMCS raises VMXERR_VMRESUME_NONLAUNCHED_VMCS (error 5) (uapi/asm/vmx.h, v6.12).

Why have two instructions at all? Because VMRESUME lets the CPU assume the VMCS has not changed structurally since the last exit and skip some validation, making the common case (re-enter the same guest you just exited) cheaper than the cold-start case. KVM remembers the launch state in software, in the launched boolean of struct loaded_vmcs (vmcs.h, v6.12), and the assembly entry path branches on it. From the real KVM VMX entry routine (arch/x86/kvm/vmx/vmenter.S, v6.12):

;; __vmx_vcpu_run: %arg3 (-> EBX) carries the run flags, including the VMRESUME bit
	mov  %_ASM_ARG3L, %ebx
	...
	bt   $VMX_RUN_VMRESUME_SHIFT, %ebx   ; test the "should resume?" bit
	jnc  .Lvmlaunch                      ; carry clear -> not launched -> VMLAUNCH
.Lvmresume:
	vmresume                             ; the launched path
.Lvmlaunch:
	vmlaunch                             ; the first-entry path

The bt (bit-test) sets the carry flag from the VMRESUME bit; jnc (“jump if no carry”) takes the VMLAUNCH branch when the bit is clear. This is the entire launch-vs-resume decision, in hardware-adjacent assembly. AMD sidesteps it: VMRUN is one instruction with no launch-state distinction — it does not matter whether this is the first or thousandth entry, the hypervisor always issues VMRUN (AMD 33047 §2.2).

The VM entry, step by step

On Intel, the entry instruction operates on the current VMCS (the one a prior VMPTRLD selected — see VMCS and VMCB (Virtual Machine Control Structure)). The hardware sequence is (Intel SDM Vol 3C §27, summarized; corroborated by felixcloutier):

  1. Check the VM-execution and VM-entry control fields and the host-state area for consistency. If a control field is malformed, the entry fails before touching guest state and the instruction returns to the hypervisor with VMXERR_ENTRY_INVALID_CONTROL_FIELD (error 7) or ..._INVALID_HOST_STATE_FIELD (error 8) — a “VM-instruction error,” distinct from a VM exit.
  2. Save host state into the host-state area (host CR3/RSP/RIP/segment selectors, etc.).
  3. Load guest state from the guest-state area and load any MSRs listed in the VM-entry MSR-load area.
  4. Consistency-check the loaded guest state. If this fails, it is treated as a VM exit (not a VM-instruction error) with the high bit set in the exit reason (VMX_EXIT_REASONS_FAILED_VMENTRY = 0x80000000), e.g. EXIT_REASON_INVALID_STATE = 33.
  5. Inject a pending event if the VM-entry interruption-information field requests one (this is how the hypervisor delivers a virtual interrupt or exception on the way in).
  6. Begin executing guest code in non-root operation.

The GPRs are the subtle part: the VMCS does not save/restore general-purpose registers (RAX, RBX, RCX, …). Only the “special” architectural state (control/segment/system registers, RIP, RSP, RFLAGS) is in the VMCS. KVM therefore loads and saves the guest GPRs in software, by hand, in the assembly wrapper around the entry instruction (vmenter.S, v6.12):

	;; immediately before VMLAUNCH/VMRESUME: load guest GPRs from the vcpu struct
	mov VCPU_RCX(%_ASM_AX), %_ASM_CX
	mov VCPU_RDX(%_ASM_AX), %_ASM_DX
	mov VCPU_RBX(%_ASM_AX), %_ASM_BX
	...
	mov VCPU_RAX(%_ASM_AX), %_ASM_AX   ; RAX loaded last (it held the pointer)
	;; ... VMLAUNCH / VMRESUME here; guest runs; on exit we land at vmx_vmexit ...
	pop           VCPU_RAX(%_ASM_AX)   ; save guest RAX (pushed on the exit path)
	mov %_ASM_CX, VCPU_RCX(%_ASM_AX)   ; save guest RCX, RDX, ... back to vcpu
	mov %_ASM_DX, VCPU_RDX(%_ASM_AX)
	...

After saving the guest GPRs, the routine XORs every register to zero before returning to C, to prevent the host from speculatively using leftover guest register values — a Spectre-class mitigation baked into the entry/exit path.

On AMD the entry is simpler to describe. VMRUN takes the VMCB’s physical address as an implicit rAX operand (AMD 33047 §2.2.1): “Software must load RAX … with the physical address of the VMCB, a 4-Kbyte-aligned page.” VMRUN then “saves some host processor state information in the host state save area in main memory at the physical address specified in the VM_HSAVE_AREA MSR; it then loads corresponding guest state from the VMCB state-save area,” consistency-checks the guest state, and — if legal — runs the guest “until an intercept event occurs, at which point the processor … resumes host execution at the instruction following the VMRUN.” Note the AMD host-state target is a separate page (the VM_HSAVE_PA area), not the VMCB.

Critically, AMD’s VMRUN “saves or restores a minimal amount of state” — enough to resume the hypervisor for a simple intercept. The host’s FS, GS, TR, LDTR (with hidden state), KernelGsBase, and the SYSENTER/STAR MSRs are not handled by VMRUN; the hypervisor manages those with the separate VMLOAD/VMSAVE instructions (AMD 33047 §2.11). This deliberate split lets a hypervisor skip the expensive segment/MSR save when an intercept can be handled without touching that state — a performance lever Intel does not expose as cleanly.

What triggers a VM exit

A VM exit happens when the guest, running in non-root/guest mode, hits something the hypervisor configured to be intercepted (via the control fields / intercept bitmaps in the control structure) or something that unconditionally exits. The Intel reason codes (uapi/asm/vmx.h, v6.12) and AMD exit codes (uapi/asm/svm.h, v6.12) enumerate them. The categories that matter:

  • Sensitive / privileged instructions the guest is not allowed to execute transparently. CPUID (EXIT_REASON_CPUID = 10 / SVM_EXIT_CPUID = 0x072) always exits so the hypervisor can lie about CPU features. RDMSR/WRMSR (EXIT_REASON_MSR_READ = 31/MSR_WRITE = 32) exit if the targeted MSR is marked in the MSR bitmap. INVD, XSETBV, VMCALL/VMMCALL (the hypercall instruction), and the nested-VMX instructions (VMLAUNCH, VMREAD, VMRESUME, …) all exit.
  • Control-register and debug-register access. MOV to/from CRn exits if intercepted (EXIT_REASON_CR_ACCESS = 28); without two-dimensional paging, CR3 writes must trap so the hypervisor can resync shadow page tables — with NPT they need not, which is one of the biggest exit-eliminations in the stack.
  • HLT. A halting guest exits with EXIT_REASON_HLT = 12 / SVM_EXIT_HLT = 0x078 (if HLT-exiting is enabled), letting the hypervisor sleep or schedule the vCPU thread rather than spin the CPU — see the halt-polling discussion in KVM vCPU Run Loop.
  • Exceptions raised in the guest, if the matching bit is set in the exception-intercept bitmap (EXIT_REASON_EXCEPTION_NMI = 0). For example, page faults can be intercepted; debug exceptions (#DB), breakpoints (#BP), and others are selectable.
  • External interrupts and NMIs. A physical interrupt arriving while the guest runs forces EXIT_REASON_EXTERNAL_INTERRUPT = 1 (if external-interrupt exiting is set), so the host can service its own devices and decide whether to inject a virtual interrupt into the guest. This is historically the highest-frequency exit for I/O-heavy guests, which is exactly what APIC Virtualization (APICv and AVIC) and Posted Interrupts exist to eliminate.
  • Second-level page-table faults. When the guest accesses a guest-physical address that the hypervisor’s NPT table does not map (or maps without the needed permission), the CPU exits with EXIT_REASON_EPT_VIOLATION = 48 (Intel) / SVM_EXIT_NPF = 0x400 (AMD). This is how demand-paging of guest RAM, dirty-tracking for live migration, and MMIO trapping all work — the hypervisor catches the fault, maps the page (or emulates the MMIO), and resumes.
  • I/O instructions. IN/OUT to a port marked in the I/O bitmap exit with EXIT_REASON_IO_INSTRUCTION = 30 / SVM_EXIT_IOIO = 0x07b, routing legacy port I/O to device emulation in the VMM (see MMIO and Port IO Emulation).
  • Triple fault and other fatal conditions (EXIT_REASON_TRIPLE_FAULT = 2), which KVM turns into a guest shutdown.

On exit, the hardware writes the reason (and a reason-specific exit qualification / EXITINFO1/EXITINFO2) into the control structure and lands the hypervisor at the host RIP. On AMD the VMEXIT additionally “writes back to the VMCB the current guest state … saves the reason for exiting the guest in the VMCB’s EXITCODE field … clears all intercepts … reloads the host state previously saved by the VMRUN instruction” (AMD 33047 §2.3). What the hypervisor does with the reason is VM Exit Reasons and Handling.

The cost of a transition

A VM entry/exit is not free, because the CPU must serialize, swap a large architectural state, and re-establish translation and interrupt context. Direct measurement with the VMXbench UEFI microbenchmark (which uses RDTSC around bare VMLAUNCH/VMRESUME and a forced exit) reports, on its test hardware, a VM exit of ~326–332 cycles and a VM entry of ~290–330 cycles — so an entry+exit round trip is roughly 600 cycles, often quoted as “a few hundred cycles each way” / “~200 cycles” as a rule of thumb on the newest cores (VMXbench README; cf. the vmlatency driver; MIT 6.828 lecture notes put “trap-and-emulate exit” cost in the hundreds-of-cycles range (MIT 6.828 Virtualization II)).

Uncertain

Verify: the specific cycle figures (~326–332 cycles VM exit, ~290–330 cycles VM entry). Reason: these are from the VMXbench README’s sample output on unspecified test hardware and are highly microarchitecture-dependent — bare hardware transition cost has trended down across CPU generations (Nehalem ~1000+ cycles historically, modern cores far less), and the full KVM exit cost (including the software handler, GPR save/restore, and speculation barriers) is substantially higher than the bare instruction cost. To resolve: run VMXbench / vmlatency on the target CPU, and separately measure KVM’s end-to-end exit cost via the kvm:kvm_exit tracepoint timing. uncertain

The headline cost is only the bare transition. The real per-exit cost in KVM is higher: the assembly must save/restore GPRs, run the Spectre/L1TF mitigations (e.g. the l1tf_vmx_mitigation L1-data-cache flush on affected parts, uapi/asm/vmx.h v6.12), then run the C exit handler. This is why the entire subsystem is engineered to avoid exits rather than make them faster: virtio batches I/O so notifications are rare (virtio Device Model); vhost handles them in-kernel (vhost (In-Kernel virtio Backend)); VFIO passthrough removes device exits entirely (VFIO Framework); APICv/posted interrupts remove interrupt exits (APIC Virtualization (APICv and AVIC), Posted Interrupts); and EPT/NPT removes the CR3-write and shadow-page-table exits (Two-Dimensional Paging (EPT and NPT)). Every one of those optimizations is “delete a round trip through this 300-cycle gate.”

Failure Modes and Subtleties

  • VM-instruction error vs VM exit. A failed entry due to a bad control or host-state field is a VM-instruction error — the entry instruction simply returns to the hypervisor with RFLAGS.CF/ZF set and an error number readable via VMREAD of VM_INSTRUCTION_ERROR. A failed entry due to bad guest state is a VM exit with the 0x80000000 failed-entry bit set. KVM distinguishes these; conflating them leads to misdiagnosed entry bugs. The relevant error numbers (VMXERR_VMLAUNCH_NONCLEAR_VMCS, etc.) live in enum vm_instruction_error_number (uapi/asm/vmx.h v6.12).
  • Launch/resume mismatch. Issuing VMRESUME on a never-launched VMCS (error 5) or VMLAUNCH twice (error 4) is the classic nested-VMX and vCPU-migration bug — after VMCLEAR the launch state resets to “clear,” so the next entry must be VMLAUNCH again, and KVM’s loaded_vmcs.launched flag must be reset in lockstep.
  • Host segment/LDTR reload on AMD. Because VMRUN saves only minimal host state, and because “immediately after VMEXIT, the processor still contains the guest value for LDTR” (AMD 33047 §2.3), the hypervisor must VMLOAD its own FS/GS/TR/LDTR after exit before relying on them — skipping it reads guest descriptor state.
  • Lost interrupt across the entry. A physical interrupt that becomes pending in the tiny window between the hypervisor’s “should I exit?” check and the actual entry instruction must not be silently swallowed. KVM’s vcpu_enter_guest() orders an IN_GUEST_MODE store before re-checking pending requests precisely so a racing kick forces an immediate exit rather than being lost — the software side of this is detailed in KVM vCPU Run Loop.
  • The “300 cycles” is a moving target. Quoting a fixed exit cost is misleading across CPU generations and across exit reasons (an EPT violation that KVM resolves in-kernel is far cheaper than one that returns to the userspace VMM for MMIO emulation). Always frame cost as a distribution over exit reasons, not a single number.

Alternatives and Comparisons

The trap-and-exit model is the post-hardware-virtualization design. Before VT-x/AMD-V, x86 hypervisors could not rely on a clean exit on every sensitive instruction (some did not trap when run unprivileged — see Why x86 Needed Hardware Virtualization), so VMware used binary translation (rewriting guest kernel code to insert traps) and Xen used paravirtualization (modifying the guest OS to call the hypervisor explicitly). Both avoided the missing-trap problem in software. Hardware VM entry/exit replaced them with a clean architectural transition, at the cost of the per-exit latency this note quantifies — which is why, paradoxically, paravirtual devices (virtio Device Model) live on top of hardware virtualization: the CPU virtualization is hardware, but the I/O path is still cooperatively paravirtual to keep exit counts down. The two vendors’ entry/exit designs differ mainly in ergonomics (Intel’s launch-state two-instruction model and opaque VMCS vs AMD’s single VMRUN and transparent VMCB) rather than in the fundamental cost structure; see Intel VMX and AMD SVM for the side-by-side.

Production Notes

The single most actionable production metric is the VM-exit rate by reason, read from the kvm:kvm_exit tracepoint or the per-vCPU vcpu->stat counters. A guest pinning a host core with high EXTERNAL_INTERRUPT or MSR_WRITE exits is paying the transition tax on its I/O or timekeeping path — the fix is APICv/posted interrupts or moving the device path to vhost/VFIO, not “make exits faster.” A guest with a burst of EPT_VIOLATION exits at boot is demand-faulting its RAM (benign) or thrashing huge-page promotion (tunable). Hyperscalers tune their microVMs (Firecracker microVM) by removing entire device models so those exit reasons never occur. The mental discipline, repeated throughout this MOC: a VM is fast when it rarely crosses the entry/exit gate, and the way to make a slow VM fast is almost never to optimize the gate — it is to stop walking through it.

See Also

Graph View

Backlinks