vault

Intel VMX and AMD SVM

14 min read

Intel VMX and AMD SVM

Intel VMX (Virtual Machine Extensions, marketed as VT-x) and AMD SVM (Secure Virtual Machine, marketed as AMD-V and code-named Pacifica) are the two x86 hardware-virtualization instruction-set extensions that made efficient virtualization of an unmodified guest operating system possible. Before them, x86 was not classically virtualizable: a handful of sensitive instructions silently behaved differently in user mode instead of trapping, defeating the trap-and-emulate model (see Why x86 Needed Hardware Virtualization and Trap-and-Emulate and the Popek-Goldberg Requirements). Both extensions solve this the same way — they add a new processor operating mode in which the guest runs directly on the silicon with its own full ring 0–3 privilege model, but any “sensitive” operation traps out to the hypervisor through a hardware-defined exit. Intel shipped VT-x first (Pentium 4 models 662/672, 14 November 2005); AMD shipped SVM in May 2006 (Athlon 64 “Orleans”/“Windsor”) (Wikipedia: x86 virtualization). The two are mechanically analogous but not binary compatible: they use different instructions and different in-memory control structures, which is exactly why Linux’s KVM ships two separate vendor modules — kvm-intel for VMX and kvm-amd for SVM — on top of the architecture-neutral kvm core.

This note compares the two extensions at the instruction-and-mechanism level: how each is turned on, how each enters and leaves a guest, and how each hands a guest hypercall to the hypervisor. Two closely related topics are deliberately deferred to siblings to keep this note atomic (the vault’s One-Idea-Per-Note rule): the in-memory control structure that holds the entire guest CPU state lives in VMCS and VMCB (Virtual Machine Control Structure), and the orthogonal privilege axis these extensions add — root/host versus non-root/guest, each with its own rings 0–3 — lives in Root Mode and Non-Root Mode. The precise step-by-step of a single entry-and-exit is in VM Entry and VM Exit Mechanics.

Mental Model

flowchart TB
  subgraph CORE["kvm.ko (architecture-neutral core)"]
    API["/dev/kvm ioctl API, vCPU run loop, memory slots"]
  end
  subgraph INTEL["kvm-intel.ko (Intel VMX)"]
    VMXON["VMXON: enter VMX root operation"]
    VMLAUNCH["VMLAUNCH / VMRESUME: VM entry"]
    VMEXIT_I["VM exit -> reason in VMCS"]
    VMCALL["VMCALL: guest hypercall"]
  end
  subgraph AMD["kvm-amd.ko (AMD SVM)"]
    EFER["set EFER.SVME: enable SVM"]
    VMRUN["VMRUN: world switch into guest"]
    VMEXIT_A["#VMEXIT -> EXITCODE in VMCB"]
    VMMCALL["VMMCALL: guest hypercall"]
  end
  API --> INTEL
  API --> AMD
  VMXON --> VMLAUNCH --> VMEXIT_I
  EFER --> VMRUN --> VMEXIT_A

The two vendor extensions sit behind one common KVM core. What it shows: Intel and AMD expose the same conceptual operations — turn the extension on, enter a guest, run until a trap, take the exit, service a hypercall — but through entirely different instructions (VMXON/VMLAUNCH/VMCALL versus EFER.SVME/VMRUN/VMMCALL). The insight to take: the two extensions are isomorphic at the level of “what virtualization needs,” but disjoint at the level of “what opcode does it,” which is precisely why KVM abstracts them behind struct kvm_x86_ops and ships two distinct kernel modules rather than one.

Intel VMX — VMXON, VMLAUNCH/VMRESUME, VMCALL

Intel’s VMX introduces a small family of new instructions. The Intel Software Developer’s Manual (SDM) Volume 3C enumerates them: VMPTRLD, VMPTRST, VMCLEAR, VMREAD, VMWRITE, VMCALL, VMLAUNCH, VMRESUME, VMXOFF, VMXON, INVEPT, INVVPID, and VMFUNC (Wikipedia: x86 virtualization, citing SDM Vol 3C). The lifecycle splits into three phases: arming the processor, entering/leaving guests, and disarming.

Arming with VMXON. Before any guest can run, the logical CPU must enter VMX operation — the SDM’s term: “Software enters VMX operation by executing a VMXON instruction” (SDM Vol 3C §23). Three preconditions must hold. First, software sets the VMXE bit (bit 13) in control register CR4. Second, the BIOS/firmware-locked MSR IA32_FEATURE_CONTROL (MSR 0x3A, called MSR_IA32_FEAT_CTL in the kernel) must have its lock bit set and the appropriate VMX-enable bit set; if firmware locks this MSR with VMX disabled, VMXON faults. Third, software hands VMXON the physical address of a 4 KB-aligned, naturally page-sized VMXON region initialized with the VMCS revision identifier. KVM does exactly this in kvm_cpu_vmxon():

cr4_set_bits(X86_CR4_VMXE);                  /* CR4.VMXE = 1 */
asm goto("1: vmxon %[vmxon_pointer]\n\t"     /* VMXON <phys addr of VMXON region> */
         _ASM_EXTABLE(1b, %l[fault])

(from arch/x86/kvm/vmx/vmx.c, v6.12). The _ASM_EXTABLE entry installs a kernel exception-table fixup so that if VMXON faults (e.g. because firmware disabled VMX), control jumps to a fault label instead of crashing — KVM then logs "VMXON faulted, MSR_IA32_FEAT_CTL (0x3a) = 0x%llx", exposing the real cause. The per-CPU VMXON region is the per-cpu vmxarea pointer (static DEFINE_PER_CPU(struct vmcs *, vmxarea);), allocated by alloc_kvm_area(). The wrapping function vmx_enable_virtualization_cpu() first checks whether CR4.VMXE is already set and returns -EBUSY if so (you cannot double-arm a CPU).

Entering a guest with VMLAUNCH / VMRESUME. Once in VMX root operation, the hypervisor makes a VMCS current with VMPTRLD and then performs a VM entry. Two instructions do this, and which one you must use depends on the launch state of the VMCS: VMLAUNCH is used for the first entry into a freshly cleared VMCS (it transitions the VMCS launch state from “clear” to “launched”); VMRESUME is used for every subsequent entry into an already-launched VMCS. Using the wrong one faults. KVM tracks this with a single boolean — loaded_vmcs->launched — and computes a run flag accordingly:

if (vmx->loaded_vmcs->launched)
    flags |= VMX_RUN_VMRESUME;

(from __vmx_vcpu_run_flags(), arch/x86/kvm/vmx/vmx.c, v6.12). The assembly entry routine then branches on that flag bit:

bt   $VMX_RUN_VMRESUME_SHIFT, %ebx   ; test the VMRESUME flag bit
jnc  .Lvmlaunch                       ; if clear -> first entry -> VMLAUNCH
.Lvmresume:
    vmresume
    jmp .Lvmfail
.Lvmlaunch:
    vmlaunch
    jmp .Lvmfail

(from arch/x86/kvm/vmx/vmenter.S, v6.12). The remarkable part is the control-flow “magic” after a successful entry: execution does not fall through to .Lvmfail. The comment in the source says it plainly — “After a successful VMRESUME/VMLAUNCH, control flow ‘magically’ resumes below at vmx_vmexit due to the VMCS HOST_RIP setting.” The CPU has loaded the host instruction pointer from the VMCS HOST_RIP field, so when the guest later triggers a VM exit, the silicon resumes the host at the vmx_vmexit label, not at the instruction after vmlaunch. The jmp .Lvmfail lines are reached only when the entry instruction fails to enter the guest at all (a malformed VMCS), which is a different and rarer error path than a clean VM exit.

The guest-initiated hypercall with VMCALL. The guest, running in non-root operation, can voluntarily call into the hypervisor with VMCALL. The SDM describes it: it “allows a guest in VMX non-root operation to call the VMM for service. A VM exit occurs, transferring control to the VMM.” On the KVM side, the exit is reported as EXIT_REASON_VMCALL and dispatched to kvm_emulate_hypercall() (the same handler the AMD side reuses) — this is how paravirtual features and KVM hypercalls are invoked from inside a guest.

Disarming with VMXOFF. Tearing down leaves VMX operation. KVM’s kvm_cpu_vmxoff() executes the instruction and clears CR4.VMXE:

asm goto("1: vmxoff\n\t" _ASM_EXTABLE(1b, %l[fault]) ::: "cc", "memory" : fault);
cr4_clear_bits(X86_CR4_VMXE);

(from arch/x86/kvm/vmx/vmx.c, v6.12; comment: “Eat all faults as all other faults on VMXOFF faults are mode related”). Before this, vmx_disable_virtualization_cpu() calls vmclear_local_loaded_vmcss() so no VMCS is left dangling as current on the CPU. There is also an emergency path, vmx_emergency_disable_virtualization_cpu(), used during a kernel crash/kexec, which walks loaded_vmcss_on_cpu and vmcs_clears each one so the next kernel inherits a clean state.

AMD SVM — EFER.SVME, VMRUN, VMSAVE/VMLOAD, VMMCALL, VMEXIT

AMD’s SVM exposes a different and somewhat smaller instruction set: VMRUN, VMLOAD, VMSAVE, CLGI, VMMCALL, INVLPGA, SKINIT, and STGI (Wikipedia: x86 virtualization). The design is built around a single all-in-one “world switch” instruction, VMRUN, rather than Intel’s launch/resume pair.

Enabling with EFER.SVME. AMD does not have a VMXON instruction. Instead, SVM is enabled by setting the SVME bit in the Extended Feature Enable Register MSR — EFER.SVME — and registering a host state-save area via the VM_HSAVE_PA MSR. KVM’s svm_enable_virtualization_cpu() does both:

rdmsrl(MSR_EFER, efer);
if (efer & EFER_SVME)
    return -EBUSY;                              /* already enabled */
...
wrmsrl(MSR_EFER, efer | EFER_SVME);             /* enable SVM */
wrmsrl(MSR_VM_HSAVE_PA, sd->save_area_pa);      /* host state-save area */

(from arch/x86/kvm/svm/svm.c, v6.12). The VM_HSAVE_PA area is where the processor automatically stashes a subset of host state on VMRUN and restores it on #VMEXIT. Note the analogy: EFER.SVME is to SVM what CR4.VMXE + VMXON is to VMX — the “arm the CPU” step. KVM identifies SVM-capable CPUs with X86_MATCH_FEATURE(X86_FEATURE_SVM, NULL) and labels the module MODULE_DESCRIPTION("KVM support for SVM (AMD-V) extensions").

Entering a guest with VMRUN. A single instruction performs the entire host→guest world switch. VMRUN takes the physical address of the 4 KB-aligned Virtual Machine Control Block (VMCB) in rAX, saves the host’s processor state to the VM_HSAVE_PA area, loads the guest state from the VMCB, and begins executing in guest mode (AMD’s term, the SVM analogue of Intel non-root operation — see Root Mode and Non-Root Mode). KVM’s assembly:

mov SVM_current_vmcb(%_ASM_DI), %_ASM_AX
mov KVM_VMCB_pa(%_ASM_AX), %_ASM_AX   ; rAX = guest VMCB physical address
...
sti
3:  vmrun %_ASM_AX                    ; world-switch into the guest
4:  cli                              ; on #VMEXIT, resume right here

(from arch/x86/kvm/svm/vmenter.S, v6.12). Unlike Intel’s “resume at HOST_RIP” trick, AMD’s #VMEXIT returns control to the instruction immediately after VMRUN — which is why the cli at label 4: is the first host instruction to run after a guest exit. The sti/cli bracketing manages the interrupt window across the switch.

The VMSAVE/VMLOAD complication. This is the most important mechanical difference from Intel, and a frequent source of confusion. VMRUN and #VMEXIT save/restore only a core set of state automatically. A second, larger set of system state — FS, GS, TR, LDTR, KernelGsBase, the SYSCALL MSRs (STAR, LSTAR, CSTAR, SFMASK), and the SYSENTER MSRs — is not handled by VMRUN alone and must be saved/restored explicitly with the VMSAVE and VMLOAD instructions (AMD APM Vol 2 Ch. 15). KVM does this around every entry. Host state is saved before the switch in svm_prepare_switch_to_guest() with vmsave(sd->save_area_pa); the source comment is explicit: “Save additional host state that will be restored on VMEXIT (sev-es) or subsequent vmload of host save area.” The vmenter assembly then vmloads the guest’s extra state before VMRUN and vmsaves it back after, then vmloads the host’s percpu state:

mov SVM_vmcb01_pa(%_ASM_DI), %_ASM_AX
1:  vmload %_ASM_AX                   ; load guest FS/GS/TR/LDTR/MSRs
...
5:  vmsave %_ASM_AX                   ; save them back to the guest VMCB
pop %_ASM_AX
7:  vmload %_ASM_AX                   ; reload the host's percpu state

(from arch/x86/kvm/svm/vmenter.S, v6.12). Intel’s VMCS, by contrast, has dedicated host-state and guest-state fields that the hardware loads/stores on every VM entry/exit, so VMX has no VMSAVE/VMLOAD analogue — it is “batteries included” where SVM is “bring the extra registers yourself.”

The guest hypercall VMMCALL and #VMEXIT. The guest calls the hypervisor with VMMCALL (the SVM analogue of Intel’s VMCALL). KVM maps it in the dispatch table identically to the Intel side: [SVM_EXIT_VMMCALL] = kvm_emulate_hypercall. More generally, every exit is reported through an EXITCODE field written by the processor into the VMCB control area on #VMEXIT; KVM looks it up in svm_exit_handlers[] via svm_invoke_exit_handler(). AMD’s intercept model is configured as a bit-vector of intercept codes in the VMCB (INTERCEPT_VMRUN, INTERCEPT_VMMCALL, INTERCEPT_CPUID, INTERCEPT_HLT, INTERCEPT_INVLPG, …; from arch/x86/include/asm/svm.h), which is conceptually the same as Intel’s VM-execution control fields in the VMCS but laid out differently.

Disabling SVM. KVM’s kvm_cpu_svm_disable() zeroes the host save area and clears EFER.SVME, executing STGI first to restore the global interrupt flag:

wrmsrl(MSR_VM_HSAVE_PA, 0);
rdmsrl(MSR_EFER, efer);
if (efer & EFER_SVME) {
    stgi();
    wrmsrl(MSR_EFER, efer & ~EFER_SVME);
}

(from arch/x86/kvm/svm/svm.c, v6.12). STGI/CLGI (Set/Clear Global Interrupt Flag) are SVM-only instructions that gate all interrupts globally — a finer hammer than the ordinary IF flag, used to make the host’s post-exit critical section uninterruptible.

Side-by-Side: the Same Job, Different Opcodes

ConceptIntel VMX (VT-x)AMD SVM (AMD-V)
Arm the CPUVMXON after CR4.VMXE=1, IA32_FEATURE_CONTROL unlockedset EFER.SVME, set VM_HSAVE_PA
Per-CPU regionVMXON region (per-cpu vmxarea)host state-save area (save_area_pa)
Control structureVMCS, accessed via VMREAD/VMWRITEVMCB, ordinary memory reads/writes
Enter guestVMLAUNCH (first) / VMRESUME (subsequent)VMRUN (always)
Extra system statehandled by VMCS host/guest fieldsexplicit VMSAVE / VMLOAD
Guest hypercallVMCALL (EXIT_REASON_VMCALL)VMMCALL (SVM_EXIT_VMMCALL)
Exit reasonVMCS exit-reason fieldVMCB EXITCODE field
Post-exit resume pointat VMCS HOST_RIP (vmx_vmexit label)instruction after VMRUN
Global IRQ gate(none; uses IF / event windows)STGI / CLGI
DisarmVMXOFF, clear CR4.VMXEclear EFER.SVME, zero VM_HSAVE_PA
Linux modulekvm-intel (X86_FEATURE_VMX)kvm-amd (X86_FEATURE_SVM)

The single most consequential difference is the launch/resume distinction (VMX) versus the all-in-one VMRUN (SVM) and the corresponding VMCS auto-save versus VMSAVE/VMLOAD manual save. Intel front-loads complexity into the VMCS (many fields, accessed only through VMREAD/VMWRITE so the CPU can cache the structure in an implementation-defined format); AMD keeps the VMCB as a plain in-memory struct you can read with normal loads but makes you manage the extra register file by hand.

Failure Modes and Common Misunderstandings

“Guest ring 0 runs in ring 1.” No. A persistent myth (carried over from pre-hardware paravirtualization tricks like Xen’s ring deprivileging) is that VT-x/SVM run the guest kernel in ring 1. They do not. The guest runs in non-root/guest mode with a complete, genuine ring 0–3; privilege is enforced by the new orthogonal mode dimension, not by shuffling rings. This is the core point of Root Mode and Non-Root Mode, and getting it wrong leads to a fundamentally incorrect mental model of how exits are triggered.

VMXON fails with MSR_IA32_FEAT_CTL showing VMX disabled. The classic symptom is kvm-intel: VMX (outside TXT) disabled by BIOS or a VMXON faulted log. The cause is firmware that locked IA32_FEATURE_CONTROL with the VMX-enable bit clear — VT-x is physically present but administratively disabled in BIOS/UEFI. KVM cannot override a locked MSR; the fix is a firmware setting, not a kernel one.

Using VMLAUNCH on an already-launched VMCS (or VMRESUME on a clear one). This faults. KVM avoids it by tracking loaded_vmcs->launched; bespoke hypervisors that mismanage this flag see VM-entry failures. There is no analogue on AMD because VMRUN is stateless in this respect.

Forgetting VMSAVE/VMLOAD on AMD. A naive SVM hypervisor that only issues VMRUN will silently corrupt FS/GS/LDTR/syscall-MSR state across the switch, because those are not part of the VMRUN auto-saved set. The symptom is bizarre host or guest misbehavior after the first entry — a class of bug that simply cannot occur on Intel because the VMCS covers that state. Google Project Zero’s “An EPYC escape” KVM-breakout write-up is a good real-world tour of how subtle SVM state handling can be (Project Zero, 2021).

Uncertain

Verify: the exact membership of the set of state that AMD VMRUN/#VMEXIT save automatically versus what requires VMSAVE/VMLOAD (this note lists FS, GS, TR, LDTR, KernelGsBase, STAR/LSTAR/CSTAR/SFMASK, SYSENTER_CS/ESP/EIP). Reason: drawn from the AMD APM Vol 2 Ch. 15 / SVM reference PDF and a secondary summary; the PDF was fetched but the precise per-field VMRUN-vs-VMSAVE split was not transcribed table-by-table. To resolve: cross-check against the “VMSAVE and VMLOAD Instructions” and “Saved State” subsections of AMD64 APM Vol 2 §15.5–15.6. uncertain

Alternatives and When Each Applies

The two extensions are not a choice a hypervisor author makes — they are dictated by the CPU vendor. Code must support both, which is precisely why KVM splits along kvm-intel/kvm-amd behind the common kvm.ko and struct kvm_x86_ops. The historical alternatives that VMX/SVM replaced are covered in Why x86 Needed Hardware Virtualization: dynamic binary translation (VMware’s original approach, rewriting guest kernel code on the fly to trap sensitive instructions) and paravirtualization (Xen’s original approach, patching the guest kernel to call the hypervisor explicitly). Both impose either runtime overhead or guest-source modification; hardware extensions let an unmodified guest run at near-native speed, which is why every modern hypervisor uses them. For the wider taxonomy see Hypervisor Types and Virtualization Models.

Production Notes

On Linux, the presence of these extensions surfaces as CPU flags in /proc/cpuinfo: vmx for Intel VT-x and svm for AMD-V (Wikipedia: x86 virtualization). KVM’s vendor module is auto-loaded by the device-table match (MODULE_DEVICE_TABLE(x86cpu, …)) — kvm-intel matches X86_FEATURE_VMX, kvm-amd matches X86_FEATURE_SVM — so a machine loads exactly one of the two. Both vendor modules MODULE_AUTHOR("Qumranet"), a fingerprint of KVM’s origin at the startup Qumranet (acquired by Red Hat in 2008). Nested virtualization (a guest hypervisor running its own guests) requires KVM to emulate the vendor extension for the L1 guest — merging the L1’s VMCS/VMCB with KVM’s own — which is why nesting is both vendor-specific and slow; see Nested Virtualization. Because the extensions are per-logical-CPU state, KVM arms and disarms VMX/SVM on every CPU as the module loads/unloads and around CPU hotplug, and provides the emergency-disable path so a kexec/crash kernel never inherits a CPU stuck in VMX/SVM operation.

See Also

Graph View

Backlinks