vault

Why x86 Needed Hardware Virtualization

16 min read

Why x86 Needed Hardware Virtualization

The Popek-Goldberg theorem of 1974 gives a clean test for whether a CPU can be virtualized by the classical trap-and-emulate technique: every sensitive instruction (one that touches or observes the machine’s real configuration) must also be privileged (one that traps when run unprivileged). The 32-bit Intel x86 architecture (IA-32) flunks this test. In their definitive 2000 analysis of the Intel Pentium, Robin and Irvine examined “all of the approximately 250 instructions” and found that seventeen of them are sensitive yet not privileged: when a deprivileged guest executes them, the CPU neither traps nor refuses — it silently does the wrong thing (Robin & Irvine 2000, §3). Those seventeen holes meant a hypervisor could never see, and so never correctly emulate, those operations. The result was a decade of ingenious software workarounds — VMware’s binary translation and Xen’s paravirtualization — until Intel VT-x and AMD-V closed the gap in hardware (2005–2006) by adding a whole new privilege axis.

Mental Model: A CPU That Lies About Its Own Privilege

The cleanest way to think about the problem is that on x86 a deprivileged guest kernel can ask the hardware about its own state and get an honest answer it was supposed to be lied to about. A VMM works by running the guest at a lower privilege than the guest expects and maintaining the illusion that the guest is still the most-privileged software on the machine. That illusion survives only if every instruction that could reveal or alter the true privilege configuration is intercepted. On x86, several such instructions execute happily in user mode and return the real state — puncturing the illusion (an equivalence violation) — while others that should change protected state silently fail to do so (a resource-control violation).

x86 has four privilege levels called rings, numbered 0 (most privileged, where operating systems run) through 3 (least privileged, where applications run). A hypervisor must take ring 0 for itself and push the guest down — but it cannot push the guest all the way to ring 3, because the guest’s own applications need ring 3 and the guest kernel needs to be protected from them. So the guest kernel is squeezed into ring 1 (or, on 64-bit x86 where rings 1 and 2 are largely unusable, into ring 3 alongside applications). This squeezing has a name and a family of consequences.

flowchart TB
  subgraph NATIVE["Native (no hypervisor)"]
    R0n["Ring 0: Guest kernel<br/>(believes it owns the machine)"]
    R3n["Ring 3: Applications"]
    R0n --> R3n
  end
  subgraph VIRT["Trap-and-emulate attempt on x86"]
    R0v["Ring 0: HYPERVISOR"]
    R1v["Ring 1: Guest kernel (deprivileged)"]
    R3v["Ring 3: Applications"]
    R0v --> R1v --> R3v
    LEAK["17 sensitive instructions<br/>(SGDT, PUSHF/POPF, LAR...)<br/>run in ring 1 with NO trap<br/>-> hypervisor never sees them"]
    R1v -.->|"silently leaks/ignores"| LEAK
  end

Ring deprivileging on x86 and where it leaks. What it shows: natively the guest kernel runs in ring 0; under a hypervisor it is demoted to ring 1 while the hypervisor takes ring 0. The insight: the demotion only works if every privileged operation traps at ring 1 — but seventeen sensitive x86 instructions execute at ring 1 (or ring 3) without trapping, either silently failing or silently returning the hypervisor’s real state. Those non-trapping instructions are the reason classical trap-and-emulate is impossible on stock IA-32.

The Four Obstacles

The Intel architects who later built VT-x catalogued the x86 virtualization holes into four named obstacles (Uhlig, Neiger et al. 2005). They are worth naming precisely because the hardware fix is organized around eliminating each one.

Ring deprivileging is the demotion itself: the guest OS, written to run at ring 0, must instead run at a less-privileged ring so the VMM can keep ring 0. This is necessary but not sufficient — it is what enables trap-and-emulate, but it also causes the next three problems.

Ring aliasing is the failure that follows when “software is run at a privilege level other than the level for which it was written” (x86 virtualization, Wikipedia). A guest can discover its true ring. The most cited example: on x86 the code-segment register %cs has the current privilege level (CPL) encoded in its low two bits, and an unprivileged guest can simply read %cs (via a PUSH %cs or a far-call/return sequence) and see “ring 1” or “ring 3” where it expected “ring 0.” The illusion shatters: the guest now knows it is virtualized, violating equivalence.

Address-space compression is the problem that the VMM must live somewhere in the address space the guest also wants to use, and the guest must be prevented from reading or corrupting the VMM’s region — yet the guest believes it owns the entire address space. This is why Xen reserves the top of the address range for itself (below) and why hardware later added separate guest/host address-space control.

Non-faulting access to privileged state is the deepest hole and the direct Popek-Goldberg violation: instructions that read or write privileged machine state “without faulting when the processor operates with insufficient privilege” — the architects’ examples are exactly SGDT, SIDT, PUSHF, and POPF (Uhlig/Neiger 2005). These are the seventeen instructions Robin & Irvine enumerated.

The Seventeen Problem Instructions

Robin & Irvine’s central finding is exact: “After examining each member of the Pentium instruction set, it was found that seventeen instructions violate Requirement 3 [the trap-on-sensitive requirement] and make the Intel processor non-virtualizable” (Robin & Irvine 2000, §3). They sort the seventeen into two families.

Uncertain

Verify: the headline count is exactly seventeen. Reason: Robin & Irvine (2000) state “seventeen instructions” in prose, and Adams & Agesen (2006) likewise reference “the seventeen problem instructions” — but the mnemonics listed below total eighteen names because some entries (e.g. PUSH/POP of segment registers, far CALL/JMP/RET, and the two MOV variants) are counted per the paper’s grouping, and secondary sources sometimes report “18.” The discrepancy is a counting convention (mnemonics vs. distinct opcodes/violations), not a factual disagreement about which instructions are problematic. To resolve: page through §3.1–§3.2 of the primary PDF and tally exactly how the paper attributes the count of seventeen to the listed mnemonics. uncertain

The first family is sensitive register instructions (violating the rule that instructions reading or changing sensitive registers must trap):

  • SGDT, SIDT, SLDT — store the Global Descriptor Table Register, Interrupt Descriptor Table Register, and Local Descriptor Table Register into a general register or memory. The x86 allows these reads at any privilege level. “Since the Intel processor only has one LDTR, IDTR, and GDTR, a problem arises when multiple operating systems try to use the same registers… if an OS in a VM uses SGDT, SLDT, or SIDT to reference the contents of the IDTR, LDTR, or GDTR, the register contents that are applicable to the host OS or Type I VMM will be given” (Robin & Irvine §3.1.1). The guest reads the host’s descriptor-table pointers — a textbook equivalence leak.
  • SMSW — store machine status word (the low bits of control register CR0). “Although this instruction only stores the machine status word, it is sensitive and unprivileged.” A guest that the VMM is running in virtual real mode could execute SMSW, see the Protection-Enable (PE) bit set, and “incorrectly see that the machine is in protected mode” (§3.1.2).
  • PUSHF / POPF — push and pop the flags register (EFLAGS). These are the famous interrupt-flag holes. The flags register holds the interrupt-enable flag (IF) and the I/O privilege level (IOPL). When POPF runs at a privilege level below IOPL, “the interrupt flag (IF) is altered only when executing at a level at least as privileged as the IOPL… an exception does not occur, but the privileged bits do not change” (Intel POPF reference). A deprivileged guest kernel trying to disable interrupts with POPF silently fails and gets no trap, so the VMM cannot even emulate the intended effect. PUSHF symmetrically lets the guest read the real flags. As Robin & Irvine put it, “the PUSHF/POPF instructions also prevent processor virtualization because they allow modification of certain bits in the EFLAGS register that control the operating mode and state of the processor” (§3.1.3).

The second family is protection-system references (instructions that consult the segment-protection machinery and thereby leak the true CPL/RPL):

  • LAR, LSL, VERR, VERW — load access rights, load segment limit, verify-for-read, verify-for-write. All four “perform the following check during their execution: (CPL → DPL) OR (RPL → DPL)” against a segment’s descriptor privilege level (DPL). Because the guest runs at CPL 3 instead of the CPL 0 it expects, the privilege comparison can give the wrong answer and “it is likely that the instruction will not execute properly” (§3.2.1).
  • POP / PUSH (of segment registers) — because the CS and SS segment registers carry the CPL in their low two bits, a process can PUSH %cs, then “examine the contents of the CS register on the stack to check its CPL. Upon finding that its CPL is not 0, the process may halt” (§3.2.2, §3.2.3). This is ring aliasing in instruction form.
  • CALL, JMP, INT n, RET (far/inter-privilege variants) — far calls, jumps, software interrupts, and far returns all perform privilege-level checks against call gates and the CPL/RPL. “Since the VM normally operates at user level (CPL 3), these checks will not work correctly when a VMOS tries to access call gates or task gates at CPL 0” (§3.2.4).
  • STR — store task register; it writes the task-state-segment selector, which contains a requested-privilege-level (RPL) field. A guest can read it and “find that it is not running at the privilege level at which it expects to run” (§3.2.5).
  • MOV (from/to segment registers) — “The MOV opcode that stores segment registers allows all six of the segment registers to be stored… This is a problem because the CS and SS registers both contain the CPL in bits 0 and 1” (§3.2.6).

The common thread of the second family is the same as ring aliasing: x86 encodes privilege in segment selectors that unprivileged code can read, so a guest can always discover its demotion.

The Pre-Hardware Workarounds

Because the architecture violates Popek-Goldberg, the only way to virtualize stock x86 before 2005 was to break one of the theorem’s assumptions. Two production approaches did so in opposite ways.

VMware: Binary Translation (keep the guest unmodified)

VMware’s approach kept the guest operating system unmodified and instead rewrote its kernel code on the fly. Their own retrospective is blunt: “Until recently, the x86 architecture has not permitted classical trap-and-emulate virtualization. Virtual Machine Monitors for x86, such as VMware Workstation and Virtual PC, have instead used binary translation of the guest kernel code” (Adams & Agesen 2006). The translator scans guest kernel instructions just before they execute and produces a safe, cached translation: innocuous instructions are copied through nearly verbatim so they still run natively (preserving efficiency), but the seventeen problem instructions are replaced with sequences that either trap into the VMM or call it directly, so the VMM regains control. Adams & Agesen describe two layers: “simple binary translation [to] eliminate privileged instruction traps and adaptive binary translation to eliminate sensitive non-privileged instruction traps” — the adaptive layer is precisely what handles the Popek-Goldberg holes that would otherwise execute silently. The cost is enormous engineering complexity (a JIT-like x86-to-x86 translator with a translation cache), but the win is transparency: any off-the-shelf operating system runs unchanged.

Xen: Paravirtualization (modify the guest to cooperate)

Xen took the opposite trade. Rather than rewrite an unmodified guest, it modified the guest’s source so the guest never issues a problematic instruction in the first place. The Xen authors state the motivation directly: “Support for full virtualization was never part of the x86 architectural design. Certain supervisor instructions must be handled by the VMM for correct virtualization, but executing these with insufficient privilege fails silently rather than causing a convenient trap” (Barham et al. 2003, §2). Their fix, paravirtualization, requires the guest kernel to be ported to a modified instruction-set-architecture-like interface:

  • The guest kernel runs in ring 1, not ring 0: “OS code typically executes in ring 0… rings 1 and 2 have not been used by any well-known x86 OS since OS/2. Any OS which follows this common arrangement can be ported to Xen by modifying it to execute in ring 1. This prevents the guest OS from directly executing privileged instructions, yet it remains safely isolated from applications running in ring 3” (§2.1.2).
  • Privileged operations become hypercalls — explicit software traps into the hypervisor analogous to system calls: “Privileged instructions are paravirtualized by requiring them to be validated and executed by Xen — this applies to operations such as installing a new page table, or yielding the processor when idle (rather than attempting to hlt it). Any guest OS attempt to directly execute a privileged instruction is failed by the processor, either silently or by taking a fault, since only Xen executes at a sufficiently privileged level” (§2.1.2).
  • Address-space compression is solved by reservation and segmentation: “The top 64MB region of each address space, which is reserved for Xen, is not accessible or remappable by guest OSes” (§2.1.1).

The cost is that you need a ported guest — the guest source must be modified, which is fine for open-source Linux/BSD but impossible for a closed operating system you cannot recompile. The payoff is near-native speed without any hardware support, and Xen reported “at most a few percent” overhead. Crucially, paravirtualization changes only the kernel: “we do not require changes to the application binary interface (ABI), and hence no modifications are required to guest applications.”

How VT-x and AMD-V Fixed It

The decisive fix was to stop fighting the architecture in software and instead extend the hardware so that x86 does satisfy Popek-Goldberg. Intel VT-x (Virtualization Technology, 2005) and AMD AMD-V/SVM (Secure Virtual Machine, 2006) added a new, orthogonal axis of operation rather than another ring.

The CPU gains two new forms of operation: VMX root operation (where the hypervisor runs) and VMX non-root operation (where the guest runs). Critically, “non-root mode grants the guest OS full ring 0–3 privilege levels, allowing the guest to execute at its intended privilege level safely” (Uhlig/Neiger 2005). This is the key: the guest kernel no longer has to be deprivileged into ring 1 or ring 3, so ring deprivileging and ring aliasing simply vanish — the guest believes it is in ring 0 because it is in ring 0, just within non-root operation. Transitions between the two forms are explicit hardware events: a VM entry drops into the guest, and a VM exit returns to the hypervisor when the guest does something the hypervisor configured to be intercepted. The entire guest CPU state is saved and restored across these transitions through an in-memory control structure, the VMCS (Virtual Machine Control Structure on Intel; VMCB, Virtual Machine Control Block, on AMD). See Intel VMX and AMD SVM and VMCS and VMCB (Virtual Machine Control Structure) for the mechanics.

The seventeen problem instructions stop being a problem in two ways. First, because the guest runs at its real intended privilege, instructions like PUSHF/POPF and the segment reads behave correctly with respect to the guest’s own ring-0 context — there is no demotion to leak. Second, for the genuinely sensitive operations the hypervisor still wants to mediate, it can configure them to cause a VM exit, so the hardware now provides exactly the trap that the classical architecture lacked. Trap-and-emulate, the technique the theorem said required sensitive ⊆ privileged, finally works on x86 — not because the old instructions changed, but because the new privilege axis made the guest’s privilege honest and gave the hypervisor a hardware interception hook. This is the foundation KVM is built on; KVM exists because VT-x/AMD-V exist.

A neat coda: Intel later added User-Mode Instruction Prevention (UMIP), a control-register-4 bit (X86_CR4_UMIP) that makes SGDT, SIDT, SLDT, SMSW, and STR finally fault when executed at CPL > 0 — directly plugging five of the original Popek-Goldberg holes at the instruction level. The v6.12 LTS kernel’s software instruction emulator enforces exactly this: in arch/x86/kvm/emulate.c, the handlers em_sgdt(), em_sidt(), and em_smsw() check (get_cr(4) & X86_CR4_UMIP) && cpl > 0 and inject a fault (Linux v6.12 emulate.c).

Uncertain

Verify: that VT-x non-root operation provides “full ring 0–3” to the guest and that this is what eliminates ring deprivileging/aliasing, as stated. Reason: drawn from a model-summarized fetch of the Uhlig/Neiger 2005 IEEE Computer paper PDF rather than line-by-line reading, and corroborated only by the secondary x86 virtualization Wikipedia article. To resolve: read the primary VT-x section of the Intel paper (or the Intel SDM Vol. 3C VMX chapter) directly and confirm the non-root-operation privilege model. uncertain

Common Misunderstandings

“x86 couldn’t be virtualized at all before VT-x.” False — it could not be virtualized by classical trap-and-emulate. VMware shipped commercial x86 virtualization in 1999 using binary translation, years before any hardware support. The theorem rules out one clean technique, not virtualization as such (see the alternatives discussion).

“VT-x made everything faster overnight.” Not initially. Adams & Agesen’s surprising 2006 result was that early hardware-assisted VMMs “often [suffer] lower performance than the pure software VMM,” because first-generation VT-x offered no help for memory-management-unit virtualization and forced an expensive VM exit on every intercepted operation, whereas a mature binary translator could avoid many traps entirely (Adams & Agesen 2006). The hardware only decisively won once NPT) and later interrupt virtualization removed the hot VM-exit sources.

“Ring 1 is where all hypervisors put the guest.” That was the paravirtualized (Xen pre-HVM) arrangement on 32-bit x86. With VT-x/AMD-V the guest runs in non-root operation at its native ring 0, and on 64-bit x86 rings 1 and 2 are largely unusable anyway, so the ring-1 trick is historical.

See Also

Graph View

Backlinks