vault

Nested Virtualization

13 min read

Nested Virtualization

Nested virtualization is running a hypervisor inside a virtual machine — letting a guest, which is itself running on a hypervisor, run its own guests. The motivating scenario is everywhere now: a cloud VM that needs to run KVM or Hyper-V, a Windows guest whose security features (Virtualization-Based Security) require a hypervisor, Windows Subsystem for Linux 2 (WSL2) inside a Windows VM, or a CI runner that builds and tests VM images. The difficulty is fundamental: a CPU offers only one hardware virtualization layer. Intel VT-x (Virtual Machine Extensions, VMX) and AMD AMD-V (Secure Virtual Machine, SVM) let a hypervisor run guests, but those guests cannot themselves execute VMX/SVM instructions — “in VMX, guests cannot use VMX instructions” (nested-vmx.rst, v6.12). KVM’s nested-virtualization feature closes this gap by emulating the virtualization hardware for the guest hypervisor, multiplexing the single physical layer by merging the guest hypervisor’s control structures with its own. The design originates in IBM’s 2010 OSDI paper “The Turtles Project: Design and Implementation of Nested Virtualization” (Ben-Yehuda et al., 2010), and is enabled by default in mainline Linux since v4.20. This note is pinned to the 6.12 LTS kernel (released 2024-11-17).

The L0 / L1 / L2 Model

The terminology is a vertical stack of trust and privilege. The bare-metal hypervisor — KVM running directly on the physical CPU — is L0. A guest of L0 that is itself a hypervisor is L1 (the guest hypervisor). A guest that L1 runs is L2 (the nested guest). The crucial fact is that L1’s virtualization instructions do not run on hardware: when L1 executes a VMX or SVM instruction, the physical CPU traps to L0, and L0 emulates the instruction. Only L0 ever truly drives the hardware virtualization extensions. The nested-vmx documentation names the three levels precisely: “the host (KVM), which we call L0, the guest hypervisor, which we call L1, and its nested guest, which we call L2.”

flowchart TB
  subgraph L2box["L2 — nested guest"]
    L2["Guest OS + apps"]
  end
  subgraph L1box["L1 — guest hypervisor (e.g. KVM/Hyper-V in a VM)"]
    L1["Hypervisor code<br/>builds vmcs12 for L2"]
  end
  subgraph L0box["L0 — bare-metal KVM on physical CPU"]
    L0["KVM emulates VMX/SVM for L1<br/>builds vmcs02 to actually run L2"]
  end
  HW["Physical CPU: VT-x / AMD-V, EPT/NPT"]
  L2 -->|"privileged op → VM exit"| L0
  L1 -->|"VMX/SVM instr → VM exit (trap)"| L0
  L0 -->|"merge vmcs01 + vmcs12 → vmcs02"| HW
  HW -->|"runs L2 directly in guest mode"| L2

The nesting stack and the structure merge. What it shows: L2 runs in hardware guest mode but on a control structure (vmcs02) that L0 builds by merging L1’s intent (vmcs12) with L0’s own requirements (vmcs01). Both L1’s virtualization instructions and L2’s privileged operations trap down to L0. The insight to take: there is only ever one hardware virtualization layer — L0 owns it. “Nesting” is L0 cleverly time-sharing that single layer between running L1 and running L2, reconstructing the right control structure each time.

The Three VMCSes — vmcs01, vmcs12, vmcs02

The mechanism is clearest on Intel. A Virtual Machine Control Structure (VMCS) is the in-memory block the CPU uses to hold an entire guest’s CPU state and the host’s control settings across a VM entry/exit (see VMCS and VMCB (Virtual Machine Control Structure)). Nesting involves three of them, and the documentation names them exactly (nested-vmx.rst, v6.12):

  • vmcs01 — “the VMCS that L0 built for L1.” This is the ordinary VMCS L0 uses to run L1 as a normal guest.
  • vmcs12 — “the VMCS that L1 builds for L2.” L1, believing it is a real hypervisor, constructs this to describe how it wants L2 to run. To L0 this is just data in L1’s memory; to L1 it is an opaque region accessed only through VMREAD/VMWRITE. In KVM’s source it is struct vmcs12.
  • vmcs02 — “the VMCS which L0 builds to actually run L2.” This is the real, hardware-loaded VMCS used when L2 executes. L0 synthesizes it.

The heart of nesting is the merge that produces vmcs02. When L1 issues VMLAUNCH/VMRESUME to run L2, the CPU traps to L0, and L0 calls prepare_vmcs02 (nested.c, v6.12). The function’s own comment states the principle:

/*
 * prepare_vmcs02 is called when the L1 guest hypervisor runs its nested
 * L2 guest. L1 has a vmcs for L2 (vmcs12), and this function "merges" it
 * with L0's requirements for its guest (a.k.a. vmcs01), so we can run the L2
 * guest in a way that will both be appropriate to L1's requests, and our
 * needs.
 */

The merge is not a copy — it is a careful combination of two policies. Consider control fields that decide which events cause a VM exit. L1 wants L2 to exit on certain events (its own hypervisor logic); L0 wants L2 to exit on other events (so L0 retains control). The merged vmcs02 must trap on the union. The source shows this for the exception bitmap and control-register masks:

/* EXCEPTION_BITMAP and CR0_GUEST_HOST_MASK should basically be the
 * bitwise-or of what L1 wants to trap for L2, and what we want to
 * trap. ...
 */
vmx_update_exception_bitmap(vcpu);
vcpu->arch.cr0_guest_owned_bits &= ~vmcs12->cr0_guest_host_mask;
vmcs_writel(CR0_GUEST_HOST_MASK, ~vcpu->arch.cr0_guest_owned_bits);

This bitwise-or is the recurring theme: when an L2 event fires, hardware exits to L0; L0 inspects the cause and decides whether this exit “belongs to” L1 (in which case L0 reflects it up — synthesizing an L2→L1 exit by writing the relevant vmcs12 fields and resuming L1 as if its own hardware had trapped) or whether L0 handles it directly (e.g. a host page fault L1 should never see). This reflect-or-handle decision on every L2 exit is the dominant cost of nesting.

Shadow VMCS — Avoiding a Trap on Every VMREAD/VMWRITE

A naive emulation traps to L0 on every VMREAD and VMWRITE L1 executes — and a hypervisor builds a VMCS by issuing dozens of VMWRITEs per entry. That is brutally slow. Intel added a hardware feature, VMCS shadowing, that lets L1’s VMREAD/VMWRITE instructions read and write a shadow VMCS directly in non-root mode without trapping to L0. KVM enables it by default (enable_shadow_vmcs = 1, module parameter enable_shadow_vmcs).

The mechanism uses two bitmaps — vmx_vmread_bitmap and vmx_vmwrite_bitmap — that designate which VMCS fields L1 may touch without a trap. KVM clears the bits for fields it can safely let L1 access directly and leaves set (trapping) the bits for fields that need L0 intervention. L1’s accesses then go to a hardware shadow VMCS, and KVM periodically synchronizes that shadow with the software vmcs12 it actually reasons about. The two sync directions are copy_shadow_to_vmcs12 and copy_vmcs12_to_shadow (nested.c, v6.12):

static void copy_shadow_to_vmcs12(struct vcpu_vmx *vmx)
{
    struct vmcs *shadow_vmcs = vmx->vmcs01.shadow_vmcs;
    struct vmcs12 *vmcs12 = get_vmcs12(&vmx->vcpu);
    ...
    vmcs_load(shadow_vmcs);
    for (i = 0; i < max_shadow_read_write_fields; i++) {
        field = shadow_read_write_fields[i];
        val = __vmcs_readl(field.encoding);
        vmcs12_write_any(vmcs12, field.encoding, field.offset, val);
    }
    vmcs_clear(shadow_vmcs);
    vmcs_load(vmx->loaded_vmcs->vmcs);
}

Reading it: KVM loads the hardware shadow VMCS, iterates the list of shadowable read/write fields, copies each from the hardware shadow into the software vmcs12, then restores the active VMCS. The reverse function pushes vmcs12 values into the shadow before resuming L1 so its VMREADs see fresh data. The net effect: L1 executes hundreds of VMREAD/VMWRITEs entirely in guest mode, and KVM pays the trap cost only at the boundaries (when L1 actually launches/resumes L2), not per field access. This single optimization is what makes nested KVM tolerable rather than merely possible.

Nested EPT and NPT — Three Levels of Address Translation

Memory virtualization compounds under nesting. A non-nested guest already uses two-dimensional paging — guest-virtual → guest-physical (the guest’s own page tables) → host-physical (the hypervisor’s Nested Page Tables, NPT, on AMD). Nesting adds a third logical translation: L2-virtual → L2-physical → L1-physical → L0-physical. But the hardware MMU still performs only two dimensions in a single walk. KVM bridges the gap by composing the page tables in software: it takes L1’s EPT (the EPT12, mapping L2-physical to L1-physical, which L1 built for L2) and combines it with L0’s own mapping (L1-physical to L0-physical) to produce a single shadow EPT (EPT02) that the hardware actually loads, mapping L2-physical straight to L0-physical.

KVM sets up a dedicated MMU context for this. When vmcs12 requests EPT for L2, prepare_vmcs02 calls nested_ept_init_mmu_context (nested.c, v6.12):

static void nested_ept_init_mmu_context(struct kvm_vcpu *vcpu)
{
    WARN_ON(mmu_is_nested(vcpu));
    vcpu->arch.mmu = &vcpu->arch.guest_mmu;
    nested_ept_new_eptp(vcpu);
    vcpu->arch.mmu->get_guest_pgd     = nested_ept_get_eptp;
    vcpu->arch.mmu->inject_page_fault = nested_ept_inject_page_fault;
    vcpu->arch.mmu->get_pdptr         = kvm_pdptr_read;
    vcpu->arch.walk_mmu              = &vcpu->arch.nested_mmu;
}

It swaps in a separate guest_mmu whose get_guest_pgd returns L1’s EPT pointer (nested_ept_get_eptp) and whose page-fault injection (nested_ept_inject_page_fault) knows to surface EPT violations to L1 rather than L0. The nested_mmu is used to walk L2’s translations. When an L2 access misses, the resulting EPT violation either reflects up to L1 (if L1’s EPT12 lacks the mapping) or is fixed by L0. Composing EPT12 with L0’s mapping in software is expensive, which is why nested memory-heavy workloads suffer most.

The AMD Side — Nested SVM with vmcb01/vmcb02/vmcb12

AMD’s design mirrors Intel’s with different names. The control structure is the Virtual Machine Control Block (VMCB), and KVM uses an analogous triplet: vmcb01 (L0’s VMCB for L1), vmcb12 (L1’s VMCB for L2, in L1’s memory), and vmcb02 (the real VMCB L0 builds to run L2). The merge functions are nested_vmcb02_prepare_save and nested_vmcb02_prepare_control (svm/nested.c, v6.12). AMD’s analogue of the trap-merge is recalc_intercepts, which ORs L1’s desired intercepts with L0’s. For memory, nested_svm_init_mmu_context sets up nested NPT (nested_npt_enabled(svm)), the SVM counterpart to nested EPT. One architectural difference: AMD’s VMCB is not opaque the way Intel’s VMCS is — L1 can read/write VMCB fields with ordinary memory accesses rather than special instructions — so AMD has no exact equivalent of Intel’s “shadow VMCS instruction-trapping” problem, though the merge cost on VMRUN remains.

Configuration and Enablement

Nesting is on by default in modern kernels but is exposed as a module parameter so it can be disabled. On Intel the module is kvm-intel; on AMD it is kvm-amd. In the v6.12 source the Intel default is static bool __read_mostly nested = 1; and the AMD default is static int nested = true; (vmx.c, svm.c). To check or set it:

# Check whether nesting is enabled (Y = yes)
cat /sys/module/kvm_intel/parameters/nested      # Intel
cat /sys/module/kvm_amd/parameters/nested        # AMD
 
# Enable explicitly (older kernels, or after disabling):
# add to /etc/modprobe.d/kvm.conf, then reload the module
options kvm-intel nested=1
options kvm-amd   nested=1

The nested-vmx documentation confirms the default and the older-kernel path: “The nested VMX feature is enabled by default since Linux kernel v4.20. For older Linux kernel, it can be enabled by giving the ‘nested=1’ option to the kvm-intel module.”

A second, easily-missed requirement is on the L1 guest’s CPU model. The hardware virtualization feature must be advertised to L1, but QEMU’s default emulated CPU (qemu64) does not expose VMX/SVM. The documentation spells out the fix: pass -cpu host (give L1 all the physical CPU’s features) or -cpu qemu64,+vmx (add just the VMX feature). Without this, L1 will not even see hardware virtualization and will refuse to act as a hypervisor (or fall back to slow software emulation).

Uncertain

Verify: that the sysfs module-parameter file is /sys/module/kvm_intel/parameters/nested (underscore) while the modprobe/insmod name is kvm-intel (hyphen). Reason: Linux normalizes hyphens to underscores in /sys/module/ paths, so the two forms coexist, but I did not directly observe both on a 6.12 system during this task. To resolve: run ls /sys/module/ | grep kvm and modinfo kvm-intel on a 6.12 host. uncertain

Why It’s Clever but Slow — and the Failure Modes

The cost model follows directly from the mechanism. Every L2 VM exit traps to L0, which must decide whether to handle it or reflect it to L1; reflecting means synthesizing an L2→L1 exit by writing vmcs12 fields and resuming L1. So an exit that a single-level guest handles in one transition can, under nesting, cost several transitions (L2→L0, then L0→L1, and L1’s eventual resume of L2 is L1→L0→L2). The Turtles paper measured this and concluded that the dominant overhead is exactly this exit multiplication, which is why eliminating exits matters even more under nesting — see the cross-cutting “eliminate the VM exit” theme in Linux Virtualization MOC. Shadow VMCS attacks the VMREAD/VMWRITE multiplication; nested EPT composition attacks the page-fault multiplication; but the irreducible reflect-or-handle decision remains.

Practical failure modes: (1) L1 sees no virtualization — almost always the missing -cpu host/+vmx flag, or nested=0 on L0. (2) L2 boots but is extremely slow — usually memory-intensive workloads paying the nested-EPT composition tax, or a CPU without shadow-VMCS support forcing per-VMREAD traps. (3) Migration of an L1 that is running L2 is notoriously fragile, because L0 must capture and restore the full nested state (vmcs12 contents, nested MMU); KVM’s nested-state save/restore ioctls exist for this but version-skew between source and destination KVM can break it — the documentation warns that changing struct vmcs12 “can break live migration across KVM versions,” which is why VMCS12_REVISION must be bumped on any layout change. (4) Triple-faults or guest hangs after a KVM update historically traced to subtle merge bugs where L0 failed to OR in an intercept L1 needed, letting L2 escape L1’s control.

Real Uses

Nesting is no longer exotic. Cloud-in-cloud: developers run KVM/QEMU or minikube inside a cloud VM for testing — AWS, Azure, and GCP all expose nested virtualization on specific instance types. Nested Hyper-V and WSL2: Windows’ security features (Credential Guard, Virtualization-Based Security) and WSL2 itself run a Hyper-V hypervisor; to run such a Windows image as an L1 guest on KVM requires nested SVM/VMX (Microsoft nested virtualization docs, WSL overview). CI runners: continuous-integration systems that build and boot VM images, or test hypervisor changes, need an L1 hypervisor inside their ephemeral build VMs. Kata Containers / KubeVirt style workloads that put VMs inside cloud VMs also lean on nesting. In each case the value is the same: a layer that needs “real” hardware virtualization can get a faithful (if slower) emulation of it one level up.

See Also

Graph View

Backlinks