vault

Two-Dimensional Paging (EPT and NPT)

15 min read

Two-Dimensional Paging (EPT and NPT)

Two-dimensional paging (TDP) is the hardware mechanism that lets a guest run its own page tables and lets the hypervisor independently control where guest memory actually lives — by making the CPU’s Memory-Management Unit (MMU) walk two page-table hierarchies on every Translation-Lookaside-Buffer (TLB) miss instead of one. The guest’s own tables map a guest-virtual address (GVA) to a guest-physical address (GPA), exactly as on bare metal; a second, hypervisor-owned table — Intel’s Extended Page Tables (EPT) or AMD’s Nested Page Tables (NPT), also marketed as Rapid Virtualization Indexing (RVI) — then maps that GPA to the real host-physical address (HPA). The MMU resolves GVA → GPA → HPA in a single nested traversal, so the guest never traps when it edits its own page tables (KVM mmu.rst, v6.12). This replaced the older software technique of Shadow Page Tables, trading a cheaper page-table update for a more expensive page walk — up to roughly 24 memory accesses on a full miss (Gandhi, Hill & Swift, ISCA 2016). Crucially, the EPT/NPT layer sits on top of the host’s own memory management, so guest pages remain ordinary host pages — swappable, dedup-able with KSM, and backed by transparent huge pages.

Mental Model

The cleanest way to think about TDP is two completely separate page-table owners that the silicon stitches together at walk time. The guest operating system owns the first table; it believes it is running on bare metal and manipulates its CR3 register and page tables freely. The hypervisor owns the second table (the EPT/NPT structure) and registers its root with the CPU through a dedicated pointer — the EPT Pointer (EPTP) on Intel or the nested CR3 (nCR3) in the guest’s Virtual Machine Control Block on AMD. Neither owner can see the other’s table; the MMU is the only entity that reads both.

The defining property is non-interference: when the guest writes its own page tables, nothing traps, because those tables only produce GPAs, which are still “fake” addresses the hardware will translate again. Contrast this with Shadow Page Tables, where the hypervisor had to intercept every guest page-table edit to keep a merged GVA → HPA table consistent. TDP eliminates that interception entirely at the cost of a longer walk.

flowchart LR
  subgraph GUEST["Guest-owned (no traps on edit)"]
    GVA["GVA<br/>(guest virtual)"]
    GPT["Guest page tables<br/>(4 levels, guest CR3)"]
    GPA["GPA<br/>(guest physical)"]
    GVA --> GPT --> GPA
  end
  subgraph HOST["Hypervisor-owned (EPT / NPT)"]
    EPT["EPT / NPT tables<br/>(4 levels, EPTP / nCR3)"]
    HPA["HPA<br/>(host physical)"]
    GPA --> EPT --> HPA
  end
  HOST -.->|"sits on top of"| MM["Host MM: reclaim, KSM,<br/>THP, NUMA, swap"]

The two-dimensional translation. What it shows: the guest’s four-level walk produces a GPA, which is not a real address; the hypervisor’s EPT/NPT walk then translates that GPA to the real HPA. The guest never sees the second stage and never traps editing the first. The insight to take: the GPA is the seam between two independently-managed worlds — guest page tables above it, host memory management (swap, KSM, huge pages, NUMA) below it. A “guest physical page” is really just an entry in the EPT pointing at a host page the kernel is free to move.

Why a Second Table at All — The Problem TDP Solves

A guest cannot be given real host-physical addresses. If two guests each believed they owned physical page 0, they would collide; if a guest could name a host-physical address directly, it could read another tenant’s RAM. So the hypervisor invents a per-guest fiction — guest-physical address space — that looks like contiguous RAM starting at 0 but is actually a sparse, relocatable mapping onto scattered host pages (the memory-slot machinery defines which GPA ranges map to which host virtual-memory regions).

Before hardware help, KVM bridged this gap in software with Shadow Page Tables: it built a merged table mapping GVA → HPA directly, loaded that into the real CR3, and trapped every guest write to the guest’s own page tables (and every guest CR3 reload) to keep the merge consistent. The walk was native-fast — four memory accesses on x86-64 — but the maintenance was brutal: marking a page copy-on-write or flushing a TLB cost thousands of cycles in VM-exit traps (Gandhi et al. §II.B). TDP inverts that trade: zero maintenance traps, longer walks.

The Two-Dimensional Walk, Step by Step

On a TLB hit, TDP costs nothing — the cached entry already maps GVA → HPA directly, tagged with the guest’s identity (see VPID/ASID below), so the translation is as fast as native (Gandhi et al. Table I). The cost appears only on a TLB miss, and it is the central performance fact of memory virtualization.

On bare metal, an x86-64 TLB miss requires the page-walker to read 4 page-table entries (one per level: PML4 → PDPT → PD → PT). Each read is itself a physical memory access, because page-table entries store physical addresses.

Under TDP, the guest’s page-table entries store guest-physical addresses, not real ones. So every physical access the guest walker would have made must itself be translated through the EPT/NPT table — which is its own four-level walk. The arithmetic, stated verbatim by Gandhi, Hill and Swift, is:

“the page table memory references grow from a native 4 to a virtualized 24 references: 4 access to translate gptr (since each gPA requires access to host page table) and each of the 4 levels of the guest page table (guest page table holds gPA) plus 4 references for the guest page table itself to obtain the final hPA: 4 × 5 + 4 references.” (ISCA 2016, §II.A)

Walking that symbol-by-symbol: there are 5 “rows” in the walk that each hold a guest-physical pointer needing translation — the guest CR3/page-table pointer (gptr) plus the four guest page-table levels. Four of these (gptr and the upper three guest levels) each cost a 5-access nested step (1 access to read the guest entry + 4 EPT accesses to translate the GPA it contains into an HPA), and the final guest page-table entry (the leaf PTE) plus the final data-page GPA costs the remaining 4 EPT accesses. The paper’s Table II makes the per-level breakdown explicit: the page-table pointer costs 4 (native 0), and each of the four guest levels costs 5 (native 1) — summing to 24 for nested paging versus 4 for native.

Uncertain

Verify: the precise “up to ~24 memory accesses” worst-case figure and its 4 × 5 + 4 decomposition. Reason: the 24 figure is well-corroborated by a peer-reviewed primary source (Gandhi, Hill & Swift, Agile Paging, ISCA 2016, with an explicit per-level Table II) and is the standard textbook number, but it assumes a specific configuration — four-level paging on both the guest and the EPT/NPT side, a cold walk with no MMU page-walk-cache hits, and no large pages. With five-level paging (host or guest CR4.LA57=1, which the TDP MMU supports via role.passthrough, see mmu.rst) the worst case is larger; with page-walk caches or huge pages it is much smaller. To resolve: confirm against Intel SDM Vol. 3C §28.2 (EPT) and AMD APM Vol. 2 §15.25 (NPT) that 24 is the documented 4-level worst case, and note the 5-level figure separately. uncertain

In practice the full 24-access walk is rare. The MMU caches intermediate translations in page-walk caches and paging-structure caches, and the EPT itself can use huge pages (2 MiB or 1 GiB) that collapse levels — the kernel’s TDP MMU explicitly maps and splits these, e.g. it can “replace [a] page table with an equivalent 1GiB hugepage” (tdp_mmu.c, v6.12). Backing guest RAM with huge pages is the single most effective way to cut TDP walk cost, because it shrinks both dimensions of the walk simultaneously.

TLB Tagging — Why Transitions Don’t Flush

Without help, every VM-entry and VM-exit would have to flush the TLB, because a cached GVA → HPA entry for the guest is meaningless to the host and vice-versa. That flush would make virtualization transitions ruinously expensive. Both vendors solve this by tagging every TLB entry with an identifier so host and guest (and different guests) coexist in the same TLB:

  • Intel — Virtual-Processor Identifier (VPID). Added with Nehalem (2008), VPID tags each cached linear-address translation with a per-vCPU identifier, so “no TLB flushes occur for VM-entries or VM-exits.” KVM enables it by default — enable_vpid = 1 (vmx.c, v6.12) — and flushes selectively with the INVVPID instruction (vpid_sync_vcpu_single, vpid_sync_vcpu_global in vmx_flush_tlb_all).
  • AMD — Address-Space Identifier (ASID). AMD’s SVM tags each TLB entry with an ASID identifying the VM, distinguishing host from guest entries so a host↔guest switch need not wipe the TLB.

These are distinct from EPT/NPT: VPID/ASID tag the combined GVA → HPA result; EPT/NPT define how that result is computed on a miss. A useful mental split: VPID/ASID make the hit path cheap across transitions, EPT/NPT make the miss path correct without traps.

EPT Violations vs EPT Misconfigurations

When the EPT/NPT walk fails, the CPU raises one of two distinct VM exits, and KVM treats them very differently. The Intel VMX handlers in vmx.c (v6.12) make the distinction concrete:

An EPT violation is a permission or presence fault — the guest touched a GPA whose EPT entry is not present, or lacks the requested read/write/execute right. This is the normal, expected fault that populates guest memory lazily (demand paging of guest RAM) or implements write-tracking (for dirty logging). handle_ept_violation() reads the exit qualification, decodes which access bits were violated, and routes the fault into the MMU:

static int handle_ept_violation(struct kvm_vcpu *vcpu)
{
	unsigned long exit_qualification;
	gpa_t gpa;
	u64 error_code;
 
	exit_qualification = vmx_get_exit_qual(vcpu);
	...
	/* Is it a read fault? */
	error_code = (exit_qualification & EPT_VIOLATION_ACC_READ)
		     ? PFERR_USER_MASK : 0;
	/* Is it a write fault? */
	error_code |= (exit_qualification & EPT_VIOLATION_ACC_WRITE)
		      ? PFERR_WRITE_MASK : 0;
	/* Is it a fetch fault? */
	error_code |= (exit_qualification & EPT_VIOLATION_ACC_INSTR)
		      ? PFERR_FETCH_MASK : 0;
	...
	gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
	...
	return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);
}

Line by line: vmx_get_exit_qual() reads the exit qualification, the VMCS field that encodes why the fault happened; the three EPT_VIOLATION_ACC_* bits say whether the offending access was a read, write, or instruction fetch, and each is folded into an error_code using the same PFERR_* bit layout the kernel uses for ordinary page faults; vmcs_read64(GUEST_PHYSICAL_ADDRESS) retrieves the faulting GPA (the CPU helpfully records it); finally kvm_mmu_page_fault() hands the GPA and decoded error code to the MMU, which walks its own struct kvm_mmu_page tables and installs the missing SPTE. The TDP MMU’s kvm_tdp_mmu_map() is documented as handling exactly this: “Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing” the mapping (tdp_mmu.c, v6.12).

An EPT misconfiguration is a malformed entry fault — the EPT entry itself has an illegal bit combination (reserved bits set, or a permission encoding the hardware forbids, such as write-without-read). KVM deliberately creates such “misconfigured” entries to trap MMIO: a GPA that maps to an emulated device has no real backing page, so KVM installs a special non-memory SPTE that triggers a misconfig on access. handle_ept_misconfig() routes it as a reserved-bit fault:

static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
{
	gpa_t gpa;
	...
	gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
	if (!is_guest_mode(vcpu) &&
	    !kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
		trace_kvm_fast_mmio(gpa);
		return kvm_skip_emulated_instruction(vcpu);
	}
 
	return kvm_mmu_page_fault(vcpu, gpa, PFERR_RSVD_MASK, NULL, 0);
}

The key distinction in code: the violation handler passes the decoded R/W/X error code (it is real memory, populate it), while the misconfig handler passes PFERR_RSVD_MASK — a reserved-bit fault — because the entry was intentionally malformed to flag “this GPA is MMIO, not RAM.” The misconfig path can short-circuit into the fast-MMIO bus (KVM_FAST_MMIO_BUS) for zero-length notification writes, skipping full instruction emulation. The comment in the handler is explicit that “[a] nested guest cannot optimize MMIO vmexits, because we have an nGPA here instead of the required GPA” (vmx.c, v6.12).

Uncertain

Verify: that an EPT misconfiguration is defined by the hardware as a reserved/illegal-encoding fault (vs. a violation being a present/permission fault), and the exact illegal encodings (e.g. write-without-read, or specific reserved bits) per Intel SDM Vol. 3C §28.2.3.1. Reason: the KVM code paths clearly use misconfig for MMIO via PFERR_RSVD_MASK, which strongly implies the reserved-bit semantics, but the authoritative hardware definition of which bit patterns the CPU classifies as “misconfiguration” was read from KVM code and a secondary reverse-engineering write-up, not the SDM directly. To resolve: read Intel SDM Vol. 3C “EPT-Induced VM Exits”. uncertain

TDP Sits On Top of Host Paging

The most consequential property for the rest of the kernel is that the EPT/NPT entry’s output — the HPA — is just a host page frame, and the host memory manager owns it. Guest “physical” RAM is, from the host’s point of view, ordinary anonymous memory in the VMM process’s address space (see Anonymous vs File-Backed Memory). That means:

  • Swapping. The host can reclaim a guest page to swap; KVM tears down the EPT entry, and the next guest access raises an EPT violation that re-faults the page in. The guest never knows.
  • Kernel Samepage Merging. Identical guest pages across VMs can be merged to one read-only host page; a guest write triggers an EPT violation → copy-on-write.
  • Transparent Huge Pages and NUMA. The host can back guest RAM with 2 MiB/1 GiB pages and migrate it between NUMA nodes; the TDP MMU adjusts its mapping level accordingly (kvm_mmu_hugepage_adjust() in kvm_tdp_mmu_map).

This is the seam where memory virtualization meets the core MM subsystem, and where the subtle bugs live — e.g. an EPT entry must be invalidated before the host page is reused, or the guest reads stale data. KVM hooks the MM notifier chain (mmu_notifier) so that any host-side page movement zaps the corresponding SPTEs.

The TDP MMU — KVM’s Default Implementation

KVM has two MMU implementations. The legacy shadow MMU code (mmu.c) handles both true Shadow Page Tables (when hardware TDP is unavailable) and TDP, using a unified page-table data structure. Since 2021 there is a separate, scalable TDP MMU (tdp_mmu.c) used only when hardware nested paging is present; it uses read-copy-update-style fine-grained locking instead of a single big mmu_lock, which matters for VMs with hundreds of vCPUs faulting concurrently. As of v6.12 it is enabled by default: bool __read_mostly tdp_mmu_enabled = true; with a read-only module parameter module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0444) (mmu.c, v6.12).

Hardware support itself is exposed through two module flags, both defaulting on:

  • Intel: tdp_enabled is set when EPT is available.
  • AMD: bool npt_enabled = true; module_param_named(npt, npt_enabled, bool, 0444); (svm.c, v6.12).

The KVM documentation describes the resulting translation chains compactly: for a non-nested guest with TDP, the MMU resolves (gva->)gpa->hpa; for a nested guest (a hypervisor inside a guest, see Nested Virtualization) it resolves (ngva->)ngpa->gpa->hpa (mmu.rst, v6.12). The doc also explains the direct vs shadow distinction: “When the number of required translations matches the hardware, the mmu operates in direct mode; otherwise it operates in shadow mode” — TDP enables direct mode.

Common Misunderstandings

“EPT/NPT replaces the guest’s page tables.” No — it adds a second table. The guest’s own page tables are untouched and fully functional; EPT/NPT only translates the GPAs they produce. A guest with paging disabled still goes through EPT.

“TDP is always faster than shadow paging.” Not for the walk itself — a TDP miss can cost ~24 accesses versus shadow’s 4. TDP wins because it eliminates the VM-exit storms that shadow paging suffered on page-table updates and context switches. Workloads with enormous TLB pressure but rare page-table changes are precisely where shadow paging’s cheap walk would still win, which is the gap the Agile Paging paper exploits.

“A guest-physical address is a real address.” It is a fiction the hypervisor manufactures; the corresponding host page can move, be swapped, or be shared at any moment.

“VPID/ASID and EPT/NPT are the same thing.” They are orthogonal: VPID/ASID tag TLB entries (the hit path); EPT/NPT define the walk (the miss path). A CPU can have one without the other.

Production Notes

The performance signature of TDP is DTLB/STLB miss-heavy, page-walk-heavy workloads — large in-memory databases, big heaps, anything with a working set far larger than TLB reach. The standard mitigations, in order of impact: back guest RAM with huge pages (1 GiB pages give the largest TLB reach and shallowest walks); pin the guest’s memory to a single NUMA node to avoid cross-node walk latency; and ensure VPID/EPT (or ASID/NPT) are enabled (they are by default — confirm with cat /sys/module/kvm_intel/parameters/{enable_vpid,...} or the AMD equivalents). On Intel, perf kvm stat and the ept_violation / ept_misconfig exit counters reveal whether a VM is thrashing its EPT. For nested guests, the doubled translation (ngva→ngpa→gpa→hpa) makes huge pages and large memory pinning even more important.

See Also

Graph View

Backlinks