vault

Guest Physical Memory and Memory Slots

15 min read

Guest Physical Memory and Memory Slots

A guest virtual machine sees a flat guest-physical address (GPA) space — what looks to it like RAM starting at address 0 — but that space is a host fiction assembled from ordinary host memory. The userspace virtual-machine monitor (VMM) mmap()s a chunk of its own address space and registers it with KVM via the KVM_SET_USER_MEMORY_REGION ioctl (or its successor KVM_SET_USER_MEMORY_REGION2), telling KVM “the guest-physical range [guest_phys_addr, guest_phys_addr + memory_size) is backed by my host-virtual memory starting at userspace_addr.” Each such registration is a memory slot (memslot): a tuple mapping a GPA range to a host-virtual-address (HVA) range (KVM API, v6.12). Because guest RAM is just host process memory — anonymous pages, a file, or hugetlbfs — it is lazily faulted in, reclaimable, swappable, dedupable, and NUMA-placed by the host’s memory manager, exactly like any other allocation. Any guest-physical address with no covering slot is treated as a hole and, on access, becomes a O exit to the VMM.

This note is about the bottom layer of memory virtualization — how guest RAM exists at all and how the GPA → HVA correspondence is registered and looked up. The hardware/software techniques that translate GPA further down to host-physical addresses (HPA) are Two-Dimensional Paging (EPT and NPT) and Shadow Page Tables; this note supplies the GPA → HVA half that both of those sit on top of.

The Three-Address Chain: GVA → GPA → HVA → HPA

Memory in a KVM guest passes through more translation layers than bare metal, and it is worth naming all of them precisely because they are routinely conflated.

  1. Guest-virtual address (GVA): what a process inside the guest uses. The guest kernel’s own page tables map GVA → GPA.
  2. Guest-physical address (GPA): the guest’s notion of “physical RAM.” This is the address the guest puts in its page-table leaves and on its (virtual) memory bus.
  3. Host-virtual address (HVA): the address in the VMM process’s address space where the backing memory lives. The memslot records the GPA → HVA correspondence.
  4. Host-physical address (HPA): the real RAM frame. The host kernel’s page tables map HVA → HPA, and may move, swap, or never-yet-allocate that frame.

The memslot owns exactly one link in that chain: GPA → HVA, a simple affine (offset-based) map. Everything left of it (GVA → GPA) is the guest’s business; everything right of it (HVA → HPA) is the host kernel’s ordinary virtual-memory machinery. The clean separation is the whole point: KVM does not allocate or own guest RAM — it borrows the VMM’s mmap’d memory and lets Linux manage it. That is why “a VM is just a process” and guest pages show up in the VMM’s RSS, can be swapped, can be KSM-merged, and obey cgroup memory limits.

Mental Model

flowchart LR
  subgraph VMM["VMM process address space"]
    MM["mmap'd region<br/>at HVA base"]
  end
  subgraph SLOT["memslot (registered with KVM)"]
    S["base_gfn, npages,<br/>userspace_addr, flags"]
  end
  subgraph GUEST["Guest-physical space"]
    G["GPA range<br/>guest_phys_addr .. +memory_size"]
  end
  HOST["Host kernel MM: HVA to HPA<br/>(fault, swap, NUMA, THP)"]
  G -->|"base_gfn"| S
  S -->|"userspace_addr"| MM
  MM -->|"page fault on demand"| HOST
  HOLE["GPA with no slot = MMIO hole"]
  G -.->|"unmapped access"| HOLE
  HOLE -.->|"KVM_EXIT_MMIO"| VMM

A memslot binds a guest-physical range to a host-virtual range. What it shows: the VMM mmaps memory (top-left), then registers a memslot describing which guest-physical frames (base_gfn) that memory covers and where it lives (userspace_addr). A GPA inside a slot resolves to an HVA by simple offset arithmetic, and the host kernel lazily backs that HVA with a real frame on first touch. A GPA that falls in no slot is a hole — accessing it traps out to the VMM as an MMIO exit so a device model can emulate the register. The insight to take: guest RAM is not a special kind of memory; it is the VMM’s own mmap’d pages, and the memslot is nothing more than a lookup table from guest-physical to host-virtual that KVM consults on every fault.

The KVM_SET_USER_MEMORY_REGION ioctl

The registration call passes a small structure (kvm.h, v6.12):

struct kvm_userspace_memory_region {
        __u32 slot;            /* low 16 bits: slot id; high 16 bits: address space id */
        __u32 flags;           /* KVM_MEM_LOG_DIRTY_PAGES, KVM_MEM_READONLY */
        __u64 guest_phys_addr; /* start GPA of this region */
        __u64 memory_size;     /* bytes; 0 means DELETE this slot */
        __u64 userspace_addr;  /* start of the VMM's mmap'd backing memory (HVA) */
};

Field by field, per the v6.12 API doc and __kvm_set_memory_region() in kvm_main.c:

  • slot — “Bits 0-15 of ‘slot’ specify the slot id and this value should be less than the maximum number of user memory slots supported per VM. The maximum allowed slots can be queried using KVM_CAP_NR_MEMSLOTS.” If KVM_CAP_MULTI_ADDRESS_SPACE is available, “bits 16-31 of ‘slot’ specifies the address space which is being modified” — KVM splits this in code as as_id = mem->slot >> 16; id = (u16)mem->slot; (kvm_main.c, v6.12). On x86 the second address space exists for System Management Mode (SMM), which sees a different memory map.
  • flags — two bits are valid: KVM_MEM_LOG_DIRTY_PAGES (1<<0) and KVM_MEM_READONLY (1<<1) (the latter gated on KVM_CAP_READONLY_MEM); KVM_MEM_GUEST_MEMFD (1<<2) is only legal through the REGION2 variant. check_memory_region_flags() rejects any other bit with -EINVAL.
  • guest_phys_addr — the GPA the region starts at. Must be page-aligned (guest_phys_addr & (PAGE_SIZE-1) must be 0). The doc recommends “the lower 21 bits of guest_phys_addr and userspace_addr be identical. This allows large pages in the guest to be backed by large pages in the host” — i.e. align to 2 MiB so transparent huge pages line up.
  • memory_size — size in bytes, page-aligned. Zero means delete the slot. It must also satisfy (memory_size >> PAGE_SHIFT) <= KVM_MEM_MAX_NR_PAGES, which is ((1UL << 31) - 1) pages (kvm_host.h, v6.12).
  • userspace_addr — the HVA of the backing memory, “which must point at user addressable memory for the entire memory slot size. Any object may back this memory, including anonymous memory, ordinary files, and hugetlbfs.” It must be page-aligned and untagged; __kvm_set_memory_region() validates it with access_ok() so KVM can later read guest memory with __copy_from_user()-style accessors (api.rst & kvm_main.c, v6.12).

A worked minimal example — registering 256 MiB of guest RAM at GPA 0:

void *mem = mmap(NULL, 256 << 20, PROT_READ | PROT_WRITE,
                 MAP_SHARED | MAP_ANONYMOUS, -1, 0);   /* host anon memory */
 
struct kvm_userspace_memory_region region = {
        .slot            = 0,
        .flags           = 0,
        .guest_phys_addr = 0,                 /* guest sees RAM at physical 0 */
        .memory_size     = 256 << 20,         /* 256 MiB, page-aligned */
        .userspace_addr  = (uint64_t)mem,     /* HVA of our mmap */
};
ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region);

Notice there is no allocation of “guest physical memory” anywhere — the VMM mmaps ordinary memory and declares it to be the guest’s RAM. KVM stores the slot and is done; the pages are not touched until the guest faults on them.

What Can and Cannot Change on an Existing Slot

The API doc is precise: “Deleting a slot is done by passing zero for memory_size. When changing an existing slot, it may be moved in the guest physical memory space, or its flags may be modified, but it may not be resized.” The code in __kvm_set_memory_region() enforces exactly this by computing a change enum (kvm_main.c, v6.12):

  • If no old slot exists at that id → KVM_MR_CREATE.
  • If base_gfn differs → KVM_MR_MOVE (the slot’s GPA moves; the HVA and size must be unchanged).
  • If only flags differ → KVM_MR_FLAGS_ONLY (e.g. toggling dirty logging).
  • Crucially, the modify path returns -EINVAL if userspace_addr changed, if npages (the size) changed, or if the KVM_MEM_READONLY bit flipped — those require a delete-then-recreate. And mem->memory_size == 0 with an existing slot → KVM_MR_DELETE.

This immutability is why VMMs that need to grow guest RAM at runtime (memory hotplug, ballooning resize) add or remove whole slots rather than resizing one — a deliberate simplification of KVM internals.

How a Lookup Works: GPA → HVA

Internally each registered region becomes a struct kvm_memory_slot holding base_gfn (the first guest frame number, i.e. guest_phys_addr >> PAGE_SHIFT), npages, userspace_addr, flags, and an id (kvm_host.h struct kvm_memory_slot, v6.12). The slots are stored in per-address-space kvm_memslots, indexed for fast lookup by guest frame number through red-black/interval trees and a hash on slot id — the structure is built for O(log n) lookup by GPA because it is consulted on the page-fault hot path.

The actual GPA → HVA arithmetic is one line, __gfn_to_hva_memslot() (kvm_host.h, v6.12):

unsigned long offset = gfn - slot->base_gfn;
offset = array_index_nospec(offset, slot->npages);  /* Spectre hardening */
return slot->userspace_addr + offset * PAGE_SIZE;

In words: take the guest frame number, subtract the slot’s base frame to get a page offset within the slot, clamp it against speculative out-of-bounds reads (the array_index_nospec guards against a malicious guest building a Spectre gadget from page-table walks), and add that byte offset to the slot’s userspace_addr. The result is the HVA. From there KVM resolves HVA → HPA through the host’s page tables with hva_to_pfn(), faulting the page in if necessary — which is where guest RAM gets physically allocated, lazily, on first access (kvm_main.c gfn_to_hva/hva_to_pfn, v6.12). A function family — gfn_to_hva, gfn_to_pfn, kvm_read_guest, kvm_write_guest — layers on top so that KVM (and in-kernel device emulation) can read and write guest memory by GPA.

Slot Flags

  • KVM_MEM_LOG_DIRTY_PAGES — “Instruct KVM to keep track of writes to memory within the slot.” KVM allocates a dirty_bitmap for the slot (one bit per guest page) and write-protects the pages (or uses the PML hardware feature) so the first write to each page is recorded. This is the foundation of live migration: the migration loop repeatedly enables logging, copies dirtied pages, and converges. kvm_prepare_memory_region() allocates/reuses/frees the bitmap as the flag toggles across a KVM_MR_FLAGS_ONLY change (kvm_main.c, v6.12).
  • KVM_MEM_READONLY — makes the slot read-only (requires KVM_CAP_READONLY_MEM). Per the doc, “writes to this memory will be posted to userspace as KVM_EXIT_MMIO exits.” This is how a VMM maps a ROM (e.g. firmware/BIOS, option ROMs) into the guest: reads come from real memory, but a write traps so the VMM can decide what to do. In the translation helpers, memslot_is_readonly(slot) makes __gfn_to_hva_many(..., write=true) return KVM_HVA_ERR_RO_BAD, routing the write to emulation (kvm_main.c, v6.12).

MMIO Holes — GPAs With No Slot

A guest-physical address space is not wall-to-wall RAM. Between (and around) the RAM regions sit device registers: the local APIC, PCI BARs, the I/O APIC, framebuffers. The VMM deliberately leaves those GPA ranges uncovered by any memslot. When the guest accesses such an address, KVM’s page-fault path finds no slot, marks the access as MMIO, and exits to userspace with KVM_EXIT_MMIO (or, for an in-kernel-emulated device, dispatches to the in-kernel handler). The VMM’s device model then reads or writes the emulated register and resumes the guest. This is the mechanism behind the classic PCI hole below 4 GiB on x86: RAM that would otherwise sit at, say, 3–4 GiB is relocated above 4 GiB so the 3–4 GiB GPA window can host memory-mapped device space — implemented purely by which GPA ranges the VMM does and does not register as slots. See MMIO and Port IO Emulation.

KVM_SET_USER_MEMORY_REGION2 and guest_memfd

Linux 6.12 carries the newer KVM_SET_USER_MEMORY_REGION2 ioctl (capability KVM_CAP_USER_MEMORY2), “an extension to KVM_SET_USER_MEMORY_REGION that allows mapping guest_memfd memory into a guest. All fields shared with KVM_SET_USER_MEMORY_REGION identically” (api.rst §4.140, v6.12). It adds guest_memfd (a file descriptor created via KVM_CREATE_GUEST_MEMFD), guest_memfd_offset, and the KVM_MEM_GUEST_MEMFD flag. The motivation is confidential computing (Intel TDX, AMD SEV-SNP): such a region “must have a valid guest_memfd (private memory) and userspace_addr (shared memory),” and KVM “selects shared vs. private … based on the gfn’s KVM_MEMORY_ATTRIBUTE_PRIVATE state,” toggled via KVM_SET_MEMORY_ATTRIBUTES. The crucial difference from ordinary slots: private guest memory backed by guest_memfd is not mappable into the VMM’s address space (the host must not be able to read encrypted guest RAM), which is precisely why it cannot be expressed as a plain userspace_addr-only slot. In kvm_main.c, REGION and REGION2 share one handler; the v1 ioctl simply zeroes the extra fields and restricts flags to KVM_SET_USER_MEMORY_REGION_V1_FLAGS (kvm_main.c ioctl dispatch, v6.12). For non-confidential VMs, the plain REGION ioctl remains correct and is what most VMMs still use.

Guest RAM Is Ordinary Host Memory — The Consequences

Because userspace_addr points at the VMM’s own mmap, guest RAM inherits everything the host MM does to anonymous (or file-backed) memory:

  • Lazy faulting. Registering a 16 GiB slot allocates no physical pages; the guest’s RSS grows only as it touches pages. This is what makes “give the VM 16 GiB” cheap until the guest uses it.
  • Swap and reclaim. Idle guest pages can be reclaimed/swapped by the host under pressure. KVM registers an mmu_notifier so that when the host unmaps/swaps an HVA, KVM invalidates the corresponding EPT/shadow mappings — this is the KVM_CAP_SYNC_MMU guarantee: “changes in the backing of the memory region are automatically reflected into the guest. For example, an mmap() that affects the region will be made visible immediately. Another example is madvise(MADV_DROP)” (api.rst, v6.12).
  • Deduplication and huge pages. Kernel Same-page Merging can collapse identical guest pages across VMs; transparent huge pages and explicit hugetlbfs backing reduce TLB and EPT pressure (the 21-bit alignment recommendation exists for this).
  • NUMA placement. Guest pages are placed and balanced like any host allocation; pinning a VMM and its memory to a NUMA node is standard for latency-sensitive guests. See Huge Pages and NUMA for Guests.

Failure Modes and Common Misunderstandings

  • -EINVAL on registration. Almost always an alignment or bounds violation: guest_phys_addr, memory_size, or userspace_addr not page-aligned; userspace_addr not actually mapped/accessible (access_ok fails); size exceeding KVM_MEM_MAX_NR_PAGES; or an out-of-range slot id / address space id. The validation block at the top of __kvm_set_memory_region() rejects each with -EINVAL (kvm_main.c, v6.12).
  • -EEXIST on overlap. “Slots may not overlap in guest physical address space” — kvm_check_memslot_overlap() enforces it per address space on CREATE/MOVE. A common bug is forgetting to delete the old slot before remapping the same GPA.
  • Trying to resize a slot. Returns -EINVAL; you must delete and recreate. VMMs that “grow RAM” do so by adding adjacent slots.
  • “The guest can see host memory it shouldn’t.” Only the bytes in registered slots are reachable by GPA; everything else is an MMIO hole. The isolation property is that the guest’s EPT/shadow tables only ever encode HPAs derived from slot HVAs, so a guest cannot name host memory outside its slots. The array_index_nospec hardening in __gfn_to_hva_memslot() exists to keep this true even under speculative execution.
  • Stale mappings after host reclaim. If KVM_CAP_SYNC_MMU is absent (some non-x86/older configs), host MM changes are not reflected and swapping guest memory is unsafe; on x86 with the mmu_notifier wired up this is handled. Diagnose via the KVM debugfs counters and the host’s reclaim/swap stats.

Alternatives and When to Choose Them

The memslot API is the only way to give a KVM guest RAM, so there is no alternative mechanism — but there are choices in how the backing is allocated. Anonymous mmap is the default: lazy, swappable, overcommit-friendly. hugetlbfs/MAP_HUGETLB backing trades flexibility (huge pages are not swappable or KSM-merged and must be reserved up front) for fewer TLB/EPT misses on large-memory guests — chosen for databases and HPC guests. File-backed slots (e.g. mmap of a tmpfs or persistent file) are used for shared memory between VMs or for persistent guest state. guest_memfd is mandatory, not optional, for confidential VMs where the host must be unable to map guest private memory. The relevant trade-off axis is flexibility/overcommit (anonymous) versus performance/determinism (hugetlbfs) versus confidentiality (guest_memfd).

Production Notes

Real VMMs register many slots, not one: a typical QEMU x86 guest has separate slots for low RAM (below the PCI hole), high RAM (above 4 GiB), firmware/ROM (read-only slots), and assorted device-adjacent regions, with the holes in between left unregistered for MMIO. Minimal microVMs like Firecracker register just a couple of large slots and rely on virtio-only device models, which is part of why they boot fast and stay small. The maximum number of slots is queryable via KVM_CAP_NR_MEMSLOTS and is generously large (on the order of thousands on x86, derived from KVM_MEM_SLOTS_NUM = SHRT_MAX minus internal slots, per kvm_host.h, v6.12) — but very high slot counts slow lookups, so VMMs coalesce contiguous RAM into few large slots. The memslot generation counter (slots->generation) is bumped on every update and is how KVM invalidates cached GPA→HVA translations (and MMIO sptes) after a slot change — a subtle correctness mechanism worth knowing when debugging stale-mapping bugs across hot-plug events.

Uncertain

Verify: the exact maximum usable user memory slot count on x86 in v6.12. The header defines KVM_MEM_SLOTS_NUM = SHRT_MAX (32767) and KVM_USER_MEM_SLOTS = KVM_MEM_SLOTS_NUM - KVM_INTERNAL_MEM_SLOTS, but the precise number of internal slots reserved on x86 (and thus the value KVM_CAP_NR_MEMSLOTS reports) was not pinned to a single source here. Reason: the deduction is correct in form but KVM_INTERNAL_MEM_SLOTS is arch-defined and was not directly read. To resolve: grep KVM_INTERNAL_MEM_SLOTS in arch/x86/include/asm/kvm_host.h for v6.12 and query a running host’s KVM_CHECK_EXTENSION(KVM_CAP_NR_MEMSLOTS). uncertain

See Also

Graph View

Backlinks