vault

KVM Architecture Overview

15 min read

KVM Architecture Overview

The Kernel-based Virtual Machine (KVM) is the Linux kernel’s hardware-virtualization subsystem: a small in-kernel driver that exposes the CPU’s virtualization extensions (Intel VT-x/VMX — Virtual Machine Extensions; AMD -V/SVM — Secure Virtual Machine) to userspace through the /dev/kvm character device and a set of ioctl() calls. KVM’s defining design choice is radical minimalism: it virtualizes only the CPU and memory and nothing else. Everything that looks like a device to the guest is emulated by a separate userspace program (the virtual-machine monitor, or VMM), accelerated by the kernel’s vhost, or handed straight to the guest by VFIO. Because KVM is “just” a kernel module rather than a standalone hypervisor, a guest virtual machine is an ordinary host process, each virtual CPU (vCPU) is an ordinary host thread, and guest RAM is ordinary host mmap’d memory — so the Linux scheduler, memory manager, and cgroup controllers all apply to VMs unchanged (Kivity et al., OLS 2007; KVM API, v6.12). That single decision — make Linux itself the hypervisor — is the source of almost every property of the rest of the subsystem.

This note covers the shape of KVM: the modules, the device node, and the “a VM is a process” consequence. The exact split of responsibility between the kernel and the userspace VMM is its sibling, The Userspace VMM and KVM Division of Labor; the per-vCPU execution loop is KVM vCPU Run Loop; the ioctl() surface is The KVM ioctl API.

Mental Model

The cleanest way to think about KVM is to invert the usual picture of a hypervisor. A classic Type-1 (bare-metal) hypervisor — Xen, VMware ESXi, Microsoft Hyper-V — is its own small operating system that boots first, owns the hardware, and schedules guests on top of itself; the host OS, if any, is just another guest. KVM does the opposite. Linux boots normally as a full general-purpose operating system, and KVM is a module loaded into that running kernel that turns the already-booted kernel into a hypervisor. There is no separate hypervisor kernel to write, debug, or schedule with: the guest scheduler is the Linux scheduler, the guest memory manager is the Linux memory manager, and the guest’s “device drivers” on the host side are just userspace and kernel code reusing Linux’s enormous driver base. This is why KVM is sometimes called a “Type-2-ish” or hosted hypervisor, though the categories blur because the CPU runs guest code natively rather than emulating it (contrast Hypervisor Types and Virtualization Models).

flowchart TB
  subgraph US["Host userspace"]
    VMM["VMM process (QEMU / Firecracker / Cloud Hypervisor)<br/>vCPU thread 0 ... vCPU thread N<br/>+ device-emulation / IO threads"]
  end
  subgraph K["Host kernel"]
    CORE["kvm.ko (arch-independent core)<br/>kvm_main.c, eventfd, mmu, lapic, i8254/i8259, ioapic"]
    ARCH["arch module: kvm-intel.ko (VMX)<br/>OR kvm-amd.ko (SVM)"]
    REST["Linux scheduler / MM / cgroups / drivers<br/>(shared with everything else on the host)"]
    CORE --- ARCH
  end
  DEV["/dev/kvm (misc char device)"]
  HW["CPU virtualization HW: VT-x / AMD-V, EPT/NPT, APICv"]
  VMM -->|"open + ioctl()"| DEV
  DEV --> CORE
  ARCH -->|"VMLAUNCH / VMRUN, VM-entry/exit"| HW
  VMM -.->|"vCPU threads are scheduled by"| REST

KVM’s module shape and where a VM lives. What it shows: the userspace VMM opens /dev/kvm and drives KVM through ioctl(); KVM is two cooperating modules — an arch-independent core (kvm.ko) and exactly one vendor module (kvm-intel.ko or kvm-amd.ko) that knows how to issue the hardware VM-entry instruction; the guest’s CPU and memory are virtualized by KVM, while everything else (scheduling the vCPU threads, backing guest RAM, emulating devices) is just normal Linux machinery shared with the rest of the host. The insight to take: KVM is deliberately tiny. The hard parts — scheduling, memory reclaim, device drivers — are not reimplemented inside a hypervisor; they are the host kernel’s existing subsystems, which a VM uses by virtue of being a process.

The Two-Module Structure

On x86, KVM ships as two loadable modules that work together, a split the original paper describes precisely: “Base kvm functionality is placed in a module, kvm.ko, while the architecture-specific functionality is placed in the two arch-specific modules, kvm-intel.ko and kvm-amd.ko” (Kivity et al., OLS 2007, §3.2). The mechanism gluing them is a classic Linux idiom — a function-pointer vector. The paper notes KVM “handles this difference in the traditional Linux way of introducing a function pointer vector, kvm_arch_ops, and calling one of the functions it defines whenever an architecture-dependent operation is to be performed.” Intel VMX and AMD SVM are not mutually compatible instruction sets, so the vendor-specific entry/exit code, the in-memory control structure layout (Virtual Machine Control Structure / VMCS on Intel, Virtual Machine Control Block / VMCB on AMD — see VMCS and VMCB (Virtual Machine Control Structure)), and nested-paging handling all live in the vendor module; the generic memory-slot logic, the in-kernel device models, and the ioctl() plumbing live in the core.

The actual object composition is worth seeing because it tells you what KVM emulates in the kernel. As of v6.12, arch/x86/kvm/Makefile builds the core module from (v6.12 x86 KVM Makefile):

kvm-y			+= x86.o emulate.o i8259.o irq.o lapic.o \
			   i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
			   debugfs.o mmu/mmu.o mmu/page_track.o \
			   mmu/spte.o
kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o
kvm-intel-y		+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \
			   vmx/nested.o vmx/posted_intr.o vmx/main.o
kvm-amd-y		+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o

Reading this line by line: x86.o is the x86 vCPU core; emulate.o is the in-kernel x86 instruction emulator (used to decode the faulting instruction on an MMIO exit); lapic.o is the in-kernel local APIC (Advanced Programmable Interrupt Controller) model; i8254.o is the in-kernel PIT (Programmable Interval Timer); i8259.o is the in-kernel PIC (legacy 8259 Programmable Interrupt Controller); ioapic.o is the in-kernel I/O APIC; mmu/ holds the two-dimensional paging machinery including tdp_mmu.o (the Two-Dimensional-Paging MMU — see Two-Dimensional Paging (EPT and NPT)). The presence of lapic.o, i8254.o, i8259.o, and ioapic.o in the kernel module is the concrete reason the interrupt-hot devices can be emulated without bouncing out to userspace — the rationale for which lives in The Userspace VMM and KVM Division of Labor. The vendor lists confirm the same split: vmx/vmenter.o and svm/vmenter.o are the assembly stubs that execute the actual VMLAUNCH/VMRESUME (Intel) and VMRUN (AMD) instructions.

The arch-independent code that both x86 and other architectures share lives under virt/kvm/ and is pulled in via include $(srctree)/virt/kvm/Makefile.kvm, whose core list is kvm-y := $(KVM)/kvm_main.o $(KVM)/eventfd.o $(KVM)/binary_stats.o, with optional pieces such as coalesced_mmio.o, async_pf.o, irqchip.o, dirty_ring.o, and guest_memfd.o compiled in by config (v6.12 virt/kvm/Makefile.kvm). kvm_main.c is the heart — it registers the device node, implements the file-descriptor hierarchy, and contains the generic run loop.

Uncertain

Verify: that on x86 v6.12 the built module name for the core is kvm.ko (driven by obj-$(CONFIG_KVM_X86) += kvm.o) rather than the older obj-$(CONFIG_KVM). Reason: the obj-$(CONFIG_...) line lives in arch/x86/kvm/Makefile but was not captured verbatim in the fetched excerpt; the v6.12 tree renamed the x86 KVM Kconfig symbol to CONFIG_KVM_X86. To resolve: read the final lines of arch/x86/kvm/Makefile at tag v6.12 (the obj-$(CONFIG_KVM_X86) += kvm.o line). The user-visible module name kvm.ko is stable across this and is not in doubt. uncertain

The /dev/kvm Character Device

KVM exposes itself to userspace as a single misc character device, /dev/kvm. The 2007 paper states it plainly: “kvm is structured as a fairly typical Linux character device. It exposes a /dev/kvm device node which can be used by userspace to create and run virtual machines through a set of ioctl()s” (Kivity et al., §3.1). In kvm_main.c the node is registered as a misc device — a single shared minor-number device rather than a per-VM device file — using misc_register() with MISC_DYNAMIC_MINOR and a kvm_chardev_ops file-operations table (v6.12 kvm_main.c). There is exactly one /dev/kvm on the system regardless of how many VMs are running; it is the system-level handle from which all VMs are spawned.

The operations the paper lists for /dev/kvm are exactly the API’s three tiers in miniature: “Creation of a new virtual machine; Allocation of memory to a virtual machine; Reading and writing virtual cpu registers; Injecting an interrupt into a virtual cpu; Running a virtual cpu.” Concretely, ioctl(KVM_GET_API_VERSION) on the device returns the stable constant 12 (#define KVM_API_VERSION 12), and applications “should refuse to run if KVM_GET_API_VERSION returns a value other than 12” — KVM froze its base API version two decades ago and evolves through capabilities (KVM_CHECK_EXTENSION) instead, covered in The KVM ioctl API (KVM API, v6.12; v6.12 kvm.h). Issuing ioctl(KVM_CREATE_VM) on the /dev/kvm fd returns a brand-new VM file descriptor — “a VM file descriptor which can be used to issue VM ioctls” — that “initially has no vCPUs or memory.”

Because access is mediated by a device node, VM creation is gated by ordinary filesystem permissions on /dev/kvm: a process can create a VM only if it can open that node (typically group kvm). This is also why container runtimes that want to launch microVMs must pass /dev/kvm into the container — there is no other entry point.

Why “A VM Is a Process” Is the Whole Story

The single most load-bearing fact about KVM is that a virtual machine is a normal Linux process and a vCPU is a normal Linux thread. The 2007 paper makes the consequences explicit on three levels. On the developer level there are “opportunities for reusing existing functionality within the kernel, for example, the scheduler, NUMA support, and high-resolution timers.” On the user level, “one can reuse the existing Linux process management infrastructure, e.g., top(1) to look at cpu usage and taskset(1) to pin virtual machines to specific cpus. Users can use kill(1) to pause or terminate their virtual machines” (Kivity et al., §6). And on the SMP (Symmetric Multi-Processing) level, “In the same way that a virtual machine maps to a host process under kvm, a virtual cpu in an SMP guest maps to a host thread.”

Mechanically, this is enforced from the moment the VM is created. In kvm_create_vm(), KVM pins the creating process’s address space to the VM with mmgrab(current->mm); kvm->mm = current->mm; (v6.12 kvm_main.c). The mm_struct is Linux’s per-process memory-descriptor — the kernel object that owns a task’s page tables and virtual-memory areas. By grabbing a reference to it, KVM ties guest memory to the VMM process’s own address space. This is what makes the next two facts true.

Guest RAM is host mmap’d memory. Userspace does not ask KVM to allocate guest memory; it mmaps memory in its own address space — typically anonymous memory or a memfd — and then registers that region with KVM via ioctl(KVM_SET_USER_MEMORY_REGION), passing a struct kvm_userspace_memory_region whose userspace_addr field is “a userspace backing memory pointer” and whose guest_phys_addr is where that memory should appear in the guest’s physical address space (KVM API, v6.12):

struct kvm_userspace_memory_region {
    __u32 slot;            /* which memory slot (0..N) this region occupies */
    __u32 flags;           /* e.g. KVM_MEM_LOG_DIRTY_PAGES, KVM_MEM_READONLY */
    __u64 guest_phys_addr; /* where in *guest* physical space this appears */
    __u64 memory_size;     /* size in bytes; 0 deletes the slot */
    __u64 userspace_addr;  /* host virtual address of the backing memory */
};

The 2007 paper’s Figure 1 (the “kvm Memory Map”) shows this directly: “Like user memory in Linux, the kernel allocates discontiguous pages to form the guest address space. In addition, userspace can mmap() guest memory to obtain direct access. This is useful for emulating dma-capable devices.” The profound consequence is that guest physical memory is, from the host’s point of view, just more of the VMM process’s anonymous memory — and therefore it is subject to every memory-management mechanism the host already has: it can be swapped to disk, merged by Kernel Same-page Merging (KSM), backed by Transparent Huge Pages (THP), NUMA-balanced, and reclaimed under pressure. The paper anticipated all of this: “Linux provides a vast array of memory management features: demand paging, large pages (hugetlbfs), and memory-mapped files. We plan to allow a kvm guest address space to directly use these features.” (It now does — see KSM and Memory Overcommit for VMs and Huge Pages and NUMA for Guests.) The translation from guest-physical to host-physical is performed by hardware nested paging on top of the host’s own page tables, the subject of Two-Dimensional Paging (EPT and NPT).

A vCPU is a host thread that blocks inside an ioctl(). The VMM creates one host thread per vCPU; each thread calls ioctl(vcpu_fd, KVM_RUN), which “executes guest code.” That thread is fully visible to the host scheduler — it appears in ps -eLf, can be pinned with taskset or sched_setaffinity(), throttled by a CPU cgroup, given a real-time policy, or migrated between cores by the Completely Fair / EEVDF scheduler exactly like any other thread. The deep dive on what happens inside KVM_RUN is KVM vCPU Run Loop; the scheduling implications are vCPU Threads and Host Scheduling. The point here is structural: KVM did not invent a scheduler for vCPUs. It reuses Linux’s, by making vCPUs threads.

Failure Modes and Common Misunderstandings

“KVM is a hypervisor like Xen.” It is not standalone. Without a userspace VMM, KVM does nothing useful — it has no way to load a guest kernel, no device emulation, no console. It is the CPU/memory engine; the policy and the devices come from QEMU/Firecracker/Cloud Hypervisor. Confusing KVM with the VMM is the single most common conceptual error; the corrective is The Userspace VMM and KVM Division of Labor.

“Guest RAM is allocated by KVM.” No — userspace allocates it (mmap) and registers it. If the VMM under-allocates or the host is low on memory, guest pages can be swapped or, under the OOM (Out-Of-Memory) killer, the VMM process itself can be killed — taking the whole VM with it — because the VM is a process and obeys process-level memory accounting. Operators routinely set oom_score_adj or use cgroups to protect VMM processes.

“KVM_GET_API_VERSION tells me what features exist.” It does not — it has returned 12 for many years and is a compatibility tripwire, not a feature catalog. Feature detection is KVM_CHECK_EXTENSION with KVM_CAP_* constants (KVM Capabilities and Extension Negotiation). New userspace built against an old kernel must check capabilities, never the version.

Module-load failures. If /dev/kvm is absent, the cause is usually that virtualization is disabled in firmware (BIOS/UEFI), the CPU lacks VT-x/AMD-V, or the wrong vendor module is loaded. modprobe kvm-intel failing with “operation not supported” on an AMD box (or vice versa) is the symptom of trying to load the wrong arch module. Nested setups additionally require the nested module parameter (Nested Virtualization).

Alternatives and When to Choose Them

The architectural alternative to KVM is a Type-1 hypervisor. Xen and ESXi own the hardware directly and can, in principle, present a smaller trusted computing base and more deterministic scheduling because they are not carrying a full general-purpose kernel. KVM’s counter-argument is leverage: by being part of Linux it inherits two decades of scheduler, memory-manager, filesystem, and driver work for free, and a fix to the Linux block layer instantly benefits VMs. For userspace-only isolation without hardware virtualization there is pure emulation (QEMU’s TCG — Tiny Code Generator — binary translation), which runs any guest architecture on any host but is orders of magnitude slower; and there are containers (Linux Containers and Isolation MOC), which share the host kernel and trade isolation strength for near-zero overhead. KVM sits in the middle: hardware-enforced isolation at near-native CPU speed, at the cost of a per-VM memory footprint and a full guest kernel. The microVM movement (Firecracker, Cloud Hypervisor) exists precisely to push that cost down — see microVMs vs Containers vs Full VMs.

Production Notes

In production the “VM is a process” model is the foundation of every density and isolation story. AWS Lambda and Fargate run each function/task in a Firecracker microVM — a single host process with a handful of vCPU threads and a few tens of MiB of registered guest memory — and rely on the host kernel’s cgroup and scheduler controls to pack thousands of them per machine (Firecracker design). Because guest RAM is host anonymous memory, cloud operators overcommit memory across VMs using KSM and ballooning, and they live-migrate VMs by walking the dirty-page log of that same host memory (Dirty Page Tracking and Live Migration). The flip side is that anything that can perturb a host process can perturb a VM: a host-side kill -STOP, a cgroup CPU throttle, or memory pressure that swaps guest pages all manifest inside the guest as stalls or “steal time.” Diagnosing a slow VM therefore starts with treating it as a process — top, taskset, cgroup accounting — before reaching for VM-specific tools, because at the host level that is literally all it is.

See Also

Graph View

Backlinks