vault

KVM Capabilities and Extension Negotiation

15 min read

KVM Capabilities and Extension Negotiation

The Kernel-based Virtual Machine (KVM) interface deliberately does not version its application binary interface (ABI) the way most APIs do. After Linux 2.6.22 froze the core ABI — “no backward incompatible change are allowed” — KVM stopped adding features by bumping a version number and instead grew an extension mechanism: each new optional feature is assigned a stable integer identifier of the form KVM_CAP_* (a capability), and userspace asks at runtime whether a given capability is present by issuing the KVM_CHECK_EXTENSION ioctl, which returns 0 if the feature is absent and a positive integer (often just 1, sometimes a meaningful count or size) if it is present (KVM API §3–§4.4, v6.12). Some capabilities are merely queryable (the feature is on once detected); others must be explicitly turned on with a second ioctl, KVM_ENABLE_CAP, before the guest may use them. The single KVM_GET_API_VERSION number — frozen at 12 — exists only to confirm “this is the stable KVM ABI”; it is emphatically not a feature dial.

This is one of the most important and most copied design decisions in the KVM userspace API, and understanding why version numbers were rejected in favor of capability negotiation explains almost every quirk of how a virtual-machine monitor (VMM) probes the host. This note is pinned to the 6.12 long-term-support (LTS) kernel (released 2024-11-17); the mechanism is unchanged in 6.18 LTS, and individual capabilities are dated where their introduction matters. For the fd levels at which these ioctls are legal see KVM File Descriptor Hierarchy; for the full ioctl catalogue see The KVM ioctl API; for the worked startup sequence see Creating a Minimal VM with the KVM API.

Mental Model — Probe, Don’t Version

The mental shift is from “what version am I talking to?” to “does this specific feature exist here, right now?” A monolithic version number implies a total order: version 14 has everything version 13 has, plus more. KVM’s reality violates that assumption badly. Features land in mainline at different times, get backported to different stable trees, exist on some CPU architectures but not others (x86, ARM64, s390, PowerPC, RISC-V, LoongArch all share the core API but support disjoint feature sets), and depend on host hardware (a capability may report “absent” simply because the silicon lacks the extension). There is no single integer that could encode “this kernel, on this arch, on this CPU, supports KVM_CAP_DIRTY_LOG_RING but not KVM_CAP_X86_BUS_LOCK_EXIT.” So KVM exposes a vector of feature bits instead of a scalar version, and userspace queries each bit it cares about.

flowchart TD
  START["VMM startup"] --> VER["ioctl(sys_fd, KVM_GET_API_VERSION)<br/>== 12 ? else refuse"]
  VER --> PROBE["ioctl(fd, KVM_CHECK_EXTENSION, KVM_CAP_X)"]
  PROBE -->|"returns 0"| ABSENT["feature absent<br/>-> fall back / disable path"]
  PROBE -->|"returns N > 0"| PRESENT{"does cap need<br/>enabling?"}
  PRESENT -->|"no (query-only)"| USE["use the feature;<br/>N may carry a count/size"]
  PRESENT -->|"yes"| ENABLE["ioctl(fd, KVM_ENABLE_CAP, &kvm_enable_cap)<br/>cap=KVM_CAP_X, args[]=params"]
  ENABLE --> USE

The capability-negotiation flow. What it shows: a one-time API-version sanity check, then per-feature probing where the return value is 0/N, and a fork between query-only capabilities and those that require a second KVM_ENABLE_CAP step. The insight to take: the return value is overloaded — for some caps it is a boolean, for others it is data (a maximum count, a buffer size, a flags mask) — so userspace must read the documentation for each capability to know what a positive value means.

Mechanical Walk-through — How a Probe Resolves in the Kernel

KVM_CHECK_EXTENSION is unusual in being legal on two fd levels: the system fd (/dev/kvm) and the VM fd, the latter gated behind KVM_CAP_CHECK_EXTENSION_VM (KVM API §4.4, v6.12). In the source, both paths funnel into one function, kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg). The system-fd dispatcher kvm_dev_ioctl() calls it with kvm = NULL; the VM-fd dispatcher kvm_vm_ioctl() calls it with the real struct kvm * (kvm_main.c:5532, 5354, v6.12). That kvm pointer is the whole reason to query on the VM fd: a few capabilities give a more accurate, VM-specific answer when the VM is known. The documentation states this directly: “Based on their initialization different VMs may have different capabilities. It is thus encouraged to use the vm ioctl to query for capabilities.”

Inside kvm_vm_ioctl_check_extension_generic() is a large switch (arg). The structure of that switch reveals the three flavours of return value:

  • Boolean “present” capabilities return 1: KVM_CAP_USER_MEMORY, KVM_CAP_CHECK_EXTENSION_VM, KVM_CAP_ENABLE_CAP_VM, KVM_CAP_IRQFD, KVM_CAP_HALT_POLL, KVM_CAP_DEVICE_CTRL, and many more all fall through to a shared return 1; (kvm_main.c:4863, v6.12).
  • Count/size-carrying capabilities return a meaningful number. KVM_CAP_NR_MEMSLOTS returns KVM_USER_MEM_SLOTS (the maximum number of guest memory regions); KVM_CAP_COALESCED_MMIO returns the page offset of the coalesced-MMIO ring; KVM_CAP_IRQ_ROUTING returns KVM_MAX_IRQ_ROUTES; KVM_CAP_DIRTY_LOG_RING returns KVM_DIRTY_RING_MAX_ENTRIES * sizeof(struct kvm_dirty_gfn) — the maximum ring byte size — when the architecture supports it, and 0 otherwise (kvm_main.c:4902–4912, v6.12).
  • VM-dependent capabilities consult the kvm pointer: KVM_CAP_MULTI_ADDRESS_SPACE returns kvm_arch_nr_memslot_as_ids(kvm) when kvm is non-NULL and a static maximum otherwise; KVM_CAP_GUEST_MEMFD returns !kvm || kvm_arch_has_private_mem(kvm) (kvm_main.c:4897, 4928, v6.12). This is exactly the case where querying on the VM fd matters.

Anything the generic switch does not recognize falls through to kvm_vm_ioctl_check_extension(kvm, arg), the architecture-specific handler. On x86 that lives in arch/x86/kvm/x86.c, and it is where the vCPU-count capabilities are answered: KVM_CAP_NR_VCPUS returns min(num_online_cpus(), KVM_MAX_VCPUS) and KVM_CAP_MAX_VCPUS returns KVM_MAX_VCPUS (x86.c:4742, v6.12). On x86 with the default Kconfig, KVM_MAX_VCPUS is 1024 (arch/x86/include/asm/kvm_host.h, v6.12). The two-tier dispatch — generic first, arch-specific fallback — is why the same capability identifier can be meaningful on one architecture and return 0 (unsupported) on another: the generic switch has no case for it, and the arch handler either recognizes it or returns 0.

The capability identifiers themselves are plain #defines in include/uapi/linux/kvm.h, assigned monotonically increasing integers as features were added over the project’s history: KVM_CAP_IRQCHIP 0, KVM_CAP_HLT 1, KVM_CAP_USER_MEMORY 3, KVM_CAP_NR_VCPUS 9, KVM_CAP_NR_MEMSLOTS 10, … up through KVM_CAP_DIRTY_LOG_RING 192 and well beyond (kvm.h:680+, v6.12). The numbers are never reused or reordered — that stability is the whole point. A capability number, once assigned, means the same thing forever, so a VMM compiled against an old header still probes the same feature against a new kernel. (There is even an instructive scar in the header: a comment “Do not use 1, KVM_CHECK_EXTENSION returned it before we had flags” marks a number that must stay reserved for compatibility.)

Query-Only vs Must-Enable: KVM_ENABLE_CAP

Detecting a capability is sometimes enough — KVM_CAP_USER_MEMORY being present means KVM_SET_USER_MEMORY_REGION simply works. But some features are off by default and must be activated, because turning them on changes guest-visible behavior, costs resources, or must happen at a specific point in setup. For those, KVM provides KVM_ENABLE_CAP, which takes a struct kvm_enable_cap:

struct kvm_enable_cap {
    __u32 cap;       /* the KVM_CAP_* identifier to enable        */
    __u32 flags;     /* must be 0 for now (reserved)              */
    __u64 args[4];   /* feature-specific initial parameters       */
    __u8  pad[64];
};

The cap field names the capability; args[] carries parameters the feature needs to initialize (KVM API §4.37, v6.12). KVM_ENABLE_CAP itself comes in two scopes — a vCPU-fd variant (capability KVM_CAP_ENABLE_CAP) for per-vCPU features and a VM-fd variant (capability KVM_CAP_ENABLE_CAP_VM) for VM-wide ones — and the documentation says plainly: “The vcpu ioctl should be used for vcpu-specific capabilities, the vm ioctl for vm-wide capabilities.” The protocol the docs prescribe is: to check if a capability can be enabled, first use KVM_CHECK_EXTENSION — then enable it. The two ioctls are partners: probe, then (if needed) enable.

The canonical must-enable example is the dirty-log ring, KVM_CAP_DIRTY_LOG_RING (and its weakly-ordered-architecture sibling KVM_CAP_DIRTY_LOG_RING_ACQ_REL). Userspace probes it with KVM_CHECK_EXTENSION — a positive return is the maximum permitted ring size in bytes — then calls KVM_ENABLE_CAP with args[0] set to the desired ring size. The kernel’s kvm_vm_ioctl_enable_cap_generic() routes both ring capabilities to kvm_vm_ioctl_enable_dirty_log_ring(), which enforces a cluster of rules that show why this feature must be explicitly enabled and cannot be merely detected: the size must be a power of two, at least one page, and no larger than the maximum; and crucially, enabling “is only allowed before creating any vCPU” — the code checks if (kvm->created_vcpus) return -EINVAL; (kvm_main.c:4937, 5052, v6.12; KVM API §8.29, v6.12). A boolean “is it present” probe could never carry the ring size, and the ordering constraint (before any vCPU exists) means the feature has to be turned on at a precise moment — both reasons a passive query is insufficient and an active enable step is required.

Concrete Capability Examples and What Their Return Values Mean

A few capabilities are worth walking through because they exhibit the full range of the mechanism’s behavior.

KVM_CAP_USER_MEMORY (id 3) — the foundational capability. It gates KVM_SET_USER_MEMORY_REGION, the ioctl by which userspace tells KVM “this range of my mmap’d host memory is the guest’s physical RAM at this guest-physical address.” It is query-only and returns 1 when present, which is effectively always on any modern kernel; a VMM that cannot find it is talking to something that predates the user-memory model and should refuse to proceed. The detailed memory-slot mechanics live in Guest Physical Memory and Memory Slots.

KVM_CAP_NR_VCPUS (id 9) and KVM_CAP_MAX_VCPUS (id 66) — count-carrying, and a textbook illustration of why the return value is data, not a boolean. KVM_CAP_NR_VCPUS returns the recommended maximum vCPUs (on x86, the number of online host CPUs, capped at KVM_MAX_VCPUS), while KVM_CAP_MAX_VCPUS returns the hard maximum (KVM_MAX_VCPUS itself). The documentation even specifies the fallback contract for old kernels: “If the KVM_CAP_NR_VCPUS does not exist, you should assume that max_vcpus is 4 cpus max. If the KVM_CAP_MAX_VCPUS does not exist, you should assume that max_vcpus is same as the value returned from KVM_CAP_NR_VCPUS” (KVM API §4.7, v6.12). This graceful-degradation rule is the capability model at its purest: a feature’s absence has a defined meaning, so userspace can run correctly against any vintage of kernel.

KVM_CAP_DIRTY_LOG_RING (id 192) — must-enable, size-carrying, with the ordering and power-of-two constraints described above. It is the modern alternative to the older bitmap-based dirty tracking (KVM_GET_DIRTY_LOG) used during live migration: instead of scanning a per-slot bitmap, userspace harvests a per-vCPU ring of struct kvm_dirty_gfn entries, and a full ring forces a vCPU exit with KVM_EXIT_DIRTY_LOG_FULL (KVM API §8.29, v6.12). The migration story is in Dirty Page Tracking and Live Migration. The capability number 192 versus KVM_CAP_USER_MEMORY’s 3 is itself the historical record: user-memory was there near the beginning; the dirty-ring is a relatively recent addition (merged for Linux 5.11) bolted onto the same probing mechanism with no version bump.

Uncertain

Verify: that KVM_CAP_DIRTY_LOG_RING first merged in Linux 5.11. Reason: I am pinned to the v6.12 tree and did not fetch the 5.11 merge commit or git log for the introducing patch; the v6.12 header only shows the capability exists (id 192) and is documented in §8.29, not the release it landed in. To resolve: check git log --oneline -- include/uapi/linux/kvm.h around the KVM_CAP_DIRTY_LOG_RING define, or the git.kernel.org tag for the introducing commit (Peter Xu’s dirty-ring series). uncertain

Why Capabilities Beat Version Numbers

The argument for capability negotiation over a version scalar is concrete and is worth stating mechanically, because it recurs throughout Linux uapi design.

First, features land independently and out of order across releases. A version number imposes a total order (”≥ N implies has-everything-up-to-N”), but KVM features merge on their own schedules, and stable trees backport selectively. Two kernels reporting the same notional “version” could differ in their feature sets; conversely, a feature might be backported to an old stable kernel without changing any version it could advertise. A vector of independently-set capability bits has no such consistency burden.

Second, the same API spans many architectures with disjoint feature sets. x86’s KVM_CAP_X86_BUS_LOCK_EXIT, ARM64’s GIC device types, s390’s KVM_CAP_S390_UCONTROL, PowerPC’s KVM_CAP_PPC_* — these coexist in one uapi header but are meaningful only on their architectures. A single version number cannot describe “the s390 features of kernel X” and “the x86 features of kernel X” simultaneously, because they are not the same set. Capability probing, which can return 0 for “not on this arch,” handles this without contortion.

Third, support is gated on host hardware, not just kernel code. A capability can report absent because the CPU lacks the extension even though the kernel knows about it. Two-dimensional paging, posted interrupts, confidential-computing modes — all are “present” only if the silicon cooperates. Runtime probing naturally captures the (kernel × arch × CPU) cross-product that a static version could not.

Fourth, graceful degradation is built in. Because the absence of a capability has a defined fallback (the 4-vCPU default, the “assume same as NR_VCPUS” rule), a VMM compiled today can run correctly on a kernel from years ago and a kernel from years hence: it probes what it needs and adapts. The ABI freeze at API version 12 guarantees that everything that worked still works; capabilities add the new without subtracting the old. The KVM documentation captures the whole philosophy in one line: “The extension mechanism is not based on the Linux version number” (KVM API §3, v6.12).

Failure Modes and Common Misunderstandings

The most damaging misuse is treating KVM_GET_API_VERSION as a feature dial. It is not. It is frozen at 12, and the docs say it “is not expected that this number will change” — applications “should refuse to run if KVM_GET_API_VERSION returns a value other than 12” (KVM API §4.1, v6.12). Using the API version to gate any feature is a category error; the answer is always 12, and the feature you want may or may not be present regardless.

A second trap is misreading a positive return value as a boolean for capabilities that carry data. If a VMM checks KVM_CAP_NR_VCPUS and treats any non-zero result as “I may create unlimited vCPUs,” it ignores that the number is the limit. Likewise treating the KVM_CAP_DIRTY_LOG_RING return as a yes/no loses the maximum ring size encoded in it. The rule is: read the per-capability documentation to learn what a positive value means.

A third is forgetting to enable must-enable capabilities, or enabling them at the wrong time. Probing KVM_CAP_DIRTY_LOG_RING and then using the ring without first calling KVM_ENABLE_CAP does nothing; calling KVM_ENABLE_CAP for it after creating a vCPU returns -EINVAL because of the created_vcpus guard. The documentation’s prescribed order — enable “right after KVM_CREATE_VM… before creating any vCPU” — is not advisory for this feature; it is enforced (KVM API §8.29, v6.12).

A fourth is probing on the wrong fd for VM-dependent capabilities. Querying KVM_CAP_MULTI_ADDRESS_SPACE or KVM_CAP_GUEST_MEMFD on the system fd returns the generic/maximum answer, not the answer for a specific VM whose machine type might restrict it. The fix is to query on the VM fd (after confirming KVM_CAP_CHECK_EXTENSION_VM), which passes the real kvm pointer into the handler. The fd-scope rules are detailed in KVM File Descriptor Hierarchy.

Alternatives and Contrast

The capability pattern is pervasive across Linux uapi precisely because version scalars age badly. io_uring advertises features through an IORING_FEAT_* flags field returned at setup and opcode-probing via IORING_REGISTER_PROBE; the Berkeley Packet Filter (bpf()) syscall probes program-type and helper availability by attempting feature-specific operations; CPUID on x86 is itself a hardware capability vector rather than a “CPU version.” All share KVM’s insight: expose a set of independent bits and let the consumer adapt, rather than a single number that pretends features are totally ordered. Where KVM is distinctive is the dual-fd query (system vs VM) and the explicit enable step for stateful features — refinements that fall out of KVM’s object hierarchy and its need to configure features at precise points in VM setup.

Production Notes

Every mature VMM is a capability-probing machine at startup. QEMU’s accel/kvm/kvm-all.c wraps KVM_CHECK_EXTENSION in kvm_check_extension() / kvm_vm_check_extension() helpers and probes dozens of capabilities while building each VM, branching its device and memory setup on the results; it logs and degrades when a capability is missing rather than failing outright. Firecracker and Cloud Hypervisor (both Rust, via the kvm-ioctls crate) expose check_extension() and enable_cap() methods on their Kvm/VmFd/VcpuFd wrappers, mirroring the kernel’s fd scoping. The operational consequence for anyone debugging a VMM that “won’t start a feature” is a reliable first diagnostic: run a KVM_CHECK_EXTENSION for the capability in question (the kvm-ioctls crate or a tiny C probe will do) — a 0 return localizes the problem to the host (kernel too old, wrong arch, or hardware lacking the extension) rather than the VMM’s code. The dirty-ring migration path is the example most operators meet first, where a missing KVM_CAP_DIRTY_LOG_RING silently forces the slower bitmap path explored in Dirty Page Tracking and Live Migration.

See Also

Graph View

Backlinks