VFIO Framework

VFIO (Virtual Function I/O) is the Linux kernel framework that lets an unprivileged userspace process take direct, safe ownership of a physical device — most often to hand that device straight to a virtual-machine guest (device passthrough). Its founding insight is that a device only becomes dangerous in userspace because of Direct Memory Access (DMA): a device writes memory by physical address, so a userspace driver that could program arbitrary DMA targets would be a kernel-memory read/write primitive. VFIO closes this hole by building its entire user API on top of the IOMMU (the I/O Memory Management Unit), so a userspace-driven device can only touch the memory its owner has explicitly mapped (VFIO docs, v6.12). The classic API is built around a three-tier model — a container (the DMA/IOMMU context), one or more groups (the hardware isolation unit), and the per-device file descriptor that exposes the device’s regions and interrupts. A newer subsystem, iommufd, merged in Linux 6.2 (December 2022), replaces the container with a cleaner /dev/iommu-based design and is the path QEMU now prefers (Phoronix, 2022).

This note covers the VFIO framework — the device-ownership and userspace-driver machinery. The hardware that actually makes it safe (translation tables, isolation domains) is its sibling The IOMMU and DMA Remapping; the grouping of devices into indivisible isolation units is IOMMU Groups and Device Isolation; and the technique for slicing one card into many passable devices is SR-IOV and Virtual Functions. The parent map is Linux Virtualization MOC.

This note is pinned to Linux 6.12, a maintained long-term-support (LTS) release (2024-11-17); mainline has since moved into the 7.x series. Every structural fact — every ioctl number, module parameter, Kconfig default and quoted comment — was read out of the v6.12 tree, not out of documentation about it. That distinction turns out to matter more than usual here: VFIO’s own in-tree documentation is measurably behind its own code at this tag, in at least two places identified below. Where a claim concerns when something appeared or disappeared, it is pinned by existence check — fetching the same path at several tags and recording status codes and grep counts.

---

Why VFIO Exists — the DMA Trust Problem

To drive a device from userspace at all, the kernel must hand the process two capabilities: the ability to read and write the device’s registers (its memory-mapped I/O regions and PCI configuration space), and the ability to set up DMA so the device can move data to and from the process’s buffers without the CPU copying every byte. The first is harmless on its own — a register poke affects only the device. The second is catastrophic if unguarded, because a PCI device performs DMA by emitting bus transactions that carry a raw address. Historically that address was a host-physical address. A malicious or buggy userspace driver that could program a device’s DMA engine to write to physical address 0 could therefore scribble over the kernel’s own page tables — a full privilege escalation with no system call involved.

The kernel’s older userspace-driver framework, UIO (Userspace I/O), simply did not solve this: it exposed device memory and interrupts but offered no DMA protection, so it was only ever safe for devices that did not do DMA, or in trusted/embedded contexts. VFIO’s whole reason for being is that it refuses to expose DMA-capable devices to userspace unless an IOMMU is present to confine that DMA. The IOMMU sits between the device and RAM and translates the addresses a device emits (I/O virtual addresses, IOVAs) through a per-device page table the kernel controls. VFIO programs that page table so that the only addresses a passed-through device can reach are the buffers its owning process explicitly mapped — and nothing else. The userspace driver becomes, in effect, sandboxed by hardware (VFIO docs, v6.12).

flowchart LR
  subgraph UNSAFE["Userspace driver with NO IOMMU (UIO, or VFIO no-iommu mode)"]
    direction TB
    U1["userspace process"] -->|"mmap BAR, program DMA engine"| D1["PCI device"]
    D1 -->|"bus transaction carries a<br/>RAW HOST-PHYSICAL address"| M1["all of RAM"]
    M1 -.->|"nothing checks this"| K1["kernel page tables,<br/>other processes,<br/>other devices"]
  end
  subgraph SAFE["Userspace driver WITH an IOMMU (VFIO)"]
    direction TB
    U2["userspace process"] -->|"mmap BAR, program DMA engine"| D2["PCI device"]
    D2 -->|"bus transaction carries an<br/>I/O VIRTUAL ADDRESS (IOVA)"| I2["IOMMU"]
    I2 -->|"walks the per-domain page table<br/>the kernel built from<br/>VFIO_IOMMU_MAP_DMA"| M2["only the pages<br/>this process mapped"]
    I2 -.->|"unmapped IOVA -&gt;<br/>DMAR / IOMMU fault"| F2["transaction blocked,<br/>logged in dmesg"]
  end

Why VFIO refuses to expose a DMA-capable device without an IOMMU. What it shows: the same userspace driver, the same device, and the only difference is what the address in a DMA transaction means. Without an IOMMU it is a host-physical address and the device is an unrestricted read/write primitive on all of memory. With an IOMMU it is an I/O virtual address that must be translated through a page table the kernel controls, so anything the owner did not explicitly map simply faults. The insight: VFIO does not make the device trustworthy — it makes the device’s reach enumerable. Everything else in the framework, including the awkward group and container objects, exists to guarantee that this picture cannot be subverted by a second device sitting next to the first one.

Two details make this stricter than it first appears. First, the confinement covers interrupts as well as data: on x86 a Message Signalled Interrupt is literally a memory write to an address in the local-APIC range, so a device that could forge arbitrary DMA could forge arbitrary interrupts. The type1 backend therefore refuses to attach a group unless the platform provides interrupt isolation — if (!allow_unsafe_interrupts && !iommu_group_has_isolated_msi(iommu_group)) returns -EPERM with the log line “No interrupt remapping support. Use the module param “allow_unsafe_interrupts” to enable VFIO IOMMU support on this platform” (drivers/vfio/vfio_iommu_type1.c, v6.12). Note the modern spelling: this is iommu_group_has_isolated_msi() in 6.12, not the older IOMMU_CAP_INTR_REMAP capability check that older write-ups describe.

Second, VFIO also has to care about cache coherency, for a reason that is not obvious. A PCIe transaction can carry a “no-snoop” bit that bypasses the CPU’s caches. The type1 backend’s comment explains what it does about it: “If the IOMMU can block non-coherent operations (ie PCIe TLPs with no-snoop set) then VFIO always turns this feature on because on Intel platforms it optimizes KVM to disable wbinvd emulation.” A guest that can issue non-coherent DMA forces the hypervisor to emulate cache-flush instructions; blocking no-snoop at the IOMMU lets KVM skip that emulation entirely. Security and performance point the same way here, which is rare enough to be worth noticing.

The single escape hatch from all of this, VFIO_NOIOMMU_IOMMU, taints the kernel when used and is discussed under Failure Modes.

This is exactly the property a hypervisor needs. When a Virtual Machine Monitor (VMM) such as QEMU gives a guest a real NIC or GPU, the guest’s driver will program the device’s DMA with guest-physical addresses. VFIO lets the VMM install IOMMU mappings of the form “guest-physical → host-physical” so that when the device DMAs to what the guest thinks is its own RAM, the IOMMU silently redirects it to the right host pages — and a compromised guest still cannot make the device touch any host memory the VMM did not map. Passthrough thereby reaches near-native device speed while preserving isolation (cross-link the virtualization map’s “passthrough is the safety-vs-speed dial” theme).

---

Mental Model — Container, Group, Device

flowchart TB
  subgraph US["Userspace VMM (e.g. QEMU)"]
    C["container fd<br/>= open('/dev/vfio/vfio')<br/>the DMA / IOMMU context"]
    GF["group fd<br/>= open('/dev/vfio/N')<br/>N = IOMMU group number"]
    DF["device fd<br/>= GROUP_GET_DEVICE_FD('0000:06:0d.0')<br/>regions + IRQs + reset"]
  end
  subgraph K["Kernel: drivers/vfio"]
    VC["vfio container +<br/>vfio_iommu_type1 backend"]
    VG["vfio_group<br/>(viability check)"]
    VPCI["vfio-pci driver<br/>(bus driver bound to the device)"]
  end
  IOMMU["IOMMU hardware<br/>(per-group domain / page table)"]
  DEV["Physical PCI device"]
  C -->|VFIO_SET_IOMMU<br/>VFIO_IOMMU_MAP_DMA| VC
  GF -->|VFIO_GROUP_SET_CONTAINER| VC
  GF -->|VFIO_GROUP_GET_DEVICE_FD| VG
  DF -->|read/write/mmap regions<br/>SET_IRQS eventfd| VPCI
  VC -->|programs IOVA->HPA| IOMMU
  VG --- VPCI
  VPCI --- DEV
  IOMMU --- DEV

The legacy VFIO three-tier file-descriptor model. What it shows: userspace opens three nested fds — the container is the DMA address space (you map guest memory into it), the group is the indivisible hardware isolation unit you attach to that container, and the device fd is how you actually talk to one device’s registers and interrupts. The insight to take: the group, not the device, is the unit of ownership, because the IOMMU can only isolate at the granularity the PCIe topology allows; the container is where DMA mappings live and is the thing the IOMMU enforces. iommufd later collapses this into “open /dev/iommu, bind a device fd to it” — same enforcement, fewer fds.

The three-tier model exists because of a hardware reality the kernel cannot wish away. An IOMMU does not always isolate at single-device granularity. Several devices behind a PCIe switch without Access Control Services (ACS), or the multiple functions of one multi-function card, may be able to issue peer-to-peer transactions to each other that never reach the IOMMU to be checked. Such devices are mutually unisolatable, so the kernel bundles them into one IOMMU group — “a set of devices which is isolatable from all other devices in the system. Groups are therefore the unit of ownership used by VFIO” (VFIO docs, v6.12). You cannot pass through part of a group; you take the whole group or nothing.

This is the single most surprising thing about VFIO in practice, so it is worth understanding why groups form the way they do rather than treating group membership as arbitrary. vfio.rst names four distinct mechanisms, and they are different failure modes of isolation, not variations on one:

Cause	What happens	Typical hardware
Multi-function device with backdoors	Functions of one card talk to each other internally; the transaction never leaves the package, so the IOMMU never sees it	Dual-port NICs, a GPU and its companion HDMI audio function
Bridge without ACS	A PCIe switch or root port lacking Access Control Services may redirect a peer-to-peer transaction between downstream ports without sending it upstream to the IOMMU	Consumer chipsets, many desktop motherboards
PCIe-to-PCI bridge	The bridge “masks the devices behind it, making transaction appear as if from the bridge itself” — the IOMMU cannot tell which device originated a request, so it must treat them as one	Legacy PCI slots behind a bridge
IOMMU design	“Obviously IOMMU design plays a major factor as well” — some IOMMUs simply cannot distinguish at device granularity	Varies by platform

The four reasons a group contains more than one device, quoted and paraphrased from Documentation/driver-api/vfio.rst (v6.12). What it shows: every cause is a place where a transaction can reach another device without passing the IOMMU. The insight: a group is not an administrative grouping the kernel chose — it is the kernel’s honest report of the smallest set of devices the hardware can actually keep apart. Which is why the kernel will not negotiate about it.

flowchart TB
  IOMMU["IOMMU — the only checkpoint"]
  IOMMU --> RP0["Root port 0<br/>ACS capable"]
  IOMMU --> RP1["Root port 1<br/>NO ACS"]
  RP0 --> N1["06:00.0 NIC"]
  RP1 --> SW["PCIe switch<br/>NO ACS"]
  SW --> G0["01:00.0 GPU"]
  SW --> G1["01:00.1 GPU HDMI audio"]
  SW --> NVME["02:00.0 NVMe SSD"]
  G0 <-.->|"peer-to-peer, never<br/>reaches the IOMMU"| G1
  G0 <-.->|"peer-to-peer"| NVME
  N1 --- GRPA["IOMMU group 12<br/>1 device — assignable alone"]
  G0 --- GRPB
  G1 --- GRPB
  NVME --- GRPB["IOMMU group 15<br/>3 devices — ALL OR NOTHING<br/>the SSD comes with the GPU"]

How PCIe topology decides what you are allowed to assign. What it shows: the NIC sits behind an ACS-capable root port, so the IOMMU can isolate it and it forms a group of one. The GPU, its audio function and an unrelated NVMe SSD sit behind a switch with no ACS, so they can exchange peer-to-peer transactions the IOMMU never inspects — and the kernel therefore bundles all three into one group. The insight to take away, and the one that catches people out: wanting to pass through the GPU is not the question. The question is what else shares its group, and the answer is decided by your motherboard, not by your configuration. On this topology you must hand the guest the SSD as well, or you get nothing.

Check yours before planning anything: ls /sys/kernel/iommu_groups/*/devices/ enumerates every group on the machine, and /sys/bus/pci/devices/<addr>/iommu_group is the symlink from a device to its group. The detailed mechanics of group formation live in IOMMU Groups and Device Isolation.

A container is the IOMMU context — concretely, one or more IOMMU domains sharing a single IOVA→host-physical mapping. You attach one or more groups to a container, then program DMA mappings on the container; every device in every attached group then sees that same IOVA layout. For a single VM you typically have one container holding all the guest’s passed-through groups, so a single VFIO_IOMMU_MAP_DMA per guest-RAM region suffices for all of them.

---

Mechanical Walk-through — How a VMM Claims a Device

The end-to-end sequence, traced against the v6.12 documentation and drivers/vfio/vfio_main.c, runs as follows.

Step 1 — Unbind the device from its normal driver and bind it to vfio-pci. A PCI device is normally claimed by its functional driver (e.g. ixgbe for an Intel NIC). To pass it through, the operator unbinds it and binds it to vfio-pci (or vfio-platform, vfio-mdev, etc.), the VFIO bus driver. This is done through sysfs — writing the device’s address to /sys/bus/pci/drivers/vfio-pci/bind, usually after telling vfio-pci to claim the device’s vendor/device ID. Once vfio-pci owns the device, the kernel creates the group’s character device /dev/vfio/$GROUP.

Step 2 — Create the container. Userspace opens container = open("/dev/vfio/vfio", O_RDWR). This is an empty DMA context with no IOMMU model selected yet. The VMM sanity-checks the ABI with ioctl(container, VFIO_GET_API_VERSION) (which must equal VFIO_API_VERSION, defined as 0 in v6.12) and probes which IOMMU backends the kernel offers with ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU) (vfio.h, v6.12).

Step 3 — Open the group and check viability. group = open("/dev/vfio/26", O_RDWR) opens the device’s group (group 26 here). The crucial gate is ioctl(group, VFIO_GROUP_GET_STATUS, &group_status): the returned flags must include VFIO_GROUP_FLAGS_VIABLE. A group is viable only when every device in it is bound to a VFIO driver, or to no driver at all. If even one sibling is still bound to a host driver, the group is non-viable and the kernel refuses to let you use it — because that host-driven sibling could DMA wherever it likes, defeating the isolation the group is supposed to provide. This is the kernel enforcing “take the whole group” at runtime.

There is a nuance here that is widely got wrong, including by the earlier version of this note. Siblings do not all have to be bound to vfio-pci. vfio.rst states the weaker requirement explicitly: “If the IOMMU group contains multiple devices, each will need to be bound to a VFIO driver before operations on the VFIO group are allowed (it’s also sufficient to only unbind the device from host drivers if a VFIO driver is unavailable; this will make the group available, but not that particular device).” So an unwanted sibling — a PCI bridge, a device with no VFIO driver — can simply be left driverless and the group becomes viable. You do not get to use that device, but you do get to use the one you wanted. Practically this is the difference between “I cannot pass through my GPU because a bridge is in its group” and “unbind the bridge and carry on.”

Step 4 — Attach the group to the container, then set the IOMMU model. ioctl(group, VFIO_GROUP_SET_CONTAINER, &container) places the group inside the DMA context. Only after a group is attached can you select the IOMMU backend: ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU). The ordering matters because the type1 backend needs a real IOMMU group to attach its domain to. VFIO_TYPE1_IOMMU (value 1) is the original x86/ARM model; VFIO_TYPE1v2_IOMMU (value 3) is the refined version most code uses today; VFIO_SPAPR_TCE_IOMMU (value 2) is the PowerPC sPAPR variant; VFIO_NOIOMMU_IOMMU (value 8) is the deliberately-unsafe no-IOMMU mode discussed under Failure Modes (vfio.h, v6.12).

Step 5 — Map guest memory for DMA. The VMM now tells the IOMMU which host memory the device is allowed to reach, with VFIO_IOMMU_MAP_DMA. The argument is a struct vfio_iommu_type1_dma_map:

struct vfio_iommu_type1_dma_map {
	__u32 argsz;     /* sizeof(struct) — the VFIO forward-compat convention */
	__u32 flags;     /* VFIO_DMA_MAP_FLAG_READ (1<<0), VFIO_DMA_MAP_FLAG_WRITE (1<<1) */
	__u64 vaddr;     /* process virtual address of the buffer (e.g. guest RAM) */
	__u64 iova;      /* I/O virtual address the device will use to reach it    */
	__u64 size;      /* length of the mapping in bytes */
};

For a VM, vaddr is the host userspace address where the VMM mmap’d the guest’s RAM, iova is the corresponding guest-physical address, and size is the region length. After this ioctl the IOMMU’s page table contains an entry translating that IOVA to the underlying host-physical pages, with the read/write permissions requested. The kernel pins these pages (so they cannot be swapped or moved out from under a device that may DMA at any instant) and charges them against the process’s locked-memory limit. VFIO_IOMMU_UNMAP_DMA (with struct vfio_iommu_type1_dma_unmap) tears a range back down and unpins the pages. The matching translation hardware is the subject of The IOMMU and DMA Remapping.

Step 6 — Get the device fd. device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:06:0d.0") returns a new fd for one specific device, identified by its PCI address string. In vfio_main.c this drives vfio_df_open(), which increments device->open_count, and on the first open calls the bus driver’s open_device callback and arms IOMMU usage; the file’s read/write/mmap/ioctl handlers only become live after access_granted is published with smp_load_acquire(&df->access_granted), a barrier ensuring the device is fully set up before userspace can touch it (vfio_main.c, v6.12).

Step 7 — Discover and use regions and IRQs. ioctl(device, VFIO_DEVICE_GET_INFO, &device_info) returns the device’s capability flags (VFIO_DEVICE_FLAGS_PCI, VFIO_DEVICE_FLAGS_RESET, …) and the count of regions and IRQs. Each region — a PCI BAR (Base Address Register window), the configuration space, the option ROM — is then described by VFIO_DEVICE_GET_REGION_INFO, which reports the region’s size and the offset into the device fd at which it can be read()/write()/mmap()’d. The PCI regions are indexed by a fixed enum: VFIO_PCI_BAR0_REGION_INDEX … VFIO_PCI_BAR5_REGION_INDEX, VFIO_PCI_ROM_REGION_INDEX, VFIO_PCI_CONFIG_REGION_INDEX, VFIO_PCI_VGA_REGION_INDEX. The VMM mmaps the MMIO BARs so the guest’s accesses to those registers hit the real hardware directly (no VM exit per register), while configuration-space accesses go through read()/write() so VFIO can emulate and police sensitive config registers (it must not let the guest reprogram the device’s real BAR addresses, for instance).

Interrupts are described by VFIO_DEVICE_GET_IRQ_INFO, indexed VFIO_PCI_INTX_IRQ_INDEX, VFIO_PCI_MSI_IRQ_INDEX, VFIO_PCI_MSIX_IRQ_INDEX. The VMM wires them up with VFIO_DEVICE_SET_IRQS, passing an eventfd via the VFIO_IRQ_SET_DATA_EVENTFD flag. When the device raises that interrupt, the kernel signals the eventfd; KVM’s irqfd plumbing can be hooked to that same eventfd to inject a virtual interrupt into the guest with no VM exit to userspace at all (see Linux Virtualization MOC’s interrupt-virtualization section). VFIO_IRQ_SET_DATA_NONE and VFIO_IRQ_SET_DATA_BOOL are the alternative data modes for masking/unmasking. Finally VFIO_DEVICE_RESET performs a function-level reset so the device is returned to a clean state between owners.

sequenceDiagram
    autonumber
    participant OP as Operator (sysfs)
    participant VMM as QEMU / DPDK
    participant K as VFIO core
    participant T1 as vfio_iommu_type1
    participant IOM as IOMMU hardware
    participant DEV as PCI device
    OP->>K: unbind from host driver,<br/>bind 0000:06:0d.0 to vfio-pci
    K-->>OP: /dev/vfio/26 appears (group 26)
    VMM->>K: open("/dev/vfio/vfio") — container fd
    VMM->>K: VFIO_GET_API_VERSION (must be 0)
    VMM->>K: VFIO_CHECK_EXTENSION(VFIO_TYPE1v2_IOMMU)
    VMM->>K: open("/dev/vfio/26") — group fd
    VMM->>K: VFIO_GROUP_GET_STATUS
    K-->>VMM: flags must include VFIO_GROUP_FLAGS_VIABLE
    Note over K: non-viable = a sibling still<br/>has a host driver bound
    VMM->>K: VFIO_GROUP_SET_CONTAINER(container_fd)
    VMM->>T1: VFIO_SET_IOMMU(VFIO_TYPE1v2_IOMMU)
    Note over T1,IOM: ordering matters — the backend needs a<br/>real group to attach a domain to
    T1->>IOM: check iommu_group_has_isolated_msi()
    IOM-->>T1: no interrupt remapping, so EPERM
    VMM->>T1: VFIO_IOMMU_MAP_DMA (vaddr, iova, size, R/W)
    T1->>T1: pin pages, charge RLIMIT_MEMLOCK,<br/>decrement dma_avail
    T1->>IOM: install IOVA to host-physical translations
    VMM->>K: VFIO_GROUP_GET_DEVICE_FD("0000:06:0d.0")
    K->>DEV: bus driver open_device() — publish access_granted
    K-->>VMM: device fd
    VMM->>DEV: VFIO_DEVICE_GET_INFO / GET_REGION_INFO
    VMM->>DEV: mmap BAR regions (guest MMIO hits hardware directly)
    VMM->>DEV: VFIO_DEVICE_SET_IRQS with eventfd
    DEV-->>VMM: interrupts now signal that eventfd
    VMM->>DEV: VFIO_DEVICE_RESET before handing the device on

The complete legacy claim sequence, in the order the kernel requires it. What it shows: three file descriptors opened in a fixed nesting order, a viability gate before the group may join a container, an IOMMU model selected only after a group is present, DMA mappings programmed on the container, and only then a per-device fd. The insight: almost every step is a check rather than a configuration. The API is shaped like an argument the userspace process has to win — prove the group is whole, prove the platform isolates interrupts, prove the memory is pinnable within your limit — and only at the end does it get something it can actually drive.

Step 6 has one detail worth calling out because it is the kind of thing that goes wrong silently. In vfio_main.c, vfio_df_open() increments device->open_count and, on the first open, calls the bus driver’s open_device callback; the file’s read/write/mmap/ioctl handlers only become live once access_granted is published and read back with smp_load_acquire(&df->access_granted). That acquire/release pair is a memory barrier ensuring every side effect of device setup is visible before any userspace access can observe the fd as usable — without it, a sufficiently aggressive CPU could let a read() on the device fd race ahead of the setup that made the read safe.

---

Interrupt Delivery — eventfd, irqfd, and Why INTx Is Hard

Getting a device’s data into a guest is DMA. Getting its interrupts into a guest is a separate problem with its own machinery, and the code is unusually instructive because the three PCI interrupt types need three different amounts of work.

The universal primitive is the eventfd. Userspace registers one per interrupt with VFIO_DEVICE_SET_IRQS and the VFIO_IRQ_SET_DATA_EVENTFD flag; when the device raises that interrupt, the kernel signals the eventfd. The other data modes are VFIO_IRQ_SET_DATA_NONE (trigger or mask/unmask immediately) and VFIO_IRQ_SET_DATA_BOOL (conditional). The IRQ indices are fixed for PCI: VFIO_PCI_INTX_IRQ_INDEX, VFIO_PCI_MSI_IRQ_INDEX, VFIO_PCI_MSIX_IRQ_INDEX, plus ERR and REQ indices for error and device-request notifications.

For MSI and MSI-X, that is nearly the whole story, and the handler shows it (drivers/vfio/pci/vfio_pci_intrs.c, v6.12):

static irqreturn_t vfio_msihandler(int irq, void *arg)
{
	struct eventfd_ctx *trigger = arg;
 
	eventfd_signal(trigger);
	return IRQ_HANDLED;
}

Three lines. MSI is edge-triggered and not shared, so there is nothing to mask, nothing to disambiguate, and no state to keep: signal the eventfd and return. The interrupt is registered with a plain request_irq(irq, vfio_msihandler, 0, ctx->name, trigger).

INTx is the opposite, and the difference is entirely about level-triggering. A legacy INTx line stays asserted until the device’s own driver clears the condition — and that driver is inside the guest, which has not run yet. If the host handler simply signalled an eventfd and returned, the line would still be asserted, the handler would be re-entered immediately, and the machine would live-lock in an interrupt storm. So vfio_intx_handler() masks the interrupt before signalling:

spin_lock_irqsave(&vdev->irqlock, flags);
if (!vdev->pci_2_3) {
	disable_irq_nosync(vdev->pdev->irq);   /* no hardware mask available */
	ctx->masked = true;
	ret = IRQ_HANDLED;
} else if (!ctx->masked &&  /* may be shared */
	   pci_check_and_mask_intx(vdev->pdev)) {
	ctx->masked = true;
	ret = IRQ_HANDLED;
}
spin_unlock_irqrestore(&vdev->irqlock, flags);
 
if (ret == IRQ_HANDLED)
	vfio_send_intx_eventfd(vdev, ctx);

Two paths, chosen by whether the device supports PCI 2.3 interrupt disabling. If it does, pci_check_and_mask_intx() both confirms this device is the one asserting the shared line and masks it in one operation — necessary because INTx lines are shared and the handler is registered with IRQF_SHARED. If it does not, the whole host IRQ is disabled with disable_irq_nosync(), which is heavier because it silences the line for every device on it. Only after masking succeeds is the eventfd signalled.

That leaves the question of who unmasks. The guest’s driver will eventually service the device and expect the line to be usable again — but a syscall per interrupt would destroy the performance passthrough exists to provide. VFIO’s answer is virqfd: ctx->unmask and ctx->mask are struct virqfd objects the VMM arms with vfio_virqfd_enable(), so the guest can unmask by writing to an eventfd that the kernel watches, with no ioctl and no exit to the VMM.

The last piece closes the loop into KVM. CONFIG_VFIO_PCI_CORE carries select IRQ_BYPASS_MANAGER alongside select VFIO_VIRQFD (drivers/vfio/pci/Kconfig, v6.12). The IRQ bypass manager is what lets KVM’s irqfd be wired to the same eventfd VFIO signals, so a physical device interrupt is injected into the guest’s virtual interrupt controller — on capable hardware, via posted interrupts, without the host CPU exiting to the VMM at all.

sequenceDiagram
    participant HW as Physical device
    participant HOST as Host IRQ handler (vfio-pci)
    participant EFD as eventfd
    participant KVM as KVM irqfd / IRQ bypass
    participant G as Guest driver
    rect rgb(240,240,240)
    Note over HW,G: MSI / MSI-X — edge-triggered, the easy case
    HW->>HOST: MSI write
    HOST->>EFD: eventfd_signal(trigger) — 3 lines, no state
    EFD->>KVM: irqfd fires
    KVM->>G: virtual interrupt injected<br/>(no exit to the VMM)
    end
    rect rgb(230,238,248)
    Note over HW,G: INTx — level-triggered and shared, the hard case
    HW->>HOST: line asserted, stays asserted
    HOST->>HOST: pci_check_and_mask_intx()<br/>— is it us? mask it<br/>(else disable_irq_nosync)
    HOST->>EFD: only now vfio_send_intx_eventfd()
    EFD->>KVM: irqfd fires
    KVM->>G: virtual interrupt injected
    G->>G: guest driver services the device,<br/>clears the condition
    G->>HOST: writes the unmask virqfd<br/>(no ioctl, no VMM exit)
    HOST->>HW: line unmasked, ready for the next
    end

Interrupt delivery for the two interrupt models. What it shows: for MSI the path is a straight line from hardware to guest with the host doing essentially nothing; for INTx the host must mask the line before signalling, because nothing will deassert it until the guest’s driver runs, and the guest must later unmask through a second eventfd. The insight: the asymmetry is not VFIO being inconsistent — it is the difference between edge- and level-triggered interrupts surfacing in code. It is also a concrete reason to prefer MSI-X-capable devices for passthrough: an INTx device costs an extra masking operation and an extra eventfd round trip on every single interrupt.

Two platform notes from the same Kconfig: VFIO_PCI_INTX is def_bool y if !S390 and VFIO_PCI_MMAP likewise — s390x has neither legacy INTx nor mmap’able BARs in this framework and uses VFIO_PCI_ZDEV_KVM instead.

The Modern Path — iommufd

The container/group model has aged poorly. Its limitations are structural: a container is a single, flat DMA address space (one IOVA map shared by everything attached), which made advanced features awkward — nested translation (a guest running its own IOMMU, needing two-stage page tables), PASID (Process Address Space ID, for sharing a device among multiple address spaces), and fine-grained per-device control all fought against the “one container, one IOMMU model” assumption. The group fd was also a clumsy intermediary.

iommufd is the replacement, merged into the mainline kernel in 6.2 (its pull request went out for the 6.2 merge window in December 2022; the design had been in development for roughly two years prior) and authored largely by Jason Gunthorpe (Phoronix, 2022; LWN: Connect VFIO to iommufd, 2022). It is a standalone subsystem reachable through /dev/iommu whose job is, in the documentation’s words, “to control the IOMMU subsystem as it relates to managing IO page tables from userspace using file descriptors” (iommufd docs, v6.12). Its core user-visible objects are:

• IOMMUFD_OBJ_IOAS — an I/O Address Space, the map/unmap target. “The IOAS is a functional replacement for the VFIO container” and copies an IOVA map into a list of iommu_domains held within it.
• IOMMUFD_OBJ_HW_PAGETABLE (an hwpt) — an actual hardware I/O page table, i.e. a single struct iommu_domain managed by the IOMMU driver.
• IOMMUFD_OBJ_DEVICE — a device bound to iommufd by an external driver (here, vfio-pci).

The device-centric flow drops the group fd from userspace’s view. With CONFIG_VFIO_DEVICE_CDEV=y the device appears directly as /dev/vfio/devices/vfioX; userspace opens that, opens /dev/iommu, calls VFIO_DEVICE_BIND_IOMMUFD to attach the device fd to the iommufd context, allocates an IOAS with IOMMU_IOAS_ALLOC, attaches the device’s page table with VFIO_DEVICE_ATTACH_IOMMUFD_PT, and maps memory with IOMMU_IOAS_MAP/IOMMU_IOAS_UNMAP instead of VFIO_IOMMU_MAP_DMA. The group’s viability and isolation rules still apply underneath — iommufd checks them when binding — but the API no longer makes you juggle three fds.

Crucially, iommufd ships a VFIO compatibility layer so old VMMs keep working: it can directly implement the /dev/vfio/vfio container ioctls by mapping them onto iommufd’s internal io_pagetable operations, even allowing /dev/vfio/vfio to be symlinked to /dev/iommu. The IOMMU_VFIO_IOAS ioctl ties a legacy container’s IOVA space to an iommufd IOAS.

The Kernel Says So Out Loud

This is not a matter of interpretation. Documentation/driver-api/vfio.rst at v6.12 states the intent in two sentences that are worth quoting exactly, because they settle the “is the container model deprecated?” question:

IOMMUFD is the new user API to manage I/O page tables from userspace. It intends to be the portal of delivering advanced userspace DMA features (nested translation, PASID, etc.) while also providing a backwards compatibility interface for existing VFIO_TYPE1v2_IOMMU use cases. Eventually the vfio_iommu_type1 driver, as well as the legacy vfio container and group model is intended to be deprecated. … Long term, VFIO users should migrate to device access through the cdev interface described below, and native access through the IOMMUFD provided interfaces.

So: intended to be deprecated, not deprecated. Which is exactly the distinction that matters operationally, and it is reinforced by the same file’s warning about the compatibility layer: “at the time of writing, the compatibility mode is not entirely feature complete relative to VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface. Therefore it is not generally advisable at this time to switch from native VFIO implementations to the IOMMUFD compatibility interfaces.”

What Your Kernel Actually Built — the Kconfig Reality

Here is the fact most likely to waste an afternoon, and it is invisible from the documentation. On a stock distribution kernel, /dev/vfio/devices/vfioX probably does not exist, even though iommufd does. Read drivers/vfio/Kconfig (v6.12):

menuconfig VFIO
	select VFIO_GROUP if SPAPR_TCE_IOMMU || IOMMUFD=n
	select VFIO_DEVICE_CDEV if !VFIO_GROUP
	select VFIO_CONTAINER if IOMMUFD=n

config VFIO_DEVICE_CDEV
	bool "Support for the VFIO cdev /dev/vfio/devices/vfioX"
	depends on IOMMUFD && !SPAPR_TCE_IOMMU
	default !VFIO_GROUP        # <-- the trap

config VFIO_GROUP
	bool "Support for the VFIO group /dev/vfio/$group_id"
	default y                  # <-- ...because this is y

config VFIO_CONTAINER
	bool "Support for the VFIO container /dev/vfio/vfio"
	depends on VFIO_GROUP
	default y

VFIO_GROUP defaults to y, described as “the traditional model for accessing devices through VFIO and is used by the majority of userspace applications.” VFIO_DEVICE_CDEV defaults to !VFIO_GROUP — which, given the line above, is n. And VFIO_CONTAINER defaults to y with the guidance “Unless testing IOMMUFD say Y here.” The device-centric cdev path is therefore default-off in 6.12, and a distribution has to opt into it deliberately.

Config symbol	Default in v6.12	Provides	Note
VFIO_GROUP	y	/dev/vfio/$group_id	“used by the majority of userspace applications”
VFIO_CONTAINER	y (requires VFIO_GROUP)	/dev/vfio/vfio	“Unless testing IOMMUFD say Y here”
VFIO_DEVICE_CDEV	n (default !VFIO_GROUP)	/dev/vfio/devices/vfioX	Requires IOMMUFD; “does not support noiommu”
VFIO_NOIOMMU	n (requires VFIO_GROUP)	unsafe no-IOMMU mode	Taints the kernel

Kconfig defaults as shipped in v6.12. What it shows: the legacy path is the default-on path and the modern cdev path is default-off. The insight: “iommufd was merged in 6.2” and “your kernel exposes the device cdev” are entirely different claims. Check before designing around it: ls /dev/vfio/devices/ and grep -E 'VFIO_(GROUP|CONTAINER|DEVICE_CDEV)' /boot/config-$(uname -r).

And even with the cdev enabled, group semantics have not gone away — they have gone underneath. vfio.rst is blunt: “vfio device cdev access is still bound by IOMMU group semantics, ie. there can be only one DMA owner for the group. Devices belonging to the same group can not be bound to multiple iommufd_ctx… A violation of this ownership requirement will fail at the VFIO_DEVICE_BIND_IOMMUFD ioctl, which gates full device access.” Two more constraints from the same file: “The cdev only works with IOMMUFD”, and “cdev interface does not support noiommu devices, so user should use the legacy group interface if noiommu is wanted.” So DPDK on a machine without a usable IOMMU is stuck on the legacy path by construction.

flowchart TB
  subgraph LEG["Legacy: container / group / device — default-on in v6.12"]
    direction TB
    L1["open /dev/vfio/vfio<br/>= container = one flat IOVA space"]
    L2["open /dev/vfio/$GROUP<br/>check VFIO_GROUP_FLAGS_VIABLE"]
    L3["VFIO_GROUP_SET_CONTAINER"]
    L4["VFIO_SET_IOMMU(VFIO_TYPE1v2_IOMMU)"]
    L5["VFIO_IOMMU_MAP_DMA"]
    L6["VFIO_GROUP_GET_DEVICE_FD('0000:06:0d.0')"]
    L1 --> L2 --> L3 --> L4 --> L5 --> L6
  end
  subgraph NEW["iommufd + cdev — merged 6.2, default-OFF in v6.12"]
    direction TB
    N1["open /dev/iommu<br/>= iommufd ctx"]
    N2["open /dev/vfio/devices/vfioX<br/>device fd directly — no group fd"]
    N3["VFIO_DEVICE_BIND_IOMMUFD<br/>claims DMA ownership;<br/>group rules enforced HERE"]
    N4["IOMMU_IOAS_ALLOC<br/>= IOAS, replaces the container"]
    N5["VFIO_DEVICE_ATTACH_IOMMUFD_PT"]
    N6["IOMMU_IOAS_MAP / IOAS_UNMAP"]
    N1 --> N2 --> N3 --> N4 --> N5 --> N6
  end
  LEG -.->|"compat layer: CONFIG_IOMMUFD_VFIO_CONTAINER,<br/>or symlink /dev/vfio/vfio -&gt; /dev/iommu,<br/>or IOMMU_VFIO_IOAS ioctl"| NEW
  NEW --> ADV["what the new shape unlocks:<br/>nested translation (IOMMU_HWPT_ALLOC_NEST_PARENT),<br/>hardware dirty tracking (HWPT_SET_DIRTY_TRACKING),<br/>userspace fault queues (FAULT_QUEUE_ALLOC),<br/>PASID, IOAS sharing between VFIO and VDPA"]

The two ABIs side by side. What it shows: the legacy path opens three nested descriptors in a required order and puts the IOMMU model selection in the middle; the iommufd path opens two, binds them, and treats the device — not the group — as the thing userspace names. The insight: the group did not disappear, it moved. In the legacy model the group is an object you hold; in iommufd it is an invariant the kernel checks at VFIO_DEVICE_BIND_IOMMUFD. That is the whole ergonomic win — the same isolation guarantee, with the hardware’s awkwardness no longer projected into your file-descriptor bookkeeping.

Documentation Lag — Verified at One Tag

The v6.12 iommufd.rst closes with a “Future TODOs” list that includes “Userspace page tables, for ARM, x86 and S390”, “Dirty page tracking in the IOMMU”, and “PRI support with faults resolved in userspace”; earlier it states flatly that a device can “attach to at most one IOAS object (no support of PASID yet)”, and that the two VFIO-compatibility approaches are “still work-in-progress.”

Those statements are contradicted by include/uapi/linux/iommufd.h in the same tree at the same tag, which already defines:

Command / flag in v6.12 uapi	Corresponds to a doc “future TODO”
IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING (0x8b), HWPT_GET_DIRTY_BITMAP (0x8c)	“Dirty page tracking in the IOMMU”
IOMMU_HWPT_ALLOC_DIRTY_TRACKING flag	ditto
IOMMU_HWPT_ALLOC_NEST_PARENT flag, IOMMU_HWPT_DATA_VTD_S1 (Intel VT-d stage-1, i.e. a guest-managed page table)	“Userspace page tables” / nested translation
IOMMUFD_CMD_FAULT_QUEUE_ALLOC (0x8e), IOMMU_HWPT_FAULT_ID_VALID flag	“PRI support with faults resolved in userspace”
IOMMUFD_CMD_HWPT_INVALIDATE (0x8d)	“Kernel bypass’d invalidation of user page tables”

In-tree prose versus in-tree uapi, both read at v6.12. What it shows: nesting, hardware dirty tracking, page-table invalidation and a userspace fault queue all have shipped ioctl numbers while the documentation still lists them as unstarted. The insight, and the general lesson: kernel prose documentation rots faster than kernel code, because the code has a compiler and the prose does not. When a doc and a header disagree, the header is the fact. This is why the note you are reading cites .h files rather than .rst files for anything an application would call.

The word “PASID” (Process Address Space ID) appears 6 times in include/uapi/linux/iommufd.h at v6.12 and 0 times in include/uapi/linux/vfio.h — consistent with PASID being an iommufd-native concept that the legacy VFIO ABI never gained. For a forward look: v6.18 adds IOAS_MAP_FILE (0x8f), VIOMMU_ALLOC (0x90), VDEVICE_ALLOC (0x91), IOAS_CHANGE_PROCESS (0x92), VEVENTQ_ALLOC (0x93) and HW_QUEUE_ALLOC (0x94) — a virtual-IOMMU object model that did not exist in 6.12.

Uncertain

Verify: whether a specific advanced feature is production-ready end to end on a specific platform and kernel. Reason: the presence of an ioctl number in the uapi header proves the ABI exists, which is what is asserted above — it does not prove that a given IOMMU driver (Intel VT-d, AMD-Vi, ARM SMMUv3) implements it, nor that QEMU exposes it. Only IOMMU_HWPT_DATA_VTD_S1 is defined in 6.12, for instance, which suggests Intel-only nesting data at that tag. To resolve: read drivers/iommu/intel/, drivers/iommu/amd/ and drivers/iommu/arm/arm-smmu-v3/ at the exact tag you ship to see which iommu_domain_ops hooks are populated, and check QEMU’s release notes for backend support. The core claims here — that iommufd exists, replaces the container model, is usable for basic passthrough since 6.2, and has shipped ABI for nesting and dirty tracking by 6.12 — are verified against the tree. uncertain

On the QEMU Side

The two backends coexist in QEMU, and selection is per-device. Interactions with /dev/iommu are abstracted by a new iommufd object (compiled in with CONFIG_IOMMUFD), and “any QEMU device (e.g. VFIO device) wishing to use /dev/iommu must be linked with an iommufd object”:

# new backend: link the vfio-pci device to an iommufd object
-object iommufd,id=iommufd0 \
-device vfio-pci,host=0000:02:00.0,iommufd=iommufd0
 
# fd passing: a privileged management layer opens /dev/iommu and the
# VFIO cdev, then hands the descriptors to an unprivileged QEMU
-object iommufd,id=iommufd0,fd=22 \
-device vfio-pci,iommufd=iommufd0,fd=23
 
# legacy: omit the iommufd object entirely
-device vfio-pci,host=0000:02:00.0

“If no iommufd object is passed to the vfio-pci device, iommufd is not used and the user gets the behavior based on the legacy VFIO container” — so legacy remains QEMU’s default, and iommufd is opt-in per device. The doc lists x86, Arm and s390x as supported platforms, and carries three caveats worth knowing before you migrate (QEMU, IOMMUFD BACKEND usage with VFIO):

• PCI peer-to-peer DMA is unsupported, “as IOMMUFD doesn’t support mapping hardware PCI BAR region yet.” It surfaces as a warning, not an error: IOMMU_IOAS_MAP failed: Bad address, PCI BAR? — and the doc says explicitly “it’s not a bug.” This is the same gap vfio.rst refers to as “DMA mapping MMIO”.
• fd passing breaks mdev detection. QEMU decides a backend is an mdev by checking the sysfsdev property; with fd passing there is no way to know, so an mdev is treated like a real PCI device and x-balloon-allowed=on is rejected.
• Intel VT-d with fsts=on requires the iommufd backend — the legacy container path fails outright with “Need IOMMUFD backend when fsts=on”. So the migration is already mandatory for some configurations, not merely advisable.

Failure Modes and Common Misunderstandings

Non-viable group (“device is in use”). The single most common passthrough failure: VFIO_GROUP_SET_CONTAINER or GET_DEVICE_FD fails because another device in the same IOMMU group is still bound to a host driver. List the group with ls /sys/kernel/iommu_groups/N/devices/ and deal with every member — but recall the nuance from the walk-through above: each sibling must be bound to a VFIO driver or to no driver at all. Unbinding a bridge or an unwanted function is sufficient to make the group viable; you do not have to find a vfio-pci binding for it. On consumer hardware whose chipset lacks ACS, an entire PCIe slot complex can collapse into one group, forcing you to surrender far more than intended — hence the unofficial “ACS override” kernel patch, which fakes isolation and is a genuine security downgrade. The Arch wiki documents both the technique and the surrounding practice, including the warning that PCI root ports and bridges grouped this way “should neither be bound to vfio at boot, nor be added to the virtual machine” (Arch Wiki: PCI passthrough via OVMF). See IOMMU Groups and Device Isolation.

GPUs specifically resist late rebinding. Unbinding most devices from their host driver can be done seconds before the VM starts. GPU drivers are the exception: “due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well,” so the practical recipe is to bind vfio-pci early in boot (via modprobe.d options or an initramfs hook) rather than switching drivers on a running system (Arch Wiki). The cost is that the GPU is unavailable to the host from boot.

Running out of DMA mapping entries. A container has a hard cap on how many separate mappings it may hold, independent of their total size. dma_entry_limit defaults to U16_MAX and is documented as “Maximum number of user DMA mappings per container (65535)” (drivers/vfio/vfio_iommu_type1.c, v6.12); the running count is iommu->dma_avail, decremented per successful map and incremented per unmap, and a map attempt at zero fails. This bites workloads that map many small regions rather than a few large ones — some vIOMMU and userspace-driver patterns do exactly that. It is a writable module parameter (mode 0644), and the current headroom is queryable through the VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL capability on VFIO_IOMMU_GET_INFO.

No-IOMMU mode is genuinely unsafe. VFIO_NOIOMMU_IOMMU exists for high-performance userspace drivers (DPDK on hardware with no usable IOMMU) but provides zero DMA protection — the device can DMA anywhere. The kernel taints itself when it is used and the cdev interface refuses to support it at all. It is not a passthrough mechanism; it is “I accept that this userspace driver can corrupt all of memory.”

Forgetting to map all guest RAM. If the VMM maps only part of guest memory for DMA, the guest’s device will work until its driver hands the device a buffer in an unmapped region; the device’s DMA then faults in the IOMMU (an IOMMU page fault / DMAR fault in the host logs) and the I/O silently fails or the device wedges. The symptom is a device that works for small/early transfers and dies under load.

Pinned memory and overcommit. This one is real and unavoidable. A device may DMA at any instant, so every mapped page must be resident: VFIO_IOMMU_MAP_DMA pins the pages and charges them against the owning process’s locked-memory limit. A passthrough VM’s entire RAM is therefore locked and cannot be swapped, ballooned, or KSM-merged. A VMM that does not raise RLIMIT_MEMLOCK hits -ENOMEM from VFIO_IOMMU_MAP_DMA as soon as the mapping crosses the limit.

The accounting is more careful than “count the pages,” and the reason is instructive. Each vfio_dma records dma->task = current->group_leader (with a get_task_struct reference), dma->lock_cap = capable(CAP_IPC_LOCK) captured at map time, and dma->mm (with mmgrab). The comment explains: “pinning can be asynchronous via the external interfaces for mdev devices. RLIMIT_MEMLOCK requires a task_struct. Save the group_leader so that all DMA tracking uses the same task, to make debugging easier.” So the limit is charged to a stable task even when the pinning happens later and from a different context — and CAP_IPC_LOCK is evaluated once, at map time, not re-checked afterwards.

But “you cannot live-migrate a passthrough VM” is no longer true, and this note used to say it was. VFIO grew a migration v2 ABI, and it is fully present in v6.12’s include/uapi/linux/vfio.h: VFIO_DEVICE_FEATURE_MIGRATION (1) and VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE (2) expose an eight-state device model — ERROR, STOP, RUNNING, STOP_COPY, RESUMING, RUNNING_P2P, PRE_COPY, PRE_COPY_P2P — with PRE_COPY supporting the iterative dirty-page phase that makes migration downtime bounded. Alongside it, VFIO_DEVICE_FEATURE_DMA_LOGGING_START / _STOP / _REPORT (6, 7, 8) let the VMM ask the device itself which pages it dirtied, which is the piece the pinned-memory argument said was impossible. The legacy v1 names survive only as VFIO_DEVICE_STATE_V1_* defines.

The catch is that this is per-driver, not generic. Migration requires a variant driver — a driver built on vfio-pci-core that knows how to serialise its particular device’s internal state. The in-tree acceptance criteria explain the split: “the vfio-pci driver does include some device specific support, [but] further extensions for yet more advanced device specific features are not sustainable. The vfio-pci driver has therefore split out vfio-pci-core as a library that may be reused to implement features requiring device specific knowledge, ex. saving and loading device state for the purposes of supporting migration” (vfio-pci-device-specific-driver-acceptance.rst, v6.12). The same document sets a notably high review bar, because these drivers may reach outside their assigned device — “it’s expected that some device specific variants may interact with parent devices (ex. SR-IOV PF in support of a user assigned VF)… Authors of such drivers should be diligent not to create exploitable interfaces via these interactions.”

v6.12 ships six variant drivers under drivers/vfio/pci/, and — a detail worth checking rather than assuming, since it is easy to read the directory listing as a migration-support list — only four of them implement migration:

Variant driver	Kconfig symbol	Implements vfio_migration_ops?	Purpose
mlx5	MLX5_VFIO_PCI	yes	“provides migration support for MLX5 devices”
hisilicon	HISI_ACC_VFIO_PCI	yes	HiSilicon accelerator (ARM64) migration
pds	PDS_VFIO_PCI	yes	AMD/Pensando DSC migration
qat	QAT_VFIO_PCI	yes	“provides migration support for Intel(R) QAT Virtual Function”
virtio	VIRTIO_VFIO_PCI	no	Emulates an I/O BAR in software so a virtio-net VF “be seen as a transitional device”, since “based on PCIe spec, VFs do not support I/O Space”
nvgrace-gpu	NVGRACE_GPU_VFIO_PCI	no	Assignment support for the GPU in the NVIDIA Grace Hopper Superchip

The v6.12 variant drivers, with migration support determined by grepping each driver for vfio_migration_ops rather than by inferring it from the directory name. What it shows: two of the six exist for reasons entirely unrelated to migration — virtio works around a PCIe specification limitation, nvgrace-gpu handles a coherent-memory GPU topology. The insight: “variant driver” means “needs device-specific knowledge”, not “supports migration”. Conflating the two overstates how broadly passthrough VMs can be migrated.

So the honest statement is not “passthrough cannot be migrated” but “passthrough can be migrated only on devices whose vendor wrote and upstreamed a migration variant driver” — as of 6.12 that is four device families, not all PCI devices and not none. Memory overcommit, by contrast, remains genuinely off the table: pinning is inherent to letting hardware DMA without the CPU’s involvement.

stateDiagram-v2
    [*] --> RUNNING: device in normal operation
    RUNNING --> PRE_COPY: VMM starts migration;<br/>device keeps running while<br/>state is streamed out
    PRE_COPY --> PRE_COPY_P2P: quiesce peer-to-peer DMA<br/>while still running
    PRE_COPY_P2P --> STOP_COPY: device stopped;<br/>drain the remaining state
    RUNNING --> RUNNING_P2P: (path without pre-copy)
    RUNNING_P2P --> STOP: device stopped
    STOP --> STOP_COPY: serialise full device state
    STOP_COPY --> [*]: state handed to the destination
    [*] --> RESUMING: destination loads state
    RESUMING --> STOP: load complete
    STOP --> RUNNING: resume execution
    RUNNING --> ERROR: unrecoverable
    STOP_COPY --> ERROR: unrecoverable
    note right of PRE_COPY
        DMA_LOGGING_START / _REPORT let the
        VMM ask the DEVICE which pages it
        dirtied - the piece that makes
        bounded-downtime migration possible
        despite every page being pinned.
    end note

The VFIO migration v2 device state machine, from the VFIO_DEVICE_STATE_* enum in include/uapi/linux/vfio.h (v6.12). What it shows: the device is a participant in migration with its own lifecycle, not a passive lump of pinned memory — it can be asked to keep running while its state streams out (PRE_COPY), to stop initiating peer-to-peer traffic before it stops entirely (_P2P variants), and to report its own dirty pages. The insight: the separate _P2P states exist because a migrating device must stop DMA-ing to other devices before it stops DMA-ing to memory — otherwise a peer could keep dirtying pages nobody is tracking. That subtlety is why migration needed a real state machine rather than a stop/start pair, and why it needs a per-device variant driver to implement.

MSI/MSI-X interrupt remapping requirement. On x86, safely delivering device interrupts to a guest requires interrupt remapping in the IOMMU; without it the kernel by default refuses passthrough (or requires the allow_unsafe_interrupts module option), because an unremapped MSI is just a memory write to the local-APIC address range that a malicious device could forge to inject arbitrary interrupts. This is why VFIO depends on the IOMMU for interrupts, not only DMA — detailed in The IOMMU and DMA Remapping.

---

Alternatives and When to Choose Them

VFIO is one point on the I/O spectrum (the Linux Virtualization MOC decision framework runs emulated → virtio → vhost/vDPA → VFIO/SR-IOV). Full device emulation (QEMU pretending to be an e1000 NIC) is maximally compatible and supports migration/overcommit but is slow — a VM exit per register access. virtio paravirtualization is the sensible default for most guests. VFIO passthrough is for the cases paravirtualization cannot serve: GPUs for compute/graphics, NVMe and RDMA NICs where every microsecond of latency matters, FPGAs and other accelerators with no virtio analogue. Within passthrough, SR-IOV and Virtual Functions is the middle ground — one physical card exposes many lightweight virtual functions, each its own VFIO device, so dozens of VMs can share a NIC at near-native speed (each VF still goes through VFIO). Mediated devices (mdev) time-slice one physical device (notably for vGPU) into software-defined virtual devices fronted by vfio-mdev.

Against the older UIO framework, VFIO wins whenever the device does DMA and an IOMMU exists, because UIO offers no DMA confinement. UIO survives for DMA-less devices and IOMMU-less embedded systems.

Approach	Isolation	Speed	Overcommit / migration	Use when
Full emulation (e.g. QEMU e1000)	Total — guest never touches hardware	Poor: a VM exit per register access	Full	Compatibility, ancient guests, no host device to spare
virtio paravirtualisation	Total	Good	Full	The sensible default for almost everything
vhost / vDPA	Total (datapath in kernel or hardware)	Very good	Mostly retained	High-throughput networking without giving up a whole device
SR-IOV VF via VFIO	Hardware, per virtual function	Near-native	Pinned; migration if a variant driver exists	Many guests sharing one NIC at line rate
VFIO passthrough (PF)	Hardware, per IOMMU group	Native	Pinned; migration only with a variant driver	GPUs, accelerators, NVMe, anything with no virtio analogue
mdev (vfio-mdev)	Hardware + vendor software partitioning	Near-native per slice	Vendor-dependent	Time-slicing one device into many (vGPU, vfio-ap, vfio-ccw)
UIO	None for DMA	Native	N/A	DMA-less devices, or fully trusted embedded systems

The I/O virtualisation spectrum, ordered by how much hardware the guest actually touches. What it shows: isolation is roughly constant across the top four rows — what varies is speed and what you give up to get it. The insight: the real dial is not “safe versus fast” but “flexible versus fast.” Every step down this table trades away host-side flexibility — overcommit, migration, live reconfiguration — rather than safety, because VFIO’s whole design premise is that safety is non-negotiable and hardware-enforced.

On mdev’s status, which is routinely misreported. Mediated devices are frequently described as deprecated. As of the tags checked here, they are not. drivers/vfio/mdev/Kconfig at v6.12 is four lines containing a bare config VFIO_MDEV / tristate with no deprecation text; Documentation/driver-api/vfio-mediated-device.rst at v6.12 contains zero occurrences of “deprecat” or “obsolete”; and an existence check finds drivers/vfio/mdev/mdev_core.c, drivers/vfio/mdev/Kconfig and the samples/vfio-mdev/mtty.c sample all returning HTTP 200 at v6.18, with the Kconfig still unchanged. What is true is that mdev has been reframed rather than removed: iommufd’s documentation describes an mdev as an “In-kernel user — refers to something like a VFIO mdev that is using the IOMMUFD access interface to access the IOAS,” reached through an iommufd_access object. Mechanically, mdev devices register via vfio_register_emulated_iommu_dev() and land on the type1 backend’s emulated_iommu_groups list, which carries real limits: VFIO_UPDATE_VADDR is disabled outright when mdevs are present (“They cannot safely pin/unpin/rw while vaddrs are being updated”), “an emulated IOMMU group cannot dirty memory directly”, and “a container with a single mdev device will have an empty [IOVA] list.”

Uncertain

Verify: whether any specific mdev-based product line is being wound down, as distinct from the vfio-mdev framework being removed. Reason: the framework’s presence is verified at v6.12 and v6.18 by existence check and by reading its Kconfig and documentation, but vendor-level deprecations (particular vGPU product generations, for instance) are announced outside the kernel tree and were not researched here. The two get conflated constantly. To resolve: check the specific vendor’s driver documentation and the drivers/gpu/drm/ or vendor out-of-tree driver release notes for the hardware in question. uncertain

---

Production Notes

VFIO is the foundation under essentially all production GPU and high-performance-NIC passthrough: cloud GPU instances, network-function-virtualisation (NFV) data planes, and the GPU-passthrough gaming-VM community all sit on it. DPDK and SPDK use VFIO’s userspace-driver path — with a real IOMMU, or with VFIO_NOIOMMU_IOMMU and an accepted risk — to drive NICs and NVMe controllers entirely from userspace polling loops, bypassing the kernel network and block stacks. That is the second face of VFIO worth remembering: it is not only a hypervisor mechanism. The same container-and-mapping machinery serves a userspace driver with no guest anywhere in sight.

Reading Your Own Machine

A short, mechanical checklist, in the order that answers questions fastest:

Question	Command
What can I actually assign?	ls /sys/kernel/iommu_groups/*/devices/
Which group is this device in?	readlink -f /sys/bus/pci/devices/0000:06:0d.0/iommu_group
Is the IOMMU even on?	dmesg | grep -iE 'DMAR|AMD-Vi|IOMMU'; check intel_iommu=on / amd_iommu=on in /proc/cmdline
Which VFIO ABIs did my kernel build?	grep -E 'VFIO_(GROUP|CONTAINER|DEVICE_CDEV|NOIOMMU)' /boot/config-$(uname -r)
Does the modern cdev exist here?	ls /dev/vfio/devices/ — often absent, see the Kconfig table above
Is a device bound to VFIO?	lspci -nnk -s 06:0d.0 and look at “Kernel driver in use”
Did DMA get blocked?	dmesg | grep -iE 'DMAR:.*fault|AMD-Vi.*IO_PAGE_FAULT'
Am I hitting the locked-memory limit?	ulimit -l, and the VMM’s own RLIMIT_MEMLOCK setting
Did I taint the kernel with no-iommu?	cat /proc/sys/kernel/tainted

The last one is worth internalising: VFIO_NOIOMMU_IOMMU sets a taint flag, and the Kconfig text says exactly why — “Use of this mode will result in an unsupportable kernel and will therefore taint the kernel. Device assignment to virtual machines is also not possible with this mode since there is no IOMMU to provide DMA translation.” If you find a production host tainted and nobody remembers why, this is a candidate.

There is also a CONFIG_VFIO_DEBUGFS option in v6.12 which “allows exposure of VFIO device internals” under debug/vfio when a driver populates it — useful mainly for variant-driver migration debugging rather than day-to-day operation.

The Migration Story, Honestly Stated

The forward path is unambiguous in the kernel’s own words — the container and group model “is intended to be deprecated,” and users should move to cdev plus native iommufd. The timeline is not. As of the 6.12 LTS:

• The legacy container/group path is the default-on, feature-complete, universally supported option, and QEMU still defaults to it.
• iommufd is merged, real, and required for some configurations (Intel VT-d with fsts=on), but its VFIO compatibility layer is documented as not feature-complete, with PCI peer-to-peer DMA mapping an explicit gap.
• The device cdev that makes iommufd ergonomic is default-off in the Kconfig.

The sensible operational posture on an LTS kernel is therefore: treat the container path as the stable default, adopt iommufd where you need something only it provides (nesting, hardware dirty tracking for migration, PASID), and re-evaluate at each LTS rather than assuming the transition has completed. Validate feature availability against the exact tag you ship — the uncertainty callouts above mark the two places where in-tree documentation was measurably behind in-tree code at v6.12, which is a good reason not to take any single document’s word for it, this one included.

See Also

• The IOMMU and DMA Remapping — the hardware that makes VFIO safe: IOVA→physical translation, per-device page tables, interrupt remapping
• IOMMU Groups and Device Isolation — why the group (not the device) is VFIO’s unit of ownership, and how ACS forms groups
• SR-IOV and Virtual Functions — slicing one physical device into many passable VFIO devices
• Linux Virtualization MOC — parent map; VFIO is the “native speed, no migration/overcommit” end of the I/O dial
• Linux Device Drivers and Device Model MOC — the PCI device model and DMA API underneath VFIO (ghost link; not yet written)
• DPDK and Userspace Networking — VFIO’s other consumer: a userspace driver with no guest involved
• KVM Architecture — the irqfd / IRQ-bypass plumbing that receives VFIO’s eventfds