vault

GPU and NIC Passthrough

15 min read

GPU and NIC Passthrough

GPU and NIC passthrough is the practice of handing an entire physical PCI device — a graphics card or a network interface card (NIC) — directly to a guest virtual machine so the guest’s own driver talks to the real silicon with no host involvement on the data path. On Linux this is done through the VFIO (Virtual Function I/O) framework’s PCI driver, vfio-pci, which you bind the device to instead of its native host driver. Once bound, the device’s memory-mapped registers, interrupts, and DMA are exposed to a userspace virtual-machine monitor (VMM) such as QEMU, which maps them into the guest’s address space behind the IOMMU. The reward is near-native device performance; the price is steep and non-negotiable — no live migration and no memory overcommit, because the device holds DMA references to specific host-physical pages and carries opaque internal state the hypervisor cannot snapshot. This note covers the mechanics (binding, unbinding, the IOMMU-group constraint), the hard problems unique to GPUs (device reset, the AMD reset bug, the NVIDIA “Code 43” saga), and when NIC passthrough beats paravirtualized virtio.

Mental Model

Think of passthrough as physically unplugging the device from the host and plugging it into the guest — except the cable is the IOMMU’s address-translation table. The host kernel must first let go of the device (unbind its native driver), a stub driver (vfio-pci) must claim it so nothing else grabs it, and the IOMMU must be programmed so that when the device issues a DMA write to “physical address X,” the hardware silently rewrites X into the host-physical page backing the guest’s RAM. The guest sees a real PCI device at a real-looking BAR (Base Address Register) layout; the host sees a process holding file descriptors it must never touch.

flowchart TB
  subgraph HOST["Host kernel"]
    NATIVE["Native driver<br/>(nouveau / nvidia / ixgbe)"]
    VFIO["vfio-pci stub driver"]
    IOMMU["IOMMU (VT-d / AMD-Vi)<br/>per-device translation table"]
  end
  subgraph USER["Host userspace"]
    VMM["VMM (QEMU)<br/>opens /dev/vfio fds"]
  end
  subgraph GUEST["Guest VM"]
    GDRV["Guest's own driver<br/>(nvidia.ko / ixgbe.ko)"]
    GMEM["Guest RAM"]
  end
  DEV["Physical GPU / NIC"]
  NATIVE -.->|"unbind"| DEV
  VFIO -->|"claims device"| DEV
  VMM -->|"ioctl on device fd"| VFIO
  VMM -->|"maps BARs into guest"| GDRV
  GDRV -->|"MMIO direct"| DEV
  DEV -->|"DMA by guest-physical addr"| IOMMU
  IOMMU -->|"remapped to host page"| GMEM

Whole-device passthrough. What it shows: the native host driver is detached and vfio-pci claims the device purely to hold it; the VMM maps the device’s registers (BARs) straight into the guest so the guest’s driver does memory-mapped I/O (MMIO) without trapping, while the IOMMU confines the device’s direct memory access (DMA) to the guest’s pages. The insight: on the hot path the host is not in the loop at all — that is why it is fast, and also why the host cannot migrate or overcommit it, since it has no idea what the device and guest are doing to each other.

Mechanical Walk-through

Step 1: Enable the IOMMU and confirm groups

Passthrough is only safe because the IOMMU (Intel VT-d, AMD-Vi) confines the device’s DMA. Without it, a passed-through device could issue a DMA write to any host-physical address and scribble over kernel memory — VFIO will refuse to operate. The IOMMU is enabled with intel_iommu=on or (on AMD, usually on by default) amd_iommu=on on the kernel command line.

The IOMMU does not isolate individual functions; it isolates IOMMU groups — the smallest set of devices the hardware can guarantee are isolated from one another, dictated by PCIe topology and ACS (Access Control Services) support on the upstream bridges. This is the single most important constraint in passthrough: you cannot pass through one device in a group while the host keeps another. The VFIO documentation is explicit — “if the IOMMU group contains multiple devices, each will need to be bound to a VFIO driver before operations on the VFIO group are allowed,” and the group must pass a viability check (VFIO_GROUP_FLAGS_VIABLE) before userspace may use it (VFIO docs, v6.12). You discover a device’s group by reading the symlink:

$ readlink /sys/bus/pci/devices/0000:06:0d.0/iommu_group
../../../../kernel/iommu_groups/26

For a discrete GPU this group very often contains two functions — the VGA function (0000:01:00.0) and the GPU’s HDMI audio function (0000:01:00.1) — both of which must go to the guest together. See IOMMU Groups and Device Isolation for why groups form the way they do and the (security-eroding) pcie_acs_override patch people use to split them.

Step 2: Detach the native driver, attach vfio-pci

The host must stop using the device. There are two idioms. The robust one uses driver_override, which forces a specific device to bind only to a named driver regardless of its PCI ID:

# echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override
# echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
# echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind

The older idiom adds the device’s PCI ID to vfio-pci’s dynamic-ID table and unbinds the native driver, as the kernel VFIO docs show:

# echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
# echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id

The new_id path (here 1102 0002 is vendor device) is blunter — it makes vfio-pci claim every device with that ID — which is why driver_override is preferred for single-device targeting. The vfio-pci module itself registers as an ordinary PCI driver whose id_table matches all devices via PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_ANY_ID, PCI_ANY_ID), and its ids= module parameter is parsed by vfio_pci_fill_ids() into pci_add_dynid() calls, accepting vendor:device:subvendor:subdevice:class:class_mask format (vfio_pci.c, v6.12). The driver’s own metadata is plain: MODULE_DESCRIPTION("VFIO PCI - User Level meta-driver"), MODULE_AUTHOR("Alex Williamson <alex.williamson@redhat.com>"), and the pci_driver .name = "vfio-pci" with .driver_managed_dma = true (telling the DMA layer this driver does not do streaming DMA itself).

Because a GPU is bound at boot by the framebuffer/native driver before you get a shell, practical setups bind vfio-pci early — via softdep lines, a modprobe.d options vfio-pci ids=..., or kernel command-line vfio-pci.ids=, ensuring vfio-pci claims the card before nvidia/nouveau/amdgpu can. Detaching a NIC is easier (e.g. unbind from ixgbe), but the same group rules apply.

Step 3: The VMM consumes the device

The VMM opens the VFIO file descriptors — the container (/dev/vfio/vfio), the group (/dev/vfio/$GROUP), then a per-device fd via VFIO_GROUP_GET_DEVICE_FD — and asks VFIO for the device’s region and IRQ layout. It maps the device’s BARs into the guest’s physical-address space and registers guest RAM with the IOMMU so the device’s DMA lands in the right pages. From then on the guest’s driver does direct MMIO to the device and the device DMAs straight into guest memory; interrupts are delivered via eventfd (see irqfd and ioeventfd) and, with hardware help, can bypass the host entirely via Posted Interrupts. The full file-descriptor model and ioctl surface live in VFIO Framework; this note focuses on what is device-specific about GPUs and NICs.

Device Reset — the GPU-specific hard problem

A passed-through device must be returned to a clean state between guests (and on guest reboot), or the next guest inherits stale, half-initialized hardware. For most devices this is a quick, standardized operation; for GPUs it is the single most fragile part of passthrough.

Function-Level Reset (FLR)

The clean, standardized mechanism is PCIe Function-Level Reset (FLR): a device that advertises the FLR capability can be reset in isolation without touching siblings on the bus. In the kernel, pcie_reset_flr() checks the PCI_EXP_DEVCAP_FLR capability bit and, if present, pcie_flr() initiates the reset by setting PCI_EXP_DEVCTL_BCR_FLR in the device-control register after waiting for pending transactions to drain (pci.c, v6.12). VFIO leans on this: on enable, vfio_pci_core_enable() calls pci_try_reset_function() and records success in vdev->reset_works; on disable/close, vfio_pci_core_disable() resets the device via __pci_reset_function_locked() if reset_works is set, with the comment “If we have saved state, restore it. If we can reset the device, even better” (vfio_pci_core.c, v6.12).

The reset-method hierarchy and reset_method

The kernel maintains an ordered list of reset strategies, tried in turn until one works. In v6.12 the table pci_reset_fn_methods[] is, in order: device_specific, acpi, flr, af_flr (a conventional-PCI advanced-features FLR), pm (power-state D3hot→D0 cycle), bus (secondary-bus reset of the whole bridge), and cxl_bus (pci.c, v6.12). The administrator can read and reorder which methods apply to a given device through the sysfs reset_method attribute (added August 2021): “Reading this file gives names of the supported and enabled reset methods and their ordering. Writing a space-separated list of names of reset methods sets the reset methods and ordering… Writing an empty string disables the ability to reset the device. Writing ‘default’ enables all supported reset methods” (sysfs-bus-pci ABI, v6.12).

The AMD GPU reset bug and vendor-reset

Many consumer AMD GPUs (Polaris, Vega, Navi) do not implement a working FLR. When the guest shuts down and VFIO tries to reset the card, the standard reset leaves the GPU in a wedged state from which it cannot recover until a full host power cycle — the infamous “AMD reset bug.” The community fix is the out-of-tree vendor-reset kernel module, which implements vendor-specific reset sequences “too complex/complicated to land in pci_quirks.c,” with strategies named POLARIS10, VEGA10, VEGA20, and NAVI10 (gnif/vendor-reset). You wire it in by selecting the device-specific method:

# echo device_specific > /sys/bus/pci/devices/0000:0a:00.0/reset_method

Critically, the module must be loaded early (in the initramfs), because once the kernel’s default reset has wedged the card, vendor-reset cannot recover it (Sherlock, 2020). This maps onto the device_specific slot — pci_dev_specific_reset — sitting first in the kernel’s method list, which is exactly why some quirks ship in-tree (drivers/pci/quirks.c) and gnarlier ones live in vendor-reset.

Uncertain

Verify: the precise set of vendor-reset strategies and the exact AMD GPU generations still affected by the reset bug as of mid-2026, and whether any of these reset sequences have since been upstreamed into drivers/pci/quirks.c for 6.12/6.18. Reason: the affected-hardware list is maintained out-of-tree and evolves; this note’s list (Polaris/Vega/Navi) is sourced from the vendor-reset README and a 2020 blog, not a 6.12 kernel tree. To resolve: diff vendor-reset’s current device table against v6.12/v6.18 drivers/pci/quirks.c pci_dev_specific_reset quirks. uncertain

AER and the reset story

The other reset-adjacent concern is AER (Advanced Error Reporting): a fatal PCIe error on a passed-through device normally triggers kernel error recovery, which conflicts with the device being owned by a guest. VFIO handles this by intercepting the error: vfio_pci_core_aer_err_detected() signals the device’s error eventfd (vdev->err_trigger) and returns PCI_ERS_RESULT_CAN_RECOVER, letting the VMM/guest decide rather than the host driver (vfio_pci_core.c, v6.12).

The NVIDIA “Code 43” saga

For years, NVIDIA’s consumer GeForce Windows driver deliberately refused to initialize if it detected it was running inside a VM, throwing Windows Device Manager “Code 43” (“Windows has stopped this device because it has reported problems”). This was a driver policy, not a hardware limit — Quadro/Tesla cards were unaffected — widely understood as market segmentation to push virtualization users toward the expensive professional/vGPU lines (Hueber). The driver detected virtualization primarily via the CPUID hypervisor-present bit and the KVM/Hyper-V signature.

The community workaround was to hide the hypervisor from the guest. In libvirt/QEMU you set the KVM hidden state and (for QEMU’s own signature) spoof a vendor ID:

<features>
  <hyperv>
    <vendor_id state='on' value='whatever'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
</features>

The <kvm><hidden state='on'/> line clears the KVM CPUID leaf so the guest cannot see the KVMKVMKVM signature, and the spoofed Hyper-V vendor_id defeats the secondary check (Passthrough Post).

The saga ended in March 2021: NVIDIA added official (beta) support for GeForce GPU passthrough starting with driver release R465 (465.89), relaxing the detection so consumer cards work in a VM without the workaround (VideoCardz, 2021). NVIDIA’s own support page states the constraints plainly: the feature is beta, “one GPU is required for the Linux host OS and one GPU is required for the Windows virtual machine,” and “GPU passthrough supports only one virtual machine. vGPU (SR-IOV) or a shared GPU Passthrough for multiple VMs are not supported on GeForce.”

Uncertain

Verify: NVIDIA’s GeForce VM-passthrough support was still labeled “beta” as of the R465 announcement, and whether it has since been promoted to fully supported or had its host+guest single-GPU constraints changed in driver releases through 2026. Reason: the primary NVIDIA support page (custhelp a_id 5173) returned HTTP 403 to direct fetch during research; the “beta”, “one host + one guest GPU”, and “no vGPU/SR-IOV on GeForce” claims come from a 2021 NVIDIA-derived news report (VideoCardz) and the search-engine snippet of the NVIDIA page, not a live read. To resolve: open the NVIDIA support page in a browser and check the current driver release notes’ “GeForce GPU Passthrough” section. uncertain

GPU passthrough for gaming and ML VMs — single-GPU vs host+passthrough

The canonical desktop setup is two GPUs: a cheap card (or the CPU’s integrated graphics) drives the Linux host, and a powerful discrete GPU is bound to vfio-pci and handed to a Windows guest for gaming, or to a Linux guest for machine-learning (ML) workloads (CUDA needs the real GPU; emulation is useless for it). This is clean because the host never needs the passthrough card.

The harder variant is single-GPU passthrough: the machine has exactly one GPU, used by the host and the guest. Here you cannot leave the GPU bound to vfio-pci at boot, because the host needs it for its own display. The workaround is a pair of libvirt hook scripts that, at VM start, tear down the host’s graphical session (stop the display manager, unbind the framebuffer console with echo 0 > /sys/class/vtconsole/.../bind, detach efi-framebuffer, unbind amdgpu/nvidia, and load vfio-pci), then reverse it all at VM stop. It works but is brittle — the host loses its display entirely while the VM runs, and a crash can leave the GPU orphaned, often needing the AMD-reset dance above.

NIC passthrough — and why you’d choose it over virtio

NIC passthrough binds a network card to vfio-pci and gives the guest direct ownership of the hardware queues, so packets DMA straight from the wire into guest memory with no host networking stack on the path. The mechanics are identical to GPU passthrough (group, unbind, bind, map) but reset is rarely a problem — server NICs implement FLR correctly.

The reason to do it over paravirtualized virtio-net is latency and packet rate (packets per second, PPS). Virtio-net, even accelerated by vhost-net, still routes every packet through the host’s virtio backend and (often) a software bridge, adding microseconds of latency and a per-packet CPU cost that caps PPS. Passthrough removes the host entirely: the guest polls the NIC’s rings directly (e.g. with a DPDK — Data Plane Development Kit — driver inside the guest), achieving line-rate small-packet throughput and the lowest possible jitter, which matters for high-frequency trading, network functions virtualization (NFV), and telco data planes.

The cost is the same as for GPUs — no live migration (the guest holds the physical NIC; you cannot snapshot the card’s internal queue state) and no overcommit (one NIC, one guest). This is precisely the gap SR-IOV fills: one physical NIC presents many lightweight Virtual Functions (VFs), each a separately-passable PCI function, so a single card can serve many guests at near-passthrough speed — though VF passthrough still forfeits live migration unless paired with a fallback path.

Failure Modes

  • Group not viable. VFIO_SET_CONTAINER/group-add fails because another device in the IOMMU group is still bound to a host driver. Fix: bind every function in the group to vfio-pci, or relocate the device to a slot with finer ACS isolation.
  • AMD card wedged after VM shutdown. Second VM boot hangs or the host GPU goes dark; dmesg shows reset timeouts. Cause: AMD reset bug. Fix: vendor-reset loaded in the initramfs + reset_method=device_specific.
  • NVIDIA Code 43 on old drivers. Guest with a pre-R465 GeForce driver shows Code 43. Fix: upgrade to R465+ or hide the hypervisor (<kvm><hidden state='on'/>).
  • Host still owns the device at boot. nouveau/amdgpu/nvidia grabbed the GPU before vfio-pci. Fix: early binding via modprobe.d softdep and vfio-pci.ids= on the command line.
  • DMA faults / no traffic. The IOMMU is off or the BIOS hides VT-d/AMD-Vi. VFIO refuses with No IOMMU errors. Fix: enable intel_iommu=on/amd_iommu=on and the BIOS toggle.

Alternatives and When to Choose Them

  • virtio (the default). Maximum flexibility — live migration, overcommit, no special hardware — at a throughput/latency cost. Choose it for general-purpose VMs.
  • SR-IOV. Hardware-partition one physical device into many VFs; near-passthrough speed with many-guests density. Choose it for NICs serving many tenants.
  • Mediated devices (mdev). Software-partition one device (notably GPUs via NVIDIA vGPU) into virtual instances exposed through VFIO; the way to share a single GPU across guests when SR-IOV is unavailable.
  • Whole-device passthrough (this note). Maximum performance, zero sharing, no migration. Choose it for a single high-value workload that needs the whole device (a gaming VM, a single-tenant ML box, a line-rate appliance).

Production Notes

In clouds, bare-metal-adjacent GPU instances are usually whole-GPU passthrough (or NVIDIA vGPU via mdev for sharing); the no-live-migration property means GPU instances are scheduled as pinned, non-migratable workloads and drained explicitly for host maintenance. On Kubernetes, GPU passthrough is surfaced to VM-backed pods via KubeVirt with the NVIDIA/PCI device plugins — but that orchestration story lives in Kubernetes MOC, not here. The recurring operational lesson from the homelab and NFV communities alike is that reset is where passthrough breaks: budget for vendor-reset on AMD consumer cards, verify FLR support before buying a NIC for DPDK, and never assume a card returns to a clean state on its own.

See Also

Graph View