vault

The IOMMU and DMA Remapping

15 min read

The IOMMU and DMA Remapping

An IOMMU (Input/Output Memory Management Unit) is, for devices, what the CPU’s MMU (Memory Management Unit) is for processes: a hardware unit that sits on the path between a device and physical memory and translates the addresses a device emits before they ever reach RAM. A device performing Direct Memory Access (DMA) issues bus transactions carrying an address; without an IOMMU that address is a raw host-physical address and the device can read or write any byte of memory. The IOMMU interposes a per-device page table that maps the device’s I/O Virtual Addresses (IOVAs) to host-physical addresses, so a device’s DMA is confined to exactly the memory the kernel has chosen to map for it — and faults on anything else. Intel’s implementation is VT-d (Virtualization Technology for Directed I/O, in the kernel: DMAR / “Intel IOMMU”); AMD’s is AMD-Vi (the AMD IOMMU) (Intel VT-d spec). This single capability is what makes safe device passthrough possible, and it provides real DMA-attack protection even on machines that never run a virtual machine.

This note owns the IOMMU mechanism for the vault. Its consumer in the virtualization stack is the VFIO Framework (which programs the IOMMU on userspace’s behalf); the way devices are bundled into indivisible isolation units is IOMMU Groups and Device Isolation; and the technique that creates many DMA-isolated functions from one card is SR-IOV and Virtual Functions. The generic device-model angle will eventually live in Linux Device Drivers and Device Model MOC (not yet written). Parent map: Linux Virtualization MOC.

Mental Model — A Second MMU, For Devices

flowchart LR
  subgraph CPU["CPU side"]
    PROC["Process<br/>virtual address (VA)"]
    MMU["MMU + page tables"]
    PROC -->|VA| MMU
  end
  subgraph DEV["Device side"]
    DEVICE["PCI device DMA engine<br/>emits IOVA"]
    IOMMU["IOMMU<br/>(VT-d / AMD-Vi)"]
    DEVICE -->|IOVA + BDF| IOMMU
  end
  RAM["Physical RAM<br/>(host-physical addresses)"]
  MMU -->|VA->PA| RAM
  IOMMU -->|IOVA->PA, per-device table| RAM
  IOMMU -.->|IOVA not mapped| FAULT["DMAR / IO page fault<br/>(transaction blocked)"]

The IOMMU as a symmetric counterpart to the MMU. What it shows: just as the MMU translates a process’s virtual addresses to physical addresses using page tables the kernel controls, the IOMMU translates a device’s IOVAs to physical addresses using a per-device page table the kernel controls — and selects that table using the device’s PCI identity (its Bus/Device/Function, or BDF). The insight to take: isolation is the whole point. Two devices behind one IOMMU can be given completely disjoint page tables, so neither can reach the other’s (or the kernel’s) memory; and an IOVA the kernel never mapped produces a fault, not a stray write. The MMU protects processes from each other; the IOMMU protects memory from devices.

The analogy is exact and worth holding onto. A CPU process names memory with virtual addresses; the MMU walks the process’s page tables to find the physical page, and an unmapped address faults. A device, under an IOMMU, names memory with an IOVA; the IOMMU walks that device’s page table to find the physical page, and an unmapped IOVA faults. The kernel populates the device’s page table exactly as it populates a process’s — with only the pages that device is allowed to touch. The difference is the lookup key: the MMU implicitly knows which page table to use (it’s the current process’s CR3); the IOMMU must identify the device first, which it does from the requester ID carried in every PCIe transaction.

Why Passthrough Requires an IOMMU

The reason an IOMMU is non-negotiable for device passthrough follows directly from how DMA works. When you hand a real device to a virtual-machine guest, the guest’s own driver programs the device’s descriptors and DMA engine using guest-physical addresses — because as far as the guest knows, it owns the machine. But guest-physical addresses are a fiction maintained by the hypervisor; the actual data lives at scattered host-physical pages. If the device DMA’d directly to the guest-physical addresses it was given, it would write to whatever host memory happens to live at those physical addresses — almost certainly not the guest’s RAM, and possibly the host kernel’s. Two things are therefore required: the device’s DMA must be redirected from guest-physical to the correct host-physical pages, and it must be confined so a compromised guest cannot make the device reach beyond its own RAM.

The IOMMU does both with one mechanism. The hypervisor (via the VFIO Framework) installs a page table for the passed-through device that maps guest-physical-address → host-physical-address for every page of guest RAM. Now when the guest’s driver tells the device “DMA to guest-physical 0x1000,” the device emits IOVA 0x1000, the IOMMU translates it through the guest’s table to the real host page backing that guest page, and the transfer lands correctly. Equally important, any IOVA not in that table — i.e. any host memory the hypervisor did not deliberately expose — simply faults. The guest cannot escape its memory through the device. This is precisely the property that lets a hypervisor give a guest native device speed without giving it the keys to the host.

Without an IOMMU there is no safe passthrough at all, only the dangerous no-IOMMU escape hatch (VFIO_NOIOMMU_IOMMU), which grants the device unrestricted physical-memory access and exists only for trusted high-performance userspace drivers (VFIO docs, v6.12).

The Page-Table Walk — From BDF to Physical Page

The IOMMU’s translation is a two-phase lookup: first find this device’s page table using its identity, then walk that page table using the IOVA. On Intel VT-d the structures are named precisely (Intel VT-d spec; Project ACRN VT-d HLD).

Every PCIe transaction carries a requester ID equal to the device’s Bus/Device/Function (BDF) — 8 bits of bus, 5 of device, 3 of function. The IOMMU uses this to find the right page table through two indexed tables held in main memory:

  1. The root table is indexed by the bus number (256 entries). Each present root-entry points to a context table for that bus.
  2. The context table is indexed by the device-and-function part of the BDF (256 entries per bus, one per function). Each present context-entry holds a second-level page-table pointer — the host-physical address of the page-table root to use for this specific function (terenceli, IOMMU introduction).

So the IOMMU first does root_table[bus] → context_table, then context_table[devfn] → page_table_root. This is what lets different devices have different page tables: two functions with different BDFs index to different context-entries pointing at different page-table roots, hence different (or shared, if the kernel chooses) IOVA→physical mappings.

Having found the page-table root, the IOMMU walks it exactly like a CPU page-table walk. The IOVA is split into multi-level index fields plus a page offset, and the hardware descends a multi-level radix tree — for 48-bit addressing, the familiar four-level layout (PML4 → PDPT → PD → PT) borrowed from the CPU’s own paging, optionally a fifth level (PML5) for 57-bit addressing — until it reaches a leaf entry giving the host-physical page; the page offset is then appended. Intel calls these the second-level page tables (the device-side analogue of guest paging). In a nested/scalable-mode setup a first-level (PASID-tagged) table can sit in front of them for two-stage translation, used when a guest runs its own IOMMU.

Because walking these tables on every DMA would be ruinous, the IOMMU caches aggressively, mirroring the CPU’s TLB. It keeps a context-cache (recently used device→page-table-root mappings) and an IOTLB (I/O Translation Lookaside Buffer, recently used IOVA→physical translations) (terenceli). When the kernel changes a mapping it must invalidate the relevant IOTLB entries, and the cost and timing of that invalidation is a real performance knob — see iommu.strict below.

Interrupt Remapping — The Other Half of Isolation

Confining DMA is necessary but not sufficient, because on x86 a device interrupt is itself a memory write. A Message Signaled Interrupt (MSI) is delivered by the device performing a DMA write to a magic address in the local-APIC range (0xFEEx_xxxx); the value written encodes the destination CPU and vector. This means that without protection, “an IOMMU cannot distinguish between genuine MSIs from the device and a DMA that pretends to be an interrupt” (ACRN VT-d) — a malicious passed-through device could forge a write to that range and inject any interrupt vector at any CPU, including ones that drive the host kernel.

Interrupt remapping closes this. With it enabled, an interrupt request from a device no longer carries a raw APIC address/vector; instead it carries a remapping index into a memory-resident interrupt-remapping table the kernel controls. The IOMMU looks up the entry, validates that this device is allowed to raise this interrupt, and substitutes the real destination/vector. A device can therefore only trigger the interrupts the kernel has provisioned for it. This is why the kernel, by default, refuses device passthrough on systems without interrupt remapping (requiring an explicit allow_unsafe_interrupts override): without it, passthrough re-opens an interrupt-injection hole even if DMA is perfectly confined. VT-d further builds posted interrupts on top of remapping, letting an interrupt from a passed-through device be delivered straight into a running guest vCPU’s virtual APIC with no VM exit — covered in the Linux Virtualization MOC interrupt-virtualization section.

Configuration — Enabling and Tuning the IOMMU

The IOMMU is controlled by kernel command-line parameters, quoted here verbatim from the v6.12 tree (kernel-parameters.txt, v6.12).

intel_iommu=on

“[DMAR] Intel IOMMU driver (DMAR) option on - Enable intel iommu driver.” Other values: off disables it; igfx_off leaves the integrated GPU mapped as a normal device but bypasses its dedicated DMAR unit; sm_on enables scalable mode if the hardware advertises it (scalable mode is the modern format that underpins PASID and nested translation). On AMD the driver is enabled by firmware/ACPI presence, with amd_iommu=off to disable it.

iommu.passthrough={ "0" | "1" }

“[ARM64,X86,EARLY] Configure DMA to bypass the IOMMU by default. 0 - Use IOMMU translation for DMA. 1 - Bypass the IOMMU for DMA.” This is the crucial protection-vs-overhead dial for the host’s own devices. With iommu.passthrough=1 the kernel installs an identity (1:1 IOVA=physical) mapping for in-kernel device DMA — the IOMMU is on, but it does not actually constrain the host’s trusted drivers, avoiding per-map IOTLB churn and translation overhead. With iommu.passthrough=0 even the host’s own drivers’ DMA is fully translated and confined, which costs performance but means a buggy or compromised in-kernel driver (or a malicious hot-plugged Thunderbolt/PCIe device) cannot DMA outside its mapped buffers. The same iommu=pt / iommu=nopt spellings select pass-through vs forced translation. Note that VFIO passthrough to a guest is unaffected by this default — VFIO always installs real translating mappings for the device it owns.

iommu.strict={ "0" | "1" }

“0 - Lazy mode. Request that DMA unmap operations use deferred invalidation of hardware TLBs, for increased throughput at the cost of reduced device isolation. 1 - Strict mode. DMA unmap operations invalidate IOMMU hardware TLBs synchronously.” In lazy mode a freed IOVA may remain valid in the IOTLB for a short window before a batched flush, so a device that erroneously re-uses a just-unmapped address could still reach the page — a small isolation hole traded for far fewer (expensive) IOTLB invalidations. Strict mode flushes on every unmap, closing the window at a throughput cost. (amd_iommu=fullflush is the deprecated AMD spelling, equivalent to iommu.strict=1.)

A typical passthrough host therefore boots with intel_iommu=on iommu=pt (or the AMD equivalent): the IOMMU is enabled so VFIO can use it for the assigned device, while iommu=pt keeps the host’s own devices on the fast identity path.

DMA Bounce Buffers — The IOMMU’s Software Cousin

Even with an IOMMU, some situations need data copied through an intermediary buffer rather than translated in place. The kernel mechanism is swiotlb (the software I/O TLB), “a memory buffer allocator used by the Linux kernel DMA layer” that implements bounce buffering: “The DMA is done to/from this temporary memory buffer, and the CPU copies the data between the temporary buffer and the original target memory buffer” (swiotlb.rst, v6.12).

Its original purpose was addressing-range limits: “As physical memory sizes grew beyond 4 GiB, some devices could only provide 32-bit DMA addresses,” so the kernel allocates bounce buffers below the 4 GiB line, the 32-bit device DMAs to the low buffer, and the CPU copies to/from the real (possibly high) target. An IOMMU can solve the same problem more elegantly by mapping a high physical page to a low IOVA — which is one reason an IOMMU reduces (but does not eliminate) reliance on swiotlb.

The modern and increasingly important use is confidential computing. In an encrypted-memory VM (AMD SEV-SNP, Intel TDX), “the guest VM’s memory is encrypted by default and not accessible by the host hypervisor and VMM. For the host to do I/O on behalf of the guest, the I/O must be directed to guest memory that is unencrypted” (Confidential Computing VMs, kernel docs). Such guests “set a kernel-wide option to force all DMA I/O to use bounce buffers, and the bounce buffer memory is set up as unencrypted,” so the device DMAs into shared/unencrypted bounce pages and the guest CPU copies between those and its private encrypted memory (swiotlb.rst, v6.12). This is why I/O-heavy confidential VMs carry a real performance penalty: every DMA pays an extra CPU copy plus the address-space book-keeping. The interplay with the trust boundary is developed in Linux Virtualization MOC’s confidential-computing section.

Failure Modes and Common Misunderstandings

“The IOMMU is only for virtualization.” False, and dangerously so. The IOMMU provides DMA-attack protection outside any VM: a malicious peripheral (the classic threat is a hostile device plugged into a Thunderbolt/USB-C or PCIe port performing a DMA attack to read RAM or inject code, as in the “Thunderclap” class of attacks) is confined by the IOMMU exactly as a passed-through device is. This is why modern laptops enable the IOMMU (Intel calls the firmware feature “Kernel DMA Protection”) even with no virtualization in play, and why iommu.passthrough=0 is a meaningful hardening choice on a workstation.

DMAR/IO-page-fault floods. When a device DMAs to an unmapped IOVA, the IOMMU logs a fault (Intel: “DMAR fault”; the dmesg line names the device BDF, the faulting address, and read/write). A burst of these usually means the driver and the IOMMU mappings disagree — a buggy driver handing the device an address it never mapped, a partially mapped guest-RAM region in a VFIO setup (see VFIO Framework’s “forgetting to map all guest RAM”), or a use-after-unmap race exposed by lazy invalidation.

IOTLB invalidation cost surprises. Switching from lazy to strict mode, or onto a high-map-churn workload, can tank throughput because every unmap now stalls on a synchronous IOTLB flush. Conversely, lazy mode’s deferred flush is a documented isolation weakening, not free performance. The right setting is workload- and threat-model-dependent.

Confusing the IOMMU with the MMU’s EPT/NPT. Both translate addresses for virtualization, but they are different units doing different jobs. The CPU’s EPT/NPT (two-dimensional paging) translates guest-physical → host-physical for CPU accesses; the IOMMU translates IOVA → host-physical for device DMA. A passed-through device’s DMA goes through the IOMMU, never through EPT. (EPT/NPT is covered separately in the Linux Virtualization MOC memory-virtualization section.)

Alternatives and When to Choose Them

There is no real “alternative” to the IOMMU for the safe-passthrough job — it is the only hardware that can confine device DMA. The genuine choices are how much to use it. On a host that does no passthrough, the spectrum runs from IOMMU disabled (fastest, no DMA protection — acceptable only on physically trusted machines with no untrusted peripherals), through iommu=pt (IOMMU enabled for any device that needs it, host devices on the identity fast path — the common passthrough-host setting), to full translation iommu.passthrough=0 with strict invalidation (maximum protection against rogue peripherals and buggy drivers, at a measurable throughput cost — the hardening choice for laptops and high-security hosts).

For addressing-limit problems specifically, an IOMMU’s remapping is preferable to swiotlb bounce buffering because it avoids the CPU copy — but swiotlb remains the fallback where no IOMMU exists and is mandatory (not optional) for confidential VMs, where copying through shared memory is the only way the encrypted guest can talk to host-driven devices.

Production Notes

Every cloud GPU instance, NFV data plane, and bare-metal-with-passthrough deployment depends on the IOMMU being enabled and on the firmware exposing clean IOMMU groups (good ACS support) so devices can be assigned individually rather than in coarse bundles — see IOMMU Groups and Device Isolation. Server platforms generally ship correct ACS; consumer boards frequently do not, which is the root of most home-lab passthrough pain and the reason the (security-weakening) ACS-override patch exists (Arch Wiki: PCI passthrough). Security-conscious operators increasingly enable full IOMMU translation for host devices too, treating any PCIe/Thunderbolt port as an untrusted DMA source. And the rise of confidential VMs has made the IOMMU’s software cousin, swiotlb, a first-class performance concern: the forced bounce-buffer copy is a known, measurable tax on I/O-heavy SEV-SNP/TDX workloads that engineering teams budget for explicitly.

See Also

Graph View

Backlinks