The IOMMU and DMA Remapping

An IOMMU (Input/Output Memory Management Unit) is, for devices, what the CPU’s MMU (Memory Management Unit) is for processes: a hardware unit that sits on the path between a device and physical memory and translates the addresses a device emits before they ever reach RAM. A device performing Direct Memory Access (DMA) issues bus transactions carrying an address; without an IOMMU that address is a raw host-physical address and the device can read or write any byte of memory. The IOMMU interposes a per-device page table that maps the device’s I/O Virtual Addresses (IOVAs) to host-physical addresses, so a device’s DMA is confined to exactly the memory the kernel has chosen to map for it — and faults on anything else. Intel’s implementation is VT-d (Virtualization Technology for Directed I/O, in the kernel: DMAR / “Intel IOMMU”); AMD’s is AMD-Vi (the AMD IOMMU) (Intel VT-d spec). This single capability is what makes safe device passthrough possible, and it provides real DMA-attack protection even on machines that never run a virtual machine.

What this note owns. This note owns the IOMMU mechanism: how a device address is translated, what caches the translation, and — the part this note is built around — how and when those caches are invalidated, because that is where the security/performance trade-off actually lives. Neighbouring notes own the adjacent pieces and are not duplicated here: VFIO Framework owns the userspace-facing framework that programs the IOMMU on an application’s or hypervisor’s behalf; IOMMU Groups and Device Isolation owns the ACS-derived grouping that decides the granularity the IOMMU can enforce; SR-IOV and Virtual Functions owns splitting one card into many independently-translated functions; Two-Dimensional Paging (EPT and NPT) owns the CPU-side guest→host translation that the IOMMU is routinely confused with; get_user_pages and Page Pinning owns why guest RAM must be pinned before it can be mapped for DMA; Direct IO and O_DIRECT and virtio Device Model own the I/O paths that sit above the DMA API. The structural twin of this note on the CPU side is The Translation Lookaside Buffer and TLB Shootdowns — the IOTLB is to the IOMMU what the TLB is to the MMU, down to the strict-versus-deferred invalidation dilemma, and the two notes are best read together. Parent maps: Linux Virtualization MOC · Linux Device Drivers and Device Model MOC · Linux Memory Management MOC.

Version claims are pinned to Linux 6.12 source unless dated otherwise; 6.12 is this vault’s pin because it is a long-term-support release, though as of 2026-09-12 kernel.org/releases.json lists mainline at 7.3-rc2 and marks both 6.18 and 6.12 longterm. Hardware claims are cited to the vendor architecture specifications — Intel® Virtualization Technology for Directed I/O Architecture Specification Rev. 4.1, order number D51397-016, March 2023, and the Arm System Memory Management Unit Architecture Specification, SMMU architecture version 3, IHI 0070 H.a — and where a claim rests on the kernel driver rather than on a vendor document, the note says so.

---

Mental Model — A Second MMU, For Devices

flowchart LR
  subgraph CPU["CPU side"]
    PROC["Process<br/>virtual address (VA)"]
    MMU["MMU + page tables"]
    PROC -->|VA| MMU
  end
  subgraph DEV["Device side"]
    DEVICE["PCI device DMA engine<br/>emits IOVA"]
    IOMMU["IOMMU<br/>(VT-d / AMD-Vi)"]
    DEVICE -->|IOVA + BDF| IOMMU
  end
  RAM["Physical RAM<br/>(host-physical addresses)"]
  MMU -->|VA->PA| RAM
  IOMMU -->|IOVA->PA, per-device table| RAM
  IOMMU -.->|IOVA not mapped| FAULT["DMAR / IO page fault<br/>(transaction blocked)"]

The IOMMU as a symmetric counterpart to the MMU. What it shows: just as the MMU translates a process’s virtual addresses to physical addresses using page tables the kernel controls, the IOMMU translates a device’s IOVAs to physical addresses using a per-device page table the kernel controls — and selects that table using the device’s PCI identity (its Bus/Device/Function, or BDF). The insight to take: isolation is the whole point. Two devices behind one IOMMU can be given completely disjoint page tables, so neither can reach the other’s (or the kernel’s) memory; and an IOVA the kernel never mapped produces a fault, not a stray write. The MMU protects processes from each other; the IOMMU protects memory from devices.

The analogy is exact and worth holding onto. A CPU process names memory with virtual addresses; the MMU walks the process’s page tables to find the physical page, and an unmapped address faults. A device, under an IOMMU, names memory with an IOVA; the IOMMU walks that device’s page table to find the physical page, and an unmapped IOVA faults. The kernel populates the device’s page table exactly as it populates a process’s — with only the pages that device is allowed to touch. The difference is the lookup key: the MMU implicitly knows which page table to use (it’s the current process’s CR3); the IOMMU must identify the device first, which it does from the requester ID carried in every PCIe transaction.

---

Three Implementations of the Same Idea

“IOMMU” names a role, not a design. Three architectures fill it, with the same two-phase structure — identify the requester, then walk that requester’s tables — but different names for everything, different table shapes, and meaningfully different capabilities. Getting the vocabulary straight is a prerequisite for reading any of the kernel drivers, because drivers/iommu/ uses the generic names while each vendor’s documentation uses its own.

	Intel VT-d (drivers/iommu/intel/)	AMD-Vi (drivers/iommu/amd/)	Arm SMMUv3 (drivers/iommu/arm/arm-smmu-v3/)
Requester identity	16-bit requester ID = PCI Bus/Device/Function (BDF)	same BDF	32-bit StreamID (need not be a BDF; any device stream)
Identity → tables, level 1	Root table, 256 entries, indexed by bus number	Device Table, one 32-byte DTE per BDF (DEV_TABLE_ENTRY_SIZE 32) — a single flat array, no second level	Stream table, linear or two-level, indexed by StreamID
Identity → tables, level 2	Context table, 256 entries per bus, indexed by device+function	— (the DTE holds the page-table pointer directly)	Context Descriptor table, indexed by SubstreamID (= PASID)
Sub-device identity	PASID (Process Address Space ID), via scalable-mode PASID tables	PASID (IOMMUv2)	SubstreamID
Translation stages	first-stage (process-like) and second-stage (guest-physical-like); both = nested	host page tables and guest page tables (GCR3)	stage 1 (VA→IPA) and stage 2 (IPA→PA); both enabled = nested
Leaf table format	multi-level radix, 3/4/5 levels, 48- or 57-bit input	two formats: AMD_IOMMU_V1 (AMD-native, up to 6 levels) or AMD_IOMMU_V2 (identical to x86-64 long-mode paging)	VMSAv8-64 or VMSAv8-32 LPAE — the same descriptor format the CPU’s MMU uses
Translation caches	Context-cache, PASID-cache, IOTLB, paging-structure caches	device-table cache, IOTLB, page-walk caches	configuration cache, TLB, walk caches
Invalidation submission	Invalidation Queue (IQ): a circular buffer in memory, head/tail registers, 128- or 256-bit descriptors; plus a legacy register interface	command buffer in memory, MMIO head/tail	Command queue (CMDQ): circular buffer, CMD_TLBI_* commands terminated by CMD_SYNC
Completion barrier	Invalidation Wait Descriptor	completion-wait command	CMD_SYNC
Interrupt remapping	Interrupt Remapping Table + posted interrupts	Interrupt Remapping Table	MSI translation through the same stream tables
Enabled by	intel_iommu=on (or firmware)	firmware/ACPI (IVRS); amd_iommu=off to disable	firmware (IORT)

Table: the same architecture under three vocabularies. The Intel column is cited to the Intel® Virtualization Technology for Directed I/O Architecture Specification, Rev. 4.1 §6.2 and §6.5.2; the Arm column to the Arm SMMU Architecture Specification, SMMUv3, IHI 0070 H.a §2.1 and §3.1; the AMD column to drivers/iommu/amd/ at v6.12 (see the uncertainty note below). The insight to take: two rows carry almost all the practical consequence. The invalidation-submission row is why IOMMU invalidation is a queued, asynchronous operation rather than a single instruction — unlike the CPU’s INVLPG, you write a descriptor into a ring in memory and ring a doorbell, and you only know it completed when a wait descriptor retires. And the translation-stages row is what makes nested virtualization of the IOMMU possible at all.

Arm’s design differs in one respect that is worth pulling out of the table, because it explains why arm-smmu-v3 looks so different from the x86 drivers: the SMMU deliberately mirrors the CPU’s own translation regime. “The SMMU supports two stages of translation in a similar way to PEs supporting the Virtualization Extensions… An incoming address is logically translated from VA to IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to the output PA. Stage 1 is intended to be used by a software entity to provide isolation or translation to buffers within the entity, for example DMA isolation within an OS. Stage 2 is intended to be available in systems supporting the Virtualization Extensions and is intended to virtualize device DMA to guest VM address spaces. When both stage 1 and stage 2 are enabled, the translation configuration is called nested” (SMMUv3 spec §2). Because the descriptor format is literally the CPU’s, a device can be handed a process’s existing page tables — which is the basis of Shared Virtual Addressing.

Uncertain

Verify: every AMD-Vi row in the table above against the AMD I/O Virtualization Technology (IOMMU) Specification (publication 48882). Reason: the AMD specification could not be retrieved. www.amd.com returned HTTP 000 — a connection failure, not a 403 or 404 — on three separate document paths across four attempts including an IPv4-forced retry; docs.amd.com is a FluidTopics single-page application whose backing /api/khub/maps catalogue was fetched successfully (3,065 documents, 16 MB of JSON) and contains no IOMMU or CPU-architecture entry at all, only Vivado/Vitis product documentation. The AMD column is therefore sourced from the Linux driver, which is authoritative for what Linux does but not for what the hardware architecturally guarantees. To resolve: fetch publication 48882 from a working route and check the Device Table Entry layout, the v1/v2 page-table format definitions, and the command-buffer/completion-wait semantics. uncertain

flowchart TB
  subgraph HW["Hardware identity"]
    D1["PCI function 0000:c2:00.0<br/>(integrated GPU)"]
    D2["PCI function 0000:c2:00.1<br/>(GPU audio)"]
    D3["PCI function 0000:c1:00.0<br/>(NVMe)"]
  end
  subgraph GRP["IOMMU groups — the smallest unit that can be isolated"]
    G19["group 19"]
    G20["group 20"]
    G18["group 18"]
  end
  subgraph DOM["iommu_domain — one set of translation tables"]
    DID["type = IOMMU_DOMAIN_IDENTITY<br/>IOVA == physical, no tables<br/>requested by the driver"]
    DFQ1["type = IOMMU_DOMAIN_DMA_FQ<br/>translating + deferred invalidation"]
    DFQ2["type = IOMMU_DOMAIN_DMA_FQ<br/>translating + deferred invalidation"]
  end
  subgraph PT["Backing structures"]
    NOPT["no page table<br/>(DTE says pass-through)"]
    PT1["io-pgtable tree<br/>+ iova_domain allocator<br/>+ per-CPU flush queue"]
    PT2["io-pgtable tree<br/>+ iova_domain allocator<br/>+ per-CPU flush queue"]
  end
  D1 --> G19 --> DID --> NOPT
  D2 --> G20 --> DFQ1 --> PT1
  D3 --> G18 --> DFQ2 --> PT2

Device → group → domain → page table, using this machine’s real topology. What it shows: a device never binds to a page table directly. It binds to an IOMMU group (the isolation granularity the PCIe topology permits — see IOMMU Groups and Device Isolation), the group binds to an iommu_domain, and only the domain owns translation structures. The insight to take: the domain’s type is where every security/performance decision is encoded, and it is per-group, not global. On this machine the two functions of the same physical GPU card land in different groups and different domain types — the display function gets an identity (untranslated) domain because its driver is PASID-capable, while the audio function gets a fully translating one. “Is the IOMMU protecting me?” therefore has no single answer on a real system; it has one answer per group.

Why Passthrough Requires an IOMMU

The reason an IOMMU is non-negotiable for device passthrough follows directly from how DMA works. When you hand a real device to a virtual-machine guest, the guest’s own driver programs the device’s descriptors and DMA engine using guest-physical addresses — because as far as the guest knows, it owns the machine. But guest-physical addresses are a fiction maintained by the hypervisor; the actual data lives at scattered host-physical pages. If the device DMA’d directly to the guest-physical addresses it was given, it would write to whatever host memory happens to live at those physical addresses — almost certainly not the guest’s RAM, and possibly the host kernel’s. Two things are therefore required: the device’s DMA must be redirected from guest-physical to the correct host-physical pages, and it must be confined so a compromised guest cannot make the device reach beyond its own RAM.

The IOMMU does both with one mechanism. The hypervisor (via the VFIO Framework) installs a page table for the passed-through device that maps guest-physical-address → host-physical-address for every page of guest RAM. Now when the guest’s driver tells the device “DMA to guest-physical 0x1000,” the device emits IOVA 0x1000, the IOMMU translates it through the guest’s table to the real host page backing that guest page, and the transfer lands correctly. Equally important, any IOVA not in that table — i.e. any host memory the hypervisor did not deliberately expose — simply faults. The guest cannot escape its memory through the device. This is precisely the property that lets a hypervisor give a guest native device speed without giving it the keys to the host.

Without an IOMMU there is no safe passthrough at all, only the dangerous no-IOMMU escape hatch (VFIO_NOIOMMU_IOMMU), which grants the device unrestricted physical-memory access and exists only for trusted high-performance userspace drivers (VFIO docs, v6.12).

---

The Page-Table Walk — From BDF to Physical Page

The IOMMU’s translation is a two-phase lookup: first find this device’s page table using its identity, then walk that page table using the IOVA. On Intel VT-d the structures are named precisely (Intel VT-d spec; Project ACRN VT-d HLD).

Every PCIe transaction carries a requester ID equal to the device’s Bus/Device/Function (BDF) — 8 bits of bus, 5 of device, 3 of function. The IOMMU uses this to find the right page table through two indexed tables held in main memory:

1. The root table is indexed by the bus number (256 entries). Each present root-entry points to a context table for that bus.
2. The context table is indexed by the device-and-function part of the BDF (256 entries per bus, one per function). Each present context-entry holds a second-level page-table pointer — the host-physical address of the page-table root to use for this specific function (terenceli, IOMMU introduction).

So the IOMMU first does root_table[bus] → context_table, then context_table[devfn] → page_table_root. This is what lets different devices have different page tables: two functions with different BDFs index to different context-entries pointing at different page-table roots, hence different (or shared, if the kernel chooses) IOVA→physical mappings.

Having found the page-table root, the IOMMU walks it exactly like a CPU page-table walk. The IOVA is split into multi-level index fields plus a page offset, and the hardware descends a multi-level radix tree — for 48-bit addressing, the familiar four-level layout (PML4 → PDPT → PD → PT) borrowed from the CPU’s own paging, optionally a fifth level (PML5) for 57-bit addressing — until it reaches a leaf entry giving the host-physical page; the page offset is then appended. Intel calls these the second-level page tables (the device-side analogue of guest paging). In a nested/scalable-mode setup a first-level (PASID-tagged) table can sit in front of them for two-stage translation, used when a guest runs its own IOMMU.

Because walking these tables on every DMA would be ruinous, the IOMMU caches aggressively, mirroring the CPU’s TLB. It keeps a context-cache (recently used device→page-table-root mappings) and an IOTLB (I/O Translation Lookaside Buffer, recently used IOVA→physical translations) (terenceli). When the kernel changes a mapping it must invalidate the relevant IOTLB entries, and the cost and timing of that invalidation is a real performance knob — see iommu.strict below.

---

The walk, as a sequence

sequenceDiagram
  autonumber
  participant DRV as Driver on the CPU
  participant DEV as Device DMA engine
  participant IOMMU as IOMMU (VT-d / AMD-Vi)
  participant TBL as Translation structures<br/>in system RAM
  participant RAM as Target physical page
  DRV->>DRV: dma_map_single(dev, buf, len, DMA_FROM_DEVICE)
  DRV->>IOMMU: allocate an IOVA (iova_domain rcache),<br/>iommu_map() writes the leaf PTEs
  DRV->>DEV: program descriptor with the IOVA
  DEV->>IOMMU: memory read/write transaction<br/>{requester ID = BDF, address = IOVA, PASID?}
  IOMMU->>IOMMU: IOTLB lookup, tagged by<br/>source-id + domain-id (+ PASID)
  alt IOTLB hit
    IOMMU->>RAM: forward with the cached physical address
  else IOTLB miss
    IOMMU->>TBL: root_table[bus] -> context table
    IOMMU->>TBL: context_table[devfn] -> page-table root
    Note over IOMMU,TBL: the context-cache holds this step —<br/>a hit here skips both lookups
    IOMMU->>TBL: walk PML4 -> PDPT -> PD -> PT<br/>(paging-structure caches short-circuit<br/>the upper levels)
    alt entry present and permissions allow
      IOMMU->>IOMMU: install the translation in the IOTLB
      IOMMU->>RAM: forward with the physical address
    else not present, or a permission violation
      IOMMU-->>DEV: transaction BLOCKED
      IOMMU->>TBL: append a fault record to the fault log
      IOMMU-->>DRV: DMAR/IO-page-fault interrupt<br/>(dmesg names BDF, address, read/write)
    end
  end
  DRV->>DRV: dma_unmap_single(...)
  DRV->>IOMMU: iommu_unmap() clears the PTEs<br/>then EITHER invalidate now (strict)<br/>OR queue the IOVA for later (DMA-FQ)

One DMA transaction, from dma_map_single() to the fault or the data. What it shows: the IOMMU’s per-transaction work is a cache lookup, exactly like the CPU’s, and only a miss pays for the two identity lookups plus the multi-level walk. The four caches named in the Intel specification — context-cache, PASID-cache, IOTLB, paging-structure caches — each short-circuit a different part of this path. The insight to take: the last step is the one this note is really about. dma_unmap has two possible endings, and choosing between them is choosing between “a freed IOVA can never be reached again” and “a freed IOVA stays reachable for up to 10 ms.” Everything in the next section is about that choice.

The Intel specification frames the caching problem in terms that will be immediately familiar to anyone who has read The Translation Lookaside Buffer and TLB Shootdowns: “Remapping hardware may accelerate the address-translation process by caching data from the memory-resident paging structures. Because the hardware does not ensure that the data that it caches are always consistent with the structures in memory, it is important for software to comprehend how and when the hardware may cache such data, what actions can be taken to remove cached data that may be inconsistent, and when it should do so” (VT-d spec §6). That is the CPU TLB’s contract, restated for devices — and with the same consequence: coherence is software’s job.

The spec is also explicit that nothing may be assumed about the IOTLB’s existence or retention: “Remapping hardware need not implement any IOTLBs. Remapping hardware that do implement IOTLBs may evict or invalidate any IOTLB entry at any time. Software should not rely on the existence of IOTLBs or on the retention of IOTLB entries” (§6.2.4). Invalidation is therefore a ceiling on staleness, never a guarantee that anything was cached in the first place — which is exactly why “lazy mode is fine, entries probably got evicted anyway” is not a security argument.

Cache (Intel names)	What it holds	Tagged by	Invalidated by
Context-cache	context-entries (device → page-table root)	source-id	Context-cache Invalidate Descriptor
PASID-cache	scalable-mode PASID-table entries	PASID, domain-id	PASID-cache Invalidate Descriptor
IOTLB	the effective translation for a request — first-stage-only, second-stage-only, nested, or pass-through	source-id, domain-id, PASID	IOTLB Invalidate / P_IOTLB Invalidate Descriptor
Paging-structure caches	intermediate entries encountered during a walk	domain-id (+ PASID)	folded into the IOTLB invalidate descriptors
Device-TLB (ATS)	translations cached inside the device itself	device-local	Device-TLB Invalidate Descriptor — a round trip over the PCIe link
Interrupt Entry Cache	interrupt-remapping table entries	interrupt index	Interrupt Entry Cache Invalidate Descriptor

Table: the IOMMU’s cache hierarchy per VT-d spec §6.2 and the descriptor types that clear each, §6.5.2.1–6.5.2.7. The insight to take: the Device-TLB row is the expensive outlier. With PCIe Address Translation Services (ATS) the device caches translations, so invalidating one requires an invalidation message to travel out over the link and an acknowledgement to come back — latency measured in microseconds, and dependent on a cooperating device. This is why enabling ATS is a performance win for DMA but makes every unmap dramatically more expensive, and why “untrusted device” and “ATS” are a combination the kernel treats with suspicion.

Interrupt Remapping — The Other Half of Isolation

Confining DMA is necessary but not sufficient, and the reason is a detail of x86 interrupt delivery that the VT-d specification states without euphemism (§5.1):

“Interrupt requests appear to the Root-Complex as upstream DWORD sized memory write requests to the interrupt address range FEEx_xxxxh. Since interrupt requests arrive at the Root-Complex as write requests, interrupt-remapping is co-located with the remapping hardware units.”

A Message Signaled Interrupt (MSI) is a DMA write. There is no separate interrupt wire, no distinguishing transaction type — a device raises an interrupt by writing a doorbell value to an address in the local-APIC range, and the value written encodes the destination CPU and the vector. It follows immediately that a device which can perform arbitrary DMA writes can forge arbitrary interrupts: any vector at any CPU, including the vectors the host kernel uses. Confining a device’s data DMA while leaving FEEx_xxxxh reachable confines nothing.

Interrupt remapping closes this by applying the same identify-then-translate structure to interrupts. The hardware must first be able to attribute an interrupt to a requester — “To support domain-isolation usages, the platform hardware must be capable of uniquely identifying the requester (Source-Id) for each interrupt message” — which for PCIe devices is the requester ID from the transaction header, for legacy pin interrupts is a BIOS-assigned 16-bit source-id per I/OxAPIC reported through the ACPI MADT, and for non-discoverable message-capable devices such as the HPET is a source-id the platform must assign and report through ACPI. Having identified the source, the hardware uses a remapping index carried in the request to look up an entry in a memory-resident interrupt-remapping table that the kernel controls, validates that this source is permitted to raise this interrupt, and substitutes the real destination and vector. A device can then only trigger the interrupts the kernel has provisioned for it.

The specification is also candid about where this isolation stops, and the limitation is exactly the one that makes IOMMU groups necessary: “For message-signaled interrupt requests from devices behind conventional PCI bridges, the source-id in those interrupt requests is the requester-id of the legacy bridge device… Due to this, interrupt-remapping hardware does not isolate message-signaled interrupt requests from individual devices behind such bridges.” Devices that cannot be told apart cannot be isolated from each other — for interrupts as for DMA.

This is why the kernel refuses device passthrough on a platform without interrupt remapping, and the refusal is explicit rather than advisory (drivers/vfio/vfio_iommu_type1.c:2248–2254, v6.12):

	if (!allow_unsafe_interrupts &&
	    !iommu_group_has_isolated_msi(iommu_group)) {
		pr_warn("%s: No interrupt remapping support.  Use the module param \"allow_unsafe_interrupts\" to enable VFIO IOMMU support on this platform\n",
		       __func__);
		ret = -EPERM;
		goto out_detach;
	}

The test is iommu_group_has_isolated_msi() — note that it asks about the group, not the device, precisely because of the bridge-aliasing limitation above — and the escape hatch is a module parameter whose own description says what it is for: “Enable VFIO IOMMU support for on platforms without interrupt remapping support.” Setting vfio_iommu_type1.allow_unsafe_interrupts=1 re-opens an interrupt-injection hole even when DMA is perfectly confined, and it is one of the most commonly copy-pasted lines in home-lab passthrough guides.

VT-d builds posted interrupts on top of remapping, letting an interrupt from a passed-through device be delivered straight into a running guest vCPU’s virtual APIC with no VM exit — the specification devotes §5.2.1 to the remapping-table support this requires. That mechanism is owned by Posted Interrupts.

The IOTLB and Its Invalidation Queue — The Knob That Matters

Everything about the IOMMU’s cost is concentrated in one operation: unmapping. Mapping is cheap — write some page-table entries, no cache anywhere holds a contradicting entry, and a subsequent miss will simply walk and find the new one. Unmapping is expensive, because the IOTLB may hold the translation you just deleted, and clearing it is not an instruction. It is a descriptor written into a ring buffer in system memory, followed by a doorbell write, followed by waiting for a completion descriptor to retire.

That asymmetry is why the IOMMU’s central tuning decision is not “on or off” but when the invalidation happens relative to the unmap, and why the wrong answer is so easy to arrive at by accident: the default is not what most people assume, and it differs between distributions.

Strict mode: invalidate before returning

In strict mode, __iommu_dma_unmap() clears the page-table entries, synchronously flushes the IOTLB, and only then returns the IOVA range to the allocator (drivers/iommu/dma-iommu.c:825–846, v6.12):

	iommu_iotlb_gather_init(&iotlb_gather);
	iotlb_gather.queued = READ_ONCE(cookie->fq_domain);
 
	unmapped = iommu_unmap_fast(domain, dma_addr, size, &iotlb_gather);
	WARN_ON(unmapped != size);
 
	if (!iotlb_gather.queued)
		iommu_iotlb_sync(domain, &iotlb_gather);
	iommu_dma_free_iova(cookie, dma_addr, size, &iotlb_gather);

The whole strict/lazy decision is that one boolean, iotlb_gather.queued, set from whether the domain has a flush queue. When it is false, iommu_iotlb_sync() runs inline: the driver appends an IOTLB Invalidate Descriptor and an Invalidation Wait Descriptor to the hardware’s queue, rings the doorbell, and spins until the wait descriptor retires. Every dma_unmap_*() in the system pays that round trip. For a 10 GbE NIC retiring a million packets a second, that is a million hardware round trips a second on top of the actual work.

Lazy mode (DMA-FQ): defer, batch, and reclaim later

In lazy mode the sync is skipped entirely and the freed IOVA — together with the list of page-table pages that became free — is parked on a flush queue instead (dma-iommu.c:808–823):

	else if (gather && gather->queued)
		queue_iova(cookie, iova_pfn(iovad, iova),
				size >> iova_shift(iovad),
				&gather->freelist);

The queue is a fixed-size ring, and its dimensions are the numbers worth memorising (dma-iommu.c:105–111):

/* Number of entries per flush queue */
#define IOVA_DEFAULT_FQ_SIZE	256
#define IOVA_SINGLE_FQ_SIZE	32768
 
/* Timeout (in ms) after which entries are flushed from the queue */
#define IOVA_DEFAULT_FQ_TIMEOUT	10
#define IOVA_SINGLE_FQ_TIMEOUT	1000

	Per-CPU queue (the default)	Single shared queue
Selected when	normal hardware IOMMU	dev->iommu->shadow_on_flush — a shadowed IOTLB, i.e. a paravirtualised or emulated IOMMU where a flush traps to the hypervisor
Entries	256 per CPU	32,768 total
Timer	10 ms	1,000 ms
Why	many small batches, one timer per domain, minimal cross-CPU contention	a trap-to-hypervisor flush is so expensive that it is worth batching a thousand times more aggressively — “Shadowing IOTLB flushes do better with a single large queue” (dma-iommu.c:647–660)

An IOTLB flush then happens on one of three triggers: the 10 ms timer expires (fq_flush_timeout()), the queue fills (fq_full() forces a flush inline inside queue_iova()), or something else flushes for its own reasons and the accumulated entries are reclaimed opportunistically.

sequenceDiagram
  autonumber
  participant DRV as Driver
  participant FQ as Per-CPU flush queue<br/>(256 entries)
  participant CNT as fq_flush_start_cnt /<br/>fq_flush_finish_cnt
  participant HW as IOMMU hardware
  participant ALLOC as IOVA allocator
  DRV->>DRV: dma_unmap_single()
  DRV->>HW: iommu_unmap_fast() — clear the leaf PTEs
  Note over DRV,HW: NO iotlb sync. The IOTLB may still<br/>hold the translation we just deleted.
  DRV->>FQ: queue_iova(pfn, pages, freelist)<br/>entry.counter = read(fq_flush_start_cnt)
  FQ->>FQ: arm fq_timer for 10 ms if not already armed
  Note over FQ,ALLOC: IOVA is NOT returned to the allocator yet,<br/>and the pages are NOT freed yet
  alt queue fills first
    DRV->>HW: fq_flush_iotlb() inline, right now
  else 10 ms elapses
    FQ->>HW: fq_flush_timeout() -> fq_flush_iotlb()
  end
  HW->>CNT: atomic64_inc(fq_flush_start_cnt)
  HW->>HW: ops->flush_iotlb_all(domain)
  HW->>CNT: atomic64_inc(fq_flush_finish_cnt)
  FQ->>FQ: fq_ring_free_locked() — drain entries<br/>whose counter is BELOW fq_flush_finish_cnt
  FQ->>ALLOC: free_iova_fast() — IOVA reusable
  FQ->>DRV: iommu_put_pages_list() — page-table pages freed

The deferred-invalidation protocol. What it shows: three things are deferred together — the hardware invalidation, the reuse of the IOVA, and the freeing of the page-table pages that the unmap emptied. They are released as a unit, gated by a counter comparison. The insight to take: the counter pair is doing real work. An entry records fq_flush_start_cnt at the moment it is queued, and fq_ring_free_locked() only reclaims entries whose recorded counter is strictly below fq_flush_finish_cnt — that is, entries queued before a flush that has since completed. A flush that merely started after the entry was queued is not sufficient, because it may have sampled the hardware state before this entry’s PTE write landed. Getting that comparison wrong would produce exactly the use-after-free the whole mechanism is supposed to bound.

The ordering that makes the counter comparison sound is a single explicit barrier at the top of queue_iova(), whose comment is worth reading twice because it names both things it pairs with (dma-iommu.c:210–217): “Order against the IOMMU driver’s pagetable update from unmapping @pte, to guarantee that fq_flush_iotlb() observes that if called from a different CPU before we release the lock below. Full barrier so it also pairs with iommu_dma_init_fq() to avoid seeing partially written fq state here.”

What the window actually exposes — stated precisely

The usual summary, “lazy mode means a device can still reach a freed page”, is right but too vague to reason with. The precise statement is:

• The page-table entries are cleared immediately, on every path. A device that has never had this IOVA translated cannot get one after the unmap: it would walk, find nothing, and fault.
• What survives is only what the IOTLB already cached, for a device that was already legitimately using this IOVA. The exposure is therefore bounded to this device continuing to reach the physical page it was just told to stop using — it is a use-after-free of the target page, not a path to arbitrary memory and not a cross-device leak.
• The IOVA is not recycled until after the flush, so a different subsequent mapping cannot collide with the stale entry. This is the property that makes lazy mode defensible at all.
• The window is bounded by the flush trigger: at most 10 ms on a per-CPU queue (1,000 ms with a shadowed IOTLB), and usually far less, because 256 unmaps on one CPU force a flush early.
• The page behind the stale translation, meanwhile, may have been freed by the driver and reallocated to anything. That is the actual threat: a buggy or malicious device that keeps DMAing to a just-unmapped address writes into whatever the kernel put there next.

So the honest framing is not “lazy mode is insecure” but: lazy mode converts a hard guarantee into a 10 ms guarantee, for devices that are already inside your trust boundary. That is an entirely reasonable trade for a soldered-down NVMe controller and an entirely unreasonable one for something plugged into a Thunderbolt port — which is precisely the distinction the kernel encodes, as the next section shows.

Where the default comes from

iommu_subsys_init() derives the default domain type from two independent inputs — passthrough and strict — and the derivation is short enough to quote in full (drivers/iommu/iommu.c:191–211, v6.12):

	if (!(iommu_cmd_line & IOMMU_CMD_LINE_DMA_API)) {
		if (IS_ENABLED(CONFIG_IOMMU_DEFAULT_PASSTHROUGH))
			iommu_set_default_passthrough(false);
		else
			iommu_set_default_translated(false);
 
		if (iommu_default_passthrough() && cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
			pr_info("Memory encryption detected - Disabling default IOMMU Passthrough\n");
			iommu_set_default_translated(false);
		}
	}
 
	if (!iommu_default_passthrough() && !iommu_dma_strict)
		iommu_def_domain_type = IOMMU_DOMAIN_DMA_FQ;

Three things follow. DMA-FQ is what you get when neither passthrough nor strict is selected — it is the “translate, but lazily” middle. Memory encryption (SEV-SNP, TDX) forcibly disables passthrough, because an identity domain cannot express the C-bit/shared-page distinction those platforms require. And iommu_dma_strict itself is initialised from the build configuration: static bool iommu_dma_strict __read_mostly = IS_ENABLED(CONFIG_IOMMU_DEFAULT_DMA_STRICT); (iommu.c:44) — so the distribution’s kernel config, not the upstream default, decides what your machine does. Verify it on the live system rather than assuming; the section below does exactly that for this machine, and the answer is lazy.

There is also a runtime escalation path, iommu_set_dma_strict(), which a driver or quirk can call to downgrade an already-chosen DMA_FQ default back to plain DMA (iommu.c:706–711):

void iommu_set_dma_strict(void)
{
	iommu_dma_strict = true;
	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ)
		iommu_def_domain_type = IOMMU_DOMAIN_DMA;
}

Domain types, and the security/performance ladder they encode

iommu_domain.type is where every one of these decisions is finally recorded. The full set is enumerated in include/linux/iommu.h:177–206, v6.12, built from the flag bits __IOMMU_DOMAIN_PT, __IOMMU_DOMAIN_PAGING, __IOMMU_DOMAIN_DMA_API and __IOMMU_DOMAIN_DMA_FQ:

Domain type	Translation	IOTLB invalidation on unmap	DMA isolation	Unmap cost	Who gets it
IOMMU_DOMAIN_BLOCKED (0)	none — all DMA blocked	n/a	total (the device cannot reach anything)	n/a	a device being isolated or torn down; the safe state during driver handover
IOMMU_DOMAIN_IDENTITY	IOVA is the physical address; no page tables	n/a — nothing is cached that can go stale	none	zero	iommu.passthrough=1 / iommu=pt, or a driver whose def_domain_type asks for it
IOMMU_DOMAIN_DMA	full translation via the DMA API	synchronous, every unmap	strongest translating mode	one hardware round trip per unmap	iommu.strict=1, or CONFIG_IOMMU_DEFAULT_DMA_STRICT=y, or any untrusted PCI device
IOMMU_DOMAIN_DMA_FQ	full translation via the DMA API	deferred, ≤10 ms or 256 unmaps	translating, with a bounded staleness window	amortised to roughly one flush per batch	the default when neither passthrough nor strict is set — CONFIG_IOMMU_DEFAULT_DMA_LAZY=y
IOMMU_DOMAIN_UNMANAGED	full translation, but the mappings are owned by an API user, not the DMA API	whatever that user asks for	as strong as the user builds it	user-controlled	VFIO / IOMMUFD: “DMA mappings managed by IOMMU-API user, used for VMs”
IOMMU_DOMAIN_SVA	the device shares a process’s page tables	via MMU notifiers, driven by the CPU-side TLB invalidation	same as the process	same as a CPU TLB shootdown	Shared Virtual Addressing — “DMA addresses are shared process addresses represented by mm_struct’s”
IOMMU_DOMAIN_NESTED	two stages: a guest-owned stage over a host-owned stage	both stages, via the host	guest-controlled inner, host-controlled outer	two levels of invalidation	nested virtualization with a virtual IOMMU
IOMMU_DOMAIN_PLATFORM	driver does its own thing	driver’s problem	driver’s problem	—	“Legacy domain for drivers that do their own dma_api stuff. Do not use in new drivers.”

Table: the eight domain types, ordered roughly from most to least isolated among the ones you will actually see. The insight to take: IDENTITY and DMA_FQ are the two that get chosen by default on real systems, and they sit at opposite ends of the security range — IDENTITY provides no DMA confinement at all, DMA_FQ provides confinement with a 10 ms staleness window. Reading a machine’s /sys/kernel/iommu_groups/*/type is therefore not a formality; it is the only way to know which of these a given device actually got.

The kernel does override the default for the one case where the trade-off is clearly wrong. amd_iommu_def_domain_type() opens with it (drivers/iommu/amd/iommu.c:2771–2797, v6.12):

	/* Always use DMA domain for untrusted device */
	if (dev_is_pci(dev) && to_pci_dev(dev)->untrusted)
		return IOMMU_DOMAIN_DMA;

pci_dev->untrusted is set for devices behind an external-facing port — Thunderbolt and USB4 being the canonical case. Such a device gets IOMMU_DOMAIN_DMA, never DMA_FQ: strict, synchronous invalidation on every unmap, regardless of the system default. It also loses the option of skipping the bounce buffer: dev_use_swiotlb() returns true unconditionally for an untrusted device (dma-iommu.c:601–612), so its DMA is copied through a dedicated page even though an IOMMU is present. The reason is granularity: the IOMMU maps whole pages, so mapping a 100-byte buffer would expose the other 3,996 bytes of its page to the device. Bouncing into a page the device owns exclusively closes that. This is the mechanism behind what Intel markets as Kernel DMA Protection.

Configuration — Enabling and Tuning the IOMMU

Every quotation in this section is verbatim from the v6.12 tree (Documentation/admin-guide/kernel-parameters.txt), with the line numbers given so they can be checked. The generic iommu.* parameters are the modern interface; the driver-specific spellings still exist but most are deprecated aliases, and mixing the two is a common source of confusion because — as the documentation warns — “on x86, strict mode specified via one of the legacy driver-specific options takes precedence.”

Parameter	Values	What it does (verbatim where quoted)	Fallback when unset
iommu.passthrough= (line 2343)	0 / 1	“Configure DMA to bypass the IOMMU by default. 0 - Use IOMMU translation for DMA. 1 - Bypass the IOMMU for DMA.”	“Use value of CONFIG_IOMMU_DEFAULT_PASSTHROUGH.”
iommu.strict= (line 2328)	0 / 1	“0 - Lazy mode. Request that DMA unmap operations use deferred invalidation of hardware TLBs, for increased throughput at the cost of reduced device isolation. Will fall back to strict mode if not supported by the relevant IOMMU driver. 1 - Strict mode. DMA unmap operations invalidate IOMMU hardware TLBs synchronously.”	“Use value of CONFIG_IOMMU_DEFAULT_DMA_{LAZY,STRICT}.”
iommu.forcedac= (line 2320)	0 / 1	“0 - Try to allocate a 32-bit DMA address first, before falling back to the full range if needed. 1 - Allocate directly from the full usable range, forcing Dual Address Cycle for PCI cards supporting greater than 32-bit addressing.”	0
intel_iommu=on / off (line 2210)	—	“Enable intel iommu driver.” / “Disable intel iommu driver.”	driver/firmware decides
intel_iommu=igfx_off	—	“By default, gfx is mapped as normal device. If a gfx device has a dedicated DMAR unit, the DMAR unit is bypassed by not enabling DMAR with this option. In this case, gfx device will use physical address for DMA.”	off
intel_iommu=sm_on / sm_off	—	“Enable the Intel IOMMU scalable mode if the hardware advertises that it has support for the scalable mode translation.” Scalable mode is the format that underpins PASID and nested translation.	CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON
intel_iommu=strict	—	“Deprecated, equivalent to iommu.strict=1.”	—
intel_iommu=sp_off	—	“By default, super page will be supported if Intel IOMMU has the capability. With this option, super page will not be supported.” Disabling it costs IOTLB reach.	off
intel_iommu=tboot_noforce	—	“Do not force the Intel IOMMU enabled under tboot… Note that using this option lowers the security provided by tboot because it makes the system vulnerable to DMA attacks.”	off
amd_iommu=off (line 325)	—	“do not initialize any AMD IOMMU found in the system”	on if firmware (IVRS) reports one
amd_iommu=fullflush	—	“Deprecated, equivalent to iommu.strict=1.”	—
amd_iommu=force_isolation	—	“Force device isolation for all devices. The IOMMU driver is not allowed anymore to lift isolation requirements as needed. This option does not override iommu=pt.”	off
amd_iommu=pgtbl_v1 / pgtbl_v2	—	“Use v1 page table for DMA-API (Default).” / “Use v2 page table for DMA-API.”	v1
amd_iommu=nohugepages / v2_pgsizes_only	—	“Limit page-sizes used for v1 page-tables to 4 KiB” / “to 4KiB/2Mib/1GiB.”	off
iommu=pt / iommu=nopt (line 2305)	—	legacy x86 spellings of iommu.passthrough=1 / =0	—

Table: the parameters that actually change IOMMU behaviour, with the fallback each has when unset. The insight to take: every row’s last column says “use the value of a CONFIG_ symbol”. None of these have a hard-coded upstream default — the distribution’s kernel build decides, which is why the only reliable way to know what a machine is doing is to read /boot/config-$(uname -r) and /sys/kernel/iommu_groups/*/type, not to recall what the default “is”.

Two rows deserve more than a table cell.

iommu.passthrough is the protection-versus-overhead dial for the host’s own devices. With =1 the kernel installs an identity (IOVA = physical) domain for in-kernel DMA: the IOMMU is initialised and available to VFIO, but it does not constrain the host’s trusted drivers, so there is no IOTLB churn and no translation overhead on the host’s own I/O. With =0 even the host’s drivers’ DMA is fully translated and confined, which costs throughput but means a buggy or compromised in-kernel driver cannot DMA outside its mapped buffers. Note that VFIO passthrough to a guest is unaffected either way — VFIO always installs a real translating IOMMU_DOMAIN_UNMANAGED for the device it owns. Note also the interaction established in the previous section: on a confidential-computing platform, iommu.passthrough=1 is silently ignored, because iommu_subsys_init() calls iommu_set_default_translated() when cc_platform_has(CC_ATTR_MEM_ENCRYPT).

iommu.strict=0 has a failure mode that its documentation names and most summaries omit: “Will fall back to strict mode if not supported by the relevant IOMMU driver.” Asking for lazy mode is a request. If the driver cannot build a flush queue — iommu_dma_init_fq() failing, or a driver that does not advertise deferred-flush capability — you silently get strict mode and its throughput. The symptom is a performance regression with no log line mentioning your parameter; the diagnosis is to read /sys/kernel/iommu_groups/*/type and see DMA where you expected DMA-FQ.

A typical passthrough host therefore boots with intel_iommu=on iommu=pt (or the AMD equivalent): the IOMMU is enabled so VFIO can use it for the assigned device, while iommu=pt keeps the host’s own devices on the fast identity path. A hardening-oriented laptop or workstation wants the opposite end — iommu.passthrough=0 iommu.strict=1 — and pays for it in I/O throughput.

This Machine’s Actual IOMMU State

Rather than describe a hypothetical configuration, here is what the machine this note was written on is actually doing — an AMD Ryzen AI MAX+ 395 (“Strix Halo”) running Fedora’s 7.1.13-200.fc44.x86_64. Everything below is read from the live system.

There are no IOMMU parameters on the kernel command line at all. /proc/cmdline is:

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-7.1.13-200.fc44.x86_64 root=UUID=953e3fa7-… ro
rootflags=subvol=root rd.luks.uuid=luks-a586454e-… rhgb quiet

So every IOMMU behaviour on this box comes from the build configuration, and /boot/config-7.1.13-200.fc44.x86_64 settles it:

CONFIG_AMD_IOMMU=y
CONFIG_IOMMU_DMA=y
CONFIG_IOMMU_DEFAULT_DMA_LAZY=y
# CONFIG_IOMMU_DEFAULT_DMA_STRICT is not set
# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
CONFIG_IRQ_REMAP=y
CONFIG_SWIOTLB=y
# CONFIG_SWIOTLB_DYNAMIC is not set
CONFIG_IOMMUFD=m
CONFIG_VFIO=m
CONFIG_VFIO_NOIOMMU=y

Fedora ships CONFIG_IOMMU_DEFAULT_DMA_LAZY=y — deferred IOTLB invalidation, i.e. the equivalent of iommu.strict=0, out of the box. Tracing that through iommu_subsys_init(): passthrough is off (so iommu_set_default_translated()), iommu_dma_strict initialises to false, therefore iommu_def_domain_type = IOMMU_DOMAIN_DMA_FQ. And that is exactly what the system reports:

Observation	Command	Result
IOMMU present and which vendor	ls /sys/class/iommu/	ivhd0 -> ../../devices/pci0000:00/0000:00:00.2/iommu/ivhd0 — an AMD IVHD (I/O Virtualization Hardware Definition) unit, i.e. AMD-Vi
Capability register	cat /sys/class/iommu/ivhd0/amd-iommu/cap	190b640f
Extended feature registers	cat /sys/class/iommu/ivhd0/amd-iommu/features	246577efa2254afa:10
Number of isolation groups	ls /sys/kernel/iommu_groups/ | wc -l	31
Domain type per group	cat /sys/kernel/iommu_groups/*/type | sort | uniq -c	29 × DMA-FQ, 2 × identity
Which groups are identity	iterate …/type	group 19 → 0000:c2:00.0; group 25 → 0000:c3:00.1
What those devices are	lspci -nns c2:00.0 / c3:00.1	the integrated Radeon 8060S GPU (driver amdgpu) and the Strix Halo Neural Processing Unit (driver amdxdna)
The NVMe, for contrast	group 18 → 0000:c1:00.0, driver nvme	DMA-FQ — translated, lazily invalidated
Reserved IOVA regions	cat /sys/kernel/iommu_groups/0/reserved_regions	0x00000000fee00000 0x00000000feefffff msi and 0x000000fd00000000 0x000000ffffffffff reserved

Table: the real IOMMU state of one ordinary Fedora workstation. The insight to take: the two identity groups are not a misconfiguration and not a firmware quirk — they are a driver request, and the code that makes it is quoted below. Everything else on the machine is translated with deferred invalidation, which means every non-GPU device on this box has a bounded-but-nonzero IOTLB staleness window by default, and nobody chose that: it came from Fedora’s kernel config.

The identity domains trace to amd_iommu_def_domain_type() (drivers/iommu/amd/iommu.c:2783–2794, v6.12):

	/*
	 * Do not identity map IOMMUv2 capable devices when:
	 *  - memory encryption is active, because some of those devices
	 *    (AMD GPUs) don't have the encryption bit in their DMA-mask
	 *    and require remapping.
	 *  - SNP is enabled, because it prohibits DTE[Mode]=0.
	 */
	if (pdev_pasid_supported(dev_data) &&
	    !cc_platform_has(CC_ATTR_MEM_ENCRYPT) &&
	    !amd_iommu_snp_en) {
		return IOMMU_DOMAIN_IDENTITY;
	}

The GPU and the NPU are PASID-capable (IOMMUv2) devices; memory encryption is not active on this bare-metal desktop and SNP is not enabled; so both get an identity domain. The consequence is concrete and worth stating plainly: on this machine the integrated GPU and the NPU can DMA to any physical address. That is a deliberate kernel policy for PASID-capable devices that manage their own translations, not an oversight — but it means “my laptop has an IOMMU, so a DMA attack through the GPU is impossible” is false here, and reading /sys/kernel/iommu_groups/*/type is the only way to have known.

Note also the interaction with confidential computing that the same comment encodes: on an SEV-SNP machine this branch is skipped, every device gets a translating domain, and the GPU’s DMA becomes fully confined. Security posture on identical silicon therefore differs based on whether memory encryption is on.

Uncertain

Verify: the field-by-field decoding of cap = 190b640f and features = 246577efa2254afa:10. Reason: these are raw AMD IOMMU Capability and Extended Feature register values; decoding them requires the AMD I/O Virtualization Technology (IOMMU) Specification (publication 48882), which could not be retrieved (www.amd.com → HTTP 000 on every attempt). They are reported verbatim above rather than interpreted. To resolve: fetch publication 48882 and decode against its Capability Register and EFR field definitions, or read drivers/iommu/amd/amd_iommu_types.h’s FEATURE_* bit definitions, which cover the subset Linux uses. uncertain

Uncertain

Verify: whether this machine’s IOMMU is actually enabled and translating at the hardware level, as distinct from the kernel having built domains for it. Reason: the definitive evidence is the AMD-Vi initialisation lines in the boot log, and dmesg on this system returns read kernel buffer failed: Operation not permitted (kernel.dmesg_restrict), while /sys/kernel/debug is mode 0700 root-owned so the IOMMU debugfs directory is unreadable and there is no sudo available. The /sys/kernel/iommu_groups/ hierarchy existing and carrying live type files is strong indirect evidence, but it is not the log line. To resolve: sudo dmesg | grep -i 'AMD-Vi\|iommu' — expect AMD-Vi: Interrupt remapping enabled and a Default domain type: Translated line from iommu_subsys_init()’s pr_info. uncertain

The DMA API — The Interface That Hides All of This

A driver author never calls iommu_map(). They call the DMA API, whose entire purpose is to be identical whether there is an IOMMU, a swiotlb bounce buffer, a partial-address-line quirk, or nothing at all behind it. The abstraction rests on one type and one rule, stated at the top of the documentation (Documentation/core-api/dma-api.rst, v6.12):

“A dma_addr_t can hold any valid DMA address for the platform. It can be given to a device to use as a DMA source or target. A CPU cannot reference a dma_addr_t directly because there may be translation between its physical address space and the DMA address space.”

That sentence is the IOMMU, expressed as a type rule. On a machine with an IOMMU the dma_addr_t is an IOVA; without one it is a physical (or bus) address; with swiotlb it is the address of a bounce buffer. The driver must not care, and the API is designed so that it cannot.

The API splits into two families with genuinely different semantics, and choosing the wrong one is the commonest DMA bug after forgetting dma_mapping_error().

Call	Family	What it maps	Lifetime	CPU/device ownership sync required?	Notes
dma_alloc_coherent(dev, size, &handle, gfp)	coherent / consistent	allocates and maps in one step	long-lived, driver-owned	no — “a write by either the device or the processor can immediately be read by the processor or device without having to worry about caching effects”	returns both a CPU pointer and a dma_handle; “consistent memory can be expensive on some platforms, and the minimum allocation length may be as big as a page”
dma_pool_alloc()	coherent	sub-page chunks carved from coherent pages	long-lived	no	the documented remedy for the page-granularity minimum above: “the simplest way to [consolidate] is to use the dma_pool calls”
dma_alloc_noncoherent()	explicitly non-coherent	allocates and maps, without the coherence guarantee	long-lived	yes — dma_sync_single_for_{cpu,device}()	for drivers that would rather sync explicitly than pay for coherent memory
dma_map_single(dev, cpu_addr, size, dir)	streaming	one physically contiguous buffer	one transfer	yes	“it will fail if the user tries to map a non-physically contiguous piece of memory… memory to be mapped by this API should be obtained from sources which guarantee it to be physically contiguous (like kmalloc)”
dma_map_page(dev, page, offset, size, dir)	streaming	a page, or part of one	one transfer	yes	“although the <offset> and <size> parameters are provided to do partial page mapping, it is recommended that you never use these unless you really know what the cache width is”
dma_map_sg(dev, sg, nents, dir)	streaming	a scatter/gather list	one transfer	yes	returns fewer entries than it was given when the IOMMU merges: “this may be shorter than <nents> passed in if some elements of the scatter/gather list are physically or virtually adjacent and an IOMMU maps them with a single entry”
dma_map_resource(dev, phys, size, dir, attrs)	streaming	device MMIO, not RAM	one transfer	yes	“The API should only be used to map device MMIO resources, mapping of RAM is not permitted.” — this is the peer-to-peer DMA primitive

Table: the DMA API’s mapping types and their sync obligations, all quotations from dma-api.rst at v6.12. The insight to take: the dma_map_sg row is where an IOMMU pays for itself most visibly. Without one, every discontiguous fragment of a scatterlist is a separate bus address the device must be programmed with; with one, the kernel can map all of them to a single contiguous IOVA range and hand the device one descriptor. This is why dma_map_sg() returns a count that the driver must use in place of nents — and why a driver that iterates the original nents instead of the returned count is a classic, silent corruption bug. The documentation flags the inverse trap too: on dma_unmap_sg, “<nents> must be the number you passed in, not the number of DMA address entries returned.”*

stateDiagram-v2
  [*] --> CpuOwned: kmalloc / alloc_page
  CpuOwned: CPU OWNS the buffer<br/>CPU may read and write freely<br/>device must not touch it
  Mapped: DEVICE OWNS the buffer<br/>an IOVA exists, IOMMU PTEs are live<br/>CPU must NOT read or write
  CpuOwned --> Mapped: dma_map_single / _page / _sg<br/>+ dma_mapping_error() CHECK
  CpuOwned --> [*]: kfree
  Mapped --> Failed: dma_mapping_error() != 0
  Failed: MAPPING FAILED<br/>no IOVA, nothing to unmap
  Failed --> CpuOwned: back off, retry, or abort the request
  Mapped --> CpuPeek: dma_sync_single_for_cpu(DMA_FROM_DEVICE)<br/>borrow it back to read
  CpuPeek: CPU may READ<br/>device must be idle
  CpuPeek --> Mapped: dma_sync_single_for_device(DMA_TO_DEVICE)<br/>hand it back
  Mapped --> Unmapped: dma_unmap_single / _page / _sg
  Unmapped: PTEs cleared.<br/>Under STRICT — IOTLB already flushed, IOVA freed.<br/>Under DMA-FQ — IOVA queued, stale IOTLB entry<br/>may persist for up to 10 ms
  Unmapped --> CpuOwned: CPU may use the buffer again

The streaming-DMA ownership lifecycle. What it shows: a streaming mapping is a transfer of ownership, not merely an address translation. Between dma_map_* and dma_unmap_* the buffer belongs to the device and the CPU must not touch it except through an explicit dma_sync_* borrow. The insight to take: the bottom state is where this note’s subject matter re-enters a driver author’s world. dma_unmap_* returns the buffer to the CPU immediately, but under DMA-FQ the hardware’s IOTLB may still be able to reach it — so a device that ignores its own descriptor rings can write into a buffer the driver has already handed back. Strict mode is the setting that makes the “Unmapped” state mean what it looks like it means.

The sync rules have exactly one correct formulation, and the documentation states it as a bulleted obligation rather than advice:

“You must do this: Before reading values that have been written by DMA from the device (use the DMA_FROM_DEVICE direction); After writing values that will be written to the device using DMA (use the DMA_TO_DEVICE direction); before and after handing memory to the device if the memory is DMA_BIDIRECTIONAL.”

Three more API details matter to anyone tuning an IOMMU-backed path:

• dma_mapping_error() is not optional. “In some circumstances dma_map_single(), dma_map_page() and dma_map_resource() will fail to create a mapping” — IOVA space exhaustion is a real cause under an IOMMU, and the failure is a sentinel value, not a negative return. For dma_map_sg() the failure indication is a return of 0, and the documentation is unusually blunt about the consequence: “It is critical that the driver do something; in the case of a block driver aborting the request or even oopsing is better than doing nothing and corrupting the filesystem.”
• dma_opt_mapping_size() exists because IOMMU mapping cost is not free. “Mapping larger buffers may take much longer in certain scenarios. In addition, for high-rate short-lived streaming mappings, the upfront time spent on the mapping may account for an appreciable part of the total request lifetime.” A driver that splits its transfers at this boundary is explicitly optimising around IOVA allocation and IOTLB behaviour.
• dma_need_sync() “Returns %true if dma_sync_single_for_{device,cpu} calls are required to transfer memory ownership. Returns %false if those calls can be skipped” — the escape hatch for hot paths on coherent platforms, added so that drivers need not choose between correctness everywhere and speed on x86.

Underneath, on a translating domain, the IOVA itself comes from a per-domain allocator with a per-CPU cache designed for exactly this workload: alloc_iova_fast() over an iova_domain whose “magazines” hold IOVA_MAG_SIZE = 127 PFNs each, sized deliberately so “iova_magazine [is] 1024 bytes, so that no memory will be wasted” (drivers/iommu/iova.c:555–575, v6.12). The IOVA allocator and the flush queue are two halves of the same design: the allocator makes dma_map cheap, the flush queue makes dma_unmap cheap, and lazy mode is what lets the second one hand IOVAs back to the first in bulk.

DMA Bounce Buffers — The IOMMU’s Software Cousin

Even with an IOMMU, some situations need data copied through an intermediary buffer rather than translated in place. The kernel mechanism is swiotlb (the software I/O TLB), “a memory buffer allocator used by the Linux kernel DMA layer” that implements bounce buffering: “The DMA is done to/from this temporary memory buffer, and the CPU copies the data between the temporary buffer and the original target memory buffer” (swiotlb.rst, v6.12).

Its original purpose was addressing-range limits: “As physical memory sizes grew beyond 4 GiB, some devices could only provide 32-bit DMA addresses,” so the kernel allocates bounce buffers below the 4 GiB line, the 32-bit device DMAs to the low buffer, and the CPU copies to/from the real (possibly high) target. An IOMMU can solve the same problem more elegantly by mapping a high physical page to a low IOVA — which is one reason an IOMMU reduces (but does not eliminate) reliance on swiotlb.

The modern and increasingly important use is confidential computing. In an encrypted-memory VM (AMD SEV-SNP, Intel TDX), “the guest VM’s memory is encrypted by default and not accessible by the host hypervisor and VMM. For the host to do I/O on behalf of the guest, the I/O must be directed to guest memory that is unencrypted” (Confidential Computing VMs, kernel docs). Such guests “set a kernel-wide option to force all DMA I/O to use bounce buffers, and the bounce buffer memory is set up as unencrypted,” so the device DMAs into shared/unencrypted bounce pages and the guest CPU copies between those and its private encrypted memory (swiotlb.rst, v6.12). This is why I/O-heavy confidential VMs carry a real performance penalty: every DMA pays an extra CPU copy plus the address-space book-keeping. The interplay with the trust boundary is developed in Linux Virtualization MOC’s confidential-computing section.

---

Failure Modes and Common Misunderstandings

“The IOMMU is only for virtualization.” False, and dangerously so. The IOMMU provides DMA-attack protection outside any VM: a malicious peripheral — the classic threat being a hostile device plugged into a Thunderbolt/USB-C or PCIe port, as in the “Thunderclap” class of attacks — is confined by the IOMMU exactly as a passed-through device is. This is why modern laptops enable the IOMMU (Intel markets the firmware feature as “Kernel DMA Protection”) with no virtualization in play, and why iommu.passthrough=0 is a meaningful hardening choice on a workstation.

“My machine has an IOMMU, so my devices are confined.” Also false, and this note’s own machine is the counter-example: 2 of its 31 IOMMU groups are identity domains, meaning the integrated GPU and the NPU can DMA to any physical address. That is a deliberate kernel policy for PASID-capable devices, not a bug — but it means the question “is this device confined?” has a per-group answer that must be read from /sys/kernel/iommu_groups/<n>/type, never assumed from the presence of /sys/class/iommu/.

“iommu.strict=0 is a small, safe optimisation” / “iommu.strict=0 is reckless.” Both are wrong, and the precise version is in the IOTLB section above: lazy mode converts a hard guarantee into a bounded ≤10 ms guarantee (or ≤1,000 ms with a shadowed IOTLB), for translations a device already legitimately held, with the IOVA held back from reuse until the flush completes. It is a use-after-free window on the target page, not a route to arbitrary memory. Whether that is acceptable is a function of whether the device is inside your trust boundary — which is why the kernel makes the decision per-device for untrusted ports rather than globally.

Nobody chose the default. The single most common real-world misconfiguration in this area is not a wrong setting but an unexamined one. iommu_dma_strict is initialised from CONFIG_IOMMU_DEFAULT_DMA_STRICT, so the answer is whatever the distribution built, and it differs between distributions and between releases of the same one. On the machine measured here the answer is lazy, set by CONFIG_IOMMU_DEFAULT_DMA_LAZY=y, with no kernel-command-line parameter anywhere in sight.

Asking for lazy mode does not mean getting it. iommu.strict=0 is documented to “fall back to strict mode if not supported by the relevant IOMMU driver”, and the fallback is silent. The symptom is a throughput regression with no log line naming your parameter; the diagnosis is cat /sys/kernel/iommu_groups/*/type showing DMA where you expected DMA-FQ.

DMAR / IO-page-fault floods. When a device DMAs to an unmapped IOVA, the IOMMU logs a fault (Intel: “DMAR fault”; the dmesg line names the device BDF, the faulting address, and read/write). A burst usually means the driver and the IOMMU mappings disagree: a driver handing the device an address it never mapped, a partially mapped guest-RAM region in a VFIO setup (see VFIO Framework’s “forgetting to map all guest RAM”), or a genuine use-after-unmap exposed by lazy invalidation. Note the asymmetry that makes the last case nasty to debug: under strict mode a use-after-unmap faults immediately and loudly; under lazy mode the same bug succeeds silently for up to 10 ms and only faults if it happens to occur after the flush. Switching a suspicious workload to iommu.strict=1 is therefore a diagnostic technique, not only a hardening one.

Confusing the IOMMU with EPT/NPT. Both translate addresses for virtualization, but they are different units doing different jobs. The CPU’s EPT/NPT translates guest-physical → host-physical for CPU accesses; the IOMMU translates IOVA → host-physical for device DMA. A passed-through device’s DMA goes through the IOMMU, never through EPT. See Two-Dimensional Paging (EPT and NPT).

Confusing the IOTLB with the CPU’s TLB. They solve the same problem and share almost all of their vocabulary, but the mechanics differ in one respect that changes everything: a CPU TLB invalidation is an instruction, while an IOTLB invalidation is a descriptor written to a queue in memory, completed asynchronously and signalled by a wait descriptor. That is why the CPU side’s answer to invalidation cost is “send fewer IPIs” while the IOMMU side’s answer is “write fewer descriptors, and don’t wait for them”. The structural comparison is developed in The Translation Lookaside Buffer and TLB Shootdowns.

Diagnosing an IOMMU problem without root

Question	Command	Works unprivileged?
Is there an IOMMU, and whose?	ls -l /sys/class/iommu/	yes — ivhd* = AMD-Vi, dmar* = Intel VT-d, smmu3.* = Arm SMMUv3
What domain type did each group get?	for g in /sys/kernel/iommu_groups/*/; do echo "$(basename $g) $(cat $g/type)"; done	yes — this is the single most informative command in this note
Which devices are in a group?	ls /sys/kernel/iommu_groups/<n>/devices/	yes
What IOVA ranges are carved out?	cat /sys/kernel/iommu_groups/<n>/reserved_regions	yes — msi and direct regions appear here
What did the kernel decide at boot?	dmesg | grep -iE 'AMD-Vi|DMAR|Default domain type'	no on this machine — kernel.dmesg_restrict; needs sudo
Change a group’s domain type at runtime	echo DMA > /sys/kernel/iommu_groups/<n>/type	no — root, and the group must have no driver bound
Per-device IOTLB / fault statistics	/sys/kernel/debug/iommu/	no — /sys/kernel/debug is 0700 root:root here
What was actually configured at build time?	grep IOMMU /boot/config-$(uname -r)	yes — and this is where the real default lives

Table: IOMMU observability from an unprivileged shell. The insight to take: the group type file and the kernel config together answer almost every practical question without root. The two things that need privilege — the boot log and debugfs — are the two that would tell you about faults, which is why IOMMU fault debugging is so much harder than IOMMU configuration checking.

Alternatives and When to Choose Them

There is no real “alternative” to the IOMMU for the safe-passthrough job — it is the only hardware that can confine device DMA. The genuine choices are how much to use it, and the ladder is best read as a single ordered spectrum:

Configuration	DMA confinement	Unmap cost	Appropriate when
IOMMU disabled (iommu=off, amd_iommu=off)	none	zero	physically secured machine, no untrusted peripherals, no passthrough — and you have measured that the IOMMU is actually costing you something
iommu.passthrough=1 (iommu=pt), identity domains for host devices	none for host devices; full for VFIO-assigned ones	zero for host devices	the classic passthrough host: the IOMMU exists for the guest’s assigned device, host I/O stays on the fast path
Translating, lazy (DMA-FQ) — the usual distribution default	full, with a ≤10 ms staleness window per unmap	amortised, ~one flush per 256 unmaps or per 10 ms	general-purpose machines with trusted internal devices; this is what most people are running without knowing it
Translating, strict (iommu.strict=1)	full, no staleness window	one hardware round trip per unmap	untrusted peripherals, high-security hosts, and as a debugging mode to turn silent use-after-unmap into an immediate fault
Strict plus forced bounce buffering	full, no staleness window, no sub-page exposure	round trip plus a CPU copy	what untrusted (external-port) devices get automatically; what confidential VMs get by necessity

Table: the IOMMU configuration ladder. The insight to take: the rows are not “security settings” in isolation — each one is a specific answer to “what is the smallest window in which a device can reach memory it should not, and what am I willing to pay to close it?” The bottom row closes the sub-page window that all the others leave open, and it is the only row where the CPU does work proportional to the data.

For addressing-limit problems specifically, an IOMMU’s remapping is preferable to swiotlb bounce buffering because it avoids the CPU copy — the IOMMU can simply map a high physical page to a low IOVA. But swiotlb remains the fallback where no IOMMU exists, is mandatory for confidential VMs (where copying through shared memory is the only way an encrypted guest can talk to host-driven devices), and is forced for untrusted devices even when an IOMMU is present, because page-granularity mapping cannot protect the rest of a page from a device given a sub-page buffer.

Production Notes

Every cloud GPU instance, NFV data plane, and bare-metal-with-passthrough deployment depends on the IOMMU being enabled and on the firmware exposing clean IOMMU groups (good ACS support) so devices can be assigned individually rather than in coarse bundles — see IOMMU Groups and Device Isolation. Server platforms generally ship correct ACS; consumer boards frequently do not, which is the root of most home-lab passthrough pain and the reason the (security-weakening) ACS-override patch exists (Arch Wiki: PCI passthrough). Security-conscious operators increasingly enable full IOMMU translation for host devices too, treating any PCIe/Thunderbolt port as an untrusted DMA source. And the rise of confidential VMs has made the IOMMU’s software cousin, swiotlb, a first-class performance concern: the forced bounce-buffer copy is a known, measurable tax on I/O-heavy SEV-SNP/TDX workloads that engineering teams budget for explicitly.

---

See Also

• The Translation Lookaside Buffer and TLB Shootdowns — the CPU-side twin: the same caching problem, the same strict-versus-deferred dilemma, solved with IPIs instead of descriptor queues. Read the two together.
• VFIO Framework — the kernel framework that programs the IOMMU on userspace’s behalf, and the owner of IOMMU_DOMAIN_UNMANAGED.
• IOMMU Groups and Device Isolation — how devices are bundled into indivisible isolation units by ACS; the granularity at which every domain-type decision in this note is actually made.
• SR-IOV and Virtual Functions — one physical device split into many DMA-isolated functions, each with its own requester ID and therefore its own IOMMU context.
• Two-Dimensional Paging (EPT and NPT) — the CPU-side guest→host translation that the IOMMU is routinely confused with; different unit, different job.
• get_user_pages and Page Pinning — why guest RAM must be pinned before an IOVA can point at it, and what happens if it moves.
• Posted Interrupts — interrupt remapping’s high-performance successor for delivering a passed-through device’s interrupts straight into a running vCPU.
• Direct IO and O_DIRECT, virtio Device Model, vhost (In-Kernel virtio Backend) — the I/O paths that sit above the DMA API described here.
• Memory Management Unit, Linux Page Table Hierarchy — the CPU-side structures the IOMMU’s tables deliberately mirror.
• MOCs: Linux Virtualization MOC · Linux Device Drivers and Device Model MOC · Linux Memory Management MOC