Memory-Mapped IO and ioremap

Most peripherals on a modern system are controlled by reading and writing a small block of device registers. Memory-mapped I/O (MMIO) is the arrangement where the hardware decodes a slice of the CPU’s physical address space as accesses to those registers rather than to dynamic random-access memory (DRAM): “a part of the CPU’s address space is interpreted not as accesses to memory, but as accesses to a device” (device-io.rst). A Linux driver cannot touch those physical addresses directly — paging means the kernel runs in virtual address space, and the addresses must be mapped with the right caching attributes. The driver calls ioremap() (or, in practice, the managed wrapper devm_ioremap() or devm_platform_ioremap_resource()) to obtain a special void __iomem * cookie, then reads and writes through the typed accessors readl()/writel() (and the byte/word/quad siblings) rather than dereferencing the pointer. Those accessors are the contract: they enforce endianness conversion, compiler-ordering, and CPU memory barriers that a raw pointer dereference would silently skip. This note covers MMIO from the bare-metal driver angle against Linux 6.12 LTS; the orthogonal question of how a hypervisor traps and emulates a guest’s MMIO lives in MMIO and Port IO Emulation.

Mental Model

Think of a device’s register block as a tiny ROM/RAM chip wired onto the system bus at a fixed physical address — say 0xfe200000. When the CPU issues a load or store to an address in that range, the memory controller does not route it to DRAM; it routes it to the device, which interprets the offset as “register number” and the data as “register contents.” The device’s documentation gives you a register map (offset 0x00 = control, 0x04 = status, 0x08 = data, …), and programming the device is nothing more than writing the right bit patterns to the right offsets and reading status back.

Three facts make this harder than a normal memory access, and ioremap plus the accessors exist to handle each:

1. The kernel runs paged. A physical address is meaningless to kernel code; you need a kernel virtual address that the memory-management unit (MMU) maps to that physical region. ioremap builds that page-table mapping.
2. Device memory must not be cached or speculated like DRAM. If the CPU cached a status register, a stale value would be returned forever; if it reordered or coalesced writes, the device would see the wrong command sequence. ioremap maps the region with device/uncached attributes, and the accessors add ordering barriers.
3. The compiler must not optimize the accesses away. A polling loop while (status_reg == 0); would be hoisted out of the loop if the compiler thought the value never changed. The accessors use volatile to forbid that.

flowchart LR
  DRV["Driver code<br/>writel(CMD, base + 0x00)"] --> ACC["Accessor readl/writel<br/>(endianness + barrier + volatile)"]
  ACC --> VA["Kernel virtual addr<br/>(void __iomem *base)"]
  VA -->|"page table from ioremap()<br/>device/uncached attrs"| PA["Physical addr 0xfe200000<br/>(a PCI BAR or SoC reg block)"]
  PA --> DEV["Device register file<br/>ctrl / status / data"]

From driver code to hardware register. What it shows: the driver never names a physical address at access time; it holds a void __iomem * cookie that ioremap mapped to the device’s physical register block with uncached attributes, and every access funnels through an accessor that adds the endianness/ordering/volatile guarantees. The insight to take: the __iomem pointer is deliberately not a normal pointer — it is an opaque token that is only legal to pass to ioremap/readl/writel. Dereferencing it directly compiles, but skips every guarantee in the chain, which is exactly the class of bug sparse exists to catch.

Where the Physical Address Comes From — BARs and Resources

A driver does not hardcode 0xfe200000. The physical base and size come from enumeration. On a Peripheral Component Interconnect (PCI / PCIe) device, the register block lives behind a Base Address Register (BAR) — a field in PCI configuration space that firmware (or the kernel) programs with the physical address the device’s MMIO window has been assigned, and which the kernel exposes as a struct resource (see PCI and PCIe Enumeration). On a System-on-Chip (SoC), the same information comes from the Device Tree reg = <...> property, surfaced to a platform device as a struct resource of type IORESOURCE_MEM. Either way the driver retrieves a struct resource describing {start, end, flags} and maps it.

The most common modern idiom collapses “get the resource and map it” into one managed call. For a platform device:

void __iomem *devm_platform_ioremap_resource(struct platform_device *pdev,
                                             unsigned int index);

Its kerneldoc reads: “call devm_ioremap_resource() for a platform device … Return: a pointer to the remapped memory or an ERR_PTR() encoded error code on failure” (platform.c). Internally it calls devm_platform_get_and_ioremap_resource() with a NULL output-resource argument, which in turn calls platform_get_resource(pdev, IORESOURCE_MEM, index) to fetch the struct resource and devm_ioremap_resource() to claim and map it. So one line does what used to be three or four, and on driver detach the mapping is released automatically.

ioremap and Its Managed Wrappers

The primitive is ioremap(phys_addr_t offset, size_t size), which establishes a page-table mapping from a freshly allocated kernel virtual range to the physical region [offset, offset+size) with device-memory attributes, returning a void __iomem *. The device-io.rst documentation is blunt about the contract around it: “Most architectures allocate new address space each time you call ioremap(), and they can run out unless you call iounmap()” — i.e. forgetting to unmap is a resource leak, not merely untidy.

There is a family of variants that change the caching/ordering attributes of the mapping, all defined in asm-generic/io.h with architecture overrides:

• ioremap_wc() — write-combining. Adjacent writes may be merged into burst transfers. Used for framebuffers and other write-heavy regions where the device tolerates merged writes. Falls back to plain ioremap if the architecture has no write-combining type: #ifndef ioremap_wc / #define ioremap_wc ioremap.
• ioremap_wt() — write-through.
• ioremap_np() — non-posted writes, “stronger semantics than regular ioremap()”; returns NULL unless the architecture explicitly implements it.
• devm_ioremap_uc() — explicitly uncached (UC), the strongest “no caching at all” mapping.

Because manual unmap is error-prone, modern drivers use the device-managed (devm_) forms, which tie the mapping lifetime to the device’s bound lifetime (see Managed Device Resources devres). lib/devres.c documents devm_ioremap() as “Managed ioremap(). Map is automatically unmapped on driver detach.” The key managed entry points:

/* Map a raw physical region, auto-unmapped on detach. Returns NULL on failure. */
void __iomem *devm_ioremap(struct device *dev, resource_size_t offset,
                           resource_size_t size);
 
/* Validate + request_mem_region() + ioremap a struct resource.
 * Returns an ERR_PTR() (not NULL) on failure. */
void __iomem *devm_ioremap_resource(struct device *dev,
                                    const struct resource *res);

The distinction matters: devm_ioremap_resource() “checks that a resource is a valid memory region, requests the memory region and ioremaps it” (devres.c) — the request_mem_region() step claims exclusive ownership of the physical range in /proc/iomem, so two drivers cannot accidentally map the same registers. Plain devm_ioremap() skips that claim. And crucially, devm_ioremap*_resource() returns an ERR_PTR()-encoded error, so the driver must check with IS_ERR(), whereas devm_ioremap() returns plain NULL on failure. Mixing these up is a recurring bug.

A realistic probe:

static int widget_probe(struct platform_device *pdev)
{
        struct widget *w;
        void __iomem *base;
 
        w = devm_kzalloc(&pdev->dev, sizeof(*w), GFP_KERNEL);  /* managed alloc */
        if (!w)
                return -ENOMEM;
 
        base = devm_platform_ioremap_resource(pdev, 0);  /* get res[0] + map it */
        if (IS_ERR(base))                                /* ERR_PTR, NOT NULL */
                return PTR_ERR(base);
 
        w->base = base;
        platform_set_drvdata(pdev, w);
 
        /* Bring the device up: enable, then read back status. */
        writel(WIDGET_CTRL_ENABLE, w->base + WIDGET_CTRL);   /* offset 0x00 */
        if (!(readl(w->base + WIDGET_STATUS) & WIDGET_READY)) /* offset 0x04 */
                return -ENODEV;
        return 0;
        /* No iounmap, no release_mem_region in error paths or .remove():
         * devres unwinds the map automatically on detach. */
}

Line by line: devm_platform_ioremap_resource(pdev, 0) fetches the first IORESOURCE_MEM resource and maps it. IS_ERR(base) / PTR_ERR(base) is the mandatory error check for the _resource family. w->base + WIDGET_CTRL is plain pointer arithmetic on the __iomem cookie — adding a byte offset is fine and stays __iomem. The writel/readl calls are where the actual hardware programming happens. The comment underlines the payoff of devm_: there is no cleanup code, because unbinding the driver releases the mapping and the memory region.

The Accessors — readl/writel and Why You Must Use Them

The cardinal rule, stated in device-io.rst: the __iomem pointer “must only be passed from and to functions that explicitly operated on an __iomem token, in particular the ioremap() and readl()/writel() functions.” You do not write *base = 1;. You write writel(1, base);.

The accessor family, by access width:

Width	Read	Write
8-bit	readb	writeb
16-bit	readw	writew
32-bit	readl	writel
64-bit	readq	writeq

(b = byte, w = word/16, l = long/32, q = quad/64 — historical x86-era naming.) Looking at the actual asm-generic/io.h definition of readl’s byte sibling shows what the accessor does beyond a load:

static inline u8 readb(const volatile void __iomem *addr)
{
        u8 val;
        log_read_mmio(8, addr, _THIS_IP_, _RET_IP_);
        __io_br();                 /* barrier BEFORE the read */
        val = __raw_readb(addr);   /* the actual volatile load */
        __io_ar(val);              /* barrier AFTER the read */
        log_post_read_mmio(val, 8, addr, _THIS_IP_, _RET_IP_);
        return val;
}

Three things happen that a raw dereference would not: (1) the access is bracketed by memory barriers (__io_br/__io_ar); (2) the actual access goes through __raw_readb, which casts to volatile so the compiler may not elide or reorder it; (3) the wider variants readw/readl/readq additionally convert from little-endian device order to CPU-native order via __le16_to_cpu() and friends — so a driver works unchanged on a big-endian CPU. writel is symmetric: a write barrier __io_bw() before the __raw_writeb, and __io_aw() after. The log_*_mmio calls feed the optional MMIO tracing infrastructure.

The __raw_readb/__raw_writeb forms are exposed too, and they are exactly the unguarded versions — “direct memory access via volatile pointer casts without endianness conversion”:

static inline u8 __raw_readb(const volatile void __iomem *addr)
{
        return *(const volatile u8 __force *)addr;
}

The __force cast is what lets raw accessors strip the __iomem annotation; ordinary driver code should never need them.

ioread32 / iowrite32

ioread8/16/32/64 and iowrite8/16/32/64 are an alternative spelling that, per device-io.rst, have “almost identical behavior” to readl()/writel() “but they can also operate on __iomem tokens returned for mapping PCI I/O space with pci_iomap() or ioport_map().” In other words they unify MMIO and port-I/O behind one call: the token they take might be a real memory mapping or an encoded port number, and the accessor dispatches accordingly. On a pure-MMIO mapping ioread32(addr) is readl(addr) (asm-generic/io.h defines ioread8 as return readb(addr); when CONFIG_GENERIC_IOMAP is unset). Port I/O proper — inb/outb on x86 — is covered in Port IO and Legacy IO Spaces.

Memory Ordering — readl vs readl_relaxed

This is the subtlest part of MMIO and the source of the nastiest bugs. The default accessors carry strong ordering guarantees; the _relaxed variants trade some away for speed.

memory-barriers.txt enumerates the guarantees the default readX()/writeX() provide on default I/O attributes. Paraphrasing its five points: all readX()/writeX() to the same peripheral are ordered with respect to each other; a writeX() issued while holding a spinlock is ordered before a writeX() to the same peripheral from another CPU that acquired the lock later; a writeX() first waits for completion of prior memory writes by the same thread; a readX() completes before any subsequent memory read by the same thread; and a readX() completes before any subsequent delay() loop. These are precisely the guarantees a driver intuitively assumes — “my writes reach the device in program order, and they interleave correctly with the spinlock protecting the register block.”

The relaxed forms readl_relaxed()/writel_relaxed() drop those barriers. asm-generic/io.h says they “are like the regular version, but are not guaranteed to provide ordering against spinlocks or memory accesses,” and their definitions simply omit the __io_br/__io_ar (and __io_bw/__io_aw) calls:

static inline u8 readb_relaxed(const volatile void __iomem *addr)
{
        u8 val;
        log_read_mmio(8, addr, _THIS_IP_, _RET_IP_);
        val = __raw_readb(addr);   /* no surrounding barriers */
        log_post_read_mmio(val, 8, addr, _THIS_IP_, _RET_IP_);
        return val;
}

memory-barriers.txt is explicit that relaxed accessors “do not guarantee ordering with respect to locking, normal memory accesses or delay() loops” but remain ordered with respect to other accesses to the same peripheral. The legitimate use is a tight inner loop that touches only one device’s registers — e.g. draining a FIFO — where you have reasoned that no DRAM access or lock interleaves and the per-peripheral ordering suffices; device-io.rst warns to “use this with care.”

The barrier macros themselves (asm-generic/io.h) explain the why: __io_ar() maps to a read memory barrier rmb() to “prevent prefetching of coherent DMA data ahead of a dma-complete,” and __io_bw() maps to wmb() to “flush writes to coherent DMA data before possibly triggering a DMA read.” So MMIO ordering is not abstract — it is exactly what makes a “kick the DMA engine” write safe after you have filled a buffer in DRAM (cross-link The DMA API). When you need an explicit barrier independent of an accessor, the kernel exposes mb() (full), rmb() (read), wmb() (write), and dma_wmb()/dma_rmb() for the DMA-buffer-vs-MMIO ordering case.

Uncertain

Verify: the precise wording of the five readX()/writeX() ordering guarantees and the exact mmiowb() semantics in 6.12. Reason: these were summarized via a fetch of memory-barriers.txt rather than read line-by-line, and the “KERNEL I/O BARRIER EFFECTS” section is dense and has been revised across releases. To resolve: read the “KERNEL I/O BARRIER EFFECTS” and “ACQUIRING FUNCTIONS” sections of Documentation/memory-barriers.txt at the v6.12 tag in full. uncertain

Posted writes and the read-back idiom

A separate ordering hazard is posted writes. On PCI, a writel() may be posted — accepted by the bus bridge and reported complete to the CPU before it has actually reached the device. device-io.rst warns: “many authors are burned by the fact that PCI bus writes are posted asynchronously. A driver author must issue a read from the same device to ensure that writes have occurred.” The idiom is to follow a critical write with a dummy readl() of any register on the same device; the read cannot complete until the bus has flushed the pending write, so it forces the write through. This is not something the barriers alone guarantee across a bridge — it is a property of the bus topology, which is why the read-back is a manual driver responsibility.

mmiowb and spinlock-guarded MMIO

Historically, drivers issued mmiowb() before releasing a spinlock to guarantee that writes from inside the critical section reached the device before another CPU’s writes from its own critical section — important on weakly ordered architectures with multiple host bridges. In modern kernels the accounting is automatic: asm-generic/io.h defines __io_aw() (the after-write hook inside writel) as mmiowb_set_pending(), and the spinlock release path flushes it. The comment in the header reads “serialize device access against a spin_unlock, usually handled there.” So a driver using normal writel() inside a spinlock no longer needs an explicit mmiowb().

Uncertain

Verify: that explicit mmiowb() is fully redundant for spinlock-guarded writel() in 6.12 (i.e. the tracking is unconditional on all architectures), versus still being recommended in some niche. Reason: the “automatic mmiowb” rework landed years ago but architecture coverage and the exact rule were not re-verified against the 6.12 tree here. To resolve: check include/asm-generic/mmiowb.h and the relevant arch overrides at v6.12. uncertain

The __iomem Sparse Annotation

__iomem is an __attribute__((noderef, address_space(...))) annotation (a no-op for gcc/clang code generation) that exists for the static analyzer sparse. device-io.rst: “The ‘sparse’ semantic code checker can be used to verify that this is done correctly.” Because __iomem pointers live in a distinct address space from normal kernel pointers, sparse flags any code that (a) dereferences an __iomem pointer directly, or (b) passes a normal pointer where an __iomem one is expected, or vice-versa. Running make C=2 drivers/foo/ builds with sparse and surfaces these as warnings. This is the compiler-assisted enforcement of “never dereference the cookie” — the annotation cannot stop you at runtime, but it makes the mistake loud at analysis time.

Failure Modes

• Dereferencing the cookie directly. u32 v = *(u32 *)base; compiles and may even work on x86 (strongly ordered, and ioremap already maps uncached), then fails on ARM/PowerPC where the missing barriers and endianness conversion produce garbage or wrong-ordered device commands. Caught by sparse.
• NULL vs ERR_PTR confusion. Checking devm_ioremap_resource() with if (!base) instead of if (IS_ERR(base)) treats an error pointer (a small negative value cast to a pointer) as success and dereferences it — instant oops.
• Stale reads from caching the wrong way. Mapping a status register with ioremap_wc() (write-combining) or, worse, treating the region as cacheable memory, returns stale values forever. Status registers want plain ioremap()/devm_ioremap() (device/uncached).
• Lost writes from posted-write timing. Writing a “go” bit and then immediately reading a result without a read-back of the device can race the posted write; the device acts on the old command. Symptom: works under a debugger (slow), fails at speed. Fix: read-back idiom.
• Endianness on big-endian hosts. Using __raw_readl() (no byte-swap) instead of readl() works on x86 little-endian and breaks on big-endian. Use the cooked accessors unless you have a specific reason.
• Forgetting iounmap with the non-managed ioremap. A leak of kernel virtual address space; on 32-bit systems the vmalloc/ioremap region is small and exhausts. The devm_ forms eliminate this.

Alternatives and When to Choose Them

• Plain ioremap() + manual iounmap() — only when devm_ is unavailable (no struct device, e.g. very early boot or some firmware paths). Otherwise prefer the managed forms.
• devm_ioremap() vs devm_ioremap_resource() — use _resource when you have a struct resource and want the request_mem_region() exclusivity claim; use plain devm_ioremap() for a bare physical range you already own (e.g. a sub-window of a region you mapped elsewhere).
• devm_platform_ioremap_resource() — the default for platform/Device-Tree drivers: one call, fetches resource by index and maps it.
• pcim_iomap_regions() / pci_iomap() — the PCI-native managed mappers that map a BAR by index and integrate with PCI’s resource accounting.
• regmap — when the device might be behind I2C/SPI as well as MMIO, or when you want register caching, range validation, and uniform read-modify-write. regmap’s MMIO backend is built directly on ioremap + readl/writel, so it is a higher abstraction over exactly this machinery.
• The DMA API — for bulk data movement, MMIO is the wrong tool: you do not stream megabytes through a 32-bit register. MMIO programs the device’s control registers; DMA moves the payload. The two cooperate (MMIO writes the DMA descriptor address and kicks the engine).

Production Notes

The migration to devm_platform_ioremap_resource() and devm_ioremap_resource() is one of the quieter cleanups in the kernel’s history: thousands of platform drivers used to open-code platform_get_resource() + request_mem_region() + devm_ioremap() + matching error/cleanup paths, and the managed one-liner removed both the boilerplate and a steady stream of unmap/release leaks on error paths. The _relaxed accessors are heavily used in performance-critical subsystems — DMA-engine drivers, high-rate NIC register paths, and clock/interrupt-controller code — precisely where the cost of a full barrier on every register poke is measurable; in those drivers you will see a deliberate mix of writel() for the command that must be ordered against DRAM and writel_relaxed() for the inner FIFO drain. The posted-write read-back idiom shows up constantly in PCI NIC and storage drivers as a comment like /* flush posted write */ after a doorbell write.

See Also

• MMIO and Port IO Emulation — the virtualization counterpart: how KVM traps and emulates a guest’s MMIO/PIO when no real device backs the address. This note is bare-metal; that one is the hypervisor side.
• Port IO and Legacy IO Spaces — the x86 legacy I/O-port mechanism (inb/outb), the other historical way to reach device registers.
• regmap and Register Access Abstraction — the bus-agnostic abstraction layered over ioremap/readl/writel (and over I2C/SPI), adding caching and validation.
• PCI and PCIe Enumeration — where MMIO physical addresses come from on PCI: the Base Address Registers (BARs).
• Platform Devices and Drivers — where SoC register-block addresses come from: Device-Tree reg properties surfaced as struct resource.
• The DMA API — the complementary mechanism for bulk data movement; MMIO programs the engine, DMA moves the bytes.
• Managed Device Resources devres — the devm_ lifetime model that makes devm_ioremap() auto-unmap on detach.
• Linux Device Drivers and Device Model MOC — parent map (§7, “Talking to Hardware”).