vault

io_uring and the Block Layer

17 min read

io_uring and the Block Layer

io_uring is the Linux kernel’s modern asynchronous-I/O interface (Jens Axboe, kernel 5.1, 2019), and storage is the workload it was born to drive. This note is about the block-storage-specific angle: how an IORING_OP_READ/WRITE/READV/WRITEV/FSYNC submission queue entry turns into a bio and reaches the multi-queue block layer (blk-mq) and the device. The key facts are that io_uring’s read/write opcodes plug straight into the same *_iter file operations that ordinary pread/pwrite use, but without blocking the submitting thread; that combined with O_DIRECT (the open flag that bypasses the page cache and DMAs data straight between user pages and the device), io_uring forms an end-to-end async fast path; that registered (fixed) buffers let the kernel pin the DMA target once instead of per-I/O; and that two ring-level switches — IORING_SETUP_IOPOLL (busy-poll for completions instead of taking a device interrupt) and IORING_SETUP_SQPOLL (a kernel thread submits the ring with no syscall at all) — strip the last fixed costs off the path, letting a single core drive over a million I/O operations per second (Axboe, kernel.dk/io_uring.pdf). The generic ring machinery — the submission queue (SQ), completion queue (CQ), the SQE/CQE structures, the memory barriers — is not re-derived here; it lives in The io_uring Submission and Completion Queues and is assumed.

Mental Model

Think of three layers that, before io_uring, each charged a separate toll on every single I/O. The syscall layer charged a read/write trap per operation; the page-cache layer charged a memcpy between the kernel’s cached page and your buffer; the interrupt layer charged a hardware IRQ, a context switch, and a wakeup per completion. io_uring’s storage fast path removes all three tolls, one mechanism per toll. Ring batching (and optionally SQPOLL) removes the syscall toll — many operations cross the boundary per io_uring_enter, or zero with the kernel poll thread. O_DIRECT removes the page-cache toll — the bio points at your pages and the device DMAs into them directly. IORING_SETUP_IOPOLL removes the interrupt toll — instead of sleeping until an IRQ fires, the submitting context busy-polls the hardware completion queue and reaps the result the instant it lands.

flowchart TB
  APP["Application fills SQE:<br/>opcode=IORING_OP_READ, fd, off, addr, len"] --> SUB["io_uring_enter (or SQPOLL thread)<br/>consumes SQE"]
  SUB --> RW["io_uring/rw.c: io_read / io_write<br/>build kiocb, set IOCB_DIRECT/IOCB_HIPRI"]
  RW --> VFS["vfs_iter_read/write ->  f_op->read_iter<br/>(ext4/xfs/blockdev)"]
  VFS --> DIO["O_DIRECT: iomap_dio_rw /<br/>__blkdev_direct_IO -> build bio over USER pages"]
  DIO --> SBIO["submit_bio() -> blk-mq"]
  SBIO --> HCTX{"IORING_SETUP_IOPOLL?"}
  HCTX -->|"no"| IRQ["device IRQ -> blkdev_bio_end_io<br/>-> kiocb->ki_complete -> CQE posted"]
  HCTX -->|"yes"| POLL["HCTX_TYPE_POLL hw queue;<br/>io_do_iopoll busy-polls bio_poll()<br/>-> CQE posted, no IRQ"]
  IRQ --> CQE["CQE in CQ ring: res = bytes, user_data echoed"]
  POLL --> CQE

The storage fast path from SQE to CQE. What it shows: a read/write SQE flows through io_uring/rw.c into the ordinary read_iter/write_iter file operation, takes the O_DIRECT branch to build a bio over the user’s own pages, and reaches blk-mq via submit_bio; completion is delivered either by a device interrupt (the default) or by busy-polling a dedicated poll hardware queue (IORING_SETUP_IOPOLL). The insight to take: io_uring does not invent a new storage path — it asynchronously drives the same read_iter/submit_bio path the synchronous syscalls use, which is why every filesystem and every block driver works under io_uring for free.

Mechanical Walk-through

From SQE to read_iter

A block-I/O submission queue entry carries an opcode from the read/write family. The v6.12 uapi header defines them in order: IORING_OP_READV and IORING_OP_WRITEV (vectored, the addr field points at an iovec array), IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED (use a pre-registered buffer, selected by buf_index), IORING_OP_FSYNC (flush the file’s dirty data to durable storage), and the later, simpler IORING_OP_READ and IORING_OP_WRITE (a single contiguous buffer at addr/len, added because the common case does not need an iovec). All of them describe the same primitive — move len bytes between offset off in file fd and buffer addr — and differ only in how the buffer is expressed.

When io_uring_enter(2) (or the SQPOLL thread) consumes such an SQE, the handler in io_uring/rw.c constructs a kernel I/O control block (struct kiocb), points it at the target file, and calls the very same f_op->read_iter / f_op->write_iter operation that a synchronous preadv2 would. The difference is the completion callback: io_uring installs its own kiocb->ki_complete (either io_complete_rw for the interrupt path or io_complete_rw_iopoll for the poll path), so that when the I/O finishes asynchronously the kernel posts a CQE rather than returning to a blocked caller. This is the structural reason io_uring “just works” with every filesystem: it reuses the file’s existing iterator operation and only swaps the completion path.

The O_DIRECT fast path to submit_bio

io_uring can drive buffered I/O too, but the storage fast path that interests us is O_DIRECT. The file is opened with O_DIRECT, which sets IOCB_DIRECT on every kiocb. On that branch the filesystem does not stage data through the page cache; instead it maps the file range to on-device extents and builds a bio whose segments point directly at the user’s pages, then calls submit_bio(). For modern filesystems this is iomap_dio_rw (fs/iomap/direct-io.c); for a raw block device opened directly (e.g. /dev/nvme0n1) it is __blkdev_direct_IO* in block/fops.c. The full mechanics of that branch — the alignment contract, the iomap engine, coherency with buffered I/O — are the subject of Direct IO and O_DIRECT and are not repeated here. What matters for this note is the async property: iomap_dio_rw returns -EIOCBQUEUED for a non-sync kiocb, and when the final bio completes the block layer runs the dio end-I/O handler, which calls iocb->ki_complete(iocb, ret) — landing back in io_uring’s completion callback. As the O_DIRECT note puts it, “the direct path was built to complete asynchronously without ever blocking the submitting thread,” which is exactly the contract io_uring needs.

The single-bio async case for a raw block device is worth seeing concretely. In block/fops.c, __blkdev_direct_IO_async builds one bio over the user iterator, then:

if (iocb->ki_flags & IOCB_HIPRI) {
        bio->bi_opf |= REQ_POLLED;          /* this bio will be polled, not IRQ-driven */
        submit_bio(bio);
        WRITE_ONCE(iocb->private, bio);     /* stash the bio so iopoll can find it */
} else {
        submit_bio(bio);                    /* IRQ-driven completion */
}
return -EIOCBQUEUED;                          /* tell the caller: not done yet */

REQ_POLLED is the per-bio flag that marks it for completion polling rather than interrupt delivery; iocb->private stashes the bio so the poll loop can later find it. When the I/O is not high-priority, the same bio is submitted without REQ_POLLED and a device interrupt will eventually run blkdev_bio_end_io to complete it. Either way the call returns -EIOCBQUEUED immediately — the submitting thread is never parked on this I/O.

How blk-mq receives the bio

submit_bio() hands the bio to blk-mq, which stages it in a per-CPU software queue, possibly merges it with adjacent requests, runs it through the I/O scheduler (typically none for NVMe), and dispatches it on a hardware dispatch queue that maps to one of the device’s submission queues. The mapping of CPUs to hardware queues is the subject of blk-mq CPU to Queue Mapping; the crucial detail for io_uring is the type of hardware queue. blk-mq supports three hardware-queue types (include/linux/blk-mq.h): HCTX_TYPE_DEFAULT (“all I/O not otherwise accounted for”), HCTX_TYPE_READ (“just for READ I/O”), and HCTX_TYPE_POLL (“polled I/O of any kind”). A driver advertises a map[HCTX_MAX_TYPES] array, one CPU-to-queue map per type it supports. A REQ_POLLED bio is steered onto an HCTX_TYPE_POLL queue — a hardware queue that the driver configured without an interrupt — so that nothing will fire an IRQ on completion; the submitter must poll for it instead. This is why IORING_SETUP_IOPOLL requires the driver to have provisioned poll queues (for the NVMe PCIe driver, via the nvme.poll_queues=N module parameter).

Polled completions: IORING_SETUP_IOPOLL

When the ring is created with IORING_SETUP_IOPOLL, completions are busy-polled rather than interrupt-delivered. Axboe’s design document is precise about what this means and why it helps:

“In this context, polling refers to performing IO without relying on hardware interrupts to signal a completion event. When IO is polled, the application will repeatedly ask the hardware driver for status on a submitted IO request… For very low latency devices, polling can significantly increase the performance. The same is true for very high IOPS applications as well, where high interrupt rates makes a non-polled load have a much higher overhead.” (kernel.dk/io_uring.pdf)

The ring setup enforces strict conditions, visible in io_uring/rw.c:

if (ctx->flags & IORING_SETUP_IOPOLL) {
        if (!(kiocb->ki_flags & IOCB_DIRECT) || !file->f_op->iopoll)
                return -EOPNOTSUPP;          /* poll requires O_DIRECT + an iopoll method */
        kiocb->private = NULL;
        kiocb->ki_flags |= IOCB_HIPRI;       /* mark this I/O high-priority/polled */
        kiocb->ki_complete = io_complete_rw_iopoll;
        req->iopoll_completed = 0;
}

So a polled ring may submit only O_DIRECT operations on files whose f_op provides an iopoll method — exactly because the kernel “cannot know if a call to io_uring_enter(2)… can safely sleep waiting for events, or if it should be actively polling for them” (kernel.dk/io_uring.pdf). Buffered I/O and non-pollable files are rejected with -EOPNOTSUPP. The application reaps polled completions by calling io_uring_enter(2) with IORING_ENTER_GETEVENTS; the kernel then runs io_do_iopoll, which walks the ring’s in-flight list and calls each file’s iopoll:

ret = file->f_op->iopoll(&rw->kiocb, &iob, poll_flags);

For a block-backed file, f_op->iopoll is iocb_bio_iopoll (block/blk-core.c), which fetches the stashed bio from iocb->private and calls bio_poll(bio, iob, flags). bio_poll reads the bio’s saved cookie (bi_cookie, the hardware-queue number recorded when the request was dispatched), checks the device actually supports polling (q->limits.features & BLK_FEAT_POLL), and calls blk_mq_poll → blk_hctx_poll, which invokes the driver’s mq_ops->poll(hctx, iob) to harvest completions directly from the hardware completion queue. No interrupt is involved at any point. The mechanism of that busy-wait spin — and why classic and hybrid polling no longer exist — is the subject of Polled IO and Hybrid Polling.

Syscall-less submission: SQPOLL against storage

The complementary switch is IORING_SETUP_SQPOLL, which spawns a dedicated kernel thread that watches the submission ring’s tail and submits new SQEs without the application calling io_uring_enter at all:

“When the application updates the SQ ring and fills in a new sqe, the kernel side will automatically notice the new entry (or entries) and submit them. This is done through a kernel thread, specific to that io_uring.” (kernel.dk/io_uring.pdf)

Combined with IORING_SETUP_IOPOLL, SQPOLL also reaps: “If the io_uring instance is setup with IORING_SETUP_IOPOLL, then the kernel thread will take care of reaping completions as well.” A storage application that keeps the queue busy therefore performs zero system calls in steady state — the SQPOLL thread submits, polls the NVMe completion queue, and posts CQEs, while the application only reads and writes shared ring memory. To avoid burning a core when idle, the thread sleeps after sq_thread_idle milliseconds (default 1 second) and sets IORING_SQ_NEED_WAKEUP in the SQ flags; the application must then issue one io_uring_enter(... IORING_ENTER_SQ_WAKEUP) to wake it. SQPOLL setup is privileged — historically it required registered files, then CAP_SYS_NICE (Linux 5.11), and from 5.13 the special privilege requirement was relaxed (io_uring_setup(2)).

Uncertain

Verify: the exact privilege requirements for IORING_SETUP_SQPOLL in 6.12. Reason: the man-page text describes the 5.11/5.13 evolution but the consulted snapshot may predate further 6.x changes, and SQPOLL privilege has interacted with the broader io_uring security hardening (io_uring_disabled sysctl, see the syscall-batching note). To resolve: read io_uring/sqpoll.c and the io_uring_setup(2) man page at the 6.12 tag. uncertain

Registered (Fixed) Buffers for Zero-Copy DMA

The single most impactful storage optimization beyond the ring itself is registered buffers. Every O_DIRECT I/O needs its target pages pinned in physical memory for the duration of the DMA, because the device writes into them by physical address and they must not be reclaimed or moved underneath an in-flight transfer. For an ordinary (non-fixed) buffer the kernel must pin the pages with get_user_pages on submission and release them on completion — a per-I/O cost that, at a million IOPS, dominates. io_uring_register(2) with IORING_REGISTER_BUFFERS lets the application pre-pin a set of buffers once; the kernel builds durable page mappings and keeps the pages pinned for the lifetime of the registration. Thereafter IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED set the SQE’s buf_index to select a registered buffer, and the kernel skips the per-I/O pinning and translation entirely — the buffer is already a ready-made DMA target. The mechanics of the registration table, the buf_index selection, and fixed files (the analogous optimization for file-descriptor references) are owned by Registered Buffers and Fixed Files and the generic CQ note; the storage takeaway is simply that fixed buffers convert per-I/O page pinning into a one-time setup cost, which is what makes O_DIRECT-over-io_uring reach hardware line rate.

Configuration and Code

A minimal O_DIRECT read against an NVMe device through io_uring, with polling, using liburing:

#include <liburing.h>
#include <fcntl.h>
#include <stdlib.h>
 
struct io_uring ring;
/* IORING_SETUP_IOPOLL: completions are busy-polled, not IRQ-delivered.       */
io_uring_queue_init(256, &ring, IORING_SETUP_IOPOLL);            /* 1 */
 
int fd = open("/dev/nvme0n1", O_RDONLY | O_DIRECT);             /* 2 */
 
void *buf;
posix_memalign(&buf, 4096, 4096);                               /* 3 */
 
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, 4096, /*offset=*/0);          /* 4 */
 
io_uring_submit_and_wait(&ring, 1);                            /* 5 */
 
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);                               /* 6 */
/* cqe->res == 4096 on success, or -errno; cqe->user_data echoes the SQE tag */
io_uring_cqe_seen(&ring, cqe);
  • Line 1 — IORING_SETUP_IOPOLL builds a polled ring. The NVMe device must have poll queues (nvme.poll_queues=N) or the I/O will be rejected when submitted, because bio_poll checks BLK_FEAT_POLL and a device with no poll queues never sets it.
  • Line 2 — O_DIRECT is mandatory for a polled ring: io_rw_init_file returns -EOPNOTSUPP if IOCB_DIRECT is not set on a poll ring. Opening the raw block device gives the rawest path; a file on a direct-I/O-capable filesystem (ext4/xfs/f2fs) works identically through iomap_dio_rw.
  • Line 3 — the buffer must satisfy the device’s O_DIRECT alignment (offset, length, and address all multiples of the logical block size, usually 512 or 4096); a misaligned buffer fails with -EINVAL. See Direct IO and O_DIRECT for the exact contract and STATX_DIOALIGN discovery.
  • Line 4 — io_uring_prep_read fills an SQE with opcode = IORING_OP_READ, the fd, the buffer, the length, and the offset. For zero-copy at scale, register the buffer once and use io_uring_prep_read_fixed (IORING_OP_READ_FIXED) instead, eliminating per-I/O page pinning.
  • Line 5 — io_uring_submit_and_wait advances the SQ tail and, on a poll ring, enters the kernel with IORING_ENTER_GETEVENTS so io_do_iopoll busy-polls the device for the completion.
  • Line 6 — the CQE carries res (the byte count, like a syscall return, or a negated errno) and user_data (the opaque tag copied verbatim from the SQE, used to match the completion to its request since completions may arrive out of order).

To drive maximal IOPS with a single core and no syscalls in steady state, combine all three switches: io_uring_queue_init_params with IORING_SETUP_IOPOLL | IORING_SETUP_SQPOLL | IORING_SETUP_SQ_AFF, pinning the SQPOLL thread to a dedicated CPU via sq_thread_cpu.

Failure Modes and Misunderstandings

  • Polled ring rejects buffered I/O. Submitting a non-O_DIRECT read on an IORING_SETUP_IOPOLL ring returns -EOPNOTSUPP (from io_rw_init_file). The kernel cannot poll a page-cache copy that completes synchronously in software. Open with O_DIRECT, or do not use a poll ring.
  • Poll ring on a device with no poll queues. Even with O_DIRECT, if the NVMe driver was not given poll_queues, bio_poll finds BLK_FEAT_POLL unset and returns 0 — the I/O never completes via polling, and a poll-only reaping loop hangs. The fix is nvme.poll_queues=N (and confirming the device exposes poll queues).
  • Misaligned O_DIRECT buffer. The most common storage-io_uring bug is reused from plain O_DIRECT: any of offset, length, or buffer address not a multiple of the logical block size yields -EINVAL in the CQE’s res. The error surfaces asynchronously, which makes it easy to overlook — always check cqe->res.
  • Freeing a buffer before completion under SQPOLL. Without SQPOLL, a non-fixed buffer is safe to reuse once io_uring_enter returns. Under SQPOLL, submission is asynchronous, so a buffer being read/written by an in-flight op must live until its CQE arrives — freeing it early corrupts data or faults during DMA.
  • Assuming IORING_OP_FSYNC orders against unrelated writes. IORING_OP_FSYNC flushes the file’s data, but io_uring completes operations out of order; to fsync after a set of writes you must chain them with IOSQE_IO_LINK (or drain with IOSQE_IO_DRAIN), or the fsync may execute before the writes it was meant to follow. Ordering is the application’s responsibility, not an implicit guarantee. See The io_uring Submission and Completion Queues for the link/drain flags.
  • Expecting O_DIRECT writes to be durable on return. A completed O_DIRECT write means the device accepted the data, not that it is on stable media if the device has a volatile write cache. Durability still requires IORING_OP_FSYNC (or O_DSYNC/RWF_DSYNC). See Write Barriers FUA and Cache Flushes.

Alternatives and When to Choose Them

  • Synchronous pread/pwrite with O_DIRECT. Simplest; one thread blocks per I/O. Fine for low queue depths, but cannot keep a fast NVMe device busy from one thread because the device finishes faster than a single blocking thread can resubmit.
  • POSIX AIO (libaio / the io_submit family). The pre-io_uring async path. It works only for O_DIRECT, has higher per-operation overhead, often falls back to synchronous behavior for buffered I/O, and — as the design doc shows — “doesn’t support polled IO,” so io_uring “driv[es] twice the amount of IOPS for the same workload” (kernel.dk/io_uring.pdf). io_uring is its successor; new code should not use libaio. See Asynchronous IO Models in Linux.
  • A thread pool doing blocking buffered I/O. Easy to reason about and benefits from the page cache for reuse-heavy workloads, but pays a thread per outstanding I/O and a memcpy per byte. Choose it when caching and simplicity matter more than peak throughput.
  • io_uring with buffered (non-direct) I/O. io_uring can batch buffered reads/writes; this keeps page-cache benefits while amortizing syscall cost, but cannot use IORING_SETUP_IOPOLL. Choose it for cache-friendly workloads that still want batched async submission.

Production Notes

The canonical production user of this path is a storage engine that owns its own cache and wants the device’s full IOPS: the design document reports io_uring driving roughly 1.7M IOPS polled (versus libaio’s cliff around 608K) on the same 4 KiB random-read test, and about 1.2M IOPS with polling disabled (kernel.dk/io_uring.pdf) — concrete evidence that the gains compound across batching, direct I/O, and polling. Real deployments include the ScyllaDB and SPDK-adjacent database engines, Ceph’s io_uring backend, and PostgreSQL’s recent asynchronous-I/O work, all of which pair O_DIRECT, registered buffers, and (where the device has poll queues) IORING_SETUP_IOPOLL. The standard tuning recipe is: open data files with O_DIRECT; register buffers once and use the *_FIXED opcodes; set the NVMe poll_queues parameter and create the ring with IORING_SETUP_IOPOLL; and for the very highest IOPS, add IORING_SETUP_SQPOLL with a pinned poll CPU so the steady state issues no syscalls at all. When diagnosing throughput shortfalls, check that the device actually advertises poll queues (/sys/block/<dev>/queue/io_poll reads 1 only when BLK_FEAT_POLL is set), that buffers are registered (otherwise per-I/O get_user_pages shows up in profiles), and that the I/O scheduler is none for NVMe (any reordering scheduler wastes CPU on a device that does not benefit from it).

See Also

Graph View