vault

Asynchronous IO Models in Linux

16 min read

Asynchronous IO Models in Linux

Linux offers no fewer than five distinct ways to do I/O without blocking the calling thread, and they fall into two fundamentally different philosophies — readiness and completion. The oldest answer is the crudest: keep doing blocking syscalls, but on a pool of worker threads, so the main thread stays responsive (this is how the standard library’s POSIX AIO and Node.js’s libuv handle files). The next is readiness notification — select/poll/epoll — where the kernel tells you when an fd is ready and you then issue a non-blocking syscall yourself; this is the basis of every high-performance network server, but it has a fatal blind spot for files, because a regular disk file is always reported ready, so readiness polling gives no asynchrony for disk I/O at all (unixism.net, select(2)). The third and fourth are true completion interfaces for files: glibc’s userspace POSIX AIO (which is really just a hidden thread pool) and the kernel’s native AIO (io_setup/io_submit/io_getevents, the libaio interface), which works asynchronously only for O_DIRECT access and behaves synchronously for ordinary buffered I/O (kernel.dk/io_uring.pdf). The fifth and modern answer, io_uring (kernel 5.1, 2019), is the completion model done right: it handles buffered and direct I/O, sockets and files, uniformly, serving cached data inline and offloading only what truly blocks. This note maps that landscape and the trade-offs that separate the models; the io_uring file path itself is detailed in io_uring and the File Path.

Uncertain

Version/behavior claims pinned where they matter: the native-AIO limitations (O_DIRECT-only, synchronous-for-buffered, blocking submission) are quoted from Jens Axboe’s io_uring design document (kernel.dk/io_uring.pdf, fetched and text-extracted this session) and corroborated by unixism.net; they describe the interface as of the 6.12 / 6.18 LTS era and have been true for the life of the interface. The “regular files are always ready” property of select/poll/epoll is a long-standing, deliberate kernel behavior; the man pages do not spell it out in a single quotable line, so I flag it as sourced to secondary-but-authoritative references rather than a single primary man-page sentence. To resolve definitively: a regular-file fd added to an epoll set always returns EPOLLIN/EPOLLOUT, observable empirically. uncertain

Mental Model — Readiness versus Completion

The whole landscape collapses onto one axis: does the kernel tell you when you may act (readiness) or that the act is finished (completion)?

  • A readiness interface (select, poll, epoll) watches a set of file descriptors and wakes you when one can proceed without blocking. You then perform the actual read()/write() yourself, non-blocking. The data movement is still your syscall; the interface only schedules when you make it.
  • A completion interface (POSIX AIO, native AIO, io_uring) takes the whole operation — “read these bytes into this buffer” — and notifies you when it has already happened. The data is in your buffer by the time you hear about it; you never issue the read yourself.

The distinction is not academic, because of a hard asymmetry between sockets and files. A socket has a meaningful “ready” state: data has either arrived in the kernel’s receive buffer or it hasn’t, and epoll can report that edge precisely. A regular file on disk has no such state — the kernel can always satisfy a read, it just might have to block on the disk to do it. So select/poll/epoll “will always tell regular files as being ready for I/O” (unixism.net); adding a file fd to an epoll set is pointless, because it reports ready instantly whether or not the data is in the page cache. This single fact is why a readiness reactor cannot do asynchronous disk I/O, and why every model below the epoll line exists.

flowchart TB
  ROOT["I want I/O without blocking this thread"] --> AXIS{"readiness or<br/>completion?"}
  AXIS -->|"readiness"| READY["select / poll / epoll<br/>'fd is ready, you go read it'"]
  AXIS -->|"completion"| COMP["'the read is done,<br/>data is in your buffer'"]
  READY -->|"sockets, pipes:<br/>works great"| NET["network reactors<br/>(nginx, Redis)"]
  READY -->|"regular files:<br/>ALWAYS ready -><br/>no asynchrony"| FILEGAP["readiness is useless<br/>for disk I/O"]
  FILEGAP -->|"so files must use..."| COMP
  COMP --> TP["thread pool of<br/>blocking syscalls<br/>(libuv, POSIX AIO)"]
  COMP --> NAIO["native AIO / libaio<br/>(O_DIRECT only)"]
  COMP --> URING["io_uring<br/>(buffered + direct,<br/>files + sockets)"]

The decision tree of Linux async-I/O models. What it shows: readiness interfaces serve sockets beautifully but are useless for regular files (always “ready”), which forces all real asynchronous file I/O onto completion-based models — thread pools, native AIO, or io_uring. The insight to take: the socket/file asymmetry, not raw performance, is the deepest fault line in the landscape; io_uring is the first interface that erases it, handling both kinds of fd through one completion ring.

Model 1 — Blocking Syscalls on a Thread Pool

The baseline async strategy uses no special kernel interface at all: dedicate a pool of worker threads, and have each one make an ordinary blocking read()/write(). The thread blocks; the application’s main thread does not. This is conceptually trivial and works for everything, including regular files, which is exactly why it remains ubiquitous — Node.js’s libuv, for instance, “use[s] a separate thread pool to deal with file I/O” precisely because readiness polling fails for files (unixism.net).

The cost is structural. Each outstanding operation needs a blocked thread, so concurrency is bounded by thread count, and each thread carries a kernel stack (typically 8 KiB or more) plus scheduler overhead. Worse, every operation pays two context switches — submit thread hands to worker, worker hands back — even when the data was already in the page cache and the read could have completed in nanoseconds. The design document calls this out directly: “an application with an IO thread pool always has to bounce requests to an async context, resulting in at least two context switches. If the data requested was already in page cache, this causes a dramatic slowdown in performance” (kernel.dk/io_uring.pdf, §9.2). A userspace pool cannot know in advance whether a read will hit the cache, so it must pessimistically offload all of them. This is the inefficiency io_uring’s inline fast path was designed to erase.

Model 2 — POSIX AIO (glibc’s Userspace Thread Emulation)

POSIX standardized an asynchronous file-I/O API — aio_read(3), aio_write(3), aio_fsync(3), with aio_error(3)/aio_return(3) to collect results, aio_suspend(3) to wait, and lio_listio(3) to submit a batch (aio(7)). Each operation is described by a struct aiocb (asynchronous I/O control block) carrying the file descriptor aio_fildes, the offset aio_offset, the buffer aio_buf, the byte count aio_nbytes, and an aio_sigevent describing how to be notified — SIGEV_NONE (poll for completion), SIGEV_SIGNAL (deliver a signal), or SIGEV_THREAD (run a callback in a new thread).

The crucial fact is how Linux implements it: it does not use the kernel. “The current Linux POSIX AIO implementation is provided in userspace by glibc” (aio(7)) — glibc spawns and manages its own pool of threads that perform ordinary blocking syscalls, exactly Model 1 hidden behind a POSIX-flavored API. The man page is candid about the consequence: this approach “has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly” (aio(7)). So POSIX AIO offers a completion-style interface but inherits the thread-pool costs — and on top of them the complexity of signal-based or thread-callback completion delivery, which is notoriously awkward (signals interrupt syscalls, callbacks race). The design document is blunt: POSIX aio_read/aio_write exist “to satisfy that need, however the implementation of those is most often lackluster and performance is poor” (kernel.dk/io_uring.pdf). POSIX AIO is portable and standardized, but on Linux it is rarely the right choice for performance.

Model 3 — Native (Kernel) AIO: io_setup / io_submit / io_getevents

Linux also has a kernel asynchronous-I/O interface, confusingly also called “AIO” but completely separate from the POSIX one. It is a true completion interface implemented in the kernel, driven by five system calls: io_setup(2) creates an aio_context_t (“an asynchronous I/O context suitable for concurrently processing n operations,” io_setup(2)); io_submit(2) queues an array of struct iocb operations; io_getevents(2) reaps struct io_event completions; and io_cancel(2)/io_destroy(2) cancel and tear down.

A vital boundary, and the single most common confusion in this whole area: these are raw syscalls, not glibc-wrapped. The recommended way to call them is the third-party libaio library — “you probably want to use the io_getevents(3) wrapper function provided by libaio” (io_getevents(2)). Do not confuse this with Model 2: glibc wraps the POSIX aio_* family (its userspace thread emulation); libaio wraps the kernel io_submit family. They are different interfaces with the same nickname.

The iocb (I/O control block) names an operation via aio_lio_opcode, supporting IOCB_CMD_PREAD/PWRITE (positioned read/write), IOCB_CMD_PREADV/PWRITEV (vectored), IOCB_CMD_FSYNC/FDSYNC, IOCB_CMD_POLL, and IOCB_CMD_NOOP (io_submit(2)). Notice how narrow that set is compared to io_uring’s: reads, writes, fsyncs, a poll — no open, no stat, no rename. Native AIO can asynchronously move data in an already-open file, but it cannot asynchronously open one.

Why native AIO is limited

The design document enumerates the limitations that relegated native AIO “to a niche corner of applications” (kernel.dk/io_uring.pdf). They are worth quoting precisely because they are the direct motivation for io_uring:

  1. It only does async I/O for O_DIRECT. “The biggest limitation is undoubtedly that it only supports async IO for O_DIRECT (or un-buffered) accesses… For normal (buffered) IO, the interface behaves in a synchronous manner” (kernel.dk/io_uring.pdf). If you submit a buffered read through io_submit, it blocks inside the submit call until the read is done — defeating the entire point. Only O_DIRECT reads/writes (cache-bypassing, with their alignment and size constraints — see Direct IO and O_DIRECT) actually run asynchronously. This is why native AIO is, in practice, a database interface: databases use O_DIRECT and manage their own cache.
  2. Submission can still block even with O_DIRECT. “There are a number of ways that the IO submission can end up blocking — if meta data is required to perform IO, the submission will block waiting for that,” and if the device’s “fixed number of request slots” are all in use, “submission will block waiting for one to become available” (kernel.dk/io_uring.pdf). So even the supposedly-async path can stall in io_submit, forcing careful applications to still offload submission to a thread.
  3. At least two syscalls per I/O. “IO always requires at least two system calls (submit + wait-for-completion), which in these post spectre/meltdown days is a serious slowdown” (kernel.dk/io_uring.pdf). After the 2018 speculative-execution mitigations inflated every syscall’s cost (see Why System Calls Are Expensive), this fixed overhead became a real ceiling for high-IOPS devices.
  4. A clumsy ABI. Each submission copies 64+8 bytes and each completion 32 bytes — “104 bytes of memory copy, for IO that’s supposedly zero copy,” and the completion ring is “hard (impossible?) to use correctly from an application” (kernel.dk/io_uring.pdf).

Uncertain

The RWF_NOWAIT flag (added to preadv2/pwritev2 and usable in iocb flags) lets a caller ask native AIO to return -EAGAIN rather than block on metadata or congestion, which mitigates limitation #2 somewhat on modern kernels (io_submit(2) mentions RWF_NOWAIT returns -EAGAIN “if the I/O will block for operations such as file block allocations, dirty page flush, mutex locks, or a congested block device”). It does not lift the O_DIRECT-only limitation, only makes blocking submission detectable. The exact set of kernels and the interaction with buffered AIO are not fully verified here; treat the “still synchronous for buffered” claim as the dominant behavior and verify RWF_NOWAIT semantics against the target kernel if it matters. uncertain

Model 4 — Readiness: select, poll, and epoll

The readiness interfaces are the workhorses of network servers, and understanding why they do not help files is the key to the whole landscape. select(2) and poll(2) take a set of fds and block until at least one is ready, returning the ready set; they are O(n) in the watched-set size and rebuild that set on every call. epoll (Linux-specific) fixes the scaling: you register fds once with epoll_ctl, then epoll_wait returns only the ready ones, in O(ready) time — the difference that lets a single thread watch hundreds of thousands of sockets (epoll(7)). epoll offers level-triggered mode (report ready as long as the condition holds) and edge-triggered mode (report only on the transition to ready, requiring the application to drain the fd fully) (epoll(7)).

For sockets and pipes, this is ideal: “ready” means data has arrived (or buffer space exists), the application issues a non-blocking recv/send, and the syscall completes without blocking. nginx, Redis, and HAProxy are all epoll reactors.

For regular files, it is useless. select/poll/epoll report a disk file as ready the instant you ask, regardless of whether the data is cached, because a file read can always be satisfied — possibly after blocking on the disk, which is exactly what readiness is supposed to predict but cannot. As the Lord of the io_uring guide puts it, “while they work really well for sockets, they always return ‘ready’ for regular files” (unixism.net). So a readiness reactor that wants to read a file has no way to avoid blocking except to hand the read to a thread pool (Model 1) — which is precisely what libuv does. Readiness gives you asynchronous networking; it gives you nothing for disk. That gap is the reason completion interfaces for files had to exist.

This file-vs-socket split is also why io_uring’s internal execution (detailed in io_uring and the File Path) uses poll for pollable fds but offloads non-pollable regular files to worker threads — it has to bridge both worlds.

Model 5 — io_uring, the Completion Model Done Right

io_uring (Jens Axboe, merged kernel 5.1, May 2019 — Wikipedia) is the synthesis that resolves every limitation above. It is a completion interface — you submit whole operations, you reap completions — but unlike native AIO it:

  • Handles buffered and direct I/O. A buffered read whose data is in the page cache completes inline, with no thread hop, making io_uring “just as efficient for IO that is already in the page cache as the regular synchronous interfaces” (kernel.dk/io_uring.pdf, §9.2). A cache miss or a metadata fetch is offloaded to an internal kernel worker pool (io-wq), so the application never blocks — solving native AIO’s “buffered is synchronous” and “submission can block” problems at once.
  • Handles files and sockets uniformly, erasing the readiness/completion split: “io_uring presents a uniform interface whether dealing with sockets or with regular files” (unixism.net). Internally it polls pollable fds and offloads non-pollable ones, but the application sees one completion ring.
  • Covers the whole operation surface — not just read/write/fsync but open, close, stat, fallocate, rename, unlink — so an event loop can be fully asynchronous.
  • Amortizes the syscall cost with shared submission/completion rings, submitting many operations per io_uring_enter, or zero syscalls with SQPOLL — directly attacking native AIO’s “two syscalls per I/O” ceiling.

The shape of how io_uring executes file operations — the inline-non-blocking attempt, the -EAGAIN-then-poll-or-offload routing, the io-wq bounded/unbounded worker pool, fixed buffers and files, write-then-fsync linking — is the subject of io_uring and the File Path. The ring mechanics it rests on are The io_uring Submission and Completion Queues, and the batching argument and its three driving syscalls are io_uring as a Syscall Batching Mechanism. The one caveat that belongs in any landscape note: io_uring’s power has made it a security-sensitive surface, and several environments (ChromeOS, Android, Docker’s default seccomp profile, the kernel.io_uring_disabled sysctl) restrict or disable it — so “use io_uring” is sometimes not available, detailed in the batching note.

Choosing a Model

  • Sockets, high connection count: epoll (level- or edge-triggered) is the proven reactor; io_uring is a viable and increasingly used alternative when you also want batched submission or unified file+socket handling.
  • Files, low concurrency: plain blocking pread/pwrite — or a small thread pool — is simplest and fast enough. Do not reach for io_uring to read a config file.
  • Files, high concurrency, buffered: io_uring is the only interface that serves cache hits inline and offloads only misses. Native AIO is wrong here (buffered = synchronous); a thread pool pays the two-context-switch tax on every cached read.
  • Files, high concurrency, O_DIRECT (databases): native AIO/libaio is the legacy answer and still widely deployed — MySQL’s InnoDB uses native AIO by default on Linux (innodb_use_native_aio, which “requires the libaio library,” MySQL docs); io_uring with IORING_SETUP_IOPOLL and fixed buffers is the modern, faster successor where available.
  • Portability across UNIXes: POSIX AIO (aio_*) is standardized, but on Linux it is a glibc thread pool with poor scaling — acceptable for portability, not for peak performance.

Production Notes

The migration is visible in real systems. PostgreSQL historically relied on buffered I/O through the kernel page cache (not O_DIRECT + libaio), which is exactly why native AIO never fit it — buffered AIO is synchronous. PostgreSQL 18 (released September 2025) added a genuine asynchronous-I/O subsystem with an io_method server variable: “On Linux io_uring can be used for AIO, a worker based implementation is available on all platforms,” initially accelerating reads such as sequential scans, bitmap heap scans, and vacuums with “up to a 2-3x performance” gain (PostgreSQL 18 release announcement, release notes). The classic O_DIRECT + libaio high-IOPS engine is instead MySQL’s InnoDB, which enables native AIO by default on Linux (MySQL docs) — and engines like it are the natural candidates to move to io_uring + IORING_SETUP_IOPOLL. On the network side, the long reign of epoll continues — its scaling is excellent and its security surface is far smaller than io_uring’s — and many systems adopt io_uring incrementally for file I/O while keeping epoll for sockets, precisely because the readiness model still wins where it has always won.

The deepest lesson of the landscape is the one the mental model captures: readiness and completion are not interchangeable, the socket/file asymmetry is why both exist, and io_uring’s significance is that it is the first interface to serve both worlds through one efficient, completion-based ring. (Axboe’s design document reports io_uring reaching roughly “1.7M 4k IOPS with polling” against native AIO’s lower ceiling, but cautions the absolute figures are by now “a bit outdated” — block-layer improvements since 2019 have moved the numbers; the relative shape, io_uring outperforming native AIO and the inline page-cache-hit win, is what endures, kernel.dk/io_uring.pdf.)

See Also

Graph View

Backlinks