Direct IO and O_DIRECT

O_DIRECT is the open flag that tells the kernel “do not cache this file’s data — move it by Direct Memory Access (DMA) straight between my user buffer and the storage device.” A normal buffered read()/write() stages every byte through the page cache; an O_DIRECT I/O skips that staging entirely, so the bytes never occupy kernel page-cache memory and the CPU never does the page-cache memcpy. The trade is steep: the application loses all kernel caching, readahead, and write-behind, and in exchange must satisfy strict alignment rules (offset, length, and buffer address must be multiples of the device’s logical block size) or the I/O fails with EINVAL. Databases want this precisely because they implement a better cache than the kernel’s generic one and need predictable, double-buffer-free I/O. This note covers O_DIRECT semantics, the alignment contract, the modern iomap direct-I/O engine (iomap_dio_rw) versus the legacy fs/direct-io.c, the per-I/O RWF_* flags, and the pitfalls — pinned to Linux 6.12 LTS (fs/iomap/direct-io.c, v6.12) and open(2)/statx(2) as documented in the Linux man-pages.

Scope, and where the neighbouring notes take over. This note owns the user-facing mechanism and its consequences: the open flag and the gate that accepts or rejects it, the alignment contract and how to discover it, the two direct-I/O engines below the flag, what the flag does and does not guarantee, and when a real system should reach for it. It deliberately does not re-derive the cache internals. What the page cache is, how folios and dirty accounting and writeback thresholds work, and the detailed reconciliation the kernel performs around every direct I/O are owned by The Page Cache — which traces kiocb_write_and_wait and kiocb_invalidate_pages from the cache’s side, and covers posix_fadvise as the softer alternative. The struct address_space object and its address_space_operations vtable, including why ->direct_IO is not where modern direct I/O happens, belong to The Page Cache and address_space. Everything below the bio — request queues, merging, schedulers, completion interrupts — belongs to The Multi-Queue Block Layer blk-mq. This note links to all three rather than restating them.

Version pin. Kernel source claims below were read at Linux v6.12, a maintained long-term-support (LTS) series: releases.json on kernel.org lists 6.12.108 with moniker longterm and iseol: false as of 2026-09-04, at which point mainline is 7.3-rc1 (kernel.org releases). This matters more than usual here, because direct I/O has moved quickly since: RWF_ATOMIC arrived in 6.11, RWF_DONTCACHE in 6.14, STATX_DIO_READ_ALIGN in 6.14 — each is called out and dated where it appears. Man-page text is from the man-pages git tree at tag man-pages-6.19, which is newer than the pin and therefore describes behaviour that may postdate 6.12; that gap is flagged where it could mislead. The measured transcripts were produced on a Fedora 44 machine running 7.1.8 with btrfs on LUKS-encrypted NVMe — a newer kernel than the pin, stated explicitly because a measurement is evidence about the kernel that produced it.

Mental Model

The cleanest way to think about O_DIRECT is as a second, parallel data path that branches off the moment the read or write enters the filesystem’s ->read_iter / ->write_iter (the exact branch point differs between modern and legacy filesystems and is traced below). The buffered path goes “syscall → page cache → (on miss) block layer,” with the page cache as an obligatory waypoint. The direct path goes “syscall → filesystem maps file range to device range → build a bio that points directly at the user’s pages → block layer,” with no page cache in between. The filesystem’s only job on the direct path is the mapping — translating the file offset to one or more on-device extents — which is exactly what the iomap library was built to do.

Two consequences fall straight out of this picture. First, because there is no intermediate kernel buffer, the user’s memory is the DMA target, so it must be aligned the way the hardware demands. Second, because the kernel is no longer the single source of truth for the file’s bytes, O_DIRECT and buffered I/O to the same file are not automatically coherent — you can have stale cached pages shadowing freshly direct-written blocks unless the kernel explicitly reconciles them (which it does, carefully, on the boundaries).

flowchart TB
  subgraph BUF["BUFFERED: read(fd, buf, 4096)"]
    direction TB
    B1["user buffer<br/>(any address, any size)"]
    B2["<b>copy #2</b><br/>copy_to_iter(): page cache --> user"]
    B3["page-cache folio<br/><i>stays resident, LRU-managed,<br/>reused on the next read</i>"]
    B4["<b>copy #1</b><br/>DMA: device --> page-cache folio"]
    B5["bio over <b>kernel</b> pages"]
    B6["block layer --> device"]
    B1 --- B2 --- B3 --- B4 --- B5 --- B6
  end
  subgraph DIR["DIRECT: read(fd, buf, 4096) on an O_DIRECT fd"]
    direction TB
    D1["user buffer<br/><b>must be DMA-aligned</b>"]
    D2["<i>(no copy — this page IS the target)</i>"]
    D3["<i>(no page-cache folio at all;<br/>nothing is retained, nothing<br/>is available to the next read)</i>"]
    D4["<b>the only copy</b><br/>DMA: device --> user page"]
    D5["bio over the <b>user's</b> pages<br/>pinned by get_user_pages()"]
    D6["block layer --> device"]
    D1 --- D2 --- D3 --- D4 --- D5 --- D6
  end
  BUF ~~~ DIR

The buffered and direct paths side by side, for one 4 KiB read. What it shows: exactly which copy disappears. Buffered I/O performs two data movements — a DMA from the device into a kernel page-cache folio, then a CPU memcpy from that folio into the user’s buffer. Direct I/O performs one: the device DMAs straight into the user’s own pages, which the kernel has pinned for the duration. The insight to take: O_DIRECT removes one CPU copy and one page of kernel memory per page of I/O — and, as the same picture makes clear, it also removes the folio that would have served the next read of the same block for free. That is the entire trade, and it is why the flag is a win only for workloads that either have no reuse or manage their own reuse better than the kernel’s LRU can. The alignment requirement is not a bureaucratic detail either: it falls directly out of the geometry, because the device’s DMA engine is now addressing the user’s memory and the user’s memory has to satisfy the device’s rules.

Mechanical Walk-through

The gate at open(): FMODE_CAN_ODIRECT

O_DIRECT is not a request the kernel can always honour, and the first thing to understand is that the acceptance decision happens at open time, not at I/O time. In do_dentry_open() (fs/open.c, v6.12) the sequence is exactly:

if (f->f_mapping->a_ops && f->f_mapping->a_ops->direct_IO)
        f->f_mode |= FMODE_CAN_ODIRECT;          /* 1: legacy auto-grant */
...
if ((f->f_flags & O_DIRECT) && !(f->f_mode & FMODE_CAN_ODIRECT))
        return -EINVAL;                          /* 2: the gate */

Line 1 grants the capability automatically to any filesystem that still implements the old address_space_operations->direct_IO method. Line 2 is the gate that produces the EINVAL from open() that open(2) documents as “the filesystem does not support the O_DIRECT flag.” A modern filesystem that does not implement ->direct_IO must therefore set the bit itself in its ->open, and that is exactly what ext4 does: filp->f_mode |= FMODE_NOWAIT | FMODE_CAN_ODIRECT; in ext4_file_open() (fs/ext4/file.c, v6.12). So does tmpfs, in shmem_file_open() (mm/shmem.c, v6.12) — which is worth knowing because “tmpfs rejects O_DIRECT” is stale folklore: an existence check across tags shows FMODE_CAN_ODIRECT absent from mm/shmem.c at v6.5 and present at v6.6, so tmpfs has accepted O_DIRECT since Linux 6.6.

Where the branch happens — and where it does not

Here is the correction that most secondary material gets wrong, including older versions of this note. For a modern filesystem, an O_DIRECT read does not go through generic_file_read_iter, and it does not call a_ops->direct_IO. The kernel’s own iomap documentation says so outright: filesystems “should call iomap_dio_rw from ->read_iter and ->write_iter, and set FMODE_CAN_ODIRECT in the ->open function for the file. They should not set ->direct_IO, which is deprecated” (Documentation/filesystems/iomap/operations.rst, v6.12).

The code agrees. ext4_aops at v6.12 (fs/ext4/inode.c) has no .direct_IO member at all — nor do ext4_da_aops or ext4_journalled_aops. Grepping the whole file operations tables at v6.12 gives a clean split: fs/xfs/xfs_aops.c, fs/btrfs/inode.c and fs/f2fs/data.c contain zero occurrences of direct_IO, while fs/fat/inode.c, fs/jfs/inode.c, fs/ocfs2/aops.c and fs/nilfs2/inode.c still do. Instead ext4’s dispatch is in its own ->read_iter:

static ssize_t ext4_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
{
        ...
        if (iocb->ki_flags & IOCB_DIRECT)
                return ext4_dio_read_iter(iocb, to);   /* --> iomap_dio_rw() */
        ...
}

generic_file_read_iter’s IOCB_DIRECT branch — with its mapping->a_ops->direct_IO(iocb, iter) call — is still there in mm/filemap.c, v6.12, but it is now reached only by filesystems that have not been converted and still use generic_file_read_iter as their ->read_iter. That branch’s kiocb_write_and_wait() call is the coherency reconciliation: before reading directly from the device, the kernel writes back and waits on any dirty page-cache folios covering the same range, so the direct read does not return stale on-disk bytes that a buffered writer had not yet flushed. The iomap path performs the same reconciliation itself, inside __iomap_dio_rw. The Page Cache traces both calls in detail from the cache’s side; this note only needs the fact that they happen.

flowchart TD
  OPEN["open(path, O_RDWR | O_DIRECT)"] --> G1{"a_ops->direct_IO exists?"}
  G1 -->|yes| CAN["FMODE_CAN_ODIRECT set automatically<br/><i>(legacy filesystems)</i>"]
  G1 -->|no| G2{"does ->open set<br/>FMODE_CAN_ODIRECT itself?"}
  G2 -->|"yes — ext4, xfs, btrfs,<br/>f2fs, tmpfs (since 6.6), block devs"| CAN
  G2 -->|no| EINV["<b>open() returns -EINVAL</b><br/>'filesystem does not support O_DIRECT'"]
  CAN --> IO["read()/write()<br/>iocb->ki_flags |= IOCB_DIRECT"]
  IO --> D{"whose ->read_iter?"}
  D -->|"the filesystem's own<br/>(modern)"| MOD["ext4_file_read_iter etc.<br/>--> iomap_dio_rw()<br/><b>a_ops->direct_IO never consulted</b>"]
  D -->|"generic_file_read_iter<br/>(legacy)"| LEG["kiocb_write_and_wait()<br/>--> a_ops->direct_IO()<br/>--> __blockdev_direct_IO()"]
  MOD --> BIO["bio over pinned user pages"]
  LEG --> BIO
  BIO --> BLK["block layer<br/><i>(see: The Multi-Queue Block Layer blk-mq)</i>"]

How an O_DIRECT request actually reaches the device at v6.12, corrected. What it shows: two independent things people conflate — the capability gate at open time (FMODE_CAN_ODIRECT), and the dispatch at I/O time, which for every modern filesystem happens in the filesystem’s own ->read_iter/->write_iter rather than through the address_space_operations vtable. The insight to take: if you go looking for O_DIRECT by grepping for direct_IO in ext4, XFS or Btrfs, you will find nothing and conclude they do not support it. The vtable slot is legacy, kept alive for FAT, JFS, OCFS2 and NILFS2, and it is also the thing that grants FMODE_CAN_ODIRECT to those filesystems for free. The Page Cache and address_space lists “expecting ->direct_IO to be where O_DIRECT happens” as one of its named misunderstandings, and this is why.

The modern engine: iomap_dio_rw

For ext4, XFS, Btrfs and F2FS — verified at v6.12 by the absence of any direct_IO vtable entry in each — the filesystem’s own ->read_iter/->write_iter calls iomap_dio_rw directly (fs/iomap/direct-io.c, v6.12; the kernel iomap operations docs define direct I/O as “file I/O that is issued directly to storage, bypassing the pagecache” and state that “the iomap_dio_rw function implements O_DIRECT (direct I/O) reads and writes for files”). For example ext4’s read path is literally iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL, 0, NULL, 0) (fs/ext4/file.c, v6.12). iomap_dio_rw is a thin wrapper over __iomap_dio_rw + iomap_dio_complete. __iomap_dio_rw sets up a struct iomap_dio and an iomap_iter with flags = IOMAP_DIRECT, then:

1. For a read, it flags the dio IOMAP_DIO_INLINE_COMP (reads can always complete inline), marks the user buffer IOMAP_DIO_DIRTY if it is user-backed (so completion knows to dirty the destination pages), and again calls kiocb_write_and_wait to flush conflicting dirty cache.
2. For a write, it sets IOMAP_WRITE | IOMAP_DIO_WRITE, decides whether the I/O needs a post-write flush (IOMAP_DIO_NEED_SYNC for O_DSYNC/O_SYNC files, with a WRITE_THROUGH/FUA optimization for datasync-only writes), and then invalidates the page cache over the write range with kiocb_invalidate_pages — if that invalidation fails because the pages are pinned, it returns -ENOTBLK so the caller falls back to buffered I/O.
3. It calls inode_dio_begin(inode) (tracking outstanding direct I/Os so truncate can wait them out), then loops iomap_iter(&iomi, ops), where the filesystem maps successive file ranges to device extents, and for each mapped chunk iomap_dio_bio_iter builds and submits a bio.

The pivotal alignment check lives in iomap_dio_bio_iter:

if ((pos | length) & (bdev_logical_block_size(iomap->bdev) - 1) ||
    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
    return -EINVAL;

This is the alignment gate. pos | length ORs the file offset and the chunk length; ANDing with (logical_block_size - 1) is zero only if both are multiples of the device’s logical block size (typically 512 bytes, sometimes 4096). Separately, bdev_iter_is_aligned verifies the memory buffer addresses in the iov_iter meet the device’s DMA alignment. If either fails, the direct I/O returns -EINVAL — the canonical “you misaligned an O_DIRECT” error. Sub-block heads/tails of an unwritten or newly allocated extent are zero-filled via iomap_dio_zero before the data bio, so a direct write into a hole does not expose uninitialized device contents.

When all bios are submitted, iomap_dio_complete reverts the iterator to the bytes actually transferred, issues the deferred fsync/cache-flush if IOMAP_DIO_NEED_SYNC was set, advances iocb->ki_pos, and returns the byte count (or -EIOCBQUEUED for an async submission completed later via iocb->ki_complete).

The synchronous vs asynchronous split is itself a key part of the engine. __iomap_dio_rw decides wait_for_completion = is_sync_kiocb(iocb) || (dio_flags & IOMAP_DIO_FORCE_WAIT). A plain blocking pread/pwrite on an O_DIRECT fd is a sync kiocb, so the thread sleeps until the device bios finish and the byte count is returned inline. An io_uring/AIO submission is not sync: the call returns -EIOCBQUEUED, and when the final bio completes, the block layer runs iomap_dio_bio_end_io, which (depending on flags) either completes the dio inline in the completion context (IOMAP_DIO_INLINE_COMP, used for reads), lets the submitter finish it to avoid a context switch (IOMAP_DIO_CALLER_COMP, an optimization for writes when the issuer “groks” deferred completion via IOCB_DIO_CALLER_COMP), or punts to a workqueue (inode->i_sb->s_dio_done_wq, lazily created by sb_init_dio_done_wq) when completion needs sleepable context — for example a post-write metadata update for a size-extending write. Either way it ultimately calls iocb->ki_complete(iocb, ret) to notify the async caller. This is why O_DIRECT pairs so naturally with io_uring: the direct path was built to complete asynchronously without ever blocking the submitting thread.

sequenceDiagram
  autonumber
  participant App as Application thread
  participant FS as ext4_file_write_iter<br/>--> __iomap_dio_rw
  participant PC as Page cache
  participant BLK as Block layer / device
  participant WQ as s_dio_done_wq

  App->>FS: pwritev2() / io_uring SQE, IOCB_DIRECT
  FS->>FS: wait_for_completion =<br/>is_sync_kiocb(iocb) || IOMAP_DIO_FORCE_WAIT
  FS->>PC: kiocb_invalidate_pages(range)
  alt invalidation fails
    PC-->>FS: error
    FS-->>App: <b>-ENOTBLK</b> --> caller falls back to BUFFERED write
  else ok
    FS->>FS: inode_dio_begin() — truncate must now wait for us
    loop for each mapped extent (iomap_iter)
      FS->>BLK: submit bio over pinned user pages
    end
    alt synchronous kiocb (plain pwrite)
      BLK-->>FS: all bios complete
      FS->>FS: iomap_dio_complete(): issue deferred fsync if NEED_SYNC
      FS-->>App: byte count returned inline
    else asynchronous (io_uring / AIO)
      FS-->>App: <b>-EIOCBQUEUED</b> — thread returns immediately
      BLK->>FS: iomap_dio_bio_end_io() in completion context
      alt IOMAP_DIO_INLINE_COMP (reads)
        FS->>App: iocb->ki_complete() right here
      else IOMAP_DIO_CALLER_COMP (pure overwrites)
        FS->>App: let the submitter finish it — no context switch
      else needs sleepable context
        FS->>WQ: queue_work() — size extension, extent conversion, cache flush
        WQ->>App: iocb->ki_complete()
      end
    end
  end

The completion machinery of iomap_dio_rw at v6.12. What it shows: the single decision (is_sync_kiocb) that splits a blocking pwrite from an io_uring submission, and the three ways an asynchronous direct I/O can be completed. The insight to take: direct I/O was rebuilt around never blocking the submitting thread and never taking an unnecessary context switch — which is why the three completion modes exist at all. IOMAP_DIO_INLINE_COMP finishes reads in the interrupt-ish completion context because reads need nothing sleepable; IOMAP_DIO_CALLER_COMP hands pure overwrites back to the submitter, avoiding a workqueue hop entirely, but only when the issuer opted in via IOCB_DIO_CALLER_COMP; the workqueue punt is the fallback for anything that must sleep, such as converting an unwritten extent or extending the file. iomap_dio_bio_iter explicitly clears IOMAP_DIO_CALLER_COMP when the write “needs zeroing or extent conversion, extend[s] the file size, or issue[s] journal IO or cache flushes during completion processing.” Note also branch 5: a failed cache invalidation returns -ENOTBLK and the caller silently does a buffered write instead — an application can believe it is doing direct I/O and not be. The Page Cache covers that trap from the cache’s side.

The legacy engine: fs/direct-io.c

Older filesystems — FAT, JFS, ReiserFS, OCFS2, NILFS2, AFFS and others — still route direct_IO through __blockdev_direct_IO in fs/direct-io.c (v6.12, a ~1,300-line library function). It predates iomap and does the same job in a more ad-hoc way: it walks the request a block at a time using the filesystem’s get_block callback to map each file block to a device block, accumulating contiguous device blocks into bios. The kernel is deliberately migrating filesystems off this code onto iomap because iomap maps whole extents per call (far fewer indirect-block lookups, better large-I/O performance) and centralizes the alignment and completion logic. As of 6.12 the legacy path is not removed — it remains the direct-I/O engine for filesystems that have not been converted. Treat “uses iomap” as a property of the specific filesystem, not of O_DIRECT in general.

Filesystem	Engine at v6.12	->direct_IO in its a_ops?	Grants FMODE_CAN_ODIRECT how?
ext4	iomap_dio_rw from ext4_file_read_iter/_write_iter	no	explicitly in ext4_file_open()
XFS	iomap_dio_rw	no	explicitly in its ->open
Btrfs	iomap_dio_rw via btrfs_dio_read/_write	no	explicitly in its ->open
F2FS	iomap_dio_rw	no	explicitly in its ->open
tmpfs	shmem’s own path (no block device)	no	shmem_file_open(), since Linux 6.6
Block devices (/dev/sdX)	blkdev_direct_IO in block/fops.c	n/a — ->read_iter is blkdev_read_iter	block-device open path
FAT, JFS, OCFS2, NILFS2, AFFS	__blockdev_direct_IO (fs/direct-io.c)	yes	automatically, from the vtable entry

Which engine each filesystem uses at v6.12, verified by reading each file rather than by reputation. What it shows: the migration is essentially complete for the filesystems anyone runs a database on, and the legacy fs/direct-io.c survives only for older formats. The insight to take: the two columns on the right explain a confusing asymmetry — legacy filesystems get O_DIRECT support for free by virtue of having the vtable entry, while every converted filesystem had to add an explicit line to its ->open to keep the flag working. That is also why tmpfs gaining O_DIRECT in 6.6 was a one-line change with no block device anywhere in sight: on tmpfs “direct I/O” cannot bypass anything, since the file’s data is page cache, so what it really means there is “accept the flag and copy,” which is worth knowing before you benchmark on /dev/shm and conclude direct I/O is fast.

The Alignment Contract

O_DIRECT imposes three separate alignment requirements, and confusing them is the leading cause of EINVAL in code that “worked on my machine.” They are:

1. The memory buffer address. The device’s DMA engine addresses your pages, so the buffer must satisfy the device’s DMA alignment. Reported by statx’s stx_dio_mem_align. Checked by bdev_iter_is_aligned().
2. The file offset. Reported by stx_dio_offset_align.
3. The transfer length. Governed by the same value as the offset — stx_dio_offset_align covers both, which is why the field is not called stx_dio_offset_only_align.

The critical property, and the one that makes hard-coded constants a bug waiting to happen, is that the requirement comes from the device and the filesystem, not from O_DIRECT. open(2) is explicit: “In Linux alignment restrictions vary by filesystem and kernel version and might be absent entirely.” It then gives the history — Linux 2.4 required multiples of the filesystem block size (typically 4096); “In Linux 2.6.0, this was relaxed to the logical block size of the block device (typically 512 bytes)” — and tells you the shell command to read that floor: blockdev --getss.

flowchart TD
  A["pread(fd, buf, len, off) on an O_DIRECT fd"] --> Q1{"buf address aligned to<br/>stx_dio_mem_align?"}
  Q1 -->|no| F1
  Q1 -->|yes| Q2{"off aligned to<br/>stx_dio_offset_align?"}
  Q2 -->|no| F1
  Q2 -->|yes| Q3{"len a multiple of<br/>stx_dio_offset_align?"}
  Q3 -->|no| F1
  Q3 -->|yes| OK["bio built over the user's pages<br/>--> device DMA"]
  F1{"what does THIS filesystem<br/>do with a misaligned DIO?"}
  F1 -->|"ext4, XFS: iomap_dio_bio_iter<br/>alignment gate"| E1["<b>-EINVAL</b><br/>loud, obvious, easy to fix"]
  F1 -->|"btrfs: check_direct_read()<br/>returns non-zero"| E2["<b>silently falls through to<br/>filemap_read() — BUFFERED</b><br/>no error, no warning"]
  F1 -->|"tmpfs, NFS client"| E3["no alignment restriction at all"]

The alignment gate, and the three different things a filesystem may do when you fail it. What it shows: the three independent checks, and that failing them is not guaranteed to produce an error. The insight to take: open(2) says the handling of misaligned direct I/O “can either fail with EINVAL or fall back to buffered I/O,” and both behaviours ship in mainline today. The silent fallback is far more dangerous than the error: an application that misaligns its buffers on Btrfs gets correct results, no diagnostics, and none of the properties it opened O_DIRECT for — no bypass, full cache pollution, double buffering. If you rely on direct I/O for a real property, you must verify you are getting it, not assume the absence of EINVAL means success.

Measured: the alignment gate is not universal

Running an alignment probe against a Btrfs file (Linux 7.1.8, Fedora 44, btrfs on a LUKS-encrypted NVMe device) produces a result that contradicts everything the “alignment or EINVAL” summary would predict:

== STATX_DIOALIGN probe ==
  .../objects/pack/pack-3d30….pack   mask&DIOALIGN=0  mem_align=0  offset_align=0
 
== alignment gate (16,777,755-byte file) ==
  aligned buf, off=0,    len=4096 -> 4096 (ok)
  buf+1,       off=0,    len=4096 -> 4096 (ok)     <- misaligned BUFFER, no error
  aligned buf, off=1,    len=4096 -> 4096 (ok)     <- misaligned OFFSET, no error
  aligned buf, off=0,    len=4095 -> 4095 (ok)     <- misaligned LENGTH, no error
  aligned buf, off=0,    len=512  ->  512 (ok)

Two findings, both traceable to source. First, statx returned with the STATX_DIOALIGN bit clear in stx_mask — Btrfs does not implement it, exactly as statx(2) predicts when it says support “is supported by ext4, f2fs, and xfs since Linux 6.1” and lists nobody else. Note the correct way to read that result: the fields being zero is not the documented “direct I/O is not supported” signal here, because the kernel never filled them in at all. Always check stx_mask, not just the values — a program that only looks at stx_dio_offset_align == 0 will wrongly conclude Btrfs cannot do direct I/O, when in fact the O_DIRECT open succeeded and every read went through.

Second, every deliberately misaligned read succeeded. That is btrfs_direct_read() in fs/btrfs/direct-io.c, v6.12 doing exactly what it is written to do:

if (check_direct_read(inode_to_fs_info(inode), to, iocb->ki_pos))
        return 0;                      /* not an error — zero bytes done */

and check_direct_read → check_direct_IO testing offset & (fs_info->sectorsize - 1) and iov_iter_alignment(iter) & blocksize_mask. Returning 0 means “I transferred nothing,” and btrfs_file_read_iter then falls straight through to filemap_read(iocb, to, ret) — the ordinary buffered read path (fs/btrfs/file.c, v6.12). The application sees a perfectly ordinary successful pread. This is the silent-fallback branch of the diagram above, observed rather than theorised.

Uncertain

Verify: whether Btrfs’s silent fallback applies symmetrically to misaligned direct writes, and under exactly which conditions XFS or ext4 also fall back instead of returning EINVAL (the -ENOTBLK invalidation-failure path is a distinct, documented fallback; a misalignment fallback on those filesystems was not observed or ruled out here). Reason: only the read path was measured, on one filesystem, on one kernel (7.1.8, not the 6.12 pin), and btrfs_direct_write() has a more complex partial-write-then-buffered-remainder structure that was read but not exercised. To resolve: run the same probe with pwrite on btrfs, ext4 and XFS on the same device, and read btrfs_direct_write() and ext4_dio_write_iter() at the pinned tag. uncertain

Discovering the alignment requirement: STATX_DIOALIGN

Historically the only portable way to learn a file’s O_DIRECT alignment was to assume the worst (the filesystem block size) or use filesystem-specific ioctls like XFS’s XFS_IOC_DIOINFO. Since Linux 6.1, statx(2) with the STATX_DIOALIGN mask returns two fields (statx.2, man-pages): stx_dio_mem_align (required alignment of the user memory buffer) and stx_dio_offset_align (required alignment of the file offset and I/O length), each 0 if direct I/O is unsupported on that file. Per the man page, STATX_DIOALIGN works on block devices since 6.1 and on regular files for ext4, f2fs, and xfs since 6.1. This is the correct modern way to size aligned buffers — query it rather than hard-coding 512 or 4096.

Three refinements, all dated. stx_mask is the real answer: statx sets a bit in stx_mask for each field it actually filled, and a filesystem with no STATX_DIOALIGN implementation leaves the bit clear and the fields zero, which is indistinguishable from the documented “direct I/O is not supported on this file” if you only look at the values (measured above on Btrfs). stx_dio_read_offset_align is newer: STATX_DIO_READ_ALIGN lets a filesystem advertise a smaller offset/length alignment for reads than for writes, and per statx(2) it “is supported by xfs on regular files since Linux 6.14” — after this note’s 6.12 pin, so it will not exist on a 6.12 kernel. If zero, stx_dio_offset_align applies to reads too. STATX_WRITE_ATOMIC is a different question: stx_atomic_write_unit_min/_max/_segments_max describe torn-write protection, not alignment, and are “supported on block devices since Linux 6.11” and “by xfs and ext4 since Linux 6.13” — again both after the pin.

Query	Fields	Answers	First available
STATX_DIOALIGN	stx_dio_mem_align, stx_dio_offset_align	“How must I align a direct I/O?”	6.1 (block devs; ext4, f2fs, xfs)
STATX_DIO_READ_ALIGN	stx_dio_read_offset_align	“May reads be aligned more loosely?”	6.14 (xfs), post-pin
STATX_WRITE_ATOMIC	stx_atomic_write_unit_min/_max/_segments_max, _max_opt	“May I use RWF_ATOMIC, and at what sizes?”	6.11 (block devs), 6.13 (xfs, ext4), post-pin
XFS_IOC_DIOINFO ioctl	XFS-specific struct	the same as STATX_DIOALIGN, XFS only	ancient; open(2) says “STATX_DIOALIGN should be used instead when it is available”

The statx queries relevant to direct I/O, with the release each became usable. What it shows: three distinct questions that are easy to conflate, and the fact that two of the three answers postdate a 6.12 LTS kernel. The insight to take: portable code needs a fallback ladder, not a single query. Ask STATX_DIOALIGN; if stx_mask says it was not answered, fall back to the block device’s logical sector size (BLKSSZGET / blockdev --getss); if that is unavailable, assume the filesystem block size. Hard-coding 512 is the historical default that stopped being safe when 4 Kn drives shipped, and hard-coding 4096 wastes I/O on 512-byte-sector devices — which is precisely why the query exists.

What O_DIRECT Does Not Guarantee

This is the single most consequential misunderstanding about the flag, and it is worth stating in one sentence before elaborating: O_DIRECT is not a durability mechanism. A successful write() on an O_DIRECT file descriptor means the device accepted the data. It does not mean the data is on stable storage, because essentially every modern storage device has a volatile write cache that acknowledges writes long before they are persisted.

Jeff Moyer’s LWN durability primer puts it plainly: “I/O operations performed against files opened with O_DIRECT bypass the kernel’s page cache, writing directly to the storage. Recall that the storage may itself store the data in a write-back cache, so fsync() is still required for files opened with O_DIRECT in order to save the data to stable storage” (Moyer, LWN, 2011-09-07 — verified 2026-09-04 to be the article “Ensuring data reaches disk,” not an LWN block page). open(2) says the same from the other direction: “The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to O_DIRECT.”

flowchart TD
  W["write() returns success on an O_DIRECT fd"] --> Q1{"Did you also set<br/>O_DSYNC / O_SYNC,<br/>or pass RWF_DSYNC / RWF_SYNC?"}
  Q1 -->|no| N1["<b>Data is NOT durable.</b><br/>It reached the device's<br/>volatile write cache at best.<br/>You still need fsync/fdatasync."]
  Q1 -->|"O_DSYNC / RWF_DSYNC"| D1["iomap sets IOMAP_DIO_NEED_SYNC<br/>+ tries IOMAP_DIO_WRITE_THROUGH"]
  Q1 -->|"O_SYNC / RWF_SYNC"| D2["IOMAP_DIO_NEED_SYNC,<br/>no WRITE_THROUGH shortcut<br/>(metadata must go too)"]
  D1 --> Q2{"Can every bio in this<br/>request use REQ_FUA?"}
  Q2 -->|yes| FUA["bio carries REQ_FUA:<br/>device must persist THIS write<br/>before acknowledging.<br/><b>No separate cache flush.</b>"]
  Q2 -->|"no — needed zeroing,<br/>extent conversion, or<br/>a size-extending write"| FLUSH["WRITE_THROUGH cleared;<br/>iomap_dio_complete() issues a<br/>full sync at completion"]
  D2 --> FLUSH
  FUA --> DUR["durable"]
  FLUSH --> DUR
  N1 --> Q3{"Did you fsync()<br/>afterwards?"}
  Q3 -->|yes| DUR
  Q3 -->|no| LOST["<b>Data can be lost on power failure.</b><br/>The write 'succeeded'."]

What it takes to make a direct write durable, traced through the flag handling in fs/iomap/direct-io.c, v6.12. What it shows: O_DIRECT alone lands in the leftmost branch, which ends in data loss; durability comes from O_DSYNC/O_SYNC/RWF_DSYNC or an explicit fsync, never from the direct flag. The insight to take: the WRITE_THROUGH/REQ_FUA optimisation on the O_DSYNC path is worth understanding, because it is the reason O_DIRECT | O_DSYNC is meaningfully cheaper than O_DIRECT followed by fdatasync(). Force Unit Access (FUA) is a per-request flag that tells the device “persist this one write before you acknowledge it,” which is far cheaper than a full cache flush of everything the device is holding. __iomap_dio_rw optimistically sets IOMAP_DIO_WRITE_THROUGH for datasync-only writes and iomap_dio_bio_opflags() clears it the moment any bio in the request cannot use FUA — the source comment notes that “any non-FUA write that occurs will clear this flag, hence we know before completion whether a cache flush is necessary.” A database that opens its log O_DIRECT | O_DSYNC is buying that FUA path deliberately. See fsync fdatasync and Durability for the general durability contract, including the error-reporting semantics that fsync has and a direct write does not.

Two corollaries worth stating because they are routinely assumed away:

• O_DIRECT does not make metadata durable. A size-extending direct write has to update the inode, and that update is journaled/written like any other metadata. O_DSYNC covers “data and the metadata needed to retrieve it”; O_SYNC covers all metadata; O_DIRECT covers neither.
• O_DIRECT does not report deferred write errors the way fsync does. A direct write is synchronous to the device, so a device error surfaces on the call — but that only covers this write. The errseq_t-based error reporting that lets fsync tell you a previous writeback failed does not have an analogue here, and the standard “check the return of close(), fsync() and the write itself” discipline still applies.

Configuration and Code

Before the code, the geometry — because “aligned” is three separate facts about three separate quantities, and seeing them on one axis is what makes the EINVAL obvious rather than mysterious. Assume a device with a 512-byte logical block size, so stx_dio_offset_align == 512 and stx_dio_mem_align == 512:

FILE OFFSET AXIS (one cell = 512 bytes, the device logical block)
        0     512    1024   1536   2048   2560   3072
        +------+------+------+------+------+------+
   (a)  |##############|                             off=0    len=1024   OK
        +------+------+
   (b)      |##############|                         off=256  len=1024   EINVAL: offset
            ^ starts mid-block
   (c)  |###########|                                off=0    len=768    EINVAL: length
                    ^ ends mid-block
   (d)  |##############|                             off=0    len=1024   EINVAL: buffer,
        but buf = malloc(...)+1  --> not 512-aligned in MEMORY, not in the file

MEMORY BUFFER AXIS
        0x...000        0x...200        0x...400
        +---------------+---------------+
   ok   ^ posix_memalign(&p, 512, len)
   bad         ^ malloc() may return any 8- or 16-byte alignment

The three alignment requirements on one picture, drawn as an ASCII grid. What it shows: cases (b), (c) and (d) each violate exactly one of the three rules — offset, length, buffer address — and each fails independently of the other two. The insight to take: the file-offset and length rules are about the file, and the buffer rule is about memory; they are checked by different code ((pos | length) & (bdev_logical_block_size(...) - 1) versus bdev_iter_is_aligned(...)) and reported by different statx fields, so a program can satisfy two and fail the third. Note the fallback used here: mermaid’s packet-beta draws the contiguous fields of a single structure and cannot overlay four alternative requests on a shared offset grid, so an RFC-style ASCII figure is the right medium for this one.

A correct O_DIRECT reader, with posix_memalign for the buffer and statx to size the alignment:

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <linux/stat.h>   /* struct statx, STATX_DIOALIGN */
 
int fd = open("/data/table.ibd", O_RDONLY | O_DIRECT);   /* 1 */
struct statx stx;
statx(fd, "", AT_EMPTY_PATH, STATX_DIOALIGN, &stx);       /* 2 */
size_t align = stx.stx_dio_offset_align;                  /* 3 */
size_t memalign = stx.stx_dio_mem_align;
size_t len = align ? align : 4096;                        /* 4 */
 
void *buf;
posix_memalign(&buf, memalign ? memalign : 4096, len);    /* 5 */
 
/* offset MUST be a multiple of align; len MUST be a multiple of align;
   buf MUST be aligned to memalign — else read() fails -EINVAL */
ssize_t n = pread(fd, buf, len, /*offset=*/ 0);           /* 6 */

• Line 1 — O_DIRECT is OR’d into the open flags; it can be combined with O_RDONLY/O_WRONLY/O_RDWR. On a filesystem with no O_DIRECT support, open itself fails -EINVAL (see the open(2) ERRORS).
• Lines 2–4 — query the real alignment instead of guessing; stx_dio_offset_align governs both the offset and the length, stx_dio_mem_align governs the buffer address. A zero means direct I/O is unsupported for this file.
• Line 5 — posix_memalign (or aligned_alloc, or a hugepage) gives a DMA-aligned buffer; a plain malloc buffer will usually fail the bdev_iter_is_aligned check.
• Line 6 — every pread must keep offset, len, and buf aligned, on every call, not just the first.

Per-I/O control without a separate open is available through preadv2/pwritev2’s RWF_* flags (readv.2, man-pages). These are not O_DIRECT substitutes but adjust durability and blocking per call:

• RWF_DSYNC (since 4.7) / RWF_SYNC (since 4.7) — per-write equivalents of O_DSYNC / O_SYNC; force the data (and, for SYNC, metadata) durable before the call returns.
• RWF_HIPRI (since 4.6) — high-priority polled I/O; “usable only on a file descriptor opened using the O_DIRECT flag,” per the man page — the low-latency polling path is direct-I/O only.
• RWF_NOWAIT (since 4.14) — return -EAGAIN rather than block; meaningful for preadv2.
• RWF_APPEND (since 4.16) — per-write O_APPEND.
• RWF_NOAPPEND (since Linux 6.9) — per-write suppression of O_APPEND.
• RWF_ATOMIC (since Linux 6.11) — torn-write protection: “for a power or any other hardware failure, all or none of the data from the write will be stored, but never a mix of old and new data” (readv(2)). This is the flag that could let a database drop its double-write buffer.

At v6.12 the header reads RWF_SUPPORTED = RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT | RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC (include/uapi/linux/fs.h, v6.12). Dating each bit by existence-checking the same header across tags gives a precise timeline, which matters because this family has changed in every recent release:

Flag	Bit	Added	Verified how	Requires O_DIRECT?
RWF_HIPRI	0x01	4.6	man-pages	yes — “usable only on a file descriptor opened using the O_DIRECT flag”
RWF_DSYNC	0x02	4.7	man-pages	no
RWF_SYNC	0x04	4.7	man-pages	no
RWF_NOWAIT	0x08	4.14	man-pages	no
RWF_APPEND	0x10	4.16	man-pages	no
RWF_NOAPPEND	0x20	6.9	man-pages	no
RWF_ATOMIC	0x40	6.11	absent in fs.h at v6.10, present at v6.11	yes — “torn-write protection only works with O_DIRECT… buffered writes are not supported”
RWF_DONTCACHE	0x80	6.14	absent at v6.12, present at v6.14	no — it is the buffered alternative
RWF_NOSIGNAL	0x100	6.18	absent at v6.17, present at v6.18	no

The preadv2/pwritev2 flag family, dated by reading include/uapi/linux/fs.h at successive tags rather than by trusting a changelog. What it shows: which per-I/O flags exist, when each appeared, and which two are direct-I/O-only. The insight to take: existence-checking a uapi header across tags is a precise way to date an ABI addition — RWF_ATOMIC is genuinely absent at v6.10 and present at v6.11, matching the man page’s “since Linux 6.11” exactly, and RWF_DONTCACHE is genuinely absent from the v6.12 RWF_SUPPORTED set, so a 6.12 LTS kernel will reject it. Note the two rows that are O_DIRECT-only for opposite reasons: RWF_HIPRI because polled completion only makes sense when there is no page cache to hide device latency, and RWF_ATOMIC because torn-write protection is a property of a single device request, which buffered writeback does not give you.

RWF_ATOMIC’s constraints are strict and are worth stating so nobody plans around a looser reading of them: the total write length must be a power of two, must lie between stx_atomic_write_unit_min and stx_atomic_write_unit_max as reported by statx’s STATX_WRITE_ATOMIC, and must be at a naturally aligned file offset with respect to that length — the man page’s own example is that “a write of length 32KiB at a file offset of 32KiB is permitted, however a write of length 32KiB at a file offset of 48KiB is not.” The vector count is capped by stx_atomic_write_segments_max. And it still is not durability: “to guarantee consistency from the write between a file’s in-core state with the storage device, O_SYNC or O_DSYNC must be specified.”

Uncertain

Verify: which filesystems actually honour RWF_ATOMIC on a v6.12 kernel. The bit and the RWF_SUPPORTED entry are in the v6.12 uapi header (read directly), and statx(2) at man-pages 6.19 states STATX_WRITE_ATOMIC is “supported on block devices since Linux 6.11” and on regular files “by xfs and ext4 since Linux 6.13” — which implies that on 6.12 the flag is usable on block devices only, with no regular-file support on any filesystem. That inference was not confirmed against v6.12 filesystem source. Reason: derived from a man page newer than the pin plus a uapi header, not from reading the v6.12 iomap/XFS atomic-write implementation. To resolve: check for IOCB_ATOMIC handling in fs/xfs/xfs_file.c and fs/iomap/direct-io.c at v6.12, and for STATX_WRITE_ATOMIC in each filesystem’s ->getattr. uncertain

(Resolved from the previous revision of this note: RWF_HIPRI is still documented as O_DIRECT-only at man-pages 6.19, checked 2026-09-04, so that half of the older callout no longer applies.)

Measured: What Bypassing the Cache Actually Costs

Adjectives are the usual currency here — “direct I/O is faster for databases,” “the page cache adds overhead.” Numbers are more useful. The following comes from 2,000 random 4 KiB preads against a 16 MiB file on btrfs over a LUKS-encrypted NVMe device (Fedora 44, Linux 7.1.8, 32 logical CPUs). The buffered passes bracket the cache: the first is preceded by posix_fadvise(POSIX_FADV_DONTNEED) over the whole file, the second immediately repeats the identical offset sequence.

Pass	Per-read latency	What it measures
Buffered, cache evicted first	43.4 µs	cold reads through the page cache — device latency plus one copy
Buffered, second pass (warm)	0.6 µs	the cache doing its job: no device I/O at all
O_DIRECT, first pass	52.4 µs	device latency, no copy, no cache
O_DIRECT, second pass	52.2 µs	the same again — nothing was retained

Direct versus buffered 4 KiB random reads on one machine. What it shows: three things at once. (1) On a workload with perfect reuse, buffered is 87× faster than direct — 0.6 µs versus 52.2 µs — because the second pass never touches the device. (2) O_DIRECT’s second pass is statistically identical to its first: 52.4 → 52.2 µs, which is the “no caching, ever” property made visible. (3) On the cold comparison, direct I/O was slower than buffered (52.4 vs 43.4 µs), not faster — because the buffered path gets readahead and this is a small file where readahead pays off, while the direct path issues one device request per 4 KiB read with nothing amortised. The insight to take: every number here argues against O_DIRECT for a general workload. The flag only wins when you can point at the specific property you are buying — a private cache that beats the kernel’s LRU, cache pollution you must avoid, or tail-latency predictability you cannot get when writeback and reclaim are scheduling against you. If you cannot name which one, you are about to make your system slower and prove it with a benchmark that has no reuse in it.

Two caveats on reading these numbers. The device is behind dm-crypt, so absolute latencies include a decryption pass and are higher than bare NVMe; the ratios are what the comparison is about. And the direct-I/O numbers are for synchronous, one-at-a-time reads — which is the configuration Linus complained about in 2002 and remains O_DIRECT’s worst case. The reason production systems get value from the flag is that they never use it this way: they submit dozens of direct reads at once through io_uring, so device latency is overlapped rather than serialised, and the 52 µs becomes throughput rather than latency. That is the whole content of the next section’s advice to pair O_DIRECT with io_uring.

Coherency With the Page Cache, and Linus’s Objection

O_DIRECT does not mean the page cache is uninvolved — it means no data is cached, while the cache is still flushed and invalidated around every operation. That reconciliation is traced in full, from the cache’s side, in The Page Cache; the short version is that a direct read calls kiocb_write_and_wait() to push and wait on overlapping dirty folios, and a direct write calls kiocb_invalidate_pages() to write back and invalidate overlapping clean folios both before and after the transfer. What matters at the user-facing level is the contract this creates and the one it does not: open(2) says “applications should avoid mixing O_DIRECT and normal I/O to the same file, and especially to overlapping byte regions in the same file. Even when the filesystem correctly handles the coherency issues in this situation, overall I/O throughput is likely to be slower than using either mode alone. Likewise, applications should avoid mixing mmap(2) of files with direct I/O to the same files.”

flowchart LR
  subgraph ACC["Three ways to reach one file's bytes"]
    R["read()/write()<br/>buffered"]
    M["mmap()<br/>MAP_SHARED"]
    D["read()/write()<br/>O_DIRECT"]
  end
  R <-->|"<b>fully coherent</b><br/>same page-cache folios"| M
  R <-->|"reconciled at each I/O:<br/>flush before reads,<br/>invalidate before+after writes"| D
  M <-->|"<b>sharpest edge</b><br/>a mapped folio may be pinned,<br/>invalidation fails, no error"| D
  D --> W["kernel's own comment on the<br/>post-write invalidation:<br/>'a pretty crazy thing to do,<br/>so we don't support it 100%.<br/>If this invalidation fails, tough,<br/>the write still worked...'"]

Which pairs of access modes are coherent with each other. What it shows: the two safe relationships and the one that the kernel explicitly declines to guarantee. The insight to take: buffered and mmap are the same cache and are trivially coherent. Buffered and direct are reconciled — not the same thing as coherent, because the reconciliation is per-I/O and can fail. mmap plus direct is the combination to design out entirely, and the quoted comment names exactly that case: pages “faulted in by get_user_pages() if the source of the write was an mmap’ed region of the file we’re writing.” The comment appears verbatim in both engines at v6.12 — in generic_file_direct_write (mm/filemap.c) and in iomap_dio_complete (fs/iomap/direct-io.c) — so it is not a legacy-path caveat. One detail worth knowing when debugging: the failure is not entirely silent. dio_warn_stale_pagecache() emits a rate-limited kernel log line (once per 24 hours per rate-limit state) and calls errseq_set(&filp->f_mapping->wb_err, -EIO), so a later fsync() on that file will report EIO. The Page Cache traces both invalidation attempts in the source; fsync fdatasync and Durability covers the errseq_t mechanism that carries the error to you.

The interface has been controversial for its entire existence, and the objection is worth reading in the original because it is more substantive than its reputation as a rant. In May 2002 on linux-kernel, Linus Torvalds wrote (message-ID <Pine.LNX.4.44.0205111047280.2355-100000@home.transmeta.com>, 2002-05-11, archived at yarchive):

The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances [*]. […] It’s simply not very pretty, and it doesn’t perform very well either because of the bad interfaces (where synchronicity of read/write is part of it, but the inherent page-table-walking is another issue).

The substantive part follows: he proposes splitting the two things O_DIRECT conflates — issuing the I/O and getting the pages into the address space — into an asynchronous readahead plus a MAP_UNCACHED mapping whose faults steal pages out of the page cache, and the mirror image for writes. When Alan Cox replied that with asynchronous I/O it is “extremely nice,” Linus’s answer was the sharpest line in the thread: “the point is that AIO is needed just to cover up the fundamental idiocy in the interface. If the interface had been properly designed, it would have been useful without AIO.”

Twenty-four years later that critique is roughly what happened, from the other end: io_uring gave O_DIRECT the asynchrony it always needed, and RWF_DONTCACHE (6.14) gives applications the “buffered but do not retain” semantics that Linus’s MAP_UNCACHED sketch was reaching for. O_DIRECT itself never got prettier — it got a good async front end and a viable alternative.

Failure Modes and Pitfalls

flowchart TB
  S{"Symptom"}
  S -->|"pread/pwrite returns EINVAL"| M1["Buffer, offset or length not a<br/>multiple of the device alignment.<br/>Query STATX_DIOALIGN; stop<br/>hard-coding 512."]
  S -->|"open() returns EINVAL"| M2["Filesystem never set<br/>FMODE_CAN_ODIRECT.<br/>Not a bug — no O_DIRECT here."]
  S -->|"everything works but<br/>the cache still fills up"| M3["Silent buffered fallback:<br/>btrfs misalignment, or<br/>-ENOTBLK from a failed<br/>page-cache invalidation."]
  S -->|"slower than buffered"| M4["The workload has reuse,<br/>or the I/O is synchronous and<br/>one-at-a-time. Direct I/O gets<br/>no readahead, no write-behind."]
  S -->|"data lost after<br/>power failure"| M5["O_DIRECT is not durability.<br/>Add O_DSYNC or fsync()."]
  S -->|"corruption in parent<br/>AND child after fork()"| M6["COW moved a private buffer page<br/>out from under an in-flight DMA.<br/>Finish the I/O, or use a<br/>MAP_SHARED buffer."]
  S -->|"stale data when mixing<br/>with read()/write()/mmap"| M7["Documented as unsupported.<br/>Pick one access mode per file."]
  S -->|"works on ext4,<br/>breaks on NFS"| M8["NFS client places no alignment<br/>restriction and cannot pass the<br/>flag to the server."]

A symptom-to-cause map for direct-I/O incidents. What it shows: the eight recurring presentations and the mechanism behind each, developed in prose below. The insight to take: only two of these (M1, M2) are the loud, obvious failures people expect from O_DIRECT. The other six are silent — wrong performance, wrong durability, wrong data — which is the real reason the man page closes by recommending “that applications treat use of O_DIRECT as a performance option which is disabled by default.”

• EINVAL on misalignment. The single most common O_DIRECT bug: offset, length, or buffer not a multiple of the logical block size. The (pos | length) & (logical_block_size - 1) gate in iomap_dio_bio_iter rejects it. Per open(2), misaligned O_DIRECT I/O “can either fail with EINVAL or fall back to buffered I/O,” and the requirement was relaxed from the filesystem block size (Linux 2.4) to the device logical block size (Linux 2.6.0).
• No caching, ever. A direct read of the same block twice issues two device I/Os; a direct write is not absorbed by write-behind. If the access pattern actually has reuse, O_DIRECT makes it slower. O_DIRECT is a win only when the application caches better than the kernel, or when cache pollution itself is the problem (a one-shot backup that would evict the working set).
• Coherency with buffered I/O is fragile. Mixing O_DIRECT and buffered access to the same file is explicitly discouraged in the kernel docs; the kernel reconciles on the I/O boundaries (kiocb_write_and_wait before direct reads, kiocb_invalidate_pages before direct writes), but the man page warns the interaction is subtle and applications should not rely on it. mmap of an O_DIRECT file is the sharpest edge.
• fork(2) data corruption. open(2) warns that O_DIRECT I/O must never run concurrently with fork(2) when the buffer is a private (MAP_PRIVATE/heap) mapping: copy-on-write can move the page out from under an in-flight DMA, corrupting data in both parent and child. Complete all direct I/O before fork, or use a MAP_SHARED/shmat buffer.
• O_DIRECT does not imply O_SYNC. A successful direct write means the device accepted the data, not that it is durable: as Jeff Moyer’s LWN durability primer notes, O_DIRECT writes go “directly to the storage” but “the storage may itself store the data in a write-back cache” (Moyer, LWN 2011). Durability still requires fsync/fdatasync or O_DSYNC/RWF_DSYNC. See fsync fdatasync and Durability.
• Filesystem may silently fall back, or accept the flag and do nothing special with it. Two distinct cases, often conflated. (a) A filesystem without direct-I/O support fails the open with -EINVAL — loud and easy to handle. (b) A filesystem that accepts the flag may still route the I/O through the buffered path: Btrfs on a misaligned request (measured above), any iomap filesystem when kiocb_invalidate_pages() fails and -ENOTBLK sends the caller back to buffered, and tmpfs, which has accepted O_DIRECT since Linux 6.6 but has no block device to bypass — its file data is page cache. Portable code must handle open failing -EINVAL, must check stx_mask rather than assuming a zero stx_dio_offset_align means “unsupported,” and — if the direct property actually matters — must verify it is being obtained rather than inferring it from the absence of an error.
• Benchmarking on the wrong filesystem. A direct consequence of the previous point: /dev/shm and /tmp are usually tmpfs, where O_DIRECT is a no-op wrapper around a copy. A measurement there (0.3–0.4 µs per 4 KiB read on the test machine, versus 52 µs on real storage) tells you nothing about direct I/O and everything about memcpy.
• RWF_HIPRI on a non-O_DIRECT fd. Polled completion is documented as usable “only on a file descriptor opened using the O_DIRECT flag,” so combining RWF_HIPRI with buffered I/O will not give you the low-latency polling path. This is one of the few places where the two features are genuinely coupled.

Alternatives and When to Choose Them

• Buffered I/O (the default). Choose it for almost everything: the page cache, readahead (Readahead and Read Path), and write-behind are large wins for typical reuse-heavy workloads. Reach for O_DIRECT only with a concrete reason.
• posix_fadvise(POSIX_FADV_DONTNEED) — keep buffered I/O but drop cached pages after use; a softer alternative to O_DIRECT when the only goal is to avoid cache pollution, without taking on alignment constraints.
• O_DIRECT + io_uring — the high-throughput database/storage pattern: aligned buffers, direct I/O, batched and asynchronous via the ring, optionally RWF_HIPRI polling on NVMe. See Asynchronous IO Models in Linux and io_uring and the File Path.
• mmap — a different bypass: it maps file pages into the address space, still through the page cache, with faults driving I/O. Coherent with buffered read/write, not with O_DIRECT. See Shared Memory via mmap for what mapped I/O gives you instead.
• RWF_DONTCACHE (Linux 6.14, after this note’s pin) — the newer middle path, and probably the right answer for most applications that reach for O_DIRECT today. Per readv(2), reads or writes with this flag “will prune instantiated page cache content when the operation completes,” except that “if ranges of the read or written I/O were already in cache before this read or write, then those ranges will not be pruned” — so it evicts what it brought in and leaves other people’s hot data alone. Writes additionally kick off writeback for the dirtied range, “similar to calling sync_file_range(2) with SYNC_FILE_RANGE_WRITE.” It is explicitly “a hint, or best effort, where no hard guarantees are given,” and returns EOPNOTSUPP on a filesystem or device that does not implement it. It does not exist on a 6.12 kernel — verified by the absence of the bit from RWF_SUPPORTED in include/uapi/linux/fs.h at v6.12 and its presence at v6.14.

	Alignment rules	Readahead	Write-behind	Cache pollution	Available
Buffered (default)	none	yes	yes	full	always
posix_fadvise(DONTNEED) after	none	yes	yes	evicted afterwards, imperfectly	always
RWF_DONTCACHE	none	yes	writeback kicked	evicts only what it added	6.14+
O_DIRECT	strict, device-dependent	none	none	none — never enters	2.4.10+

The four points on the spectrum between “use the cache” and “do not use the cache.” What it shows: O_DIRECT is the only one that costs you alignment discipline, and the only one that removes readahead and write-behind along with the caching. The insight to take: most applications that reach for O_DIRECT want just one of its properties — usually “do not evict my working set while I stream this 200 GB backup.” For that goal RWF_DONTCACHE is strictly better on a kernel new enough to have it: same goal, no alignment contract, keeps readahead, and does not silently degrade to something else. O_DIRECT remains the right answer only when you need the page cache structurally absent — because you are managing your own buffer pool and the double-buffering itself is the cost, or because you need the tail-latency predictability of a device request that no reclaim or writeback pass can get in front of. Note that posix_fadvise(POSIX_FADV_DONTNEED), the historical version of this idea, has three documented ways of quietly failing to free anything; The Page Cache traces them from mm/fadvise.c.

Production Notes

The reasoning behind using O_DIRECT for database data files is uniform wherever it is used: the engine already keeps hot pages in a tuned buffer pool, so the kernel page cache would only double-buffer the same data, wasting RAM and adding a memcpy, while readahead and write-behind interfere with the engine’s own scheduling and durability guarantees. But the two most-cited examples sit on opposite sides of the decision, and conflating them is a common error.

MySQL/InnoDB has direct I/O on by default. The documented default of innodb_flush_method on Unix-like systems is “O_DIRECT if supported, otherwise fsync” (MySQL 8.4 InnoDB parameters). The precise semantics of that setting are a good illustration of everything in the durability section above: with O_DIRECT, “InnoDB uses O_DIRECT … to open the data files, and uses fsync() to flush both the data and log files” — the direct flag and the durability call are separate, and InnoDB uses both. There is a second setting, O_DIRECT_NO_FSYNC, which “uses O_DIRECT during flushing I/O, but skips the fsync() system call after each write operation,” and the manual attaches an explicit warning to it: “Data loss is possible if redo log files and data files reside on different storage devices, and an unexpected exit occurs before data file writes are flushed from a device cache that is not battery-backed.” That is the O_DIRECT-is-not-durability trap, documented by a vendor who has watched customers fall into it.

PostgreSQL does not use direct I/O and does not recommend it. This is worth stating flatly because the opposite is widely repeated. As of PostgreSQL 18, direct I/O is exposed only as debug_io_direct, whose documentation reads: “Ask the kernel to minimize caching effects for relation data and WAL files using O_DIRECT … Currently this feature reduces performance, and is intended for developer testing only” (PostgreSQL 18 developer options — identical wording in the PostgreSQL 17 docs). The setting defaults to the empty string, is not settable at runtime, and lives among the debug parameters. What PostgreSQL 18 did ship is the asynchronous-I/O infrastructure that would make direct I/O viable — io_method with values worker (the default), io_uring and sync (PostgreSQL 18 resource configuration) — which is precisely the ordering Linus argued for in 2002: the asynchrony has to come first, because synchronous direct I/O with no readahead is a straightforward performance regression. Anyone repeating “PostgreSQL uses O_DIRECT” is describing a future, not the shipping default.

Kernel maintainers have long been skeptical of O_DIRECT as an interface, and the community’s preferred answer has been to make the buffered path good enough — better posix_fadvise, and the uncached buffered mode RWF_DONTCACHE (added after 6.12, in 6.14 — verified absent from the v6.12 RWF_SUPPORTED set) — so fewer applications need it; but for self-caching databases it remains the pragmatic choice. The migration of ext4/XFS/Btrfs onto iomap direct I/O (from the legacy fs/direct-io.c) improved large sequential direct-I/O throughput by mapping whole extents per iteration instead of block-by-block. When debugging O_DIRECT performance, STATX_DIOALIGN plus blockdev --getss (logical sector size) tell you the alignment floor, and the trace_iomap_dio_rw_begin / trace_iomap_dio_invalidate_fail / iomap_dio_complete tracepoints — all present in fs/iomap/direct-io.c, v6.12 — let you watch each direct I/O’s flags and completion path, including the invalidation failures that cause the silent buffered fallback.

A practical checklist, for the case where you have decided direct I/O is genuinely warranted:

1. Name the property you are buying — private cache, no pollution, or predictable tail latency. If you cannot, do not use the flag.
2. Query the alignment; never hard-code it. statx with STATX_DIOALIGN, checking stx_mask; fall back to BLKSSZGET.
3. Allocate with posix_memalign/aligned_alloc, not malloc, and keep offset and length aligned on every call, not just the first.
4. Pair it with io_uring (or at minimum AIO). Synchronous, one-at-a-time direct I/O is its documented worst case and was measured above at 52 µs per 4 KiB read.
5. Decide durability separately — O_DSYNC for the FUA fast path, or explicit fdatasync at your commit points.
6. Do not mix direct, buffered and mmap access to the same file.
7. Verify you are getting it. Watch Cached in /proc/meminfo or the iomap tracepoints; a silent fallback to buffered produces correct data and none of the properties you paid for.

See Also

• Open Flags and Access Modes — where O_DIRECT sits among O_SYNC/O_DSYNC/O_PATH/O_TMPFILE and how the flag reaches iocb->ki_flags.
• The Multi-Queue Block Layer blk-mq — everything below the bio this note stops at: queues, merging, schedulers, completion.
• Shared Memory via mmap — the other way to take the kernel out of the data path, from the memory-mapping side.
• tmpfs In-Memory Filesystem — why O_DIRECT on /dev/shm measures nothing useful.
• The iomap Library — the extent-mapping library that backs iomap_dio_rw.
• Asynchronous IO Models in Linux — O_DIRECT is the natural partner of io_uring/AIO; RWF_NOWAIT/RWF_HIPRI semantics.
• Readahead and Read Path — the buffered path that O_DIRECT bypasses.
• The Page Cache · The Page Cache and address_space — the cache that is absent on the direct path.
• fsync fdatasync and Durability — why O_DIRECT still needs explicit syncing on volatile-cache devices.
• io_uring and the File Path — direct I/O’s high-throughput async front end.
• MOC: Linux Filesystems and VFS MOC