vault

Block Device Cache and Durability

15 min read

Block Device Cache and Durability

A storage device with a volatile write cache acknowledges a write the moment the data lands in its on-board DRAM — long before it is on the platter or in NAND flash. That cache makes the device fast and makes its completion signal a promise, not a fact: if power is lost while data sits in that cache, the data is gone, even though write() returned success. Linux models this with a per-device flag the kernel exposes at /sys/block/<dev>/queue/write_cache, reading write back (cache is volatile, must be flushed for durability) or write through (writes are durable on completion, no flush needed) (sysfs-block ABI). The durability contract — what fsync(), fdatasync(), O_SYNC, and O_DSYNC actually guarantee against this cache — is what this note is about. The mechanism that enforces it (the REQ_PREFLUSH/REQ_FUA flags and the flush state machine) lives in Write Barriers FUA and Cache Flushes.

This note is pinned to Linux 6.12 LTS (2024-11-17), cross-checked against 6.18 LTS.

Mental Model: The Cache Is a Lie You Can Audit and Override

Think of every block device as having a switch with two positions and the kernel as keeping a belief about which position the switch is in. The hardware switch is write-back (volatile cache on, fast, writes acked early) or write-through (cache off or bypassed, slow, writes acked only when durable). The kernel’s belief is the write_cache sysfs value, derived from what the driver detected — and crucially, an admin can override the belief without touching the hardware. That mismatch is the source of most real-world durability disasters: tell the kernel “write through” on a device whose cache is actually volatile, and the kernel stops sending flushes while the device keeps losing data on power loss.

flowchart TB
  APP["Application<br/>write() then fsync()"] --> PC["Page cache (volatile RAM)"]
  PC -->|"writeback / fsync"| BL["Block layer"]
  BL --> Q{"kernel belief:<br/>write_cache =<br/>back or through?"}
  Q -- "write back" --> FLUSH["issue REQ_PREFLUSH / FUA<br/>force cache to media"]
  Q -- "write through" --> NOFLUSH["no flush issued"]
  FLUSH --> DEV
  NOFLUSH --> DEV
  subgraph DEV["Device"]
    DRAM["on-board volatile cache<br/>(DRAM)"] --> MEDIA["non-volatile media<br/>(platter / NAND)"]
  end
  MEDIA --> SAFE["survives power loss"]
  DRAM -. "power loss here = data lost<br/>unless PLP/BBU present" .-> LOSS["DATA LOST"]

The durability chain from application to media. What it shows: an fsync() forces page-cache data into the block layer, which issues a device flush only if the kernel believes the device has a volatile write-back cache; data sitting in the device’s DRAM cache is lost on power failure unless it has been flushed to media (or the device has power-loss protection). The insight to take: durability has two volatile stages — host RAM (page cache) and device DRAM — and fsync() only closes the gap if (a) the kernel’s write_cache belief is accurate and (b) the flush actually reaches non-volatile media; a wrong belief or a lying device silently breaks the whole chain.

The Two Cache Modes, and What the Kernel Records

The block layer represents a device’s cache capability as a feature bit on the request queue’s limits. In 6.12, include/linux/blkdev.h defines (blkdev.h):

/* supports a volatile write cache */
#define BLK_FEAT_WRITE_CACHE		((__force blk_features_t)(1u << 0))
/* supports passing on the FUA bit */
#define BLK_FEAT_FUA			((__force blk_features_t)(1u << 1))
/* do not send FLUSH/FUA commands despite advertising a write cache */
#define BLK_FLAG_WRITE_CACHE_DISABLED	((__force blk_flags_t)(1u << 0))

A device driver sets BLK_FEAT_WRITE_CACHE when it detects a volatile cache. The admin override is a separate flag, BLK_FLAG_WRITE_CACHE_DISABLED, layered on top. The effective decision combines them (blkdev.h):

static inline bool blk_queue_write_cache(struct request_queue *q)
{
	return (q->limits.features & BLK_FEAT_WRITE_CACHE) &&
		!(q->limits.flags & BLK_FLAG_WRITE_CACHE_DISABLED);
}

Walking the boolean: the kernel issues flushes (treats the device as write-back) only if the hardware advertised a write cache and the admin has not disabled it. This single inline is the gate consulted by blk_insert_flush() (see Write Barriers FUA and Cache Flushes) — when it returns false, no flush commands are ever sent. The design is deliberate: the hardware capability (BLK_FEAT_WRITE_CACHE) is what the driver knows; the policy (BLK_FLAG_WRITE_CACHE_DISABLED) is what the admin chooses; and the kernel never mutates the hardware feature bit when the admin flips policy.

Uncertain

Verify: that the 6.12 refactor truly separated “hardware advertised cache” (BLK_FEAT_WRITE_CACHE) from “admin disabled flushing” (BLK_FLAG_WRITE_CACHE_DISABLED) as two distinct fields, rather than the older single QUEUE_FLAG_WC/QUEUE_FLAG_FUA model. Reason: the BLK_FEAT_*/BLK_FLAG_* queue_limits model is new (circa 6.10–6.12) and replaced the older per-queue flags; the exact transition kernel was not pinned in this task. To resolve: git log for the introduction of BLK_FLAG_WRITE_CACHE_DISABLED in include/linux/blkdev.h. The 6.12 definitions above are quoted directly from the header, so the current model is verified. uncertain

The sysfs Interface: write_cache and fua

The kernel exposes both halves of the picture at /sys/block/<dev>/queue/. The write_cache attribute reports the effective state and lets the admin override it. The handler is in block/blk-sysfs.c (blk-sysfs.c):

static ssize_t queue_wc_show(struct gendisk *disk, char *page)
{
	if (blk_queue_write_cache(disk->queue))
		return sprintf(page, "write back\n");
	return sprintf(page, "write through\n");
}

So reading the file just renders blk_queue_write_cache() as the strings write back or write through. The store side parses those same strings (plus none as a synonym for write-through) and flips only the override flag (blk-sysfs.c):

static ssize_t queue_wc_store(struct gendisk *disk, const char *page, size_t count)
{
	struct queue_limits lim;
	bool disable;
 
	if (!strncmp(page, "write back", 10)) {
		disable = false;
	} else if (!strncmp(page, "write through", 13) ||
		   !strncmp(page, "none", 4)) {
		disable = true;
	} else {
		return -EINVAL;
	}
 
	lim = queue_limits_start_update(disk->queue);
	if (disable)
		lim.flags |= BLK_FLAG_WRITE_CACHE_DISABLED;
	else
		lim.flags &= ~BLK_FLAG_WRITE_CACHE_DISABLED;
	/* ... queue_limits_commit_update ... */
}

The critical caveat is in the stable ABI documentation: writing write_cache “can change the kernels view of the device, but it doesn’t alter the device state” (sysfs-block ABI). In other words, echo "write through" > write_cache only stops the kernel from issuing flushes — it does not turn off the drive’s physical cache. If the cache is still volatile, you have just removed your durability protection while keeping the risk. To actually change the hardware you need hdparm -W (ATA) or sdparm/SCSI mode pages (SCSI/SAS), discussed below.

The companion fua attribute reports whether the device can honor Force Unit Access natively: per the ABI doc it shows “Whether or not the block driver supports the FUA flag for write requests. FUA stands for Force Unit Access. If the FUA flag is set that means that write requests must bypass the volatile cache of the storage device” (sysfs-block ABI). It returns 1 if the device advertised BLK_FEAT_FUA, 0 otherwise. A 0 here means the kernel must emulate FUA with a post-write flush — same correctness, worse latency (the emulation logic is detailed in Write Barriers FUA and Cache Flushes).

Detecting and Controlling the Hardware Cache

The kernel learns a device’s cache capability from the transport. For NVMe, the driver inspects the controller’s Volatile Write Cache (VWC) identify field; if NVME_CTRL_VWC_PRESENT is set, it advertises both write-cache and FUA to the block layer (nvme/host/core.c):

if (ns->ctrl->vwc & NVME_CTRL_VWC_PRESENT)
	lim.features |= BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA;
else
	lim.features &= ~(BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA);

For ATA/SATA drives, the on-drive write cache is the IDE/SATA write-caching feature, toggled by hdparm -W: the man page describes -W as “Get/set the IDE/SATA drive’s write-caching feature” (hdparm(8)). hdparm -W0 /dev/sda disables the drive’s write cache (writes become durable on completion — slow but safe); hdparm -W1 re-enables it. There is also hdparm -F to “Flush the on-drive write cache buffer.”

Uncertain

Verify: whether hdparm -W settings persist across power cycles for SATA drives by default. Reason: the man7 rendering fetched did not include the persistence/data-loss warning text (the WebFetch summary noted the page “does not provide additional details” beyond the one-line -W description). Historically ATA write-cache settings are volatile (reset to firmware default on power cycle) unless the drive supports and is told to save the setting. To resolve: read the full hdparm man page or ATA spec for the SET FEATURES persistence behavior. uncertain

For SCSI/SAS devices, the cache lives in the caching mode page, and sdparm reads/writes the WCE (Writeback Cache Enable) bit: sdparm --get=WCE /dev/sdb reads it, --set=WCE / --clear=WCE change it (sdparm). SCSI mode pages distinguish current values (“those values that are active at this time”) from saved values (“those values that will be active after the next power cycle”). By default sdparm changes only the current page; adding --save also writes the saved page so the change survives a power cycle (sdparm). This current-vs-saved distinction is the SCSI analogue of the ATA volatility caveat above — change the current page and it reverts on reboot.

The Durability Contract: fsync, fdatasync, O_SYNC, O_DSYNC

Userspace expresses durability through four primitives, each with a precise POSIX-rooted meaning.

fsync(fd) “transfers (‘flushes’) all modified in-core data of the file referred to by the file descriptor fd to the disk device … so that all changed information can be retrieved even if the system crashes or is rebooted.” Critically, the man page states this “includes writing through or flushing a disk cache if present” (fsync(2)) — that is the device cache flush, the REQ_PREFLUSH from Write Barriers FUA and Cache Flushes. fsync() flushes both file data and the file’s metadata.

fdatasync(fd) is the cheaper variant: it “does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled” (fsync(2)). A change to the modification timestamp does not need flushing; a change to the file size does (you can’t read data you can’t locate). Databases that pre-allocate files use fdatasync() to skip needless metadata flushes.

O_SYNC makes each write synchronized: “By the time write(2) … returns, the output data and associated file metadata have been transferred to the underlying hardware (i.e., as though each write(2) was followed by a call to fsync(2))” (open(2)). It implements POSIX “synchronized I/O file integrity completion.”

O_DSYNC is the per-write analogue of fdatasync(): “By the time write(2) … return, the output data has been transferred to the underlying hardware, along with any file metadata that would be required to retrieve that data (i.e., as though each write(2) was followed by a call to fdatasync(2))” (open(2)) — POSIX “synchronized I/O data integrity completion.”

The unifying point: all four push data to the device and force the device cache flush. But the man page is blunt about the contract’s limits. In its HISTORY/CAVEAT material it warns that some older or lesser-used filesystems “do not know how to flush disk caches,” and that in those cases “disk caches need to be disabled using hdparm(8) or sdparm(8) to guarantee safe operation” (fsync(2)). It also notes fsync() “does not necessarily ensure that the entry in the directory containing the file has also reached disk” — you must separately fsync() the containing directory after creating or renaming a file. These are the sharp edges every storage engineer eventually cuts themselves on.

The Data-Loss-on-Power-Failure Model

Putting it together, durability has two volatile stages and several ways to fail.

  1. Host page cache (RAM). A plain write() only dirties a page-cache page; it is lost on a host crash or power loss until written back (see Dirty Page Writeback and Flusher Threads). fsync/O_SYNC close this gap by forcing writeback.
  2. Device write cache (DRAM). Even after data leaves host RAM, it may sit in the drive’s volatile cache. A power cut here loses it unless a flush has pushed it to media. fsync issues that flush — but only if the kernel believes the device is write back.

The failure matrix follows directly:

  • Volatile cache + flushes enabled + correct write_cache → durable after fsync returns. The normal, safe configuration.
  • Volatile cache + flushes disabled (write_cache = write through on a still-volatile device, or nobarrier/barrier=0 mount) → fsync returns but data may still be in volatile DRAM → silent corruption on power loss. This is the classic footgun.
  • Lying device — firmware that ignores the FLUSH command or reports completion before media write → no kernel setting can save you. Cheap consumer SSDs have shipped with this defect; only end-to-end power-cut testing reveals it.
  • Power-loss protection (PLP) present — enterprise SSDs with on-board capacitors flush their cache to flash on power loss; battery-backed (BBU) or capacitor-backed RAID controllers do the same for the controller cache. With genuine PLP/BBU it is safe to disable flushes for performance, because the “volatile” cache is effectively non-volatile (Thomas-Krenn).

Uncertain

Verify: the prevalence and current status of consumer SSDs that ignore FLUSH/FUA commands. Reason: this was a documented problem historically (and motivated tools like diskchecker.pl and fsync test suites), but I did not consult a current primary survey of device firmware behavior in this task. To resolve: cite a recent power-cut test study or a vendor’s PLP datasheet. The general model (flush can be honored or ignored by firmware) is sound; the current population of offending devices is what’s unverified. uncertain

Tail-Latency and Throughput Tradeoffs

Disabling the device write cache (or, equivalently, forcing every write through to media) trades throughput and especially tail latency for unconditional durability. With a write-back cache, the device can buffer, coalesce, and reorder writes, so bursts complete at DRAM speed and the slow media writes happen in the background — low and predictable latency at the cost of the volatility window. Turn the cache off (hdparm -W0, sdparm --clear=WCE) and every write waits for media: throughput collapses and p99/p999 tail latency balloons, because there is no buffer to absorb a slow erase-block or a seek.

The opposite lever — leaving the cache on but forcing a flush per fsync — is the normal safe mode, but its cost is real: Thomas-Krenn measured roughly 23 fsync writes/s with flushes versus ~195/s without on a single HDD (~8×) (Thomas-Krenn), because each flush serializes against the drive’s reordering. The block layer mitigates this by merging concurrent flushes (see Write Barriers FUA and Cache Flushes), and applications mitigate it by batching commits (group commit in databases). The decision framework:

  • Need durability, have PLP/BBU? Keep the cache on, optionally disable flushes (nobarrier) — best of both worlds, the protection is in hardware.
  • Need durability, no PLP? Keep the cache on, keep flushes on, and batch fsyncs. Do not disable flushes.
  • Latency-critical, durability handled elsewhere (replicated DB with synchronous replicas, write-ahead log on separate durable device)? You might run fdatasync instead of fsync, or rely on replication for durability and accept the local volatility window — a deliberate, documented choice, not a default.
  • Never set write_cache = write through to “go faster” — it makes the device slower on a true write-through device and unsafe on a volatile one. It is a correctness knob, not a performance knob.

Failure Modes and Common Misunderstandings

“I set write_cache to write through, so flushes are off and it’s still safe.” Only if the hardware cache is genuinely non-volatile or off. The sysfs write changed the kernel’s belief, not the device (sysfs-block ABI). On a volatile-cache drive this is a silent data-loss configuration.

“My RAID controller has a write cache, so I’m safe.” Only with a charged battery or capacitor (BBU). A BBU in learn-cycle, failed, or absent leaves the controller cache volatile, and many controllers automatically switch from write-back to write-through when the battery is unhealthy — quietly tanking performance. Monitor BBU health.

“fsync on the file is enough.” Not for newly created or renamed files: per fsync(2), the directory entry may not be durable until you also fsync() the directory (fsync(2)). The classic rename-for-atomic-replace pattern requires fsyncing both the file and its directory.

Confusing the page cache with the device cache. O_DIRECT bypasses the page cache but not the device cache — a direct write can still sit in the drive’s DRAM until flushed. O_DIRECT is not a durability guarantee; you still need fsync/O_DSYNC (or FUA) for that.

Alternatives and When to Choose Them

  • Cache on + per-fsync flush (default): correct everywhere, the right baseline. Cost: flush latency, mitigated by merging and group commit.
  • Cache on + FUA per durable write instead of flush: a finer-grained alternative the block layer uses for FUA-capable devices — only the specific commit write bypasses the cache, leaving the rest cached. Lower overhead than a full flush when only one record must be durable.
  • Cache off entirely (hdparm -W0): unconditional durability without any flush logic. Justified only when you cannot trust the flush path and have no PLP — pays a large, permanent throughput/tail-latency tax.
  • Cache on + flushes disabled (nobarrier, PLP/BBU present): maximum performance with hardware-backed durability. Correct only with verified power-loss protection.

Production Notes

The lesson repeated across two decades of incidents is that the durability chain is only as strong as its least-trustworthy link, and the links are easy to misconfigure invisibly. A write_cache = write through left over from a benchmark, a degraded RAID BBU, a nobarrier copied from a forum post, or a consumer SSD that ignores FLUSH — each produces a system that passes every fsync and loses data on the one power cut that matters. The defensive posture: keep flushes on by default, only disable them with documented PLP/BBU, verify with actual power-cut testing (pull-the-plug tests, not graceful shutdowns), and monitor BBU/PLP health continuously. The nobarrier-with-PLP configuration is legitimate and common in enterprise databases (Thomas-Krenn) — but it is a deliberate, hardware-justified exception, never a default tweak.

See Also

Graph View