vault

Dirty Page Writeback and Flusher Threads

16 min read

Dirty Page Writeback and Flusher Threads

When a buffered write(2) modifies a file, the kernel marks the affected page-cache page dirty and returns immediately; the bytes still live only in volatile RAM. Eventually those dirty pages must reach the disk, and the machinery that drives them there is the per-bdi flusher: a work item (wb_workfn) running on a shared kernel workqueue, one writeback context (struct bdi_writeback) per backing device info (bdi) (or per memory-cgroup-per-bdi under cgroup writeback), which walks the dirty-inode lists, calls each filesystem’s ->writepages to turn dirty folios into block I/O descriptors (bios), and hands them down into the multi-queue block layer. This note owns the block-dispatch side of writeback as it appears in the Linux 6.12 LTS tree (released 2024-11-17): how a dirty folio becomes a bio, what triggers the flusher, and how the request crosses into blk-mq. The memory-management side — the page lifecycle, the PG_dirty/PG_writeback flags and XArray tags — is owned by Dirty Pages and Writeback, and the writer-pacing throttle by Dirty Page Balancing and Throttling; this note cross-links rather than repeats them. (fs/fs-writeback.c, v6.12)

The reason this layer exists is a throughput mismatch: a CPU dirties memory at gigabytes per second while even a fast NVMe (Non-Volatile Memory Express) device commits at a fraction of that. The page cache makes writes fast and asynchronous; the flusher is the deferred drain that later batches those dirty pages into large sequential I/O, defers writes so that short-lived data may never hit disk, and bounds how much unwritten data can accumulate. Everything from the dirty page up to the constructed bio is the flusher’s job; everything from the bio to the platter belongs to blk-mq.

Mental Model — One Workqueue Item Per Writeback Context

The flusher is not a dedicated kernel thread per disk, despite the flush-MAJOR:MINOR names you see in ps. It is a delayed_work item that runs on a shared, unbound workqueue. Each backing device gets a bdi_writeback (universally abbreviated wb) that owns the dirty-inode lists and the work item; the workqueue’s pool of kworker threads execute it.

flowchart TB
  subgraph BDI["backing_dev_info (one per block device)"]
    WB["bdi_writeback (wb)<br/>b_dirty / b_io / b_more_io lists<br/>delayed_work dwork"]
  end
  TRIG["Triggers:<br/>periodic timer · background<br/>threshold · sync/fsync · reclaim"] -->|"queue_delayed_work(bdi_wq, dwork)"| WB
  WB -->|"workqueue runs"| WF["wb_workfn()<br/>→ wb_do_writeback()"]
  WF -->|"per dirty inode"| WSI["writeback_sb_inodes()<br/>→ __writeback_single_inode()"]
  WSI -->|"per inode mapping"| DW["do_writepages()<br/>→ a_ops->writepages()"]
  DW -->|"build bio, submit_bio()"| MQ["blk-mq<br/>(rq_qos / scheduler / driver)"]
  MQ -->|"I/O completes"| END["folio_end_writeback()<br/>page is clean"]

The dispatch path from trigger to block layer. What it shows: any of four triggers re-arms the wb’s delayed_work; the workqueue runs wb_workfn, which drills down wb_do_writeback → wb_writeback → writeback_sb_inodes → __writeback_single_inode → do_writepages, where the filesystem turns dirty folios into bios and submits them; completion eventually clears PG_writeback. The insight to take: the flusher is a chain of nested loops — over work items, over inodes, over the pages of each inode — and the per-inode ->writepages callback is the single hinge where memory-management writeback (folios, tags) becomes block-layer I/O (bios).

The Backing Device and Its Writeback Context

Every block device — and every pseudo-device that can hold dirty pages, such as an NFS (Network File System) mount or a FUSE (Filesystem in Userspace) mount — has a struct backing_dev_info (the bdi) describing its writeback characteristics. Hanging off it are one or more struct bdi_writeback (wb) contexts. On a system without cgroup writeback there is exactly one wb per bdi — the embedded root wb at bdi->wb (backing-dev-defs.h:186, v6.12). With CONFIG_CGROUP_WRITEBACK enabled there is one wb per memory cgroup per bdi, so a single disk can have many flusher contexts, each draining and accounting the dirty pages charged to its cgroup. The flusher therefore operates per-wb, not per-device; “one flusher thread per disk” is a useful simplification but is wrong under cgroup writeback.

Each wb owns the lists that drive writeback (backing-dev-defs.h:111–114):

  • b_dirty — inodes with dirty pages, ordered by the time they were dirtied.
  • b_io — inodes that have been parked and queued for the current writeback pass.
  • b_more_io — inodes that need another pass (e.g. only partially written before the pass ran out of budget).
  • b_dirty_time — inodes whose only dirty state is a timestamp (the lazytime mount option).

It also holds a struct delayed_work dwork — the work item that is the flusher — and a parallel struct delayed_work bw_dwork used to recompute the device’s estimated write bandwidth every 200 ms (BANDWIDTH_INTERVAL = max(HZ/5, 1); page-writeback.c:59). The bandwidth estimate wb->avg_write_bandwidth feeds both the writer throttle and the chunk-sizing below.

There is no per-device kernel thread. Writeback runs on a single shared workqueue created at boot in mm/backing-dev.c:

/* mm/backing-dev.c:494, v6.12 */
bdi_wq = alloc_workqueue("writeback", WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);

Walking the flags: WQ_MEM_RECLAIM guarantees a rescuer thread so writeback can make forward progress even when the system is out of memory (writeback is part of reclaim, so it must never deadlock waiting on the very memory it would free); WQ_UNBOUND lets the worker run on any CPU rather than being pinned, which suits I/O-bound work; WQ_SYSFS exposes the workqueue’s attributes under /sys. The kworker/... threads you observe running wb_workfn, renamed flush-MAJOR:MINOR via set_worker_desc("flush-%s", bdi_dev_name(wb->bdi)) (fs-writeback.c:2310), are these shared workqueue workers executing each wb’s dwork.

Uncertain

Verify: that the writeback workqueue is not freezable in 6.12. Reason: the v6.12 alloc_workqueue("writeback", WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0) call above contains no WQ_FREEZABLE flag, yet older write-ups (and a sibling note) describe bdi_wq as “freezable.” Freeze handling for writeback may now live elsewhere (e.g. suspend explicitly flushing/quiescing). To resolve: trace the suspend path and any WB_* freeze handling against the v6.12 tree. uncertain

The Flusher Entry Point — wb_workfn

wb_workfn() (fs-writeback.c:2304) is the function the workqueue runs. It recovers the wb from the dwork via container_of, sets the worker description, and then in the normal path loops wb_do_writeback(wb) until the work list is empty:

/* fs/fs-writeback.c:2304, wb_workfn() core */
do {
    pages_written = wb_do_writeback(wb);
    trace_writeback_pages_written(pages_written);
} while (!list_empty(&wb->work_list));
 
if (!list_empty(&wb->work_list))
    wb_wakeup(wb);                       /* more queued: rerun ASAP */
else if (wb_has_dirty_io(wb) && dirty_writeback_interval)
    wb_wakeup_delayed(wb);               /* reschedule periodic flush */

The final two lines implement the self-perpetuating periodic timer: as long as the wb still has dirty I/O and periodic writeback is enabled (dirty_writeback_interval != 0), wb_wakeup_delayed() re-queues the dwork to fire again after the periodic interval (queue_delayed_work(bdi_wq, &wb->dwork, timeout), fs-writeback.c:158). There is also an emergency path: if the workqueue cannot get enough workers and wb_workfn is running off the rescuer thread, it does a bounded writeback_inodes_wb(wb, 1024, WB_REASON_FORKER_THREAD) and bails, so the rescuer is not hogged.

wb_do_writeback() (fs-writeback.c:2273) is where policy lives. It sets WB_writeback_running on the wb state, then handles, in order:

  1. Explicit work items. get_next_work_item() drains wb->work_list, executing each struct wb_writeback_work. These come from sync(2), fsync(2), free-space pressure, memory reclaim, and superblock sync.
  2. wb_check_start_all() — a “flush everything” request, signalled by the WB_start_all state bit, splitting the work across cgroup wbs with wb_split_bdi_pages().
  3. wb_check_old_data_flush() — the periodic / kupdate-style flush. If dirty_writeback_interval is set and the last periodic flush is older than that interval, it writes back currently-dirty pages with for_kupdate = 1, sync_mode = WB_SYNC_NONE, reason = WB_REASON_PERIODIC.
  4. wb_check_background_flush() — the background flush, run when wb_over_bg_thresh(wb) reports that dirty memory has crossed the background threshold; for_background = 1, reason = WB_REASON_BACKGROUND.

The “kupdate” name is historical — it refers to the old kupdated kernel thread that periodically flushed dirty buffers before the modern per-bdi flusher replaced it.

wb_writeback and the Inode Loop

All four cases above funnel into wb_writeback() (fs-writeback.c:2077), which loops issuing batches of inode writeback under a single blk_plug (so the block layer can accumulate and merge requests before dispatch — see Request Merging and Plugging). Its loop has three notable exits that keep background and kupdate writeback from running forever and starving a sync:

  • It stops when work->nr_pages is exhausted.
  • For background or kupdate work, it yields the moment !list_empty(&wb->work_list) — that is, if any explicit work (a sync) arrives, the open-ended background flush steps aside and is restarted afterward.
  • For background work specifically, it stops once !wb_over_bg_thresh(wb) — i.e. once dirty memory has fallen back below the background threshold, the job is done.

For a kupdate pass, wb_writeback computes dirtied_before = jiffies - msecs_to_jiffies(dirty_expire_interval * 10) and queue_io() moves only inodes dirtied before that cutoff onto b_io. This is how dirty_expire_centisecs (default 3000 = 30 s) takes effect: periodic writeback only targets inodes that have been dirty longer than the expiry interval, so freshly written data is left in cache a while longer in case it is overwritten or deleted.

writeback_sb_inodes() (fs-writeback.c:1856) is the per-superblock inode loop. For each inode on b_io it builds a struct writeback_control (wbc) from the work item, skips new or being-freed inodes, defers an inode already under I_SYNC if this is only a WB_SYNC_NONE pass (requeueing it to b_more_io), marks the inode I_SYNC to pin it, sizes the work chunk, and calls __writeback_single_inode(). The chunk size matters for fairness: in writeback_chunk_size() (fs-writeback.c:1816), a WB_SYNC_ALL (data-integrity) pass uses nr_to_write = LONG_MAX so the inode is fully synced in one go, while a WB_SYNC_NONE pass uses roughly half the device’s measured bandwidth (min(wb->avg_write_bandwidth / 2, ...), with a MIN_WRITEBACK_PAGES floor) so no single inode monopolizes a background pass.

A subtle but important detail lives in the same loop: after writing a chunk, if need_resched() is true, the code calls blk_flush_plug(current->plug, false) before cond_resched(). The comment explains why — it is balancing “building up a nice long list of IOs to improve our merge rate” against “getting those IOs out quickly for anyone throttling in balance_dirty_pages().” Holding plugged I/O while sleeping would stall a writer that is parked waiting for the flusher to make progress, so the flusher flushes its plug before yielding the CPU. This is one of the explicit couplings between this note and Dirty Page Balancing and Throttling.

The Hinge — do_writepages Turns Folios Into bios

__writeback_single_inode() (fs-writeback.c:1647) is where memory-management writeback becomes block I/O. Its essential line is:

/* fs/fs-writeback.c:1658 */
ret = do_writepages(mapping, wbc);

do_writepages() (page-writeback.c:2672) dispatches to the filesystem’s address-space operation a_ops->writepages:

/* mm/page-writeback.c:2672, do_writepages() core */
wb = inode_to_wb_wbc(mapping->host, wbc);
wb_bandwidth_estimate_start(wb);
while (1) {
    if (mapping->a_ops->writepages)
        ret = mapping->a_ops->writepages(mapping, wbc);
    else if (mapping->a_ops->writepage)
        ret = writeback_use_writepage(mapping, wbc);   /* legacy fallback */
    else
        ret = 0;
    if (ret != -ENOMEM || wbc->sync_mode != WB_SYNC_ALL)
        break;
    reclaim_throttle(NODE_DATA(numa_node_id()), VMSCAN_THROTTLE_WRITEBACK);
}

For a modern extent-based filesystem the ->writepages callback is the iomap implementation (iomap_writepages); for ext4 it is ext4_writepages; for the page-cache generic path the legacy fallback walks PAGECACHE_TAG_DIRTY-tagged folios via write_cache_pages(). Whatever the implementation, its job is the same: iterate the dirty folios of this inode (visiting only tagged folios, so the cost is O(dirty), not O(file-size)), transition each from dirty to under-writeback via folio_start_writeback() (the flag/tag dance owned by Dirty Pages and Writeback), assemble them into one or more struct bio describing a contiguous device sector range and the in-memory pages to transfer, and call submit_bio(). From that point the page is the block layer’s responsibility.

The -ENOMEM retry loop is worth noting: a data-integrity (WB_SYNC_ALL) writeback that cannot allocate memory must not give up, so it throttles on local-node writeback activity (reclaim_throttle) and tries again, because abandoning a fsync’s writeback would silently break durability.

The Trigger to Block-Layer Bridge — REQ_BACKGROUND

The wbc->sync_mode and the work flags do not just steer the flusher; they are translated into block-layer request flags that the lower layers — the I/O scheduler and blk-wbt — read. That translation is wbc_to_write_flags() (writeback.h:104):

/* include/linux/writeback.h:104 */
static inline blk_opf_t wbc_to_write_flags(struct writeback_control *wbc)
{
    blk_opf_t flags = 0;
    if (wbc->sync_mode == WB_SYNC_ALL)
        flags |= REQ_SYNC;
    else if (wbc->for_kupdate || wbc->for_background)
        flags |= REQ_BACKGROUND;
    return flags;
}

This is the single most important bridge into the block layer. A data-integrity flush (fsync/sync, WB_SYNC_ALL) sets REQ_SYNC, so the I/O scheduler prioritizes it and blk-wbt gives it the high-priority depth. An opportunistic background or periodic flush sets REQ_BACKGROUND, which is precisely the flag Writeback Throttling and wbt’s get_limit() reads to apply the idle throttle depth — that is, the kernel deliberately tags routine flusher I/O as low-priority so the block-layer throttle can hold it back when it would harm read latency. Periodic and background writeback is, by design, the most throttle-able I/O in the system.

Submission Into blk-mq

submit_bio() ultimately calls blk_mq_submit_bio() (blk-mq.c:2940). It splits the bio to device limits, attempts to merge it with a plugged or scheduler-held request, and — when it must allocate a fresh request — calls into blk_mq_get_new_requests(), whose very first action is:

/* block/blk-mq.c:2859 */
rq_qos_throttle(q, bio);                 /* <-- wbt_wait() may sleep here */

rq_qos_throttle() walks the queue’s chain of rq_qos (request quality-of-service) policies; if blk-wbt is registered, this is where a background-writeback bio may be parked until in-flight write depth drops. After the request is built, rq_qos_track(q, rq, bio) (blk-mq.c:3012) records it, and on completion rq_qos_done(q, rq) releases the slot. The flusher does not know or care about this — it simply submits bios — but it is the reason the flusher and the read-latency throttle are coupled. The mechanism of that throttle is owned by Writeback Throttling and wbt; the request lifecycle by The Multi-Queue Block Layer blk-mq.

Configuration — Periodic and Background Knobs

Two sysctls under /proc/sys/vm/ (documented in Documentation/admin-guide/sysctl/vm.rst) set the flusher cadence; their default values come straight from the v6.12 initializers in mm/page-writeback.c:

# dirty_writeback_centisecs — how often the flusher wakes to flush old data.
#   Default 500 centiseconds = 5 s (dirty_writeback_interval = 5 * 100).
#   0 disables periodic writeback entirely.
$ cat /proc/sys/vm/dirty_writeback_centisecs
500
 
# dirty_expire_centisecs — how old a dirty page must be to be flushed periodically.
#   Default 3000 centiseconds = 30 s (dirty_expire_interval = 30 * 100).
$ cat /proc/sys/vm/dirty_expire_centisecs
3000

These combine: every dirty_writeback_centisecs a flusher wakes (wb_check_old_data_flush), and on each wake it targets inodes dirtied longer ago than dirty_expire_centisecs. So with defaults, a dirtied page that is never explicitly synced is on its way to disk within roughly 5–35 seconds. The background trigger is separate and earlier: dirty_background_ratio (default 10%) / dirty_background_bytes set the level at which wb_over_bg_thresh() returns true and wb_check_background_flush() starts draining without throttling the writer — the full dirty-limit machinery is in Dirty Page Balancing and Throttling. The per-bdi max_ratio/min_ratio knobs under /sys/class/bdi/<bdi>/ cap how much of the writeback cache a single device may hold, firewalling a slow or stuck NFS/FUSE mount so it cannot consume all of dirty RAM (Documentation/ABI/testing/sysfs-class-bdi, v6.12).

Failure Modes and Diagnosis

The “USB stick / NFS stall.” A slow or stuck device accumulates dirty pages that drain glacially. Its flusher wb makes little progress, b_more_io fills, and — without the per-bdi max_ratio firewall — dirty pages for that device count against the global dirty limit and stall writers to other, faster devices. Symptom: nr_dirty in /proc/vmstat pinned high while one device’s queue barely moves; processes stuck in D (uninterruptible) state inside balance_dirty_pages. Diagnose with grep -E 'nr_dirty|nr_writeback' /proc/vmstat and inspect /sys/class/bdi/<dev>/.

Flusher cannot keep up — the write cliff. On a large-RAM box with the default 20% dirty ratio, tens of gigabytes can be dirty before throttling bites; a sync, fsync, or umount then forces the flusher to drain it all at once, producing multi-second I/O stalls. The flusher is working correctly — the problem is the dirty pool was allowed to grow too large. The fix is on the throttle side (Dirty Page Balancing and Throttling): switch to absolute dirty_bytes/dirty_background_bytes so the pool stays small and the flusher drains continuously.

fsync amplification. A WB_SYNC_ALL writeback uses nr_to_write = LONG_MAX and on journaling filesystems (ext4 data=ordered) can drag in unrelated dirty data via the journal. Symptom: fsync latency spikes correlated with other writers. The writeback tracepoint family (/sys/kernel/debug/tracing/events/writeback/) — writeback_start, writeback_written, writeback_single_inode, writeback_pages_written, writeback_exec — exposes exactly which inodes each pass touched and how many pages it wrote, and is the authoritative way to watch the flusher.

Background writeback never starting. If dirty_background_ratio/bytes is set too high relative to the working set, wb_over_bg_thresh() rarely fires and dirty pages accumulate until the hard throttle, defeating the “drain early, drain continuously” intent. Lowering the background threshold makes the flusher start sooner.

Alternatives and Boundaries

The flusher is the asynchronous, deferred dispatch policy. The application can bypass or pre-empt it. O_DIRECT skips the page cache entirely, so there are no dirty pages and no flusher involvement — I/O is submitted as bios directly by the issuing thread. O_SYNC/O_DSYNC keep the page cache but make every write() behave like an implicit fsync, collapsing the deferral. fsync/fdatasync/sync force a WB_SYNC_ALL pass now rather than waiting for the periodic timer, then wait for in-flight writeback and (for fsync) issue a device cache flush — the durability mechanics are owned by Write Barriers FUA and Cache Flushes and the syscall semantics by Dirty Pages and Writeback. mmap + msync dirties through page tables instead of write(), but the same flusher drains the result. Choose deferred flushing (the default) for throughput; the synchronous variants only when per-write durability latency is acceptable.

Production Notes

On write-heavy servers the flusher’s behavior is usually tuned indirectly, through the dirty-limit knobs, because the flusher itself has few direct tunables. The canonical recipe (PostgreSQL, MySQL/InnoDB, large rsync/backup hosts) is to lower dirty_background_bytes so wb_over_bg_thresh() fires early and the flusher drains continuously rather than in a thundering sync burst, keeping background I/O smooth. Because routine flusher I/O is tagged REQ_BACKGROUND, leaving blk-wbt enabled (the default) lets the block layer additionally hold flusher writes back when reads are latency-sensitive — see Writeback Throttling and wbt for when to keep it on. Observability beyond the tracepoints: /proc/meminfo shows Dirty: and Writeback:; /proc/vmstat exposes nr_dirty, nr_writeback, nr_written; and bpftrace/perf probes on wb_workfn, do_writepages, and wbc_to_write_flags reveal which inodes and pages are flowing and with which priority flags.

See Also

Graph View

Backlinks