Block Layer Statistics and iostat

Every block device the Linux kernel manages keeps a small set of cumulative counters describing how much I/O it has done and how long that I/O took. They are exposed in two byte-identical formats — /proc/diskstats (one line per device, all devices) and /sys/block/<dev>/stat (one device, no major/minor prefix) — and they are the raw material from which tools like iostat, sar, and dstat compute the human-meaningful rates and latencies you actually read. The counters are deceptively simple: reads/writes/discards/flushes completed, sectors moved, milliseconds spent, requests in flight, and the device’s busy time (io_ticks). The single most important — and most misunderstood — derived metric is %util, the fraction of wall-clock time the device had at least one request outstanding. On a traditional disk that serves requests one at a time, %util near 100% means saturation; on a modern multi-queue SSD or RAID array that services many requests in parallel, %util is meaningless as a saturation signal, and trusting it is a classic performance-analysis error (iostat(1); iostats.rst). This note pins to Linux 6.12 LTS (released 2024-11-17; the stat layout is stable through 6.18).

The counters described here are the device-level accounting the block layer maintains. They are distinct from, but complementary to, the scheduler-internal per-cgroup accounting that BFQ keeps under io.bfq.* — see the cross-link note below.

Mental Model: Counters In, Rates Out

The kernel never reports rates or latencies directly. It maintains monotonically increasing counters (they only go up, except in_flight, which oscillates), and userspace tools sample them twice, subtract, and divide by the elapsed wall-clock time. So “200 reads/sec” is (reads_now - reads_before) / seconds_elapsed, computed by iostat, not stored by the kernel. This sampling model is why the first iostat line reports averages since boot and you should ignore it.

flowchart LR
  subgraph K["Kernel (per block_device, per-CPU)"]
    A["blk_account_io_start()<br/>in_flight++, update io_ticks"]
    B["blk_account_io_done()<br/>ios++, sectors+=, nsecs+=duration<br/>in_flight--, update io_ticks"]
  end
  A --> CTR["disk_stats counters<br/>(summed across CPUs at read)"]
  B --> CTR
  CTR --> PD["/proc/diskstats<br/>/sys/block/sda/stat"]
  PD -->|"sample t1"| IO["iostat reads twice"]
  PD -->|"sample t2"| IO
  IO -->|"delta / elapsed"| OUT["r/s w/s rkB/s<br/>r_await aqu-sz %util"]

The accounting pipeline. What it shows: the kernel increments per-CPU counters at request start and completion; reading /proc/diskstats sums them; iostat samples the file at two instants and converts deltas into rates and latencies. The insight: the kernel stores cumulative state, the tool computes everything else — so every iostat number is a difference over an interval, and the interval you pass (iostat -x 1) defines the resolution.

The Raw Counters

Where they live and how to read them

# All devices, one line each, with major:minor and device name prefix:
cat /proc/diskstats
#  8  0 sda 12034 880 982344 4501 ...
 
# One device, the bare statistics (no prefix):
cat /sys/block/sda/stat
#  12034  880  982344  4501  ...

The two come from the same source and “should not differ” (iostats.rst). Use /sys/block/<dev>/stat to watch a known small set of disks (one open); use /proc/diskstats to watch many at once and avoid hundreds of opens per sample.

The fields

Both files emit a single whitespace-separated line. As of 6.12, the field set produced by part_stat_show() and diskstats_show() is, in order (genhd.c, lines 951–993, 1239–1305):

#	Field	Unit	Meaning
1	read I/Os	requests	reads completed successfully
2	read merges	requests	reads merged with an already-queued request
3	read sectors	512-B sectors	sectors read
4	read ticks	ms	total time reads waited (alloc → completion)
5	write I/Os	requests	writes completed
6	write merges	requests	writes merged
7	write sectors	512-B sectors	sectors written
8	write ticks	ms	total time writes waited
9	in_flight	requests	requests issued to driver, not yet completed
10	io_ticks	ms	total wall-time the device was active (had I/O queued)
11	time_in_queue	ms	total per-request wait time, summed over read+write+discard+flush
12	discard I/Os	requests	discards completed
13	discard merges	requests	discards merged
14	discard sectors	512-B sectors	sectors discarded
15	discard ticks	ms	total time discards waited
16	flush I/Os	requests	flushes completed (whole-disk only, not partitions)
17	flush ticks	ms	total time flushes waited

Two essential subtleties:

• A “sector” here is always 512 bytes, the historical UNIX sector, regardless of the device’s real logical or physical block size (stat.rst, lines 67–73). A device with 4096-byte logical blocks still reports its I/O in 512-byte sector units here. This is why iostat divides sectors by 2 to get kibibytes.
• The “ticks” fields measure per-request residency, not device-busy time. Field 4 (read ticks) is the sum over all completed reads of completion_time − allocation_time. If 60 reads each waited 30 ms, read ticks grows by 60 × 30 = 1800, so this counter can climb faster than 1000 ms per wall-second when requests overlap (stat.rst, lines 75–82).

How the kernel actually accumulates them

The counters are per-CPU (so the hot accounting path takes no global lock) and summed only when read. The blk-mq completion path is the canonical site (blk-mq.c, lines 976–1019):

• blk_account_io_start() runs when a request is queued: it increments the per-CPU in_flight[rw] and advances io_ticks.
• blk_account_io_done() runs at completion: it increments ios[sgrp], adds the transferred sectors, adds the residency nsecs[sgrp] += now − req->start_time_ns, decrements in_flight[rw], and again advances io_ticks.

Here start_time_ns is stamped at blk_mq_alloc_request() and the completion at __blk_mq_end_request(), which is exactly the interval the docs describe for the “ticks” fields (iostats.rst, lines 77–79). Since 4.19 the timing is taken in nanoseconds and truncated to milliseconds only at display time, so the millisecond fields are rounded-down nanosecond measurements.

io_ticks — field 10, the device’s busy time and the basis of %util — is special. It is not a sum of request durations; it is wall-clock time during which the device was active. update_io_ticks() advances it by the elapsed jiffies since the last update, but only if the device had at least one request in flight at that moment (blk-core.c, lines 993–1007). So whether the device serves one request or thirty concurrently during a given millisecond, io_ticks advances by the same one millisecond. This is the structural reason %util cannot see parallelism — it measures “was the device busy at all,” never “how busy.”

io_ticks accuracy caveat (since 5.0)

The kernel doc notes that since 5.0, io_ticks “counts jiffies when at least one request was started or completed. If [a] request runs more than 2 jiffies then some I/O time might not be accounted in case of concurrent requests” (iostats.rst, lines 101–103). The busy-time measurement is sampled at request edges, so it is an approximation, not an exact integral of busy intervals.

Uncertain

Verify: that field 11 in 6.12 is a summed per-type wait time rather than the historical separately-maintained “weighted time in queue” (aveq). Reason: stat.rst still labels field 11 “time_in_queue / weighted # of milliseconds,” but in 6.12 the struct disk_stats (part_stat.h, lines 8–14) has no time_in_queue member, and both part_stat_show() and diskstats_show() compute field 11 at read time as (nsecs[READ]+nsecs[WRITE]+nsecs[DISCARD]+nsecs[FLUSH]) in ms (genhd.c lines 982–986, 1287–1291). So the implementation is the sum of the four “ticks” fields, which differs from the old weighted-by-in-flight accumulator the documentation describes. To resolve: the code is authoritative for 6.12 — field 11 == sum of read/write/discard/flush ticks; the doc text is stale. uncertain

Partitions report less

Historically partitions exposed only four fields (reads, read-sectors, writes, write-sectors). Modern kernels restore the full set for partitions, but with a subtlety: because the disk-relative address is resolved early, an operation is attributed to whichever partition contains the first sector of the (possibly merged) request, which can introduce small inaccuracy when requests merge across a partition boundary (iostats.rst, lines 188–193). Flush counts (fields 16–17) are never tracked per partition — only on the whole disk (stat.rst, lines 57–59).

Why no locking

The doc is explicit that “no locks are held while modifying these counters,” so colliding updates can introduce minor inaccuracies — summing per-partition reads will be very close to but not exactly the disk total (iostats.rst, lines 134–144). The per-CPU design makes this practically a non-issue. Field 9 (in_flight) is the only field that should return to zero when the device is idle; all others only increase and may eventually wrap on a long-lived busy system, so consumers must handle counter wrap.

iostat: From Counters to Metrics

iostat -x (“extended” statistics) is the standard way to interpret these counters. Each column is a rate or average computed from the deltas between two samples (iostat(1)):

$ iostat -x 1
Device   r/s    w/s   rkB/s   wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  %util
nvme0n1 1820.0  340.0 233000  21800   12.0    4.0   0.7   1.2    0.21    0.34   0.48   128.0    64.1   38.0

Walking the load-bearing columns:

• r/s, w/s, d/s, f/s — read / write / discard / flush requests completed per second, after merging. Derived from the delta of fields 1, 5, 12, 16.
• rkB/s, wkB/s — kibibytes transferred per second; the sector deltas (fields 3, 7) divided by 2 (512 B → KiB) per second.
• rrqm/s, wrqm/s and %rrqm, %wrqm — requests merged per second and the percentage of requests that were merged before dispatch. High merge percentages mean the workload is sequential and the block layer is coalescing adjacent requests (see Request Merging and Plugging).
• r_await, w_await, d_await — average milliseconds a request of that type took, including both queueing time and device service time. This is the latency that matters to applications, computed as Δ(ticks) / Δ(I/Os) for the type. A read await of 0.21 ms is flash-fast; tens of milliseconds suggests a saturated rotational disk or a deep queue.
• rareq-sz, wareq-sz — average request size in KiB (sectors per request / 2). Small sizes with high IOPS = random workload; large sizes = sequential.
• aqu-sz — average queue length: “the average queue length of the requests that were issued to the device” (iostat(1)). It is computed from the weighted-time field (field 11) divided by the elapsed interval, i.e. Δ(time_in_queue) / Δ(wallclock). This is by Little’s Law the average number of requests resident in the device — aqu-sz = throughput × latency. For parallel devices, aqu-sz is a far better saturation indicator than %util, because it actually grows as more requests pile up. (Older iostat called this column avgqu-sz.)
• %util — see below.

%util — the famous trap

The man page defines %util precisely:

“Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.” (iostat(1))

%util is just Δ(io_ticks) / Δ(wallclock) × 100. Because io_ticks (as shown above) advances by the same amount whether one or thirty requests were outstanding, %util saturates at 100% the moment the device is never idle — which a moderately-loaded NVMe SSD reaches at a tiny fraction of its real capacity. A drive that can do 500,000 IOPS will show %util = 100% while doing 20,000 IOPS, simply because there is always something in flight. %util = 100% on an SSD or RAID array does not mean the device is saturated. Use aqu-sz (queue depth versus the device’s parallelism), the actual r/s/w/s against the device’s rated IOPS, and *_await latency against its spec instead.

The removed svctm

Old iostat output included svctm (“service time”). Modern sysstat removed it: it was derived by attributing io_ticks per request, an estimate that is fundamentally broken once requests are serviced in parallel — the same flaw that breaks %util. Do not look for svctm; if you find it, your sysstat is old and the number is untrustworthy.

Uncertain

Verify: the exact aqu-sz formula (Δ time_in_queue / Δ interval) and that svctm is removed in the sysstat version backing the linked man page. Reason: the man page on man7.org tracks a recent sysstat but its exact version was not pinned, and the man page does not spell out the aqu-sz arithmetic — the Little’s-Law derivation is inferred from the field semantics. To resolve: check the sysstat source (iostat.c) for the target distro’s version. uncertain

Failure Modes and Common Misunderstandings

Reading the first iostat sample as live data. The first interval’s numbers are averages since boot; discard them and use the second onward.

Confusing “ticks” with device-busy time. Fields 4/8 (read/write ticks) are summed per-request residency and can exceed wall-clock time when requests overlap. Only io_ticks (field 10) is wall-clock device-busy time. Mixing them up makes await and utilization look contradictory.

Treating %util as load on flash. Covered above — the single most common block-layer observability mistake.

Forgetting the 512-byte sector. Sectors are always reported in 512-byte units even on 4 KiB-block devices; if you compute bandwidth from raw diskstats sectors, multiply by 512, not by the device block size.

Partition vs disk attribution. Per-partition counters can drift slightly from the whole-disk total because of lockless updates and cross-partition merges; do not expect them to sum exactly.

Alternatives and Related Tooling

• /proc/diskstats directly — for custom monitoring agents; cheaper than iostat for many disks (one file, one read).
• sar -d — historical disk-activity reporting from the same counters, useful for retrospective analysis.
• blktrace / blkparse and bpftrace on the block tracepoints (block:block_rq_issue, block:block_rq_complete) — when the aggregate counters are too coarse and you need per-request traces and true latency histograms. These read the same accounting events the counters summarise.
• /sys/block/<dev>/queue/ tunables — nr_requests, read_ahead_kb, scheduler selection — the knobs you adjust after the statistics tell you something is wrong.

Production Notes

The practical workflow: run iostat -x 1, ignore the first line, and read aqu-sz and *_await as the real saturation/latency signals, treating %util as a binary “is the device idle or not” flag only. For an NVMe drive, cross-reference r/s + w/s against the device’s datasheet IOPS and r_await/w_await against its rated latency; %util near 100% with low IOPS and sub-millisecond awaits means the device is fine and mostly idle, not saturated. For a rotational disk, %util near 100% combined with rising aqu-sz and double-digit-millisecond awaits genuinely does indicate saturation, because the device is serial. The asymmetry between those two readings of the identical %util value is the whole lesson of this note.

See Also

• Parent: Linux Block Layer and Storage MOC — §1 The Request Path and the bio
• Mechanism: The Multi-Queue Block Layer blk-mq — where blk_account_io_start/done live
• Related: Request Merging and Plugging — what the *rqm/s merge counters reflect
• Scheduler cross-link: BFQ Budget Fair Queueing Scheduler — its io.bfq.* cgroup counters are the scheduler-side complement to these device-level counters
• Schedulers overview: Linux IO Schedulers Overview