vault

Choosing an IO Scheduler

12 min read

Choosing an IO Scheduler

Picking a block I/O scheduler is a workload-and-hardware decision, not a “best scheduler” decision — there is no globally correct answer, only a correct answer for this device under this workload. The four candidates on modern Linux are none (FIFO pass-through, lowest CPU cost), mq-deadline (deadline-bounded general default), BFQ (Budget Fair Queueing — rich per-process fairness, high CPU cost), and Kyber (latency-target self-tuner for fast devices). The kernel itself picks a default by hardware-queue count alone (single-queue → mq-deadline, multi-queue → none); the familiar “bfq for HDDs, mq-deadline for SSDs” mapping is a userspace udev policy, not kernel behaviour. You inspect and override the choice through /sys/block/<dev>/queue/scheduler, and persist it with a udev rule — not the long-dead elevator= boot parameter, which has had no effect since kernel 5.0 (2019) (Red Hat). This note is the decision framework — the taxonomy and the scheduler internals live in Linux IO Schedulers Overview, mq-deadline Scheduler, BFQ Budget Fair Queueing Scheduler, and Kyber IO Scheduler. Pinned to Linux 6.12 LTS (2024-11-17); current LTS 6.18 (2025-11-30).

Mental Model — Three Axes, One Trade-off Triangle

Every I/O scheduler is a point in a three-way trade-off between latency, throughput, and fairness, with a fourth hidden axis — CPU cost per request — that only matters once the device is fast enough that the host CPU, not the disk, becomes the bottleneck. The decision collapses to: where is my bottleneck, and what do I most want to protect?

flowchart TD
  Q1{"Is the device<br/>very fast?<br/>(NVMe, &gt;100K IOPS,<br/>multi-queue)"}
  Q1 -->|"yes"| Q2{"Do you need a<br/>latency cap or<br/>cgroup throttling?"}
  Q1 -->|"no (HDD / SATA SSD /<br/>SD-eMMC, single queue)"| Q3{"Interactive desktop<br/>or per-process<br/>fairness needed?"}
  Q2 -->|"no — raw throughput"| NONE["none<br/>(let the device schedule;<br/>spend zero host CPU)"]
  Q2 -->|"yes — latency target"| KYBER["kyber<br/>(self-tunes depth to<br/>hit read/write latency)"]
  Q3 -->|"yes"| BFQ["bfq<br/>(proportional-share fairness,<br/>low desktop latency)"]
  Q3 -->|"no — servers, predictable"| MQD["mq-deadline<br/>(deadline-bounded,<br/>cheap, safe default)"]

A first-cut decision tree. What it shows: the first question is always “is the device fast enough that scheduling on the host is wasted effort?” — if yes, you default to none and only add kyber when you specifically need a latency knob; if no, you choose between bfq (fairness, interactivity) and mq-deadline (cheap predictability). The insight to take: scheduler choice tracks the bottleneck. On a slow disk the disk is the bottleneck, so spending CPU to reorder requests pays off (mq-deadline/BFQ); on a fast NVMe drive the host CPU becomes the bottleneck, so the cheapest scheduler (none) wins because reordering buys nothing the device controller is not already doing internally.

How the Default Is Actually Chosen

Two distinct mechanisms set the scheduler, and confusing them is the root of most “why is my scheduler X?” questions.

Mechanism one — the kernel. At queue bring-up, elevator_get_default() in block/elevator.c decides what to attach. As walked through in Linux IO Schedulers Overview, its logic is purely: if the driver set BLK_MQ_F_NO_SCHED_BY_DEFAULT, attach nothing; if the device has more than one hardware queue and unshared tags, attach nothing (none); otherwise attach mq-deadline. It never inspects whether the disk rotates. A multi-queue NVMe SSD therefore boots as none; a single-queue SATA/SAS disk or SD/eMMC card boots as mq-deadline.

Mechanism two — udev. Distributions then layer a policy on top by shipping udev rules that match on ATTR{queue/rotational} and write a chosen scheduler into ATTR{queue/scheduler}. This is where the “bfq for spinning disks” convention actually comes from. A real example is openSUSE’s 60-io-scheduler.rules, which skips zoned devices (the kernel auto-configures those), forces none on loop and multipath-component devices, and offers commented options to put bfq on single-queue rotational disks while noting that “real” multiqueue devices keep the kernel’s none default. CachyOS’s 60-ioschedulers.rules is more opinionated, assigning bfq to rotational sd*, mq-deadline to non-rotational sd*/mmcblk*, and kyber to nvme*.

Uncertain

Verify: that systemd/udev ships no default I/O-scheduler rule, so on a plain systemd install the scheduler is whatever the kernel chose (none for multi-queue, mq-deadline for single-queue) unless a distribution adds its own rule. Reason: distributions diverge — some ship aggressive 60-ioschedulers.rules, others ship none — and I could not retrieve the Arch wiki (HTTP 403) to confirm its exact current guidance. The openSUSE and CachyOS rule files cited are distribution-specific, not upstream systemd defaults. To resolve: check the actual /usr/lib/udev/rules.d/ and /etc/udev/rules.d/ on the target system (grep -r scheduler /usr/lib/udev/rules.d /etc/udev/rules.d) and read cat /sys/block/<dev>/queue/scheduler — the bracketed entry is ground truth regardless of how it got there. uncertain

The takeaway: do not reason about the default in the abstract — read it. The bracketed name in /sys/block/<dev>/queue/scheduler is what is actually running, after both kernel and udev have had their say.

Inspecting and Changing the Scheduler

The entire interface is sysfs. To see what a device is using and what is available:

# List schedulers; the bracketed one is active.
$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq
 
$ cat /sys/block/sda/queue/scheduler
[mq-deadline] none
 
# Is the device rotational (1) or solid-state (0)?
$ cat /sys/block/sda/queue/rotational
1
 
# How many hardware queues? (multi-queue → kernel default is none)
$ ls /sys/block/nvme0n1/mq/
0  1  2  3  ...

The mq/ directory’s entries are the per-queue hardware contexts; their count is the nr_hw_queues that drove the kernel’s default choice. To change the scheduler live — effective immediately, lost on reboot:

# Switch sda to BFQ. Requires bfq to be built-in or its module loaded.
$ echo bfq | sudo tee /sys/block/sda/queue/scheduler
# If bfq is a module and not yet loaded:
$ sudo modprobe bfq

Writing the name routes through elevator_change() → elevator_switch() in the kernel, which freezes the queue, tears down the old elevator, and brings up the new one (see Linux IO Schedulers Overview for the code path). Writing none detaches any scheduler. An invalid name returns -EINVAL (the shell reports “Invalid argument”), typically because the scheduler is not compiled in or its module is not loaded — bfq in particular is a module on many distributions.

To persist the choice, write a udev rule. This is the only supported persistent mechanism in the blk-mq era:

# /etc/udev/rules.d/60-ioscheduler.rules
# Spinning disks → BFQ (fairness, good for interactive HDD use)
ACTION=="add|change", KERNEL=="sd[a-z]*", ATTR{queue/rotational}=="1", \
  ATTR{queue/scheduler}="bfq"
# SATA / SAS SSDs and SD/eMMC → mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]*|mmcblk[0-9]*", ATTR{queue/rotational}=="0", \
  ATTR{queue/scheduler}="mq-deadline"
# NVMe → none (let the controller schedule)
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/rotational}=="0", \
  ATTR{queue/scheduler}="none"

Line by line: ACTION=="add|change" fires both when the device first appears and when udev re-triggers; KERNEL== matches the device-node name pattern; ATTR{queue/rotational}== filters by spinning-vs-solid; and the assignment ATTR{queue/scheduler}="..." writes the chosen scheduler into the same sysfs file you would echo to. After dropping the file in place, sudo udevadm control --reload && sudo udevadm trigger applies it without a reboot. This pattern is exactly what the distribution rules cited above implement; the real-world rule sets (CachyOS, openSUSE) add GOTO/LABEL guards so only the first matching rule fires.

Why elevator= No Longer Works

A persistent trap for people coming from older kernels: the elevator= kernel boot parameter is dead. It was tied to the legacy single-queue block layer, which was removed in 5.0. In v6.12, block/elevator.c still registers a handler for it — but the handler does nothing except warn:

static int __init elevator_setup(char *str)
{
	pr_warn("Kernel parameter elevator= does not have any effect anymore.\n"
		"Please use sysfs to set IO scheduler for individual devices.\n");
	return 1;
}
__setup("elevator=", elevator_setup);

So elevator=bfq on the kernel command line silently changes nothing and emits a dmesg warning. The Red Hat solution “Why does the ‘elevator=’ parameter no longer work in RHEL8” documents the same behaviour, and the Debian bug #914758 tracks the missing equivalent. The replacement is always: a udev rule (persistent) or a sysfs write (transient). There is also no CONFIG_DEFAULT_IOSCHED Kconfig option anymore — it was deleted with the legacy schedulers in commit f382fb0bcef4.

Workload-Driven Choice

The hardware class narrows the field; the workload picks the winner inside it.

High-IOPS NVMe (databases, object stores, anything throughput-bound). Use none. The device exposes one hardware queue per CPU and its controller already schedules across internal flash channels better than the host can guess. On a drive doing hundreds of thousands to millions of IOPS, every microsecond of per-request host scheduling is a tax you pay with no benefit — recall the BFQ docs’ own figure that BFQ costs ~1.9 µs/request versus ~0.7 µs for mq-deadline, and that BFQ’s sustainable ceiling with full accounting is only a few hundred KIOPS. A million-IOPS workload would spend more than a full CPU core just running BFQ’s scheduling logic. none lets the host get out of the way.

NVMe where you need a latency guarantee or QoS isolation. Use kyber. If a fast device is shared between a latency-sensitive consumer (synchronous reads) and a throughput-hungry one (bulk writes), Kyber’s read_lat_nsec/write_lat_nsec targets let it throttle the background traffic to protect foreground latency, at far lower CPU cost than BFQ — it was built for exactly this multi-queue regime (Kyber docs).

SATA/SAS SSD, general server. Use mq-deadline (the kernel default for these single-queue devices). It bounds worst-case latency cheaply and prevents read or write starvation without the overhead of per-process accounting. It is the conservative, predictable choice — what most server distributions effectively run.

Spinning HDD, especially interactive/desktop. Use bfq. On a slow rotating disk the device is the bottleneck, so spending CPU to schedule is worthwhile, and BFQ’s proportional-share fairness keeps a single greedy process (a backup, a dd, a compile) from making the desktop unresponsive. This is the workload BFQ was designed for, and the reason the 2018 LWN discussion “Two new block I/O schedulers for 4.12” and its successors debated making BFQ the default for single-queue devices. BFQ also integrates with the cgroups I/O controller for hard per-cgroup bandwidth limits — relevant if you need to cap a noisy container’s disk usage.

Single-queue flash (SD/eMMC, USB sticks, embedded). Either mq-deadline (the default) or bfq if you need fairness/cgroup control. These devices are single-queue, so the kernel gives them mq-deadline; on a phone or embedded board where one app must not starve the UI, BFQ is a reasonable upgrade despite its cost, since the device IOPS are modest enough that BFQ’s CPU tax is affordable.

Benchmarking Caveats

Scheduler benchmarking is notoriously easy to get wrong, and a bad benchmark will “prove” whatever the test setup biased toward.

The first caveat is match the benchmark to the real workload. A pure sequential fio run will favour throughput-maximising behaviour and make none look best even on a HDD where, under a mixed interactive workload, BFQ would feel far better. The metric that matters for an interactive system is tail latency under contention (p99/p99.9 of foreground requests while a background hog runs), not aggregate MB/s of a single stream. Measure the thing you actually care about.

The second caveat is queue depth and parallelism. A single-threaded, queue-depth-1 benchmark cannot exercise the multi-queue parallelism that makes none shine on NVMe; you need many concurrent submitters (high iodepth, numjobs) to see the scaling story. Conversely, at queue-depth-1 the scheduler choice barely matters because there is never more than one request to order.

The third caveat is the device’s own caches and internal scheduler. Modern SSDs and especially NVMe controllers reorder and cache internally; a host-side benchmark measures the combination of host scheduler and device firmware, and short runs can be dominated by the device write cache rather than steady-state behaviour. Use sustained runs, account for the device cache, and prefer --direct=1 (O_DIRECT) to bypass the page cache when you want to isolate the block layer — otherwise you are partly benchmarking writeback and the page cache, not the scheduler.

The fourth caveat is CPU accounting. On fast devices the cost that distinguishes schedulers is CPU per request, which a throughput number hides. Watch CPU utilisation (mpstat, perf) alongside IOPS — a scheduler that matches another’s IOPS while burning a whole extra core is strictly worse for a CPU-constrained server, and that difference is invisible if you only chart bandwidth.

Failure Modes and Gotchas

The classic surprise is “I set elevator=bfq and nothing changed” — covered above; it is a no-op since 5.0, use a udev rule. The second is “echo bfq says Invalid argument” — bfq (and sometimes kyber) is a module not yet loaded; modprobe bfq first, and in a udev rule the module is auto-loaded when the rule writes the name. The third is applying a rotational rule to NVMe — NVMe devices report rotational=0, so a rule matching ATTR{queue/rotational}=="0" on nvme* that assigns mq-deadline will downgrade a fast device to a single-queue scheduler it does not need; keep NVMe on none (or kyber) and reserve mq-deadline/bfq for sd*/mmcblk*. The fourth is zoned devices — host-managed SMR and ZNS namespaces require the scheduler to preserve write ordering; the kernel and distro rules deliberately leave these to none/mq-deadline and udev rules skip them, because forcing a reordering scheduler can violate the sequential-write constraint (see Zoned Block Devices and ZBD).

See Also

Graph View

Backlinks