Kyber is a deliberately minimal blk-mq I/O scheduler written by Omar Sandoval at Facebook and merged in Linux 4.12 (2017) for fast multiqueue devices — NVMe SSDs and the like — where the classic “reorder by sector” approach buys nothing. Instead of sorting requests, Kyber throttles queue depth to hit a configurable latency target: you tell it the latency you want for reads and synchronous writes (defaults 2 ms and 10 ms), and it continuously measures actual completion latency and tightens or loosens how many requests of each type it lets reach the device at once (block/kyber-iosched.c; LWN, “Two new block I/O schedulers for 4.12”). The whole scheduler is under ~1,000 lines and exposes just two tunables — read_lat_nsec and write_lat_nsec. This note describes Kyber as it exists in Linux 6.12 LTS (2024-11-17).
Uncertain
Verify: the default latency targets and per-domain depth/batch values quoted here — read_lat_nsec = 2 ms, write_lat_nsec = 10 ms, discard target 5 s; per-domain max depths {READ:256, WRITE:128, DISCARD:64, OTHER:16}; batch sizes {READ:16, WRITE:8, DISCARD:1, OTHER:1}; async reservation KYBER_ASYNC_PERCENT = 75 (i.e. 25% reserved for sync). Reason: read directly from the static const tables in block/kyber-iosched.c at the v6.12 tag during this task, so high-confidence, but these are the initial values — Kyber’s whole point is to adjust the depths at runtime, so the effective depth on a live system will differ from the table. To resolve: read /sys/block/<dev>/queue/iosched/*_lat_nsec and the debugfs *_tokens files on the running kernel. uncertain
Mental Model
The intuition is the opposite of mq-deadline’s. mq-deadline asks “in what order should I send these requests?” and reorders by sector. Kyber asks “how many requests should I dare to have in flight, so that completions stay fast?” and answers it with a token bucket per request type. A request of a given type can only be dispatched if a free token is available for that type; the number of tokens is the in-flight cap. Kyber then runs a feedback loop: it timestamps every completion, builds a latency histogram, and on a timer compares the measured 99th-percentile latency against the target. If latencies are creeping above target (the device is congested), it removes tokens — fewer requests in flight, shorter device queues, faster completions. If a type is comfortably under target, it adds tokens back. It is a software control loop steering a hardware queue toward a latency setpoint.
flowchart TB
subgraph DOM["Four scheduling domains (by request type)"]
RD["READ tokens<br/>(cap ≤ 256)"]
WR["WRITE tokens<br/>(cap ≤ 128)"]
DI["DISCARD tokens<br/>(cap ≤ 64)"]
OT["OTHER tokens<br/>(cap ≤ 16)"]
end
REQ["new request"] -->|"classify by REQ_OP"| DOM
DOM -->|"token available?"| DISP{{"kyber_dispatch_request():<br/>round-robin domains,<br/>batch then advance"}}
DISP -->|"yes: take token, dispatch"| DEV["device"]
DISP -->|"no: throttle, wait"| HOLD["held until token freed"]
DEV -->|"on completion: record latency"| HIST["per-CPU latency histograms"]
HIST -->|"timer (every 100 ms)"| CTRL["kyber_timer_fn():<br/>p90 I/O latency -> congested?<br/>p99 total latency -> resize"]
CTRL -->|"tighten / loosen"| DOM
Kyber’s token-throttling control loop. What it shows: requests are sorted into four type-based domains, each gated by a token bucket that caps in-flight depth; completions feed per-CPU latency histograms that a periodic timer reads to recompute each domain’s token count toward the latency target. The insight: Kyber controls latency not by ordering requests but by limiting concurrency — a feedback controller, not a sorter. Shallower device queues mean each request waits behind fewer others, which is the direct lever on completion latency for a fast device that has no seek penalty to optimize away.
Mechanical Walk-through
The four scheduling domains
Kyber classifies every request into one of four domains by operation type (kyber_sched_domain() switches on REQ_OP_MASK): KYBER_READ, KYBER_WRITE, KYBER_DISCARD, and KYBER_OTHER (everything else — e.g. flushes). Each domain has its own token bucket (domain_tokens[], an sbitmap_queue — a scalable bitmap allocator) with a maximum depth from the kyber_depth[] table:
The source comment explains the caps: “Even for fast devices with lots of tags like NVMe, you can saturate the device with only a fraction of the maximum possible queue depth.” So Kyber deliberately keeps device queues shallower than the hardware would allow — a deep queue is exactly what raises per-request latency, because a request sits behind everything ahead of it. Reads get the deepest bucket (most throughput headroom) because they are the latency-critical type; discards get a tiny bucket because a flood of TRIM/discard commands can clog a device’s internal bookkeeping.
Dispatch: round-robin domains with per-domain batches
kyber_dispatch_request() services domains in a round-robin batch fashion. It tracks a cur_domain and a batching counter. While batching < kyber_batch_size[cur_domain] it keeps pulling from the current domain (kyber_dispatch_cur_domain()); the batch sizes are:
so up to 16 reads dispatch before Kyber rotates to writes, etc. To dispatch a request, kyber_dispatch_cur_domain() must first acquire a token via kyber_get_domain_token() → __sbitmap_queue_get(). If a token is available, the request is dispatched and the token is held until the request completes. If no token is free, the domain is throttled: the request stays queued, the hardware queue registers on a wait queue (sbitmap_add_wait_queue()), and dispatch moves on. When a request completes and frees its token, kyber_domain_wake() re-runs the hardware queue so the waiting request can proceed. This is the throttle: a domain at its token limit simply cannot put more requests on the device until in-flight ones finish.
When the batch is exhausted (or the current domain is empty or out of tokens), Kyber advances cur_domain round-robin and resets batching = 0. This guarantees every domain gets serviced — reads can’t be starved by writes and vice versa, but reads get larger batches so they dominate throughput.
The feedback loop: measure, then resize
This is what makes Kyber Kyber. Two things happen continuously:
On every completion (kyber_completed_request()), Kyber records the latency into per-CPU histograms — both total latency (now - rq->start_time_ns, the full time including queueing) and I/O latency (now - rq->io_start_time_ns, just the device portion). add_latency_sample() buckets each sample relative to the domain’s target: there are KYBER_LATENCY_BUCKETS = 8 buckets (2 << KYBER_LATENCY_SHIFT), the first 4 of which (KYBER_GOOD_BUCKETS) are at or below target (“good”), the last 4 above (“bad”). The bucket width is target / 4. Each completion also nudges a timer: timer_reduce(&kqd->timer, jiffies + HZ/10) — so the adjustment runs at most every 100 ms while there is traffic.
On the timer (kyber_timer_fn()), Kyber sums the per-CPU histograms and decides:
Congestion detection (p90 of I/O latency). For each domain it computes the 90th percentile of I/O latency. If any domain’s p90 lands in a “bad” bucket (p90 >= KYBER_GOOD_BUCKETS), the device is deemed congested (bad = true). The source uses p90, not p99, here deliberately — “we don’t want to be too sensitive to outliers.”
Per-domain resize (p99 of total latency). For each domain it computes the 99th percentile of total latency and scales the token depth linearly:
Since KYBER_LATENCY_SHIFT = 2, this is orig_depth * (p99+1) / 4. The comment walks the arithmetic: “if the p99 is 3/4 of the target, then we throttle down to 3/4 of the current depth, and if the p99 is 2x the target, then we double the depth.” The rule applied: if the device is congested, throttle domains that currently have good latency (steal depth from the well-behaved to relieve pressure); either way, loosen domains with bad latency (give them room so they aren’t unfairly squeezed). The new depth is clamped between 1 and the domain’s kyber_depth[] maximum.
The need to compute percentiles introduces a sampling rule: calculate_percentile() waits until it has 500 samplesorone second has elapsed since the first sample, whichever comes first, before it trusts the histogram. This stops Kyber from yanking the depth around on three data points.
Reserving capacity for synchronous I/O
A flood of asynchronous writes must not starve synchronous reads (a process blocked on a read is waiting; a buffered write’s issuer has moved on). Kyber enforces this at tag allocation time via kyber_limit_depth(): asynchronous requests are capped at async_depth = (1 << shift) * KYBER_ASYNC_PERCENT / 100, i.e. 75% of the scheduler tag pool, reserving the remaining 25% for synchronous requests. So even when async writes saturate their share, a sync read can always grab a tag and get scheduled.
Configuration and Tunables
Kyber exposes exactly two knobs, under /sys/block/<dev>/queue/iosched/, both in nanoseconds:
# Select Kyber.$ echo kyber > /sys/block/nvme0n1/queue/scheduler# The only two tunables (nanoseconds).$ grep . /sys/block/nvme0n1/queue/iosched/*/sys/block/nvme0n1/queue/iosched/read_lat_nsec:2000000/sys/block/nvme0n1/queue/iosched/write_lat_nsec:10000000
read_lat_nsec (default 2000000 = 2 ms): the completion-latency target for reads. Lower it and Kyber will throttle harder (fewer in-flight requests) to keep read latency tighter, at the cost of throughput.
write_lat_nsec (default 10000000 = 10 ms): the target for synchronous writes. The kernel doc is explicit that this applies to synchronous writes; buffered async writes are governed by the 75% async cap, not a latency target.
Note what is not tunable: the per-domain depth caps, batch sizes, the percentile choices (p90/p99), the 100 ms timer, and the discard target (5 s) are all compile-time static const. Kyber’s philosophy is that you specify the goal (latency) and the scheduler tunes the mechanism (depth) itself — which is the entire reason it has only two knobs.
When the debug filesystem (debugfs, CONFIG_BLK_DEBUG_FS) is mounted, /sys/kernel/debug/block/<dev>/sched/ exposes the live state: the current token counts per domain (*_tokens), cur_domain, batching, and async_depth — invaluable for watching the feedback loop actually move the depths.
Failure Modes and Common Misunderstandings
“Kyber reorders my I/O for throughput.” It does not sort by sector at all. On a spinning disk Kyber is a poor choice — there is no seek optimization, and throttling depth on a device that benefits from a deep elevator just hurts. Kyber is for fast, seekless devices.
“Lower latency target = always better.” Setting read_lat_nsec very low forces Kyber to keep the device queue shallow, which caps throughput: a fast NVMe device needs some queue depth to stay busy. Too aggressive a target leaves the device idling between completions. The defaults (2 ms / 10 ms) are a balance; tighten only with measurement.
“It’s a hard latency guarantee.” No — it is a target a feedback loop steers toward over ~100 ms windows using percentiles. Transient spikes above target are normal; Kyber reacts to them, it does not prevent them. And it needs 500 samples (or 1 s) before it even adjusts, so it is unresponsive to very short bursts.
Discard-heavy workloads. The DISCARD domain has a depth cap of just 64 and a batch size of 1, deliberately. A filesystem issuing huge bursts of TRIM/discard (e.g. an fstrim over a large SSD) can find discards throttled — usually desirable, since unthrottled discards can stall the device and spike read/write latency, but surprising if you expected discards to fly.
none: the baseline for fast NVMe — no scheduling at all, lowest CPU overhead, maximum IOPS. Choose Kyber over none when you want a latency target enforced (e.g. keeping read tail latency bounded under a heavy write load) and can spend a little CPU on the feedback loop. If you don’t need a latency bound, none is leaner.
mq-deadline: the general default for rotational and SATA SSD media. It reorders by sector with soft deadlines — the right mechanism when seek/locality matters. Kyber’s depth-throttling is the right mechanism when it doesn’t (fast multiqueue devices). They solve different problems; they are not interchangeable.
BFQ: when you need per-process bandwidth fairness and interactive responsiveness (desktops), BFQ’s budget allocation is far richer than Kyber’s domain throttling. BFQ is much heavier per request, though, and not aimed at server-grade fast devices.
The honest summary from the LWN coverage: “Throughput-sensitive server loads are more likely to run with Kyber, while users concerned with interactive response and possibly using slower devices would likely opt for BFQ.” Kyber is the server-on-fast-flash latency-targeting choice; it is rarely a desktop or spinning-disk choice.
Production Notes
Kyber originated at Facebook for exactly the workload it suits: server fleets of fast SSDs where tail latency under mixed read/write pressure is the metric that matters (Omar Sandoval’s original patch; Phoronix on the 4.12 merge). It is built in by default (MQ_IOSCHED_KYBER, default y in block/Kconfig.iosched), whose help text captures the design in one sentence: “Given target latencies for reads and synchronous writes, it will self-tune queue depths to achieve that goal.”
In practice Kyber is not the udev default for any device class on most distributions — NVMe gets none, rotational/SATA gets mq-deadline or BFQ. Kyber is an opt-in for operators who have measured tail-latency problems on fast devices and want a depth-throttling controller without writing one. The standard workflow: switch to kyber, set read_lat_nsec/write_lat_nsec to your SLO, run the real workload, and watch the debugfs *_tokens files and your latency metrics — if throughput collapses, your target is too tight; if tail latency still spikes, your target is too loose or the device is simply overloaded. Always benchmark against none on the same hardware: on a sufficiently fast and underloaded device, none may match Kyber’s latency at lower CPU cost, in which case Kyber’s machinery is not earning its keep.