vault

Zone Append and the Zoned Block Interface

15 min read

Zone Append and the Zoned Block Interface

A zoned block device carves its logical address space into fixed-size zones that, for the sequential-write-required kind, may only be written in order starting at the zone’s write pointer — random overwrites are rejected by the hardware. The naive way to honor this is for the host to track each zone’s write pointer and issue plain REQ_OP_WRITE commands at exactly that offset. But the block layer reorders and parallelizes I/O, and multiple threads may target one zone, so two in-flight writes to the same zone race: whichever the device serves second lands at the wrong write pointer and fails. The classic fix — serialize per zone, one write in flight at a time — throttles each zone to queue depth 1 and destroys the parallelism fast flash needs. Zone append (REQ_OP_ZONE_APPEND, merged in Linux 5.8 in 2020) breaks the deadlock: the host submits a write addressed only to the start of the zone, the device picks the actual landing location at its current write pointer, writes the data atomically, and returns the assigned Logical Block Address (LBA) in the completion. Because the host no longer dictates the offset, many appends to one zone can be outstanding at once with no host-side ordering — exactly the property that lets btrfs and F2FS run on zoned storage at full speed (LWN, “Introduce Zone Append”; enum req_op in blk_types.h).

This note is pinned to Linux 6.12 LTS (released 2024-11-17). A pivotal infrastructure change — zone write plugging, which replaced the older zone write locking — landed in 6.10 and is therefore the model present in 6.12 (LWN, “Zone write plugging”).

Mental Model

Think of a sequential zone as an append-only log on a spool of tape. The tape has a head — the write pointer — and you can only lay down new data where the head currently sits; you cannot rewind and overwrite mid-tape without first rewinding to the very start (a zone reset). The problem is not writing the tape; it is coordinating many writers who all want to append to the same spool. With plain writes, every writer must know precisely where the head is and take a turn — a lock — so the head advances cleanly. Zone append removes the coordination entirely: each writer hands the device a blob and says “stick this wherever the head is right now, and tell me where you put it.” The device, which owns the head, can interleave dozens of these requests internally and reply to each with its assigned position.

flowchart TB
  subgraph PLAIN["Plain writes (REQ_OP_WRITE) — host owns the write pointer"]
    W1["Thread A:<br/>write @ WP=100"] --> ZWL["Per-zone serialization<br/>(only ONE in flight)"]
    W2["Thread B:<br/>write @ WP=100"] -.->|"must WAIT"| ZWL
    ZWL --> DEV1["Device: append @100,<br/>WP -> 108"]
  end
  subgraph APPEND["Zone append (REQ_OP_ZONE_APPEND) — device owns the write pointer"]
    A1["Thread A:<br/>append to zone Z"] --> DEVQ["Device queue<br/>(MANY in flight)"]
    A2["Thread B:<br/>append to zone Z"] --> DEVQ
    A3["Thread C:<br/>append to zone Z"] --> DEVQ
    DEVQ --> DEV2["Device assigns LBAs:<br/>A->100, B->108, C->116<br/>returns each LBA"]
  end

Two ways to write a sequential zone. What it shows: with plain writes (top) the host must pick the target LBA equal to the write pointer, so concurrent writers to one zone must be serialized to a single in-flight request — queue depth 1 per zone. With zone append (bottom) every writer addresses the zone start, the device packs them at the live write pointer, and returns each one’s assigned LBA — so the zone runs at high queue depth. The insight: moving write-pointer authority from host to device is what converts a serialization bottleneck into ordinary parallel I/O.

Why Plain Writes Don’t Scale: The Serialization Bottleneck

A sequential-write-required (SWR) zone has a hardware write pointer: “Each sequential zone has a write pointer maintained by the device” and “data must be written sequentially. Any write must align with the current write pointer position. Violation triggers device errors” (zonedstorage.io). So a REQ_OP_WRITE to a sequential zone is only valid when its start sector equals the current write pointer. After it completes the write pointer advances by the write’s length, and the next write must target the new value.

That is fine for a single writer issuing one write at a time. It collapses the moment you want concurrency, which is precisely what fast NVMe flash demands. Two problems compound:

  1. The block layer reorders. The multi-queue block layer (see The Multi-Queue Block Layer blk-mq) batches, merges, and dispatches requests across per-CPU queues; there is no global guarantee that two writes to the same zone reach the device in submission order.
  2. Multiple writers race. If thread A and thread B both compute “the write pointer is at sector 100” and both submit a write to 100, the device serves one, advances the pointer to 108, and then rejects the second as an unaligned write because 100 is no longer the pointer.

The historical defense was zone write locking (ZWL): the block layer (specifically the mq-deadline scheduler) held a per-zone lock so that only one write request per zone could be in flight, and it kept that ordering tied to the scheduler. As LWN describes, ZWL worked by “limiting each zone to one in-flight write request,” “tying zone ordering to the mq-deadline I/O scheduler exclusively,” and “operating at the request level, holding scheduling tags even while waiting.” The cost is brutal: each zone is pinned to queue depth 1, and because the wait happened at the request level, plugged writes “hold scheduling tags” — starving the whole device of dispatch slots, not just that zone.

Uncertain

Verify: the precise claim that legacy zone write locking was implemented only by mq-deadline (and so using a different scheduler silently broke ordering on pre-6.10 kernels). Reason: stated in the LWN zone-write-plugging article (LWN 968580) and the v4 patchwork cover letter, but the exact scheduler coupling is a 6.10-era retrospective and not re-checked against pre-6.10 source. To resolve: read block/mq-deadline.c zone-locking code in a pre-6.10 tree (e.g. v6.6). Not load-bearing for 6.12, which uses plugging. uncertain

How Zone Append Solves It

Zone append inverts the authority. The operation is enumerated in the block layer’s request opcodes as:

/* write data at the current zone write pointer */
REQ_OP_ZONE_APPEND	= (__force blk_opf_t)7,

(include/linux/blk_types.h, v6.12)

The comment — “write data at the current zone write pointer” — is the whole idea. A REQ_OP_ZONE_APPEND bio (see The bio Structure) is addressed to the start LBA of the target zone, not to a specific write pointer offset. As the zonedstorage docs put it: “When writing to a zoned block device using zone append, the start sector of the write is pointing at the start LBA of the zone to write to. Upon completion the block device will respond with the position the data has been placed in the zone” (zbd-api, via the patch cover letter). The btrfs write-up frames the consequence: zone append “does not require a write pointer; it returns address of the written block. This means that the order in which structures are written does not matter and a block-group lock is not required” (LWN, “Btrfs on zoned”).

Because the device — which alone knows the live write pointer — chooses the landing spot, the host can have arbitrarily many appends to one zone outstanding simultaneously. There is no per-zone serialization, no queue-depth-1 penalty, and no host-side write-pointer bookkeeping in the data path. The cover-letter analogy is apt: zone append “from a high level perspective can be seen like a file system’s block allocator, where the user writes to a file and the file-system takes care of the data placement on the device” (LWN 818709) — except the device is the allocator.

The catch the caller must handle: you don’t know where your data landed until the operation completes. The assigned LBA comes back in the bio on completion (the block layer writes it into the bio’s sector field), so the filesystem or application must read the result after the I/O finishes rather than predicting it beforehand. This is why btrfs had to add an extent-mapping fixup step: it allocates a logical address, submits the append, and on completion patches its metadata to point at the actual device LBA the drive chose.

The max_zone_append_sectors Limit

A single zone append cannot be unbounded — the device advertises a maximum. In struct queue_limits the field is max_zone_append_sectors, surfaced to user space as the read-only sysfs attribute /sys/block/<dev>/queue/zone_append_max_bytes (“Maximum size in bytes of a zone append write operation,” introduced in 5.8) (zbd-api). The kernel helpers are:

queue_max_zone_append_sectors(q)   /* returns 0 if !blk_queue_is_zoned(q) */
bdev_max_zone_append_sectors(bdev) /* queue_max_zone_append_sectors(bdev_get_queue(bdev)) */

(include/linux/blkdev.h, v6.12)

A caller that wants to append more than this in one logical write must split the payload into multiple appends — and crucially, each of those appends will get its own assigned LBA, which is exactly the “large direct I/O split across multiple requests” hazard that zonefs has to guard against (covered in zonefs and Zoned Storage Filesystems). Related limits, also in queue_limits, are max_open_zones and max_active_zones (the device’s caps on how many zones may be concurrently open/active), plus chunk_sectors, which encodes the fixed zone size in 512-byte sectors. struct gendisk separately tracks nr_zones, zone_capacity, and last_zone_capacity — note zone capacity ≤ zone size: a zone may expose fewer usable blocks than its addressing stride (zoned-storage intro).

Zone Write Plugging and Append Emulation

Not every zoned device implements zone append in hardware. NVMe Zoned Namespaces (see NVMe Zoned Namespaces ZNS) do; SMR (Shingled Magnetic Recording) hard drives speaking SCSI ZBC / ATA ZAC do not. To give every zoned device a uniform interface, the kernel emulates zone append where the hardware lacks it — and since 6.10 the machinery that does this is zone write plugging, replacing the old zone write locking.

A zone write plug (struct blk_zone_wplug) is a small per-zone object holding a FIFO of pending write bios, a tracked write-pointer offset (wp_offset), error state, and a refcount. The entry point blk_zone_plug_bio() routes every write-side bio for a zoned disk: it lets reads and zone-management commands pass, but for writes it consults the plug. As LWN explains: “A write BIO to a zone is ‘plugged’ to delay its execution if a write BIO for the same zone was already issued.” The decisive improvement over locking is that plugging works at the bio level and off the scheduler: “Plugged BIOs waiting for execution thus do not hold scheduling tags and thus do not prevent other BIOs from being submitted to the device (reads or writes to other zones).” Any scheduler — including none — now works with zoned devices, where ZWL had required mq-deadline.

Plugs are allocated lazily and freed when the zone is fully written, reset, or finished. They are kept in an RCU-protected hash table: “a zone write plug is allocated when a write BIO is received for the zone and not freed until the zone is fully written, reset or finished” (LWN 968580).

Append emulation rides on this same plug. For a device that asked for emulation, the block layer also plugs incoming REQ_OP_ZONE_APPEND bios: per the v6.12 source comments, “Zone append operations for devices that requested emulation must also be plugged so that these BIOs can be changed into regular write BIOs,” and the handler will “use a regular write starting at the current write pointer. Similarly to native zone append operations, do not allow merging” (block/blk-zoned.c, v6.12). In other words, the plug serializes the emulated appends, rewrites each one’s opcode to REQ_OP_WRITE, sets its sector to the tracked wp_offset, advances the offset, and reports the chosen LBA back — synthesizing the device-assigns-location semantics on hardware that can only do plain ordered writes. The earlier 5.8 series had done equivalent emulation in the SCSI sd driver and null_blk, caching each zone’s write pointer; the 6.10 plugging unified it into the generic block layer (LWN 818709).

On error, the plug cannot trust its cached wp_offset — the device may have advanced or rejected writes. disk_zone_wplug_handle_error() “executes a report zone to update the zone write pointer offset to the current value as indicated by the device,” then retries or aborts the queued writes accordingly (LWN 968580; blk-zoned.c, v6.12).

The Broader Zoned Block Interface (zbd-api)

Zone append is one verb in a small command set the kernel exposes for zoned devices (zbd-api). The zone-management operations, added in 5.5, share the opcode space:

REQ_OP_ZONE_OPEN	= 10,  /* Open a zone */
REQ_OP_ZONE_CLOSE	= 11,  /* Close a zone */
REQ_OP_ZONE_FINISH	= 12,  /* Transition a zone to full */
REQ_OP_ZONE_RESET	= 13,  /* reset a zone write pointer */
REQ_OP_ZONE_RESET_ALL	= 15,  /* reset all the zone present on the device */

(blk_types.h, v6.12)

A op_is_zone_mgmt() helper classifies the first four as management ops; REQ_OP_ZONE_RESET_ALL is deliberately excluded “due to its different handling in the block layer.” These are reachable from user space via ioctls — BLKOPENZONE, BLKCLOSEZONE, BLKFINISHZONE (all 5.5), BLKRESETZONE (4.10) — and in-kernel via the unified blkdev_zone_mgmt() (which replaced the old blkdev_reset_zones()) (LWN, “Zone management commands”). To discover zones there is BLKREPORTZONE (4.10), returning struct blk_zone descriptors (start, length, write pointer, type, condition, and — since 5.9 — capacity), plus BLKGETZONESZ and BLKGETNRZONES (4.20). open/close/finish exist because devices impose open/active-zone limits, and explicitly managing zone state can both improve write performance and protect data (LWN 803256).

Uncertain

Verify: the exact ioctl-to-kernel-version mapping (BLKOPENZONE/BLKCLOSEZONE/BLKFINISHZONE = 5.5, BLKGETZONESZ/BLKGETNRZONES = 4.20, BLKREPORTZONE/BLKRESETZONE = 4.10). Reason: sourced from the zonedstorage.io zbd-api table, a strong but secondary reference, not each merge commit. To resolve: confirm each against the introducing commit in include/uapi/linux/blkzoned.h git history. The 5.5/5.8/5.9 feature dates are independently corroborated by LWN. uncertain

Common Misunderstandings

  • “Zone append guarantees ordering.” No — it guarantees placement, not order. Two appends may land in either relative position; the device tells you where each went. Code that needs a specific interleaving must still impose it (e.g., a single append per logical record, then read back the LBA).
  • “You can predict the LBA before submitting.” No. The whole point is that you cannot; the assigned LBA arrives only on completion. This is the most common porting bug for code retrofitting plain-write logic.
  • “Plain writes to a sequential zone are forbidden.” They are allowed but must hit the write pointer exactly and, on 6.12, are serialized per zone by the write plug. Zone append is the scalable path, not the only path.
  • “Zone write locking and zone write plugging are the same.” They solve the same correctness problem but plugging (6.10+) works at the bio level, frees the scheduler choice, and does not hold dispatch tags while waiting — a real throughput difference.

Alternatives and When to Choose Them

The alternative to zone append is host-managed plain writes under a write plug — viable for a single sequential writer (e.g., a log-structured filesystem that already serializes its own appends) but a poor fit for high-concurrency flash. For workloads that cannot tolerate the sequential-write constraint at all, the dm-zoned device-mapper target (4.13) hides zoning behind a conventional block device by buffering and remapping, at the cost of write amplification and a conventional-zone scratch area — appropriate only for legacy filesystems on SMR drives, never for ZNS performance. The modern choice for new code is to use zone append directly via a zone-aware filesystem (zonefs and Zoned Storage Filesystems, btrfs zoned mode, F2FS zoned) and let the kernel’s emulation paper over devices that lack native append.

Production Notes

The headline production win is concurrency. Btrfs’s zoned-mode author reported a 36% improvement for 4 KB random writes once writes switched to zone append, precisely because the per-block-group write lock could be dropped (LWN 853308) — and btrfs continues to use native zone append on 6.12. The 5.8 series had also folded zonefs onto append (“we have converted zonefs to run use ZONE_APPEND for synchronous direct I/Os,” LWN 818709), but note that this is no longer the case in 6.12: verified against the v6.12 source, zonefs now issues plain ordered writes and leans on the zone write plug for ordering rather than issuing append itself (see zonefs and Zoned Storage Filesystems). On the resource-accounting side, emulation is cheap — the original SCSI write-pointer cache cost about “52156 * 4 = 208624 Bytes” (~51 pages) for a 52,156-zone drive (LWN 818709). Operationally, the move from zone write locking to plugging in 6.10 means you no longer must pin zoned devices to mq-deadline; on 6.12 you can run none and still get correct ordering — a meaningful simplification for ZNS deployments chasing maximum IOPS (see Choosing an IO Scheduler).

See Also

Graph View