vault

The NVMe Subsystem

16 min read

The NVMe Subsystem

NVM Express (NVMe) — short for Non-Volatile Memory Express — is the modern register-level interface and command set for talking to solid-state storage over a memory-mapped transport, designed from scratch for flash rather than inherited from the spinning-disk era like the SCSI/ATA stacks. In Linux the implementation lives under drivers/nvme/host/, split into a transport-independent core (core.c) that knows about controllers, namespaces and subsystems, and a set of transport drivers that move bytes: the PCIe driver pci.c, and the NVMe-over-Fabrics drivers rdma, tcp and fc. The core’s job is to discover what hardware exposes — controllers and the namespaces behind them — using the Admin command set (Identify, Set Features, queue creation), and to publish each namespace as a /dev/nvmeXnY block device wired into the multi-queue block layer (blk-mq). This note covers the architecture: the controller/namespace/subsystem object model, the driver layering, the discovery flow, and native multipath. The queue mechanism — how submission/completion queue pairs and doorbells actually carry commands — is dissected in its sibling NVMe Queue Pairs and the Driver.

This note is pinned to Linux 6.12 LTS (released 2024-11-17; the newest LTS at the time of writing is 6.18, 2025-11-30). Structural facts below are verified against the v6.12 source blobs listed in sources.

Mental Model

The single most useful mental model is a three-level object hierarchy: a subsystem contains one or more controllers, and a controller exposes one or more namespaces. Each level maps to a concrete C structure in drivers/nvme/host/nvme.h:

  • A namespace is the thing you actually read and write — a contiguous array of logical blocks (Logical Block Addresses, LBAs). It is the NVMe equivalent of a SCSI Logical Unit (LUN) or a partition-able disk. In Linux it surfaces as struct nvme_ns, and the disk you mount lives behind a struct nvme_ns_head (the “anchor” shared across controllers — more on this under multipath).
  • A controller is the management endpoint that owns the queues and processes commands. It is struct nvme_ctrl. A single physical PCIe SSD is normally one controller; a fabrics target can present many controllers.
  • A subsystem (struct nvme_subsystem) is the umbrella the spec uses to group controllers that share the same namespaces — identified by an NVMe Qualified Name (NQN) stored in subnqn. This is what makes multipath possible: two controllers in the same subsystem can each see the same namespace.
flowchart TB
  subgraph SUBSYS["nvme_subsystem<br/>(subnqn = NQN)"]
    direction TB
    CTRL0["nvme_ctrl<br/>(controller 0)<br/>admin_q + I/O queues"]
    CTRL1["nvme_ctrl<br/>(controller 1)<br/>fabrics path"]
  end
  NSH["nvme_ns_head<br/>(shared namespace anchor)<br/>gendisk -> /dev/nvme0n1"]
  CTRL0 --> NS0["nvme_ns<br/>(path via ctrl 0)"]
  CTRL1 --> NS1["nvme_ns<br/>(path via ctrl 1)"]
  NS0 --> NSH
  NS1 --> NSH
  NSH --> BLK["blk-mq<br/>/dev/nvme0n1"]
  CORE["nvme core (core.c)<br/>transport-independent"] -.discovers.-> SUBSYS
  PCIE["transport: pci.c<br/>(PCIe queue pairs)"] -.drives.-> CTRL0
  FAB["transport: tcp/rdma/fc"] -.drives.-> CTRL1

The NVMe object hierarchy as Linux models it. What it shows: one subsystem (keyed by its NQN) holds two controllers reached over different transports; each controller has its own nvme_ns view of the same namespace, and both views converge on a single nvme_ns_head that owns the gendisk exposed as /dev/nvme0n1. The insight: the nvme_ns_head is the deduplication point — it is why a dual-ported drive shows up as one device node rather than two, and where path selection happens.

Driver Layering: Core vs. Transports

The defining architectural decision in the Linux NVMe stack is the split between a transport-agnostic core and pluggable transports. core.c carries no knowledge of how a command physically reaches the controller; it builds struct nvme_commands, hands them to a request_queue, and processes the resulting completions. The mechanics of moving a command are abstracted behind const struct nvme_ctrl_ops *ops (a field of struct nvme_ctrl), which each transport fills in.

  • PCIe (pci.c) is the local case: the controller is a PCI function, its registers and doorbells are memory-mapped through a Base Address Register (BAR), and queues are rings in host DRAM that the device reaches by Direct Memory Access (DMA). This is the M.2/U.2 SSD in your laptop or server. Its data path is the subject of NVMe Queue Pairs and the Driver.
  • Fabrics drivers carry NVMe commands over a network so a host can use storage on another machine as if it were local. NVMe-over-Fabrics (NVMe-oF) has three in-tree transports: rdma (over RoCE or InfiniBand), tcp (over plain TCP/IP, the most deployable), and fc (Fibre Channel). They share fabrics.c for connection setup (the Connect command, Property Get/Set for register access) but differ in how bytes cross the wire.

Uncertain

Verify: the precise list and stability of in-tree fabrics transports (rdma, tcp, fc) and whether any additional transport exists in 6.12. Reason: this note’s research focused on core.c, pci.c, nvme.h and multipath.c; the fabrics transport source files were not individually fetched. To resolve: enumerate drivers/nvme/host/ in the v6.12 tree and read fabrics.c, tcp.c, rdma.c, fc.c headers. uncertain

Because the core is transport-independent, a fabrics namespace and a PCIe namespace look identical to the block layer above — same gendisk, same blk-mq request_queue, same /dev/nvmeXnY naming. The 6.12 tree even adds an nvme/host PCI endpoint function target so a Linux box can be an NVMe PCIe device, reusing the same core (kernel docs, NVMe PCI Endpoint Function Target).

The Object Structures, Field by Field

Reading nvme.h makes the model concrete. struct nvme_ctrl is the controller: it holds enum nvme_ctrl_state state (the lifecycle), struct request_queue *admin_q (the queue for Admin commands), const struct nvme_ctrl_ops *ops (the transport hooks), struct blk_mq_tag_set *tagset (the tag set backing the I/O queues — see Tag Sets and Request Allocation), struct list_head namespaces, a back-pointer struct nvme_subsystem *subsys, u16 cntlid (the controller ID), u32 queue_count, u64 cap (a cached copy of the Controller Capabilities register), u16 oncs (Optional NVM Command Support — tells you whether the drive supports things like Write Zeroes), and keep-alive state (u16 kas, struct delayed_work ka_work).

The controller lifecycle is the enum nvme_ctrl_state:

enum nvme_ctrl_state {
	NVME_CTRL_NEW,
	NVME_CTRL_LIVE,
	NVME_CTRL_RESETTING,
	NVME_CTRL_CONNECTING,
	NVME_CTRL_DELETING,
	NVME_CTRL_DELETING_NOIO,
	NVME_CTRL_DEAD,
};

A controller is born NVME_CTRL_NEW, becomes NVME_CTRL_LIVE once enabled and identified, drops to NVME_CTRL_RESETTING during a reset (driven by struct work_struct reset_work), NVME_CTRL_CONNECTING while a fabrics link is being re-established, and finally NVME_CTRL_DELETING/NVME_CTRL_DEAD on teardown. The state machine matters because I/O submission checks it: nvme_check_ready() in the queue path fails or requeues commands when the controller is not live.

struct nvme_subsystem carries char subnqn[NVMF_NQN_SIZE] (the NQN), char serial[20], char model[40], char firmware_rev[8], struct list_head ctrls (its controllers), struct list_head nsheads (its shared namespace anchors), u8 cmic (Controller Multi-Path I/O and Namespace Sharing Capabilities — the bit that says “this subsystem supports multiple controllers and shared namespaces”), and enum nvme_iopolicy iopolicy (the subsystem-wide multipath policy).

The namespace is split across two structures, which is the subtle part. struct nvme_ns is a per-controller path to a namespace: it has struct nvme_ctrl *ctrl, struct request_queue *queue, struct gendisk *disk, struct nvme_ns_head *head, and — when multipath is on — enum nvme_ana_state ana_state and u32 ana_grpid. struct nvme_ns_head is the shared anchor; the comment in nvme.h calls it exactly that: “Anchor structure for namespaces. There is one for each namespace in a NVMe subsystem that any of our controllers can see.” The head carries struct nvme_subsystem *subsys, struct nvme_ns_ids ids (the unique identifiers — NGUID, EUI-64, UUID — used to recognize that two nvme_ns are the same namespace), unsigned ns_id, u8 lba_shift (LBA size as a power of two), struct gendisk *disk, and the multipath routing fields struct nvme_ns __rcu *current_path[].

How Discovery Works: From PCI Probe to /dev/nvme0n1

The end-to-end flow when a PCIe NVMe drive appears is worth tracing because every abstraction shows up in order.

1. Transport probe and controller init. The PCIe nvme_probe() (in pci.c) maps the BAR, sets up the admin queue pair, enables the controller, then calls into the core to finish initialization. nvme_init_ctrl_finish() (exported from core.c) does the transport-independent work — its job, per the source, is to “Initialize the cached copies of the Identify data and various controller register in our nvme_ctrl structure.” It calls nvme_init_identify(), nvme_configure_apst() (Autonomous Power State Transition), nvme_configure_timestamp(), and nvme_start_keep_alive().

2. Identify Controller. nvme_identify_ctrl() builds an Admin command with opcode = nvme_admin_identify and cns = NVME_ID_CNS_CTRL (Controller Namespace Structure value 0x01) and submits it synchronously with nvme_submit_sync_cmd(). The 4 KiB response is the Identify Controller data structure, from which the core extracts capabilities — Optional Admin Command Support (OACS), Optional NVM Command Support (ONCS), maximum transfer size — and applies any quirks from the core_quirks[] table keyed by vendor/device ID. Quirks are how Linux works around firmware bugs without changing the generic path; the feature-and-quirk policy governs what may be added.

3. Namespace scan. nvme_scan_work (a work-queue item, so scanning runs in process context off the probe path) issues Identify with cns = NVME_ID_CNS_NS_ACTIVE_LIST (0x02) to fetch the list of active namespace IDs, then for each ID calls nvme_identify_ns() (cns = NVME_ID_CNS_NS, 0x00) and nvme_identify_ns_descs() to read the per-namespace geometry (LBA size, capacity) and the unique identifiers (NGUID/EUI-64/UUID).

4. Anchor and disk allocation. For each namespace, the core looks up or allocates a struct nvme_ns_head keyed by the unique identifiers. nvme_alloc_ns_head() initializes the head (INIT_LIST_HEAD(&head->list), init_srcu_struct(&head->srcu)) and, when multipath is enabled, calls nvme_mpath_alloc_disk() to create the shared gendisk. nvme_update_ns_info_block() then freezes the per-controller queue, updates the block-layer limits from the Identify data, and calls set_capacity_and_notify() so the kernel knows the device’s size.

5. Naming. The block device name comes from add_disk() on the head’s gendisk: the controller’s instance and the head’s instance give the familiar nvme0n1 form (nvmeX = controller instance, nY = namespace). There is also a character device for passthrough commands, named with the same scheme — nvme_add_ns_cdev() does dev_set_name(&ns->cdev_device, "ng%dn%d", ns->ctrl->instance, ns->head->instance), giving the /dev/ngXnY “generic namespace” node that nvme-cli uses for raw command submission.

Once add_disk() returns, the namespace is a first-class block device: partitions are scanned, udev fires, and /dev/nvme0n1 can be mounted. Every read and write from here on flows down through blk-mq into the queue path of NVMe Queue Pairs and the Driver.

Admin vs. I/O Command Sets

NVMe cleanly separates Admin commands (management: discovery, queue creation, feature setting, firmware, keep-alive) from I/O commands (the NVM command set: read, write, flush, and friends), and Linux mirrors that split with separate queues. There is exactly one Admin Submission/Completion Queue pair (queue ID 0), backed by admin_q and admin_tagset; the I/O queues (IDs 1..N) are backed by the controller’s main tagset.

The Admin opcodes the core relies on (include/linux/nvme.h):

enum nvme_admin_opcode {
	nvme_admin_create_sq    = 0x01,  /* create a submission queue */
	nvme_admin_create_cq    = 0x05,  /* create a completion queue  */
	nvme_admin_identify     = 0x06,  /* Identify (ctrl/ns/list)    */
	nvme_admin_set_features = 0x09,  /* e.g. number of queues      */
	nvme_admin_async_event  = 0x0c,  /* asynchronous event request */
	nvme_admin_keep_alive   = 0x18,  /* fabrics liveness           */
};

The I/O opcodes are deliberately tiny — the NVM command set’s core is just three opcodes:

enum nvme_opcode {
	nvme_cmd_flush = 0x00,
	nvme_cmd_write = 0x01,
	nvme_cmd_read  = 0x02,
};

nvme_setup_cmd() in core.c is the translator: it switches on req_op(req) (the block-layer request operation) and fills the matching nvme_command union member — REQ_OP_READ/REQ_OP_WRITE become an nvme_rw_command (carrying slba, the starting LBA, and length), REQ_OP_FLUSH becomes nvme_cmd_flush, REQ_OP_DISCARD becomes a Dataset Management (DSM) command, REQ_OP_WRITE_ZEROES becomes a Write Zeroes command, and zone operations become Zone Management commands (the link to NVMe Zoned Namespaces ZNS). It tags the command with cmd->common.command_id = nvme_cid(req) so the completion can be matched back to the originating request.

Keep-alive deserves a note because it is a fabrics-era addition that runs on PCIe too. nvme_keep_alive_work() periodically sends a nvme_admin_keep_alive command so the controller knows the host is still there; NVME_DEFAULT_KATO (Keep-Alive TimeOut) is 5 seconds, and the source comments that “The host should send Keep Alive commands at half of the Keep Alive Timeout accounting for transport roundtrip times.” If Traffic-Based Keep-Alive (NVME_CTRL_ATTR_TBKAS) is negotiated, ordinary I/O traffic counts as liveness and the explicit keep-alive cadence relaxes.

Native NVMe Multipath vs. dm-multipath

A namespace reachable through two controllers in the same subsystem is multipathed. Linux can handle this two ways, and which one runs is a build/boot decision.

Native NVMe multipath is implemented inside the NVMe driver itself (multipath.c, authored by Christoph Hellwig). The kernel parameter is multipath, exposed as nvme_core.multipath, and it defaults to true:

bool multipath = true;
module_param(multipath, bool, 0444);
MODULE_PARM_DESC(multipath,
	"turn on native support for multiple controllers per subsystem");

When on, the nvme_ns_head owns a single gendisk whose bio submit function is nvme_ns_head_submit_bio(); it calls nvme_find_path() to pick a live nvme_ns and resubmits the bio down that controller’s queue. The disk uses &nvme_ns_head_ops, and the head is allocated with blk_alloc_disk(&lim, ctrl->numa_node). The result: a dual-ported drive presents as a single /dev/nvme0n1, and failover happens transparently inside the kernel.

Path selection follows Asymmetric Namespace Access (ANA) — a spec mechanism where the controller advertises, per ANA group, whether a path is NVME_ANA_OPTIMIZED, NVME_ANA_NONOPTIMIZED, inaccessible, and so on; nvme_update_ana_state() consumes the ANA log page. Optimized paths are preferred over non-optimized (kernel docs). Within the eligible paths, the I/O policy decides:

static const char *nvme_iopolicy_names[] = {
	[NVME_IOPOLICY_NUMA] = "numa",        /* default */
	[NVME_IOPOLICY_RR]   = "round-robin",
	[NVME_IOPOLICY_QD]   = "queue-depth",
};

numa (the default) picks the path whose controller is closest to the submitting CPU’s NUMA node; round-robin cycles through paths for balanced spread; queue-depth picks the path with the fewest in-flight commands (it reads atomic_read(&ns->ctrl->nr_active)). The live policy is at /sys/module/nvme_core/parameters/iopolicy and can be set at boot with nvme_core.iopolicy=round-robin.

dm-multipath is the older, generic alternative from the device mapper — the same machinery SCSI/Fibre Channel SANs use. It lives a layer up: the NVMe driver exposes each path as a separate device and multipathd plus the dm-multipath target merge them. To use it on NVMe you must disable native multipath (nvme_core.multipath=N).

Uncertain

Verify: that disabling native multipath (nvme_core.multipath=N) is the only prerequisite for dm-multipath on NVMe, and the current distro-default stance (RHEL historically preferred dm-multipath for NVMe-oF in some releases). Reason: the kernel multipath doc fetched did not directly compare to dm-multipath, and distro policy is version- and vendor-specific. To resolve: consult the current RHEL/SUSE storage-administration guides and multipath.conf defaults for the target distro/version. uncertain

The trade-off is honest: native multipath is faster and simpler (no userspace daemon, ANA-aware, no extra dm layer in the hot path) and is the default; dm-multipath is more flexible (uniform tooling across SCSI and NVMe, richer path-checker/policy ecosystem) and is what shops with existing SAN automation often keep.

The nvme-cli Userspace Tool

nvme-cli is the standard userspace tool — “a user space utility to provide standards compliant tooling for NVM-Express drives” that “relies on the IOCTLs defined by the mainline kernel driver” (nvme.1 man page; source at linux-nvme/nvme-cli). It does not talk to the hardware directly; it issues ioctl()s on /dev/nvmeX (controller) or /dev/nvmeXnY / /dev/ngXnY (namespace) that the kernel translates into NVMe commands. Representative subcommands:

  • nvme list — list NVMe controllers and namespaces present.
  • nvme id-ctrl /dev/nvme0 — issue Identify Controller and pretty-print it (the same NVME_ID_CNS_CTRL structure the core caches at probe).
  • nvme id-ns /dev/nvme0n1 — Identify Namespace (geometry, LBA formats).
  • nvme list-ns /dev/nvme0 — the active namespace list.
  • nvme smart-log /dev/nvme0 — the SMART/health log page (wear, temperature, error counts).
  • nvme format /dev/nvme0n1 — low-level format / change LBA size.
  • nvme reset /dev/nvme0 — controller reset (drives the NVME_CTRL_RESETTING transition above).
  • nvme connect / nvme disconnect / nvme discover — NVMe-oF connection management (these build the Connect and Discovery-controller Get-Log-Page commands).

Every command “return[s] 0 on success and 1 on failure.” Because nvme-cli is just a thin shell over the kernel ioctls, it is the canonical way to inspect and verify everything described above — you can literally watch the Identify data the driver discovered.

Failure Modes and Diagnosis

  • Controller stuck resetting. If firmware hangs, reset_work cycles the controller; repeated resets in dmesg (nvme nvme0: Removing after probe failure or I/O timeout, reset controller) indicate a firmware or PCIe-link problem, not a filesystem problem. The state machine (NVME_CTRL_RESETTING → back to LIVE, or onward to DEAD) is your trace.
  • Namespace not appearing. If /dev/nvme0n1 is missing but /dev/nvme0 exists, the controller probed but the namespace scan failed or the namespace is inactive — check nvme list-ns and nvme id-ns. A namespace whose unique IDs collide with another (buggy firmware reusing NGUIDs) can be deduplicated into the wrong nvme_ns_head.
  • Wrong multipath behavior. A drive showing up as two devices when you expected one (or vice versa) is almost always a nvme_core.multipath mismatch with what your storage tooling expects. Confirm with cat /sys/module/nvme_core/parameters/multipath.
  • Keep-alive timeouts on fabrics. On NVMe-oF, a flaky network triggers keep-alive expiry → NVME_CTRL_CONNECTING reconnect loops. The kato and ctrl_loss_tmo connect-time options tune how aggressively the host gives up.

Alternatives and Contrast

The historical alternative is the SCSI stack (driving SATA via libata, SAS, USB storage): a layered upper/mid/lower design built for an era of one slow queue per device, where command translation and a deep mid-layer were acceptable overhead. NVMe discards that — a flat command set, queues sized to the core count, and a driver thin enough to keep up with millions of IOPS. In virtualization, the paravirtual contrast is virtio-blk (and virtio’s NVMe-emulating cousins): virtio reaches the same “rings in guest memory, doorbell to notify, interrupt to complete” shape, but with a hypervisor-defined virtqueue rather than a hardware NVMe controller — see virtio Device Model. The newest NVMe frontier is NVMe Zoned Namespaces ZNS, which keeps this whole subsystem but constrains namespaces to sequential-write zones.

Production Notes

In practice the NVMe subsystem is what makes a server’s storage layer “just work” at flash speeds: the per-CPU queue model means there is no global lock to tune, and the default I/O scheduler for NVMe is none (FIFO pass-through) because reordering buys nothing when the device has dozens of queues of its own — see Choosing an IO Scheduler. The fields most worth knowing in production are the multipath policy (nvme_core.iopolicy, default numa) and the keep-alive/connect timeouts for fabrics. For inspection, nvme-cli plus /sys/class/nvme/ is the toolbox; for tracing the command path, the block-layer tracepoints and blktrace see the bios, while nvme-cli’s error-log/smart-log see what the controller itself reports.

See Also

Graph View

Backlinks