vault

NVMe Queue Pairs and the Driver

17 min read

NVMe Queue Pairs and the Driver

An NVMe (Non-Volatile Memory Express) controller is driven through queue pairs — a Submission Queue (SQ) and a Completion Queue (CQ) living as circular buffers in host memory, reached by the controller over Direct Memory Access (DMA). The host writes a 64-byte command into the next SQ slot and rings a doorbell (a memory-mapped register write) to tell the controller the tail advanced; the controller fetches the command, executes it, writes a 16-byte completion into the CQ, and raises an interrupt. There are no shared locks between queue pairs, so each one can be owned by a different CPU — which is exactly why the multi-queue block layer (blk-mq) maps one NVMe I/O queue 1:1 onto a blk-mq hardware queue. This note dissects that mechanism in the Linux PCIe driver: struct nvme_queue, the doorbell and phase-tag protocol, MSI-X interrupts, and the queue_rq path that turns a block-layer request into an nvme_command. The architecture around it — controllers, namespaces, discovery, multipath — is the sibling The NVMe Subsystem.

This note is pinned to Linux 6.12 LTS (released 2024-11-17; newest LTS at writing is 6.18, 2025-11-30). Code is verified against the v6.12 drivers/nvme/host/pci.c blob in sources.

Mental Model

Picture a ring of mailboxes for outgoing mail (the SQ) and a ring of mailboxes for replies (the CQ), both in your house (host DRAM), plus a doorbell on the mail carrier’s truck. To send a command you drop it in the next SQ mailbox and press the doorbell once (or once per batch) to say “the tail moved to here.” The carrier (the controller) walks your SQ from where it last stopped up to the tail you announced, performs each errand, and drops a receipt in the next CQ mailbox. It flips a single phase bit in each receipt so you can tell new receipts from stale ones without it telling you a count, then rings your bell (an interrupt) to wake you. You read receipts until the phase bit stops matching, then press the CQ doorbell to tell the carrier “I’ve collected up to here, you may reuse those slots.”

flowchart LR
  subgraph HOST["Host DRAM (one nvme_queue per CPU)"]
    SQ["Submission Queue<br/>64-byte entries<br/>sq_tail ->"]
    CQ["Completion Queue<br/>16-byte entries<br/>cq_head -> phase bit"]
  end
  subgraph BAR["Controller BAR (MMIO doorbells)"]
    SQDB["SQ Tail Doorbell<br/>q_db"]
    CQDB["CQ Head Doorbell<br/>q_db + db_stride"]
  end
  CPU["CPU / blk-mq hctx"] -->|"1. write nvme_command"| SQ
  CPU -->|"2. writel(sq_tail)"| SQDB
  SQDB -.->|"3. DMA fetch cmd"| CTRL["NVMe controller"]
  CTRL -.->|"4. DMA write completion + flip phase"| CQ
  CTRL -->|"5. MSI-X interrupt"| CPU
  CPU -->|"6. read CQEs until phase mismatch"| CQ
  CPU -->|"7. writel(cq_head)"| CQDB

One NVMe queue pair and its doorbells. What it shows: the SQ and CQ are rings in host memory; the only registers on the device are two doorbells per queue. Submission is “write entry, ring SQ doorbell”; completion is “controller writes CQE and flips the phase bit, raises MSI-X, host reaps and rings CQ doorbell.” The insight: the controller never has to tell the host how many completions are ready — the phase bit lets the host discover that itself, which is what makes polling (and interrupt coalescing) cheap.

The nvme_queue Structure

Every queue pair is a struct nvme_queue (drivers/nvme/host/pci.c). Its fields are the protocol:

struct nvme_queue {
	struct nvme_dev *dev;
	spinlock_t sq_lock;
	void *sq_cmds;                 /* the SQ ring in host memory   */
	struct nvme_completion *cqes;  /* the CQ ring in host memory   */
	dma_addr_t sq_dma_addr;        /* SQ bus address for the ctrl  */
	dma_addr_t cq_dma_addr;        /* CQ bus address for the ctrl  */
	u32 __iomem *q_db;             /* doorbell register pointer    */
	u32 q_depth;                   /* ring length (entries)        */
	u16 cq_vector;                 /* MSI-X vector for this CQ     */
	u16 sq_tail;                   /* next SQ slot host will write */
	u16 last_sq_tail;             /* last tail value rung          */
	u16 cq_head;                   /* next CQ slot host will read  */
	u16 qid;                       /* queue ID (0 = admin)         */
	u8  cq_phase;                  /* expected phase bit           */
	u8  sqes;                      /* SQ entry size shift          */
};

sq_cmds and cqes are the two rings; sq_dma_addr/cq_dma_addr are the bus addresses the controller was told to DMA against. q_db points into the device’s doorbell region. sq_tail, cq_head, and cq_phase are the host’s view of progress. cq_vector is the MSI-X interrupt line for this CQ. Crucially, sq_lock is per queue — there is no global submission lock, which is the whole scalability story.

The SQ entry size is fixed by the spec at 64 bytes and the CQE at 16 bytes. nvme.h encodes these as power-of-two exponents: NVME_ADM_SQES = 6 (2⁶ = 64) and NVME_NVM_IOCQES = 4 (2⁴ = 16). So an SQ of depth 1024 occupies 64 KiB and the matching CQ 16 KiB of host DRAM, allocated coherently with dma_alloc_coherent().

Allocating a Queue Pair

nvme_alloc_queue() builds one queue’s memory and wiring:

static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
{
	struct nvme_queue *nvmeq = &dev->queues[qid];
 
	nvmeq->sqes = qid ? dev->io_sqes : NVME_ADM_SQES;
	nvmeq->q_depth = depth;
	nvmeq->cqes = dma_alloc_coherent(dev->dev, CQ_SIZE(nvmeq),
					 &nvmeq->cq_dma_addr, GFP_KERNEL);
	/* ... nvme_alloc_sq_cmds() allocates sq_cmds similarly ... */
	nvmeq->cq_head  = 0;
	nvmeq->cq_phase = 1;
	nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
	nvmeq->qid  = qid;
	dev->ctrl.queue_count++;
}

Two lines carry the most meaning. cq_phase = 1 initializes the expected phase to 1, because the controller writes the first generation of completions with phase 1 (the ring is implicitly zeroed, so a fresh entry reads 0 until the controller flips it). And the doorbell address:

nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];

The controller exposes a contiguous doorbell array at register offset NVME_REG_DBS = 0x1000 (the comment in nvme.h is /* SQ 0 Tail Doorbell */). Each queue gets two consecutive doorbells — the SQ tail then the CQ head — and the spacing between doorbells is the doorbell stride db_stride (from the controller’s CAP.DSTRD capability field; normally 1 native register width). So queue qid’s SQ-tail doorbell is at index qid * 2 * db_stride and its CQ-head doorbell is db_stride further on. This matches the NVMe over PCIe Transport Specification exactly: the Submission Queue y Tail Doorbell sits at 0x1000 + (2y) * (4 << CAP.DSTRD) and the Completion Queue y Head Doorbell at 0x1000 + (2y + 1) * (4 << CAP.DSTRD) (NVMe PCIe Transport Spec).

Uncertain

Verify: the exact doorbell-offset formula and that 4 << CAP.DSTRD is the per-doorbell stride in bytes. Reason: the canonical NVMe PCIe Transport Specification PDF returned HTTP 403 on direct fetch; the formula here is reconstructed from a web-search excerpt of that spec plus the driver’s dev->dbs[qid * 2 * dev->db_stride] indexing. To resolve: open the ratified NVMe PCIe Transport Specification §3 (Controller Registers / Doorbell registers) directly and confirm the offset arithmetic. uncertain

A queue pair is created on the device (not just allocated in host memory) via two Admin commands, completion queue first: adapter_alloc_cq() issues nvme_admin_create_cq (giving the controller cq_dma_addr and the MSI-X vector), then adapter_alloc_sq() issues nvme_admin_create_sq (binding the SQ to its CQ). nvme_create_queue() does exactly this ordering — CQ before SQ — because a submission queue must name an already-existing completion queue.

The Submission Path: queue_rq

When blk-mq has a request to dispatch, it calls the driver’s .queue_rq. For NVMe that is nvme_queue_rq():

static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
				  const struct blk_mq_queue_data *bd)
{
	struct nvme_queue *nvmeq = hctx->driver_data;
	struct nvme_dev *dev = nvmeq->dev;
	struct request *req = bd->rq;
	struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
	blk_status_t ret;
 
	if (unlikely(!test_bit(NVMEQ_ENABLED, &nvmeq->flags)))
		return BLK_STS_IOERR;
	if (unlikely(!nvme_check_ready(&dev->ctrl, req, true)))
		return nvme_fail_nonready_command(&dev->ctrl, req);
 
	ret = nvme_prep_rq(dev, req);          /* build nvme_command + map DMA */
	if (unlikely(ret))
		return ret;
	spin_lock(&nvmeq->sq_lock);
	nvme_sq_copy_cmd(nvmeq, &iod->cmd);    /* copy 64B cmd into SQ slot   */
	nvme_write_sq_db(nvmeq, bd->last);     /* ring the SQ tail doorbell   */
	spin_unlock(&nvmeq->sq_lock);
	return BLK_STS_OK;
}

Read top to bottom: hctx->driver_data is this queue (the 1:1 mapping established at init time, below). nvme_check_ready() enforces the controller-state machine from The NVMe Subsystem — a non-live controller fails or defers the command. nvme_prep_rq() is where the block-layer request becomes hardware: it calls nvme_setup_cmd() (in core.c) to fill an nvme_rw_command (opcode read/write, nsid, slba, length, the command_id) and nvme_map_data() to build the DMA scatter list — PRP (Physical Region Page) lists or SGLs (Scatter-Gather Lists) describing the data buffer for the controller to DMA against. Then, under the per-queue sq_lock, the 64-byte command is copied into sq_cmds[sq_tail] and the doorbell is rung. The lock is held only for the copy and doorbell, never across the DMA mapping.

The doorbell write itself is deliberately clever about batching:

static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq)
{
	if (!write_sq) {
		u16 next_tail = nvmeq->sq_tail + 1;
		if (next_tail == nvmeq->q_depth)
			next_tail = 0;
		if (next_tail != nvmeq->last_sq_tail)
			return;                 /* defer: more in the batch  */
	}
	if (nvme_dbbuf_update_and_check_event(nvmeq->sq_tail,
			nvmeq->dbbuf_sq_db, nvmeq->dbbuf_sq_ei))
		writel(nvmeq->sq_tail, nvmeq->q_db);
	nvmeq->last_sq_tail = nvmeq->sq_tail;
}

The write_sq argument is blk-mq’s bd->last — true on the final request of a dispatch batch. When it is false and more commands are coming, the function returns without ringing the doorbell, letting several commands accumulate so a single MMIO write announces the whole batch. A Memory-Mapped I/O (MMIO) writel() to device space is expensive (it stalls the CPU until the write posts), so coalescing doorbells is a real throughput win — this is the same idea blk-mq exposes through .commit_rqs/nvme_commit_rqs and the bulk .queue_rqs/nvme_queue_rqs. The nvme_dbbuf_* calls are the optional shadow doorbell buffer optimization: a virtualized controller can place doorbell values in host memory and an “event index,” letting the driver skip the actual MMIO writel() when the device indicates it does not need the notification — pure overhead removal in emulated/virtual NVMe.

The Completion Path: Phase Tags

Completions are the elegant part. The CQE (struct nvme_completion in nvme.h) is 16 bytes:

struct nvme_completion {
	union nvme_result { __le16 u16; __le32 u32; __le64 u64; } result;
	__le16 sq_head;     /* how much of the SQ may be reclaimed       */
	__le16 sq_id;       /* submission queue that generated this entry*/
	__u16  command_id;  /* of the command which completed            */
	__le16 status;      /* did the command fail, and if so, why?     */
};

The low bit of status is the phase tag (P bit). The controller toggles the phase it writes each time the CQ wraps: entries are valid when their phase bit equals the host’s expected cq_phase. So the host never needs a “how many are ready” register — it just reads the head entry’s phase:

static inline bool nvme_cqe_pending(struct nvme_queue *nvmeq)
{
	struct nvme_completion *hcqe = &nvmeq->cqes[nvmeq->cq_head];
	return (le16_to_cpu(READ_ONCE(hcqe->status)) & 1) == nvmeq->cq_phase;
}
 
static inline int nvme_poll_cq(struct nvme_queue *nvmeq, struct io_comp_batch *iob)
{
	int found = 0;
	while (nvme_cqe_pending(nvmeq)) {
		found++;
		dma_rmb();
		nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
		nvme_update_cq_head(nvmeq);
	}
	if (found)
		nvme_ring_cq_doorbell(nvmeq);
	return found;
}

The loop walks consecutive CQEs while their phase matches; nvme_handle_cqe() uses the CQE’s command_id to look up the originating request and complete it through blk-mq; nvme_update_cq_head() advances cq_head, flipping cq_phase on wrap. The dma_rmb() is a read memory barrier ensuring the rest of the CQE is read only after the phase bit is observed valid (the controller writes the data then the phase last; the barrier prevents the CPU reordering the reads). When the batch is drained, nvme_ring_cq_doorbell() writes cq_head to the CQ-head doorbell (q_db + db_stride), telling the controller it may reuse those slots. The sq_head field in each CQE doubles as flow control — it tells the host how far the controller has consumed the SQ, so the host knows how many SQ slots are free without a separate register.

MSI-X Interrupts: One Vector Per CQ

The NVMe PCIe Transport Specification recommends “the host allocate a unique MSI-X vector for each Completion Queue.” Linux follows this: each I/O CQ gets its own Message Signaled Interrupt-eXtended (MSI-X) vector, so completions on different queues raise different interrupts and land on different CPUs — no single interrupt funnels all completions. nvme_setup_irqs() allocates the vectors with CPU-affinity spreading:

static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)
{
	struct irq_affinity affd = {
		.pre_vectors = 1,            /* one reserved for admin */
		.calc_sets   = nvme_calc_irq_sets,
		.priv        = dev,
	};
	unsigned int irq_queues, poll_queues;
	unsigned int flags = PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY;
 
	poll_queues = min(dev->nr_poll_queues, nr_io_queues - 1);
	dev->io_queues[HCTX_TYPE_POLL] = poll_queues;
	irq_queues = 1;
	if (!(dev->ctrl.quirks & NVME_QUIRK_SINGLE_VECTOR))
		irq_queues += (nr_io_queues - poll_queues);
	return pci_alloc_irq_vectors_affinity(pdev, 1, irq_queues, flags, &affd);
}

.pre_vectors = 1 reserves vector 0 for the admin queue; PCI_IRQ_AFFINITY asks the IRQ subsystem to spread the rest across CPUs so each queue’s interrupt is steered to the CPU that owns that queue. Polled queues are subtracted from the interrupt count — they have no IRQ. Each queue requests its handler with queue_request_irq() → pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq, ...), naming the IRQ nvme%dq%d (controller, queue) so you can see per-queue interrupt counts in /proc/interrupts. The handler is tiny:

static irqreturn_t nvme_irq(int irq, void *data)
{
	struct nvme_queue *nvmeq = data;
	DEFINE_IO_COMP_BATCH(iob);
	if (nvme_poll_cq(nvmeq, &iob)) {
		if (!rq_list_empty(iob.req_list))
			nvme_pci_complete_batch(&iob);
		return IRQ_HANDLED;
	}
	return IRQ_NONE;
}

It just reuses nvme_poll_cq() — the same completion loop the poll path uses — and batches the completions (io_comp_batch) so blk-mq finishes many requests with one call. Interrupt and poll share one drain routine; only the trigger differs.

Mapping NVMe Queues onto blk-mq Hardware Queues

This is where the two notes meet. blk-mq divides a device into hardware queues (blk_mq_hw_ctx, “hctx”) fed by per-CPU software queues — see Software and Hardware Queues in blk-mq. NVMe maps one I/O queue to one hctx. The binding is set at init:

static int nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
			  unsigned int hctx_idx)
{
	struct nvme_dev *dev = to_nvme_dev(data);
	struct nvme_queue *nvmeq = &dev->queues[hctx_idx + 1];   /* +1 skips admin (qid 0) */
	hctx->driver_data = nvmeq;
	return 0;
}

hctx->driver_data = nvmeq is the entire 1:1 mapping: hardware context i is NVMe I/O queue i+1 (the +1 skips the admin queue). That is why nvme_queue_rq() can recover its queue with one pointer load, lock-free, and why submission scales linearly with cores. The CPU-to-hctx side of the map is nvme_pci_map_queues():

static void nvme_pci_map_queues(struct blk_mq_tag_set *set)
{
	struct nvme_dev *dev = to_nvme_dev(set->driver_data);
	int i, qoff, offset = queue_irq_offset(dev);
	for (i = 0, qoff = 0; i < set->nr_maps; i++) {
		struct blk_mq_queue_map *map = &set->map[i];
		map->nr_queues = dev->io_queues[i];
		map->queue_offset = qoff;
		if (i != HCTX_TYPE_POLL && offset)
			blk_mq_pci_map_queues(map, to_pci_dev(dev->dev), offset);
		else
			blk_mq_map_queues(map);   /* poll queues: plain CPU map */
		qoff += map->nr_queues;
		offset += map->nr_queues;
	}
}

For interrupt-driven queue types it calls blk_mq_pci_map_queues(), which builds the CPU→queue map from the MSI-X affinity — so the CPU that an I/O queue’s interrupt is steered to is the same CPU that blk-mq assigns to that hctx. Submission and completion land on the same core: best cache locality, no cross-CPU bouncing. The deep treatment of this is in blk-mq CPU to Queue Mapping. blk-mq was, in fact, designed to mirror NVMe’s queue-pair model — the per-CPU-queue architecture of blk-mq exists precisely so a device like NVMe with one queue per core has nothing to fight over (blk-mq docs).

Queue-Count Negotiation, Queue Types, and Polled Queues

How many I/O queues exist is negotiated. The driver requests up to nvme_max_io_queues() = num_possible_cpus() + nr_write_queues + nr_poll_queues, then nvme_set_queue_count() issues nvme_admin_set_features (Number of Queues) and the controller returns how many it will actually grant. The driver creates that many with nvme_create_io_queues().

NVMe defines three blk-mq queue types (HCTX_MAX_TYPES), tracked in dev->io_queues[]:

  • HCTX_TYPE_DEFAULT — general I/O (and writes when a read set exists).
  • HCTX_TYPE_READ — an optional dedicated set for reads, enabled with the write_queues module parameter (it carves out separate read queues so heavy writes do not delay reads).
  • HCTX_TYPE_POLL — queues with no interrupt, driven by busy-polling, enabled with the poll_queues module parameter.

nvme_calc_irq_sets() divides the available MSI-X vectors between DEFAULT and READ; POLL queues get none. A polled queue carries the NVMEQ_POLLED flag and is created with vector = 0 and no IRQ. blk-mq drives it through .poll:

static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
{
	struct nvme_queue *nvmeq = hctx->driver_data;
	if (!nvme_cqe_pending(nvmeq))
		return 0;
	spin_lock(&nvmeq->cq_poll_lock);
	bool found = nvme_poll_cq(nvmeq, iob);
	spin_unlock(&nvmeq->cq_poll_lock);
	return found;
}

A thread doing preadv2(..., RWF_HIPRI) or io_uring with polling spins in nvme_poll() instead of sleeping for an interrupt. For ultra-low-latency workloads this removes the interrupt-delivery and context-switch cost from the completion path entirely — at the price of burning a CPU. See Polled IO and Hybrid Polling. The default queue depth is 1024 entries (module parameter io_queue_depth), bounded by NVME_PCI_MIN_QUEUE_SIZE = 2 and NVME_PCI_MAX_QUEUE_SIZE = 4095, and further capped by the controller’s CAP.MQES (Maximum Queue Entries Supported, NVME_CAP_MQES(cap) = cap & 0xffff).

Failure Modes and Diagnosis

  • Command timeout / I/O timeout. If a CQE never arrives, blk-mq’s timeout fires nvme_timeout(), which aborts the command and, if that fails, resets the controller (dmesg: nvme nvmeX: I/O N QID Q timeout, aborting). A flood of these on one QID points at a specific queue/CPU or a wedged controller.
  • Phase-tag confusion after reset. If cq_phase is not re-initialized to 1 and cq_head to 0 on queue re-creation, the host either misses real completions or processes stale ones — a classic driver bug class. The fix is exactly the cq_head = 0; cq_phase = 1; in nvme_alloc_queue().
  • Missed doorbell / lost wakeup. Batching (bd->last) is correct only if the last request of a batch always rings; a bug that drops the final doorbell hangs all queued commands. The last_sq_tail bookkeeping exists to get this right.
  • All interrupts on CPU 0. If MSI-X affinity is not spread (BIOS/quirk NVME_QUIRK_SINGLE_VECTOR, or nvme.use_threaded_interrupts), completions pile onto one core and throughput collapses. Check /proc/interrupts for the nvmeXqY lines.

Alternatives and Contrast

The contrast that illuminates the design is virtio-blk: virtio reaches the same shape — descriptor rings in guest memory, a “kick” (doorbell-equivalent) to notify the backend, an interrupt to signal used buffers (see virtio Notifications and Virtqueue Kicks) — but the ring format is virtio’s virtqueue, not NVMe’s SQ/CQ, and the “device” is a hypervisor. The older SCSI model had a single deep request queue per device and a thick translation mid-layer; it never exposed per-core hardware queues, which is why it could not feed a million-IOPS SSD. NVMe’s queue-pair-per-core model, mirrored by blk-mq, is the structural reason modern Linux storage scales.

Production Notes

In production the levers that matter are queue count and queue type. Default behavior (one I/O queue per CPU, interrupt-driven, depth 1024, scheduler none) is right for almost everyone — the design is self-tuning. The poll_queues parameter is the knob for latency-critical databases willing to spend CPU on busy-polling; write_queues (the READ set) helps mixed read/write workloads where writes were starving reads. For diagnosis, /proc/interrupts shows the per-queue MSI-X spread, cat /sys/block/nvme0n1/mq/*/ exposes the blk-mq hardware-queue layout, and the block tracepoints plus nvme-cli error-log bracket the problem between the block layer and the controller. The single most important takeaway: there is no global lock anywhere in the NVMe submission or completion path — every contention point was designed out, which is exactly what flash speeds demanded.

See Also

Graph View

Backlinks