vault

The Virtqueue and Vring Layout

16 min read

The Virtqueue and Vring Layout

A virtqueue is the shared-memory transport at the heart of virtio — the lock-free, producer/consumer queue through which a guest driver hands I/O buffers to a host device and gets them back. It is what lets virtio batch I/O and avoid a VM exit per operation: the driver writes buffer descriptors into memory both sides can see, and notifies the device only when it must. The on-wire memory layout is the vring (“virtio ring”). There are two layouts. The split virtqueue — the classic design since the first virtio — splits the queue into three separately-located arrays: a descriptor table, an available ring (driver→device), and a used ring (device→driver). The newer packed virtqueue (added in virtio 1.1, negotiated via VIRTIO_F_RING_PACKED) folds all three into a single descriptor ring, using per-descriptor flag bits and a ring-wrap counter so both sides track ownership in one cache-friendly array. This note walks the actual struct vring_desc/vring_avail/vring_used byte layouts from the kernel UAPI and the OASIS virtio 1.3 specification, plus descriptor chaining and indirect tables. How the notification doorbell and interrupt-suppression work is the sibling note virtio Notifications and Virtqueue Kicks.

Mental Model

Think of a split virtqueue as a valet system with two clipboards and a parking lot. The descriptor table is the parking lot: a fixed array of slots, each describing one buffer (its guest-physical address, its length, and whether the device may write to it). The available ring is the driver’s clipboard: “slots 3, 7, and 12 are ready for you, device.” The used ring is the device’s clipboard back: “I’m done with the chain that started at slot 3, and I wrote 1500 bytes into it.” Each side owns one clipboard for writing and reads the other’s. Crucially, the two rings are separate cache lines / separate pages, so the producer and consumer rarely touch the same memory — that separation is the whole point of the “split” design and why it scales.

flowchart LR
  subgraph GUEST["Guest driver (producer of 'available')"]
    AVAIL["available ring (vring_avail)<br/>flags, idx, ring[]"]
  end
  subgraph SHARED["Descriptor table (vring_desc[N])"]
    D0["desc[3]: addr,len,flags,next"]
    D1["desc[7]: addr,len,flags,next"]
    D2["desc[12]: addr,len,flags,next"]
  end
  subgraph HOST["Host device (producer of 'used')"]
    USED["used ring (vring_used)<br/>flags, idx, ring[{id,len}]"]
  end
  AVAIL -->|"publishes head indices"| SHARED
  SHARED -->|"device reads/writes buffers"| HOST
  HOST -->|"publishes consumed (id,len)"| USED
  USED -.->|"driver reclaims buffers"| GUEST

The split virtqueue. What it shows: the driver adds buffers by filling descriptor slots and pushing their head indices onto the available ring; the device consumes them and pushes completions (descriptor id + bytes written) onto the used ring. The insight to take: there are three independent memory regions, and ownership flows one direction through each — driver writes descriptors and the available ring, device writes the used ring. No locks are needed because each idx field is a monotonically-increasing counter written by exactly one side.

Why a Ring At All

The motivating problem is the cost of crossing the guest/host boundary (covered in virtio Device Model). If the driver had to signal the host once per buffer, throughput would collapse under VM exits. A ring lets the driver enqueue many buffers and signal once; it lets the device dequeue many and complete many before raising one interrupt. The ring also decouples the two sides temporally — the driver can keep adding while the device keeps consuming, as long as there is space. The data structures are designed so that the only synchronization is a pair of free-running 16-bit indices (avail->idx, used->idx) plus memory barriers; there is no lock, and the structures were deliberately frozen for ABI stability — the kernel header warns “Do NOT change this since it will break existing servers and clients” (virtio_ring.h, v6.12).

The Descriptor: struct vring_desc

The fundamental unit is the descriptor, a 16-byte structure describing one contiguous buffer (virtio_ring.h, v6.12):

struct vring_desc {
        __virtio64 addr;   /* buffer address (guest-physical) */
        __virtio32 len;    /* buffer length in bytes          */
        __virtio16 flags;  /* descriptor flags (see below)    */
        __virtio16 next;   /* index of next descriptor, if F_NEXT set */
};

Walking the fields: addr is the guest-physical address of the buffer (or, under VIRTIO_F_ACCESS_PLATFORM, a DMA address the IOMMU translates). len is the buffer’s size. flags is a bitmask, and next chains descriptors. The descriptor table is an array of vring_desc[Queue Size], 16-byte-aligned (VRING_DESC_ALIGN_SIZE = 16), so a 256-entry queue’s descriptor table is exactly 4096 bytes — one page. The __virtio16/32/64 types encode the spec’s endianness rule: modern (1.0+) devices use little-endian, but the type abstracts over the transitional legacy byte order.

The three flag bits (virtio_ring.h, v6.12):

#define VRING_DESC_F_NEXT     1  /* buffer continues via the `next` field      */
#define VRING_DESC_F_WRITE    2  /* device-write-only buffer (else read-only)  */
#define VRING_DESC_F_INDIRECT 4  /* buffer is itself a table of descriptors    */

VRING_DESC_F_NEXT (the spec’s VIRTQ_DESC_F_NEXT) marks a descriptor as part of a chain: the next field holds the array index of the following descriptor, letting a single logical request scatter across many physical buffers. VRING_DESC_F_WRITE flips the direction: cleared means the buffer is device-readable (driver→device data, e.g. a packet to transmit or a write request’s payload); set means device-writable (device→driver data, e.g. a received packet or a read request’s destination). The device must not write to a read-only descriptor. VRING_DESC_F_INDIRECT is covered below.

Descriptor Chaining

A real I/O request rarely fits one buffer. A SCSI command has a header, then data, then a status byte; a network packet has a virtio-net header then the frame. The driver expresses this as a chain: descriptor 0 has F_NEXT set and next = 5; descriptor 5 has F_NEXT set and next = 9; descriptor 9 has F_NEXT clear, ending the chain. The convention is read-only descriptors first, then write-only (all F_WRITE-clear before all F_WRITE-set), so the device knows where its input ends and its output begins. Only the head index of the chain is published to the available ring; the device follows next pointers to traverse the whole chain. The kernel even chains unused (free) descriptors via the same next field to maintain its free list (per the header comment: “We chain unused descriptors via this, too”).

The Available Ring: Driver → Device

The available ring is how the driver tells the device which descriptor chains are ready (virtio_ring.h, v6.12):

struct vring_avail {
        __virtio16 flags;     /* e.g. VRING_AVAIL_F_NO_INTERRUPT */
        __virtio16 idx;       /* where the driver will put the next entry */
        __virtio16 ring[];    /* ring[i] = head descriptor index */
        /* (le16 used_event follows, only if VIRTIO_F_EVENT_IDX) */
};

idx is a free-running 16-bit counter of how many entries the driver has ever made available; it wraps naturally at 65536. To find the actual slot, the device computes idx % Queue Size. To add a buffer, the driver writes the head descriptor index into ring[idx % QueueSize], issues a write barrier, then increments idx. The device reads idx, and any entries between its last-seen position and the new idx are freshly available. flags carries VRING_AVAIL_F_NO_INTERRUPT = 1 — the driver’s hint “don’t interrupt me when you consume a buffer” (an optimization; see virtio Notifications and Virtqueue Kicks). The available ring is 2-byte aligned (VRING_AVAIL_ALIGN_SIZE = 2) and its size is 6 + 2 × QueueSize bytes (the spec’s table: flags + idx + the entries).

The Used Ring: Device → Driver

The used ring is the device’s completion path (virtio_ring.h, v6.12):

struct vring_used_elem {
        __virtio32 id;   /* head index of the used descriptor chain */
        __virtio32 len;  /* total bytes the device wrote into it    */
};
 
struct vring_used {
        __virtio16 flags;            /* e.g. VRING_USED_F_NO_NOTIFY */
        __virtio16 idx;              /* where the device put the next completion */
        vring_used_elem_t ring[];    /* completions */
        /* (le16 avail_event follows, only if VIRTIO_F_EVENT_IDX) */
};

When the device finishes a chain, it writes a vring_used_elem to ring[used->idx % QueueSize] containing the chain’s head id and len, the total number of bytes written into the device-writable portion — critical for variable-length results like a received packet whose size the driver could not know in advance. Then it increments used->idx. The driver polls or is interrupted, sees used->idx advance, reclaims the descriptors of each completed chain, and frees them back to its descriptor free list. Note id is a 32-bit field even though descriptor indices are 16-bit — the header comment notes “u32 is used here for ids for padding reasons” (alignment). flags carries VRING_USED_F_NO_NOTIFY = 1, the device’s hint “don’t kick me when you add a buffer.” The used ring is 4-byte aligned (VRING_USED_ALIGN_SIZE = 4), size 6 + 8 × QueueSize.

The full split-ring memory layout, with the legacy single-contiguous-region arrangement, is documented in the header’s comment:

struct vring {
        // The actual descriptors (16 bytes each)
        struct vring_desc desc[num];
        // A ring of available descriptor heads with free-running index.
        __virtio16 avail_flags;
        __virtio16 avail_idx;
        __virtio16 available[num];
        __virtio16 used_event_idx;   // for VIRTIO_F_EVENT_IDX
        char pad[];                  // pad to next align boundary
        // A ring of used descriptor heads with free-running index.
        __virtio16 used_flags;
        __virtio16 used_idx;
        struct vring_used_elem used[num];
        __virtio16 avail_event_idx;  // for VIRTIO_F_EVENT_IDX
};

Two design subtleties hide here. First, the used_event index lives at the end of the available ring and avail_event lives at the end of the used ring — a deliberate cross-placement so each side’s event index sits in memory it controls; the helper macros vring_used_event(vr) and vring_avail_event(vr) index past the ring arrays to reach them. Second, in modern (1.0+) virtio the three regions are independently located, not in one contiguous block — the driver tells the transport each region’s address separately (queue_desc_lo/hi, queue_avail_lo/hi, queue_used_lo/hi in the PCI common config, per virtio Device Model). The single-blob vring_init()/vring_size() helpers describe the legacy layout where all three were packed together with page alignment.

The Packed Virtqueue: One Ring to Rule Them All

The split ring has a cache-locality problem: completing one request touches three separate cache lines (descriptor, available entry, used entry), often on different pages, causing cache misses and extra reads across the guest/host boundary. The packed virtqueue, introduced in virtio 1.1 and negotiated via VIRTIO_F_RING_PACKED (feature bit 34), collapses everything into a single descriptor ring so a request and its completion reuse the same cache line.

The packed descriptor is also 16 bytes but laid out differently (virtio_ring.h, v6.12):

struct vring_packed_desc {
        __le64 addr;   /* buffer address  */
        __le32 len;    /* buffer length   */
        __le16 id;     /* buffer ID (returned in the completion) */
        __le16 flags;  /* AVAIL/USED ownership bits + NEXT/WRITE/INDIRECT */
};

There is no separate available or used ring. Instead, ownership is encoded in two flag bits of each descriptor, defined as shifts (virtio_ring.h, v6.12):

#define VRING_PACKED_DESC_F_AVAIL  7   /* bit 7  */
#define VRING_PACKED_DESC_F_USED   15  /* bit 15 */

The trick is the ring-wrap counter. Both driver and device maintain a single-bit counter, initialized to 1, that flips each time they wrap past the end of the ring. The spec: “To mark a descriptor as available, the driver sets the VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Driver Ring Wrap Counter. It also sets the VIRTQ_DESC_F_USED bit to match the inverse value.” When the device finishes, “the device sets the VIRTQ_DESC_F_USED bit in Flags to match the internal Device Ring Wrap Counter. It also sets the VIRTQ_DESC_F_AVAIL bit to match the same value” (virtio 1.3 §2.8). So a descriptor is available to the device when AVAIL == wrap_counter and USED != wrap_counter; it is used (completed) when AVAIL == USED == wrap_counter. Each side simply reads the next descriptor’s flag bits and compares them to its own wrap counter — no separate index array, no second cache line. The id field (not the ring position) identifies the buffer in the completion, because the device may complete out of order. Event suppression uses small struct vring_packed_desc_event structures (an off_wrap and flags pair) at the Driver/Device areas, with VRING_PACKED_EVENT_F_WRAP_CTR = 15 as the wrap-counter bit. Notably, the packed ring’s Queue Size need not be a power of 2 (unlike split), per the spec.

flowchart TB
  subgraph RING["Single packed descriptor ring"]
    direction LR
    P0["desc[0]<br/>AVAIL=wrap, USED=!wrap<br/>(device can take it)"]
    P1["desc[1]<br/>AVAIL=USED=wrap<br/>(completed)"]
    P2["desc[2]<br/>both = old wrap<br/>(driver owns / free)"]
  end
  DRV2["Driver wrap counter"] -->|"writes AVAIL bit"| RING
  DEV2["Device wrap counter"] -->|"writes USED bit"| RING

Packed-ring ownership via flag bits. What it shows: ownership of each descriptor is read directly from its AVAIL/USED flag bits compared against the reader’s wrap counter, with no separate available/used arrays. The insight to take: the packed ring trades the split ring’s simple free-running indices for a denser, single-cache-line layout — fewer memory accesses per request, which is why modern high-throughput backends prefer it, at the price of a more subtle ownership protocol.

Indirect Descriptors

A virtqueue’s capacity is bounded by its Queue Size — but a single request needing many scattered buffers would consume many descriptor-table slots, starving other in-flight requests. Indirect descriptors (negotiated via VIRTIO_F_INDIRECT_DESC, feature bit 28) solve this. The driver allocates a separate table of descriptors anywhere in memory and places one descriptor in the main ring with VRING_DESC_F_INDIRECT set, whose addr/len point at that table. The device, seeing the flag, treats the referenced memory as struct vring_desc[len / 16] and processes the whole indirect table as the request (virtio 1.3 §2.7.5).

The rules are strict: “The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the VIRTIO_F_INDIRECT_DESC feature was negotiated. The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only one table per descriptor)” (virtio 1.3 §2.7.5). No nesting, no indirection-within-indirection. The net effect: one ring slot can represent an arbitrarily large scatter-gather list, dramatically raising effective queue depth for storage workloads. The trade-off is an extra memory indirection (the device must fetch the table), so for tiny requests inline chaining can be faster — drivers heuristically choose.

Queue Sizing, Notifications, and EVENT_IDX

A virtqueue’s depth is Queue Size, a power of two for split rings, set during setup (queue_size in the PCI common config). The driver discovers the maximum the device supports and may choose a smaller value. Two feature bits live at this layer and recur across notes:

  • VIRTIO_F_EVENT_IDX (bit 29) replaces the coarse NO_INTERRUPT/NO_NOTIFY flags with a threshold: each side publishes an index (used_event/avail_event) saying “only notify me once you reach this point.” The kernel’s vring_need_event(event_idx, new_idx, old) helper implements the comparison (virtio_ring.h, v6.12). This is a major interrupt/kick-rate reducer.
  • VIRTIO_F_RING_RESET (bit 40) lets a single queue be reset and reconfigured without resetting the whole device.

How the actual doorbell write and interrupt delivery work — and how EVENT_IDX suppresses them — is the dedicated sibling virtio Notifications and Virtqueue Kicks; this note covers only the layout those notifications coordinate.

Failure Modes and Common Misunderstandings

Confusing idx (a counter) with a ring position. avail->idx and used->idx are free-running 16-bit counters of total entries ever produced, not array indices. You always compute the slot as idx % QueueSize. Treating idx as a slot index breaks the moment it exceeds QueueSize, and the natural 16-bit wrap is intentional — never mask it to the queue size before comparing.

Forgetting memory barriers. The driver must barrier after writing the descriptor and before incrementing avail->idx, or the device may read a published index pointing at a half-written descriptor. On the device side, read the descriptor only after reading the new idx. Missing barriers cause rare, load-dependent corruption — among the nastiest virtio bugs.

len semantics in the used ring. vring_used_elem.len is bytes the device wrote, not the buffer size. For a virtio-net receive, it tells the driver the actual packet length. Misreading it as the descriptor length truncates or over-reads received data.

Packed-ring wrap-counter desync. If a backend forgets to flip its wrap counter on ring wrap, it will misread ownership and either stall (never see new available descriptors) or double-process. Because there is no index array to cross-check, this is harder to debug than the split ring’s equivalent.

Assuming in-order completion. Unless VIRTIO_F_IN_ORDER (bit 35) is negotiated, the device may complete chains out of order; the used ring’s id (split) or packed descriptor’s id is what reassociates a completion with its request. Code that assumes FIFO completion corrupts state under reordering backends.

Alternatives and When to Choose Them

The split-vs-packed choice is the main dial. Split virtqueues are universally supported, simpler to implement, and the safe default — every virtio implementation since 1.0 speaks them. Packed virtqueues (1.1+) are denser and cache-friendlier, favoured by high-throughput software backends and especially by hardware that implements virtqueues natively (vDPA NICs), because a single ring maps cleanly onto a DMA descriptor ring. Choose packed when both sides support it and the workload is throughput-bound; stick with split for maximum interoperability and simplicity. Orthogonally, indirect descriptors raise effective queue depth for scatter-gather-heavy storage; enable them unless per-request latency on tiny buffers dominates. These are all negotiated — the driver and device fall back to the lowest common layout automatically, so a guest never breaks for asking.

Production Notes

The vring layout is frozen ABI: the header’s “Do NOT change this” warning is load-bearing, because a guest kernel from 2014 must interoperate with a host VMM from 2026 across live migration. This is why new capabilities arrive as new feature bits and new layouts (packed) rather than mutations of the old structures. In practice, default queue sizes are commonly 256 or 1024 entries; storage backends often raise this and enable indirect descriptors to sustain deep I/O. Performance investigations of “why is my VM’s I/O slow” frequently land here: a queue that is too shallow serializes I/O, missing EVENT_IDX floods interrupts, and an absent INDIRECT_DESC caps scatter-gather depth. Tools like QEMU’s info virtio-queue-status and the guest’s /sys/.../virtio*/ attributes expose the negotiated layout; confirming packed-vs-split and the queue size is the first step when a virtio data path underperforms. The same layout is what vhost reads directly from host kernel context — vhost is essentially a kernel-side consumer of these exact rings — and what vhost-user maps into a separate process’s address space.

See Also

Graph View

Backlinks