vault

vhost-user and vDPA

15 min read

vhost-user and vDPA

vhost-user and vDPA are the two evolutions of vhost that take the virtio datapath out of the host kernel entirely. vhost-user moves the backend into another userspace process — a Data Plane Development Kit (DPDK) switch, Open vSwitch with DPDK (OVS-DPDK), or the Storage Performance Development Kit (SPDK) — which talks to the virtual-machine monitor (VMM) over a UNIX domain socket and is handed the guest’s memory and notification eventfds as file descriptors; the per-packet path then runs entirely in that sidecar with the guest’s virtqueues mapped into its address space, never touching the kernel (QEMU, Vhost-user Protocol). vDPA (vhost Data Path Acceleration) goes one step further: it is a kernel framework for hardware whose ring layout is virtio-compliant but whose control path is vendor-specific, so a physical NIC’s virtual function can DMA directly into the guest’s virtqueues at SR-IOV-class speed while the control plane stays in software — preserving virtio portability and, crucially, live migration, which raw SR-IOV/VFIO passthrough cannot do (Red Hat, Introduction to vDPA kernel framework). The unifying idea is that the guest always sees the same virtio device; only the backend — userspace sidecar, or NIC silicon — changes.

Both build on the same vhost contract from vhost (In-Kernel virtio Backend): feature negotiation, a guest-memory table, ring addresses, and two eventfds (kick in, call out). What differs is where the ring is serviced. This note covers the vhost-user UNIX-socket protocol and its DPDK/SPDK consumers, the vdpa bus and subsystem, the vhost-vdpa and virtio-vdpa consumption paths, and VDUSE (a software vDPA device implemented in userspace).

Mental Model

vhost progressively pushes the datapath outward — from the VMM thread, to a kernel thread (classic vhost), to a sidecar process (vhost-user), to NIC hardware (vDPA). The guest’s virtio driver is unchanged at every step.

flowchart TB
  subgraph GUEST["Guest (unmodified virtio driver)"]
    VQ["virtqueues in guest RAM"]
  end
  subgraph A["vhost-user: backend in another process"]
    SOCK["UNIX socket<br/>(control: features, fds)"]
    SIDE["DPDK / OVS-DPDK / SPDK<br/>poll-mode, kernel-bypass"]
    SOCK --> SIDE
  end
  subgraph B["vDPA: backend in hardware"]
    BUS["vdpa bus (kernel)"]
    HW["virtio-compliant NIC VF<br/>(DMAs into guest virtqueues)"]
    BUS --> HW
  end
  VQ -->|"shared via SCM_RIGHTS memory fds"| SIDE
  VQ -->|"ring addr programmed by set_vq_address"| HW
  SIDE -. "control plane only" .-> GUEST
  HW -. "doorbell + interrupt" .-> GUEST

The two backend relocations. What it shows (top): in vhost-user the guest’s virtqueue memory is shared into a sidecar process via file descriptors passed over a UNIX socket, and that process — a poll-mode DPDK/SPDK dataplane — services the rings without ever entering the kernel. (Bottom): in vDPA the kernel vdpa bus programs a real NIC virtual function with the guest’s ring addresses, and the hardware DMAs directly into those rings. The insight to take: the control plane (feature bits, ring setup, migration state) is always software; only the data plane moves — to a process you can crash-isolate (vhost-user) or to silicon you cannot customize (vDPA). The guest cannot tell the difference because the ring layout is identical virtio in all cases.

vhost-user: the backend in another process

The classic vhost design (previous note) services the ring in a host kernel thread. vhost-user replaces the kernel backend with a userspace process and the ioctl() control interface with a message protocol over a UNIX domain socket. The QEMU spec is explicit: “This protocol is aiming to complement the ioctl interface used to control the vhost implementation in the Linux kernel. It implements the control plane needed to establish virtqueue sharing with a user space process on the same host” (QEMU, Vhost-user Protocol).

The two parties are the frontend (the VMM — QEMU, Cloud Hypervisor) and the backend (the external process). They “use communication over a Unix domain socket to share file descriptors in the ancillary data of the message.” That phrase — file descriptors in the ancillary data — is the whole mechanism. Over SCM_RIGHTS (the sendmsg() facility for passing open fds between processes), the frontend hands the backend three things:

  1. Guest memory. VHOST_USER_SET_MEM_TABLE sends “a list of vhost memory regions … Each region has two base addresses: a guest address and a user address,” and “In the ancillary data there is an array of file descriptors for each memory mapped region.” The backend mmap()s those fds and now sees the same physical pages of guest RAM the guest does — which is why the guest’s memory must be allocated as shareable. With QEMU, “the memory must be specified with share=on,” and the instance memory is allocated as shared hugepages so both QEMU and the backend (and the guest’s virtio ring) map the same region (Cloud Hypervisor docs, vhost-user-blk testing).

  2. The kick eventfd. VHOST_USER_SET_VRING_KICK passes the eventfd the guest signals when it adds buffers — the same ioeventfd KVM signals on the guest’s notify write. The backend polls it (or busy-loops) to learn there is work.

  3. The call eventfd. VHOST_USER_SET_VRING_CALL passes the eventfd the backend signals to inject the guest interrupt — the same irqfd KVM consumes. The backend writes it after placing completions in the used ring.

Feature negotiation mirrors the kernel ioctls: VHOST_USER_GET_FEATURES / VHOST_USER_SET_FEATURES exchange the virtio feature bitmask, and VHOST_USER_GET_PROTOCOL_FEATURES / SET_PROTOCOL_FEATURES negotiate vhost-user-protocol-level capabilities (multi-queue, in-band notifications, etc.) (QEMU spec). Once setup completes the per-packet loop is guest kick → ioeventfd → backend wakes (or is already polling) → backend reads ring in shared memory → backend signals call eventfd → KVM injects IRQ — the kernel is on the path only to relay the two eventfds and the doorbell trap; the data copy itself happens entirely in the sidecar’s address space.

The DPDK / OVS-DPDK / SPDK consumers

vhost-user exists because of poll-mode, kernel-bypass dataplanes:

  • DPDK provides a vhost library that implements the backend side; an application built on it becomes a vhost-user backend process.
  • OVS-DPDK (Open vSwitch with the DPDK datapath) exposes dpdkvhostuser / dpdkvhostuserclient ports. As Red Hat’s protocol write-up puts it, “OVS DPDK can directly read from and write into the instance’s virtio ring,” so “both OVS DPDK and QEMU can directly exchange packets across this reserved memory section” with no kernel networking stack in the path (Red Hat KB 3394851, A detailed view of the vhost user protocol).
  • SPDK is the storage analogue. It “move[s] all necessary drivers into userspace and operat[es] in a polled mode instead of relying on interrupts,” and its vhost target implements vhost-user-blk / vhost-user-scsi backends. SPDK requires the VM memory be share=on hugepages for exactly the reason above (SPDK, vhost Target).

The price of this design is dedicated CPU: a poll-mode backend pins cores and busy-spins on the rings rather than blocking on the eventfd, trading CPU for latency. The win is line-rate switching/storage with full kernel bypass and fault isolation — a crashing DPDK/SPDK sidecar takes down only that dataplane, not the host kernel, which is the key safety advantage over in-kernel vhost.

vDPA: the backend in hardware

vDPA generalizes vhost to real devices. Red Hat’s definition is the precise one: “A ‘vDPA device’ means a type of device whose datapath complies with the virtio specification, but whose control path is vendor specific” (Red Hat, Introduction to vDPA kernel framework). In plain terms: the NIC vendor builds silicon that lays out its descriptor rings exactly like virtio, so a guest running an unmodified virtio-net driver can drive it; but the device’s setup/config registers are vendor-specific, so a small kernel driver translates the standard virtio control operations into vendor pokes.

The payoff is the best of both prior worlds. From SR-IOV passthrough it inherits raw speed — “The virtio data plane is mapped directly from the guest application to the VF in the physical NIC,” so the hardware DMAs straight into the guest’s virtqueues at near-native throughput. From virtio it inherits portability (the guest needs no vendor driver) and, decisively, live migration: “In order to support live migration, the framework supports saving and restoring device state via the existing vhost API” — the same VHOST_GET/SET_VRING_BASE (last_avail_idx) and get_vq_state/set_vq_state checkpoint that classic vhost uses. Raw VFIO/SR-IOV passthrough exposes opaque vendor state that the host cannot snapshot, which is why passthrough cannot migrate; vDPA’s standardized virtio state can be (see Dirty Page Tracking and Live Migration).

Safety is also better than passthrough. The framework mediates the control plane and “will not allow direct hardware register mapping except for the doorbell registers (used for notifying the HW on work to do)” (Red Hat, vDPA intro). Only the doorbell — the single performance-critical write — is mapped to the guest; everything else goes through the kernel mediation layer.

The vdpa bus and subsystem

vDPA is structured as a proper Linux bus. In v6.12 the core registers a bus_type literally named "vdpa":

static const struct bus_type vdpa_bus = {
    .name       = "vdpa",
    .dev_groups = vdpa_dev_groups,
    .match      = vdpa_dev_match,
    .probe      = vdpa_dev_probe,
    .remove     = vdpa_dev_remove,
};

(drivers/vdpa/vdpa.c, lines 118–124). A parent driver for specific hardware (e.g. a Mellanox/NVIDIA ConnectX mlx5_vdpa, an Intel IFC VF, or the software vdpa_sim) allocates a struct vdpa_device with __vdpa_alloc_device() and registers it on this bus with vdpa_register_device() (lines 157, 258). The device carries a const struct vdpa_config_ops * — the contract every vDPA device must implement.

Those ops are exactly the virtio control plane, expressed as callbacks (include/linux/vdpa.h):

  • set_vq_address — tell the device where this virtqueue’s descriptor/avail/used rings live (in guest IOVA). This is what makes the hardware DMA into the guest’s rings.
  • set_vq_num, set_vq_ready/get_vq_ready — ring size and enable.
  • kick_vq (and kick_vq_with_data when VIRTIO_F_NOTIFICATION_DATA is negotiated) — the doorbell (lines 150–160).
  • set_vq_cb — install the interrupt callback for a virtqueue (line 161–165).
  • set_vq_state / get_vq_state — read/write last_avail_idx, “the state for a virtqueue” (lines 174–182). This pair is the live-migration primitive: snapshot the ring position on the source, restore it on the destination.
  • get_device_features / set_driver_features, get/set_status, reset, get_config/set_config — the standard virtio feature/status/config-space handshake (lines 222–259, 280–294).
  • set_map / dma_map / dma_unmap — program the device’s DMA translation (its on-chip IOMMU or a host IOMMU domain), so the device can only reach the guest’s pages (lines 317–345).

Two consumers: virtio_vdpa and vhost_vdpa

The same registered vDPA device can be claimed by one of two bus drivers, selected via the driver_override sysfs attribute the bus exposes (vdpa.c, lines 80–106):

  • virtio_vdpa binds the vDPA device into the in-kernel virtio stack, presenting it as an ordinary virtio_device so a host kernel driver (or a container, XDP program, etc.) uses it. It implements a virtio_config_ops (virtio_vdpa_config_ops) that forwards every virtio operation to the underlying vdpa_config_ops, then calls register_virtio_device() (drivers/virtio/virtio_vdpa.c, lines 465–513). Red Hat: “For kernel virtio drivers the vDPA framework will present a virtio device.”

  • vhost_vdpa exposes the device to a userspace VMM as a vhost character device, /dev/vhost-vdpa-N. Red Hat: “For userspace drivers, vDPA framework will present a vhost char device.” QEMU opens it and drives it with the same vhost ioctls plus vDPA extensions: VHOST_VDPA_GET_DEVICE_ID, VHOST_VDPA_GET/SET_STATUS, VHOST_VDPA_GET/SET_CONFIG, VHOST_VDPA_SET_VRING_ENABLE, VHOST_VDPA_GET_IOVA_RANGE, and the migration-oriented VHOST_VDPA_SUSPEND / VHOST_VDPA_RESUME (which “preserve all the necessary state … required for restoring in the future”) (include/uapi/linux/vhost.h, lines 149–237). The dispatch in vhost_vdpa_vring_ioctl() simply forwards each ioctl to the device’s config_ops, e.g. VHOST_VDPA_SET_VRING_ENABLE → ops->set_vq_ready(vdpa, idx, s.num) (drivers/vhost/vdpa.c, lines 655–658). It also manages IOMMU mappings: struct vhost_vdpa holds an iommu_domain *domain and per-address-space vhost_iotlb tables (lines 45–60), and supports multiple address spaces (ASIDs) via VHOST_VDPA_GET_AS_NUM / VHOST_VDPA_SET_GROUP_ASID for separating descriptor-table access from buffer access (vhost.h, lines 182–197).

This dual-consumer design is the elegant core of vDPA: one hardware abstraction, consumed either by the kernel’s own virtio stack or by a VMM, with the choice made at runtime by binding.

VDUSE: a software vDPA device implemented in userspace

vDPA does not require hardware. VDUSE (vDPA Device in Userspace) lets a userspace daemon implement a vDPA device, so the rest of the stack treats it like real vDPA hardware. The v6.12 documentation states: “the emulated vDPA device’s control path is handled in the kernel and only the data path is implemented in the userspace,” and “Currently, only virtio block device is supported” (Documentation/userspace-api/vduse.rst).

The daemon opens /dev/vduse/control, creates a device with ioctl(VDUSE_CREATE_DEV), sets up virtqueues, then services a message stream from the kernel over /dev/vduse/$NAME: VDUSE_GET_VQ_STATE (return the avail index — the migration checkpoint again), VDUSE_SET_STATUS (handle a virtio status change), and VDUSE_UPDATE_IOTLB (“userspace to update the memory mapping for specified IOVA range”). For data, VDUSE_IOTLB_GET_FD returns a file descriptor the daemon mmap()s to reach guest memory. Finally the device is attached to the vdpa bus over netlink, after which virtio_vdpa or vhost_vdpa can bind it like any other vDPA device.

VDUSE’s stated security goal is to “reduce[] security risks when the userspace process that implements the data path is run by an unprivileged user” — guest memory is reached only through the controlled IOTLB fds, and DMA is brokered through a bounce-buffer/IOVA domain rather than raw physical access. It is, in effect, the software counterpart to a hardware vDPA NIC: a userspace virtio-blk backend that the kernel and VMM see as a vDPA device.

Failure Modes and Common Misunderstandings

“vhost-user needs no special guest memory.” Wrong — and a frequent first-time failure. The backend can only see guest RAM if it is allocated shareable. With QEMU you must use -object memory-backend-file,share=on (or memory-backend-memfd,share=on) and typically hugepages; omitting share=on yields a backend that maps nothing and a VM whose vhost-user NIC/disk silently does no I/O (Cloud Hypervisor / SPDK docs; SPDK vhost).

“vDPA is just SR-IOV with extra steps.” No. SR-IOV/VFIO passthrough exposes the vendor’s device interface to the guest (needs a vendor driver, opaque migration state). vDPA exposes a virtio interface (stock guest driver) and standardizes the migration state, so live migration works and the guest is portable across vendors. The throughput is comparable because both DMA into the guest’s rings; the difference is the control-plane abstraction.

vhost-user reconnection semantics. Because the backend is a separate process, it can crash and restart. The protocol supports the frontend acting as the socket server so the backend can reconnect, but in-flight ring state can be lost across a restart unless the backend persists last_avail_idx. A backend that does not handle reconnection cleanly manifests as a VM whose device hangs after a sidecar restart.

ASID/IOVA confusion in vhost-vdpa. Multi-address-space devices separate the IOVA space used for the descriptor table from that used for buffers (VHOST_BACKEND_F_DESC_ASID, vhost_types.h line 188–192). Misconfiguring ASIDs causes the device to read descriptors or buffers from the wrong translation and fault. This matters for confidential VMs where descriptor and buffer memory have different protection.

vDPA software simulators are not for production. vdpa_sim / VDUSE are invaluable for testing the stack without hardware, but VDUSE in v6.12 supports only virtio-blk and its performance is bounded by the userspace daemon — do not mistake “vDPA works on my laptop” for hardware-accelerated vDPA.

Alternatives and When to Choose Them

  • In-kernel vhost — datapath in a host kernel thread; choose when the endpoint is something the kernel already speaks (tap on a bridge, LIO target) and you do not want to dedicate cores to polling.
  • vhost-user — datapath in a userspace sidecar; choose for poll-mode kernel-bypass throughput (DPDK switching, SPDK storage) and when crash isolation of the dataplane matters. Costs dedicated CPU and shared-hugepage setup.
  • vDPA — datapath in hardware with software control; choose for SR-IOV-class throughput plus virtio portability plus live migration. Costs vDPA-capable hardware (or a VDUSE software device).
  • Raw SR-IOV / VFIO passthrough — maximum speed, no mediation, but a vendor driver in the guest and no live migration. Choose only when migration is genuinely unneeded.

The axis remains where the ring is serviced vs how much isolation, portability, and migratability you keep. vDPA is the point that refuses the usual trade-off: hardware speed and virtio portability and migration, at the cost of needing the right silicon.

Production Notes

vhost-user is the workhorse of NFV (Network Function Virtualization) and high-performance cloud storage: OVS-DPDK + vhost-user is the standard way OpenStack/telco deployments give VMs line-rate networking, and SPDK vhost-user-blk/scsi backs low-latency virtualized storage; Kata Containers can attach SPDK vhost-user block devices to VM-isolated pods (SPDK vhost; Red Hat KB 3394851). vDPA is younger but production-deployed: NVIDIA/Mellanox mlx5_vdpa and Intel vDPA NICs offer virtio-net offload with migration, and Red Hat has driven much of the upstream framework (the vdpa.c copyright credits Intel and Red Hat, with Jason Wang as a primary author — drivers/vhost/vdpa.c, lines 1–12). The strategic narrative across the industry is convergence: vhost-user, classic vhost, and hardware vDPA all expose the same virtio device to the guest, so a workload can move between a software dataplane and a hardware-offloaded one without the guest ever knowing — which is the entire point of standardizing the datapath on virtio.

Uncertain

Verify: the precise upstream kernel release each individual vDPA feature/op landed in (e.g. multi-ASID, VHOST_VDPA_SUSPEND/RESUME, get_vq_desc_group). Reason: this note pins the presence and shape of these ioctls and config_ops to the v6.12 source it read, but it does not pin the introduction release of each — and vDPA is fast-moving, with ops added across many releases. To resolve: git log --follow each symbol on git.kernel.org for the exact merge tag before making any “since 6.x” claim. The mechanism described here is verified against v6.12; only the per-feature introduction dates are unpinned. uncertain

See Also

Graph View

Backlinks