irqfd and ioeventfd

An eventfd is the kernel’s cheapest counting semaphore between two pieces of code: a 64-bit counter behind a file descriptor, where one side writes to increment it and the other side reads (or poll()s) to consume it (eventfd(2)). KVM bolts two specialized eventfd bindings onto that primitive to get the userspace virtual-machine monitor (VMM) — QEMU, Firecracker, Cloud Hypervisor — out of the datapath. ioeventfd (KVM_IOEVENTFD) registers a guest physical address so that a matching guest MMIO/PIO write signals an eventfd instead of returning to userspace — the guest→host fast path, the cheap end of the virtio “kick.” irqfd (KVM_IRQFD) does the reverse: when some host actor signals an eventfd, KVM injects an interrupt on a guest-visible interrupt line (a GSI) with no userspace round-trip — the host→guest fast path. Together they are the glue that makes vhost, vDPA, and VFIO fast: a packet can travel guest→kernel-backend→guest as ioeventfd → kernel thread → irqfd, and the VMM thread is never scheduled. This note traces both, verified against the Linux 6.12 long-term-support (LTS) kernel (virt/kvm/eventfd.c).

Version pin

Kernel source was read at the Linux v6.12 long-term-support (LTS) tag; mainline has since moved into the 7.x series, and anything that could plausibly have changed is dated in place. QEMU source was read at v9.1.0. Historical performance numbers quoted below come from the original 2009 merge commits, fetched from git.kernel.org, and are dated accordingly — they were measured on 2009 hardware and should be read as ratios, not as absolutes for a modern machine.

A GSI — Global System Interrupt — is KVM’s abstract, controller-independent interrupt-line number; a GSI-to-controller-pin mapping table (the IRQ routing table) says whether GSI N means “IOAPIC pin N” or “an MSI message,” so the same number works whether the guest uses the legacy PIC, the IOAPIC, or message-signalled interrupts. The companion note Interrupt Injection and the Virtual APIC covers what KVM does after the GSI is asserted — how the bit lands in the local APIC and gets into the guest. This note is about the plumbing that decouples the VMM, not the injection mechanics themselves.

Mental Model: Two Doorbells, Neither Rung by the VMM

The whole point is that a virtual device has two doorbells and the VMM should answer neither on the hot path. The guest rings the outbound doorbell by writing a device “notify” register; the device backend rings the inbound doorbell by raising an interrupt. Classic emulation routes both through userspace: the guest write VM-exits into KVM, KVM returns to QEMU, QEMU emulates the register and (later) asks KVM to inject an interrupt via another ioctl. Every step is a context switch. ioeventfd collapses the outbound doorbell into “guest write → KVM signals an eventfd, returns straight to the guest”; irqfd collapses the inbound doorbell into “someone signals an eventfd → KVM asserts a GSI.” The eventfd is the shared rendezvous object that a third party — a vhost kernel thread, a VFIO hardware IRQ handler — holds, so the doorbells now connect the guest directly to that third party.

flowchart LR
  subgraph GUEST["Guest vCPU"]
    GW["MMIO write to<br/>virtio notify reg"]
    GIRQ["receives vIRQ"]
  end
  subgraph KVM["KVM (host kernel)"]
    IOEV["ioeventfd<br/>(addr match on KVM_MMIO_BUS)"]
    IRQF["irqfd<br/>(GSI asserter)"]
  end
  subgraph BACKEND["Host backend (vhost / VFIO / vDPA)"]
    EF1["kick eventfd"]
    WORK["kernel worker /<br/>hardware IRQ"]
    EF2["call eventfd"]
  end
  GW -->|"VM exit, no userspace"| IOEV
  IOEV -->|"eventfd_signal()"| EF1
  EF1 --> WORK
  WORK -->|"eventfd_signal()"| EF2
  EF2 -->|"poll wakeup"| IRQF
  IRQF -->|"kvm_set_irq(gsi)"| GIRQ

The two eventfd doorbells of a virtio fast path. What it shows: the guest’s outbound kick is trapped by an ioeventfd (an entry on KVM’s in-kernel MMIO bus) which signals the backend’s “kick” eventfd; the backend does its work and signals a “call” eventfd registered as an irqfd, which asserts the GSI and injects the interrupt. The insight to take: the VMM (QEMU) appears nowhere in this loop — it only registered the two eventfds at setup time. Both doorbells now cross the virtualization boundary inside the kernel, so the per-request cost is two eventfd signals and one kernel-thread wakeup, not four context switches.

Before the mechanism, a reference card. The two ioctls are mirror images and it helps to see the symmetry laid out before reading either one in detail:

	KVM_IOEVENTFD (guest → host)	KVM_IRQFD (host → guest)
Capability	KVM_CAP_IOEVENTFD	KVM_CAP_IRQFD
Architectures (api.rst, v6.12)	all	x86 s390 arm64
ioctl level	VM	VM
Argument	struct kvm_ioeventfd	struct kvm_irqfd
What userspace names	a guest physical address (+ optional length and value)	a GSI (irqchip pin)
Direction of the eventfd	KVM writes it (eventfd_signal)	KVM watches it (wait-queue callback)
How KVM is attached to the eventfd	holds an eventfd_ctx reference; signals it from ioeventfd_write()	vfs_poll() installs irqfd_wakeup on the eventfd’s wait queue, at priority
Registered as	a kvm_io_device on KVM_MMIO_BUS / KVM_PIO_BUS / KVM_FAST_MMIO_BUS / KVM_VIRTIO_CCW_NOTIFY_BUS	an entry on kvm->irqfds.items
Flags	DATAMATCH, PIO, DEASSIGN, VIRTIO_CCW_NOTIFY	DEASSIGN, RESAMPLE
Multiplexing	several entries at one address, distinguished by datamatch	none — one eventfd per irqfd (-EBUSY otherwise)
Precondition	none beyond a legal address	in-kernel irqchip (-EAGAIN otherwise); full irqchip for RESAMPLE
Cardinality limit	the VMM’s open-fd limit (explicitly exempt from NR_IOBUS_DEVS)	one per (eventfd); many irqfds may share a GSI
Hardware acceleration beyond it	zero-length “fast MMIO” (no instruction decode)	IRQ bypass → VT-d posted interrupts / ARM IRQ forwarding

Both were merged in 2009 by the same author for the same reason: KVM: irqfd (commit 721eecbf, Gregory Haskins, 20 May 2009) landed first, and KVM: add ioeventfd support (commit d34e6b17, 7 July 2009) followed seven weeks later. The irqfd commit message frames the goal generically — “a new mechanism to inject a specific interrupt to a guest using a decoupled eventfd mechanism: Any legal signal on the irqfd (using eventfd semantics from either userspace or kernel) will translate into an injected interrupt in the guest at the next available interrupt window” — and that phrase, from either userspace or kernel, is the whole point. It is what made vhost possible six months later.

What “Avoids a VM Exit” Actually Means — and What It Is Worth

“Avoids a VM exit” is the sentence everybody repeats about these two ioctls, and on its own it is nearly content-free, because there are three different things it could mean and only one of them is true. Getting this straight first makes the rest of the note read as engineering rather than folklore.

A guest memory-mapped I/O (MMIO) write to a trapped address always causes a hardware transition out of guest mode. On Intel VT-x that is a VM exit; on AMD-V an #VMEXIT. Nothing KVM can do prevents it — the trap is how the hypervisor learns the write happened at all. What ioeventfd changes is how far the control transfer goes after that. There are three tiers, and they differ by roughly an order of magnitude each:

Tier	Path	Who runs	Rough cost class
No exit at all	Guest write to a non-trapped page, or interrupt delivered by posted-interrupt hardware	Guest only	Nanoseconds — a memory write
Light exit — into KVM, back to the guest	VM exit → handle_ept_misconfig() / handle_io() → kvm_io_bus_write() → eventfd_signal() → VMRESUME	Host kernel, same thread, same address space	Sub-microsecond to ~1 µs
Heavy exit — all the way to userspace	VM exit → KVM → return from the KVM_RUN ioctl → QEMU decodes and emulates → ioctl(KVM_RUN) again	Host kernel plus the userspace VMM thread, with a scheduler round-trip and two syscall boundaries	Several microseconds

ioeventfd converts a heavy exit into a light one. It does not eliminate the exit. Gregory Haskins’ original merge commit says exactly this, and is worth quoting because it is the primary statement of the design intent (KVM: add ioeventfd support, commit d34e6b17, 7 July 2009):

“Normal IO requires a blocking round-trip since the operation may cause side-effects in the emulated model or may return data to the caller. Therefore, an IO in KVM traps from the guest to the host, causes a VMX/SVM ‘heavy-weight’ exit back to userspace, and is ultimately serviced by qemu’s device model synchronously before returning control back to the vcpu. However, there is a subclass of IO which acts purely as a trigger for other IO (such as to kick off an out-of-band DMA request, etc). For these patterns, the synchronous call is particularly expensive since we really only want to simply get our notification transmitted asychronously and return as quickly as possible. […] Therefore, we provide a mechanism for registration of an in-kernel trigger point that allows the VCPU to only require a very brief, lightweight exit just long enough to signal an eventfd.”

The measured numbers, from the merge commit

The same commit carries a benchmark, and this is the primary source for the “worth it” claim. Haskins built a test module called doorbell that counted signals, wired it up two ways — through QEMU’s device model, and directly through an ioeventfd — and measured round-trip time (RTT) and I/O operations per second (IOPS):

Path	IOPS	Round-trip time	Notes
qemu-mmio — heavy exit to userspace	110,000	9.09 µs	measured
ioeventfd-mmio — light exit, MMIO trap	200,100	5.00 µs	measured
ioeventfd-pio — light exit, port-I/O trap	367,300	2.72 µs	measured
qemu-pio	153,139	6.53 µs	extrapolated by the author, not measured
ioeventfd-hc (hypercall)	412,585	2.37 µs	extrapolated by the author, not measured

The author’s own conclusion: “The conclusion to draw is that we save about 4us by skipping the userspace hop.”

Uncertain

Verify: whether these ratios still hold on current hardware. Reason: the numbers above are from July 2009, on unnamed hardware, against a QEMU of that era. VM-exit latency has fallen substantially since Nehalem (and again with VMCS shadowing, KVM_CAP_IOEVENTFD_ANY_LENGTH’s fast-MMIO path, and modern QEMU’s own improvements), so the absolute microsecond figures are certainly stale; the sign and rough magnitude of the effect — a userspace hop costs single-digit microseconds and an in-kernel eventfd signal does not — is corroborated by the fact that every production VMM still builds around it. The two rows marked extrapolated were explicitly labelled as such by the author (“these are just for fun, for now, until I can gather more data”). To resolve: benchmark on current hardware with perf kvm stat or KVM_GET_STATS_FD, comparing mmio_exits/io_exits counts and wall time with ioeventfd=on and ioeventfd=off on the same virtio-blk device. uncertain

Two rows deserve a second look, because they explain a design decision that otherwise looks like legacy cruft. Port I/O is nearly twice as fast as MMIO, 2.72 µs against 5.00 µs. The reason is instruction decode: a port-I/O exit hands KVM the port number, direction, and size directly in the VMCS exit-qualification field, whereas an MMIO exit historically gave KVM only a faulting address, forcing it to fetch and decode the guest instruction to work out what was written and how wide. That single fact is why virtio-pci kept a port-I/O notify path long after PCI moved on, and why the modern answer — the zero-length “fast MMIO” ioeventfd — exists at all. It is covered in detail below.

The corresponding statement for the other direction comes from Michael Tsirkin’s vhost-net merge commit (vhost_net: a kernel-level virtio server, commit 3a4d5c94, 14 January 2010): “Compared to userspace, people reported improved latency (as I save up to 4 system calls per packet), as well as better bandwidth and CPU utilization.” Four syscalls per packet is the unit of measure to hold onto — that is what the two eventfds jointly remove from the per-packet path.

flowchart TD
    W(["Guest vCPU executes a write to the<br/>virtio notify register"]) --> TRAP["<b>Hardware trap — unavoidable</b><br/>VM exit / #VMEXIT<br/><i>the page is not mapped writable in EPT/NPT</i>"]

    TRAP --> DEC{"Which exit reason,<br/>and is the address<br/>registered?"}

    DEC -->|"EPT misconfig + a zero-length<br/>entry on KVM_FAST_MMIO_BUS"| FAST["<b>Fastest path</b><br/>read GUEST_PHYSICAL_ADDRESS from the VMCS<br/><b>no instruction fetch, no decode</b><br/>eventfd_signal&#40;&#41;<br/>kvm_skip_emulated_instruction&#40;&#41;"]
    DEC -->|"MMIO/PIO exit, address matches<br/>an ioeventfd on KVM_MMIO_BUS<br/>or KVM_PIO_BUS"| LIGHT["<b>Light exit</b><br/>decode width and value<br/>kvm_io_bus_write&#40;&#41; &rarr; bsearch<br/>ioeventfd_write&#40;&#41; &rarr; eventfd_signal&#40;&#41;<br/>return 0 = handled in kernel"]
    DEC -->|"no matching registration"| HEAVY["<b>Heavy exit</b><br/>kvm_mmu_page_fault&#40;&#41; &rarr; MMIO emulation<br/>fill vcpu-&gt;run-&gt;mmio<br/><b>return from the KVM_RUN ioctl</b>"]

    FAST --> RESUME(["VMRESUME — back in the guest.<br/>The VMM thread was never scheduled."])
    LIGHT --> RESUME
    HEAVY --> QEMU["QEMU's main/vCPU thread wakes,<br/>decodes the MMIO, calls the device model,<br/>then re-enters ioctl&#40;KVM_RUN&#41;"]
    QEMU --> RESUME2(["VMRESUME — after a full<br/>userspace round-trip"])

    style FAST fill:#e6f4ea
    style LIGHT fill:#eef4ff
    style HEAVY fill:#ffeeee

The three fates of one guest notify-register write. What it shows: the hardware trap at the top happens in all three cases — ioeventfd never prevents it — and the branching is entirely about how far the control transfer travels afterwards and how much work KVM must do to understand the write. The insight to take: the phrase “avoids a VM exit” should be read as “avoids the userspace leg of a VM exit.” The green box is the interesting one: it avoids not only userspace but also instruction decode, which is why a zero-length ioeventfd on the fast-MMIO bus is measurably cheaper than a length-matched one at the same address, and why KVM_CAP_IOEVENTFD_ANY_LENGTH exists as a distinct capability.

ioeventfd: Guest Writes Become eventfd Signals

KVM_IOEVENTFD is a VM-level ioctl that “attaches or detaches an ioeventfd to a legal pio/mmio address within the guest. A guest write in the registered address will signal the provided event instead of triggering an exit” (api.rst §4.59). The userspace argument is:

struct kvm_ioeventfd {
	__u64 datamatch;   /* optional value the write must equal */
	__u64 addr;        /* legal pio/mmio address */
	__u32 len;         /* 0, 1, 2, 4, or 8 bytes */
	__s32 fd;          /* the eventfd to signal */
	__u32 flags;
	__u8  pad[36];
};

(include/uapi/linux/kvm.h). The mechanism lives entirely in virt/kvm/eventfd.c. Registration (kvm_assign_ioeventfd_idx) grabs a reference on the eventfd (eventfd_ctx_fdget), allocates a struct _ioeventfd recording addr, length, datamatch, and the target eventfd, then registers it as a device on a KVM I/O bus:

kvm_iodevice_init(&p->dev, &ioeventfd_ops);
ret = kvm_io_bus_register_dev(kvm, bus_idx, p->addr, p->length, &p->dev);

The ioeventfd_ops has exactly one interesting method — .write = ioeventfd_write. KVM’s MMIO/PIO bus is a sorted array of (address, length) → device entries; when a guest write VM-exits and KVM walks the bus, a hit on this range dispatches to:

static int ioeventfd_write(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
			   gpa_t addr, int len, const void *val)
{
	struct _ioeventfd *p = to_ioeventfd(this);
	if (!ioeventfd_in_range(p, addr, len, val))
		return -EOPNOTSUPP;
	eventfd_signal(p->eventfd);
	return 0;
}

That is the entire datapath: a successful range/value match calls eventfd_signal() and returns 0, meaning “handled in kernel.” KVM does not return to userspace — KVM_RUN re-enters the guest immediately. The match logic in ioeventfd_in_range is precise: the address must be exact; if length == 0 the entry is a pure address trigger (any write to that address fires); otherwise the length must match, and if a datamatch was supplied (KVM_IOEVENTFD_FLAG_DATAMATCH) the written value must equal p->datamatch, else it is a “wildcard” that matches any value. Datamatch is what lets one notify-register serve many virtqueues: a virtio device with several queues registers several ioeventfds at the same address with different datamatch values (the queue index), so the guest selects which eventfd it rings by what it writes. The collision check is written to permit exactly this and nothing looser:

static bool
ioeventfd_check_collision(struct kvm *kvm, struct _ioeventfd *p)
{
	list_for_each_entry(_p, &kvm->ioeventfds, list)
		if (_p->bus_idx == p->bus_idx &&
		    _p->addr == p->addr &&
		    (!_p->length || !p->length ||
		     (_p->length == p->length &&
		      (_p->wildcard || p->wildcard ||
		       _p->datamatch == p->datamatch))))
			return true;
	return false;
}

Read the boolean carefully, because it encodes the whole sharing policy. Two registrations at the same address collide (-EEXIST) if either is zero-length (a zero-length entry claims every write to that address, so it can share with nothing), or they have the same length and either is a wildcard, or they have the same length and the same datamatch. What is therefore allowed is precisely: same address, same length, different datamatch — several queues sharing one notify register. A zero-length entry and a datamatch entry at the same address can never coexist, which is why kvm_assign_ioeventfd() also rejects the two flags together outright (“ioeventfd with no length can’t be combined with DATAMATCH”: you cannot compare a value you have refused to read).

The lookup side pays for this with a two-stage search in virt/kvm/kvm_main.c (v6.12). kvm_io_bus_get_first_dev() does a bsearch() over the sorted range array — so finding an entry for the address is O(log n) in the number of devices on that bus — then walks backwards to the first entry with an equal key, and __kvm_io_bus_write() walks forwards trying each in turn:

idx = kvm_io_bus_get_first_dev(bus, range->addr, range->len);
if (idx < 0)
	return -EOPNOTSUPP;
while (idx < bus->dev_count &&
	kvm_io_bus_cmp(range, &bus->range[idx]) == 0) {
	if (!kvm_iodevice_write(vcpu, bus->range[idx].dev, range->addr,
				range->len, val))
		return idx;
	idx++;
}
return -EOPNOTSUPP;

Each candidate’s ioeventfd_write() returns -EOPNOTSUPP on a datamatch miss, so the loop continues to the next. The cost of datamatch multiplexing is therefore a linear scan over the queues sharing one address, on the hot path, on every kick — logarithmic to find the group, linear within it. For a 16-queue device that is up to 16 comparisons per notification. This is a real (if small) argument for the modern alternative described next, in which each queue gets its own address and the scan length is one.

flowchart TD
    HIT(["kvm_io_bus_write&#40;&#41; dispatched to<br/>ioeventfd_write&#40;&#41; for this entry"]) --> A{"addr == p-&gt;addr?"}
    A -->|"no"| MISS["return -EOPNOTSUPP<br/><i>try the next entry</i>"]
    A -->|"yes"| L{"p-&gt;length == 0?"}
    L -->|"yes — zero-length entry"| FIRE["<b>eventfd_signal&#40;p-&gt;eventfd&#41;</b><br/>return 0 = handled"]
    L -->|"no"| LM{"len == p-&gt;length?"}
    LM -->|"no"| MISS
    LM -->|"yes"| WC{"p-&gt;wildcard?<br/><i>i.e. no DATAMATCH flag</i>"}
    WC -->|"yes"| FIRE
    WC -->|"no"| DM{"the written value,<br/>widened to u64,<br/>== p-&gt;datamatch?"}
    DM -->|"yes"| FIRE
    DM -->|"no"| MISS
    MISS --> NEXT(["loop continues; if nothing matches,<br/>the write falls through to<br/><b>full MMIO emulation in userspace</b>"])

    style FIRE fill:#e6f4ea
    style NEXT fill:#ffeeee

ioeventfd_in_range() as a decision tree. What it shows: four independent conditions must all pass — exact address, then length (unless the entry is zero-length), then either wildcard or exact value — and any failure means “not mine, try the next registration.” The insight to take: the red terminal is the whole class of ioeventfd misconfiguration bugs. There is no error, no warning, and no log line when nothing matches; the write simply takes the slow path it would have taken had no ioeventfd been registered at all. A device that is “mysteriously slow” and a device whose ioeventfd registration is subtly wrong look identical from inside the guest. The only reliable signal is a rising mmio_exits/io_exits count in KVM’s own statistics.

There are two important refinements in the same file. First, KVM_IOEVENTFD_FLAG_PIO routes to KVM_PIO_BUS instead of KVM_MMIO_BUS for port-I/O notify registers (legacy virtio-pci). Second, the zero-length “fast MMIO” optimization: when len == 0 and the bus is KVM_MMIO_BUS, kvm_assign_ioeventfd also registers the entry on a separate KVM_FAST_MMIO_BUS:

if (!args->len && bus_idx == KVM_MMIO_BUS) {
	ret = kvm_assign_ioeventfd_idx(kvm, KVM_FAST_MMIO_BUS, args);
	...
}

This pairs with KVM_CAP_IOEVENTFD_ANY_LENGTH, which the API documentation describes only vaguely — “the kernel will ignore the length of guest write and may get a faster vmexit. The speedup may only apply to specific architectures, but the ioeventfd will work anyway” (api.rst §4.59). The documentation’s vagueness is a good example of why in-tree docs must be checked against code: on x86 the speedup is specific, large, and easy to state once you find where KVM_FAST_MMIO_BUS is consumed. It is consumed in exactly one place — handle_ept_misconfig() in arch/x86/kvm/vmx/vmx.c (v6.12):

static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
{
	gpa_t gpa;
 
	if (vmx_check_emulate_instruction(vcpu, EMULTYPE_PF, NULL, 0))
		return 1;
 
	/*
	 * A nested guest cannot optimize MMIO vmexits, because we have an
	 * nGPA here instead of the required GPA.
	 */
	gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
	if (!is_guest_mode(vcpu) &&
	    !kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
		trace_kvm_fast_mmio(gpa);
		return kvm_skip_emulated_instruction(vcpu);
	}
 
	return kvm_mmu_page_fault(vcpu, gpa, PFERR_RSVD_MASK, NULL, 0);
}

Read the arguments to kvm_io_bus_write(): length 0 and value NULL. KVM never looks at what the guest wrote. The guest physical address comes straight out of the VMCS field GUEST_PHYSICAL_ADDRESS, which the hardware fills in for free on the exit; there is no instruction fetch from guest memory, no instruction decode, and no operand extraction. Then kvm_skip_emulated_instruction() advances RIP past the write and the vCPU re-enters the guest. Compare that with the ordinary MMIO path, which must read the guest’s instruction bytes (potentially faulting on the instruction page), run them through KVM’s x86 emulator to determine the operand width and value, and only then consult the bus. Instruction decode is the single most expensive part of MMIO emulation, and the zero-length ioeventfd removes it entirely.

Two constraints fall out of the same eleven lines. !is_guest_mode(vcpu) disables the optimisation for nested guests, and the comment says exactly why: at L2 the address in the VMCS is a nested guest physical address (nGPA), not the L1 GPA the bus is keyed on, so the lookup would be wrong. And because the path is entered from handle_ept_misconfig(), it depends on the extended page tables being programmed to misconfigure the notify page rather than merely mark it non-present — the mechanism is bound to how the KVM MMU sets up MMIO pages, which is covered in Two-Dimensional Paging (EPT and NPT). On AMD there is no EPT_MISCONFIG exit reason and no equivalent hook in this file, which is what the API doc means by “may only apply to specific architectures.”

Validation in kvm_assign_ioeventfd enforces that len is 0, 1, 2, 4, or 8 (“natural-word sized”), rejects addr + len overflow, rejects unknown flag bits, and rejects a datamatch combined with zero length.

Which form does a real VMM use? Two answers, and they are not the same

It is tempting to assume every virtio device uses datamatch. QEMU v9.1.0 says otherwise, and the two branches of one function make the whole trade-off legible (hw/virtio/virtio-pci.c):

hwaddr modern_addr = virtio_pci_queue_mem_mult(proxy) *
                     virtio_get_queue_index(vq);
hwaddr legacy_addr = VIRTIO_PCI_QUEUE_NOTIFY;
 
if (assign) {
    if (modern) {
        memory_region_add_eventfd(modern_mr, modern_addr, 0,
                                  false, n, notifier);      /* size 0, match_data false */
        if (modern_pio) {
            memory_region_add_eventfd(modern_notify_mr, 0, 2,
                                      true, n, notifier);   /* size 2, match_data true  */
        }
    }
    if (legacy) {
        memory_region_add_eventfd(legacy_mr, legacy_addr, 2,
                                  true, n, notifier);       /* size 2, match_data true  */
    }
}

The signature is memory_region_add_eventfd(mr, addr, size, match_data, data, e) (include/exec/memory.h, v9.1.0), so:

Transport	Address	size	match_data	Resulting KVM registration
virtio-pci 1.0 (modern), MMIO notify	queue_mem_mult × queue_index — a distinct address per queue	0	false	Zero-length wildcard on KVM_MMIO_BUS and KVM_FAST_MMIO_BUS — the no-decode path
virtio-pci 1.0, optional PIO notify (modern-pio-notify)	one port	2	true (data = queue index)	Datamatch on KVM_PIO_BUS
virtio-pci 0.9 (legacy)	VIRTIO_PCI_QUEUE_NOTIFY — one shared port	2	true (data = queue index)	Datamatch on KVM_PIO_BUS

virtio_pci_queue_mem_mult() returns 0x1000 when the page-per-vq property is set and 4 otherwise — so modern virtio-pci spaces its per-queue notify registers either one page or four bytes apart, and each gets its very own zero-length ioeventfd.

This is the correction worth carrying away: datamatch is the legacy multiplexing answer, and the modern answer is to stop multiplexing. Legacy virtio-pci had exactly one 16-bit notify port for the whole device, so the queue index had to travel in the data and datamatch was the only way to demultiplex it in-kernel. The virtio 1.0 layout gives each virtqueue its own notify address, which lets QEMU use a zero-length wildcard — eliminating both the datamatch comparison and, far more importantly, the instruction decode. Datamatch remains essential for the transports that still share an address: legacy virtio-pci, the optional modern PIO notify path, and virtio-ccw on s390, where “the ioevent is matched to a subchannel/virtqueue tuple instead” and addr carries the subchannel id with datamatch carrying the virtqueue index (api.rst §4.59).

The guest kick, with and without

sequenceDiagram
    autonumber
    participant G as Guest driver<br/>(virtio-net TX)
    participant HW as CPU / VT-x
    participant K as KVM (host kernel)
    participant Q as QEMU vCPU thread<br/>(userspace VMM)
    participant B as Backend<br/>(vhost worker / QEMU device model)

    rect rgb(250,238,238)
    Note over G,B: WITHOUT ioeventfd — the notify write is emulated in userspace
    G->>HW: mov [notify_reg], queue_idx
    HW->>K: VM exit (EPT violation / misconfig)
    K->>K: fetch guest instruction bytes,<br/>run the x86 emulator,<br/>extract width + value
    K->>K: kvm_io_bus_write() finds nothing
    K->>Q: fill vcpu->run->mmio,<br/><b>return from ioctl(KVM_RUN)</b>
    Note over Q: the VMM thread must be SCHEDULED.<br/>Two syscall boundaries, one context switch.
    Q->>B: device model handles the kick
    Q->>K: ioctl(KVM_RUN) again
    K->>HW: VMRESUME
    HW->>G: guest continues
    end

    rect rgb(236,244,236)
    Note over G,B: WITH ioeventfd — never leaves the kernel
    G->>HW: mov [notify_reg], queue_idx
    HW->>K: VM exit (EPT misconfig)
    K->>K: gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS)<br/><b>no instruction fetch, no decode</b>
    K->>K: kvm_io_bus_write(KVM_FAST_MMIO_BUS, gpa, 0, NULL)<br/>bsearch hit => ioeventfd_write()
    K->>B: <b>eventfd_signal(p->eventfd)</b><br/>wakes the vhost worker on another CPU
    K->>K: kvm_skip_emulated_instruction()
    K->>HW: VMRESUME
    HW->>G: guest continues
    Note over Q: <b>The VMM thread was never woken.</b><br/>It is blocked in ioctl(KVM_RUN)<br/>and stays there.
    end

The guest-to-host doorbell, both ways. What it shows: the same guest instruction and the same hardware trap, differing only in what happens between the exit and the VMRESUME. The upper lane crosses into userspace and back — two syscall boundaries and a scheduler round-trip; the lower lane stays inside kvm_io_bus_write() and returns to the guest immediately. The insight to take: the arrow that matters is the one that is absent in the lower lane. QEMU’s vCPU thread is not “made faster” — it is not involved at all; it remains blocked inside its ioctl(KVM_RUN) for the entire transaction. That is what “the VMM is out of the datapath” means concretely, and it is why the backend can be a vhost kernel thread on a completely different CPU without any userspace coordination.

irqfd: eventfd Signals Become Guest Interrupts

KVM_IRQFD “allows setting an eventfd to directly trigger a guest interrupt. kvm_irqfd.fd specifies the file descriptor to use as the eventfd and kvm_irqfd.gsi specifies the irqchip pin toggled by this event. When an event is triggered on the eventfd, an interrupt is injected into the guest using the specified gsi pin” (api.rst §4.75). The argument is:

struct kvm_irqfd {
	__u32 fd;          /* the eventfd to watch */
	__u32 gsi;         /* the GSI to assert when it fires */
	__u32 flags;
	__u32 resamplefd;  /* for level-triggered (RESAMPLE) mode */
	__u8  pad[16];
};

The clever part of irqfd is how it watches the eventfd: it does not spawn a thread to read it. In kvm_irqfd_assign, KVM installs a custom wait-queue callback onto the eventfd’s internal wait queue using the poll machinery:

init_waitqueue_func_entry(&irqfd->wait, irqfd_wakeup);
init_poll_funcptr(&irqfd->pt, irqfd_ptable_queue_proc);
...
events = vfs_poll(fd_file(f), &irqfd->pt);

vfs_poll on an eventfd calls back into irqfd_ptable_queue_proc, which does add_wait_queue_priority(wqh, &irqfd->wait). So irqfd hangs itself on the eventfd’s wait queue. Now whenever anyone calls eventfd_signal() on that fd — a vhost worker, a VFIO interrupt handler, even a userspace write() — the eventfd wakes its waiters and KVM’s irqfd_wakeup runs:

static int irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode,
			int sync, void *key)
{
	...
	if (flags & EPOLLIN) {
		eventfd_ctx_do_read(irqfd->eventfd, &cnt);
		...
		irq = irqfd->irq_entry;   /* the cached GSI routing entry */
		if (kvm_arch_set_irq_inatomic(&irq, kvm,
				KVM_USERSPACE_IRQ_SOURCE_ID, 1, false) == -EWOULDBLOCK)
			schedule_work(&irqfd->inject);
		...
	}
	...
}

It runs in the waker’s context, with interrupts disabled and the wait-queue lock held — so it tries an atomic injection first (kvm_arch_set_irq_inatomic). On x86 this fast path can land an edge-triggered MSI straight into the destination vCPU’s local APIC. If the routing cannot be resolved without sleeping (it returns -EWOULDBLOCK), KVM falls back to schedule_work(&irqfd->inject), deferring to a workqueue where irqfd_inject runs:

static void irqfd_inject(struct work_struct *work)
{
	...
	if (!irqfd->resampler) {
		kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID, irqfd->gsi, 1, false);
		kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID, irqfd->gsi, 0, false);
	} else
		kvm_set_irq(kvm, KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID, irqfd->gsi, 1, false);
}

For the common (edge-triggered) case it asserts the GSI to 1 then immediately to 0 — a pulse, because an edge interrupt has no persistent line state. kvm_set_irq (in virt/kvm/irqchip.c) looks up the GSI in the routing table and calls each routed entry’s .set() handler, which for x86 lands in the IOAPIC, PIC, or MSI delivery code — and ultimately in __apic_accept_irq (see Interrupt Injection and the Virtual APIC). The GSI routing entry is cached in irqfd->irq_entry under a seqcount (irqfd_update recomputes it whenever userspace changes IRQ routing) so the hot irqfd_wakeup path never has to walk the routing table under contention.

A crucial precondition: irqfd requires an in-kernel irqchip. kvm_irqfd_assign bails with -EAGAIN if kvm_arch_intc_initialized(kvm) is false, and kvm_arch_irqfd_allowed (in arch/x86/kvm/irq.c) requires irqchip_in_kernel(kvm) for a basic irqfd and the full kernel irqchip (irqchip_kernel) for a resampling one. You cannot use irqfd with a fully userspace interrupt controller, because there would be no in-kernel target to assert the GSI on.

KVM_IRQFD is documented as available on x86 s390 arm64 (api.rst §4.75), and the arm64 semantics are worth stating because the GSI abstraction lands differently there: “in case no routing entry is associated to this gsi, injection fails; in case the gsi is associated to an irqchip routing entry, irqchip.pin + 32 corresponds to the injected SPI ID; in case the gsi is associated to an MSI routing entry, the MSI message and device ID are translated into an LPI (support restricted to GICv3 ITS in-kernel emulation).” The + 32 is the ARM Generic Interrupt Controller’s fixed partition — INTIDs 0–15 are software-generated interrupts and 16–31 are private peripheral interrupts, so shared peripheral interrupts start at 32.

The host-to-guest doorbell, with and without

sequenceDiagram
    autonumber
    participant B as Backend<br/>(vhost worker / VFIO MSI handler)
    participant Q as QEMU I/O thread<br/>(userspace VMM)
    participant K as KVM (host kernel)
    participant L as Guest LAPIC model<br/>(__apic_accept_irq)
    participant V as Guest vCPU

    rect rgb(250,238,238)
    Note over B,V: WITHOUT irqfd — the VMM must be scheduled to raise the interrupt
    B->>Q: writes to a pipe / eventfd the VMM polls
    Note over Q: the I/O thread must WAKE:<br/>epoll_wait returns, it is scheduled,<br/>it decodes which device fired
    Q->>K: ioctl(KVM_IRQ_LINE) or ioctl(KVM_SIGNAL_MSI)<br/><b>one syscall per interrupt</b>
    K->>L: kvm_set_irq(gsi) => routing => set()
    L->>V: IRR bit set, vCPU kicked (IPI) if running
    V->>V: takes the virtual interrupt
    end

    rect rgb(236,244,236)
    Note over B,V: WITH irqfd — the signaller's own context does the injection
    B->>B: <b>eventfd_signal(call_ctx)</b>
    Note over B: the eventfd wakes its wait queue.<br/>KVM hung irqfd->wait on it at setup time<br/>via vfs_poll() + add_wait_queue_priority()
    B->>K: <b>irqfd_wakeup() runs INLINE</b><br/>in the signaller's context,<br/>IRQs off, wait-queue lock held
    K->>K: eventfd_ctx_do_read(), then<br/>irq = irqfd->irq_entry (cached route, seqcount)
    alt route resolvable atomically
        K->>L: kvm_arch_set_irq_inatomic()
        L->>V: interrupt delivered
    else -EWOULDBLOCK
        K->>K: schedule_work(&irqfd->inject)
        K->>L: irqfd_inject(): kvm_set_irq(gsi,1) then (gsi,0)
        L->>V: interrupt delivered
    end
    Note over Q: <b>The VMM thread was never woken.</b>
    end

The completion interrupt, both ways. What it shows: without irqfd the interrupt cannot be raised until the userspace VMM’s I/O thread is scheduled and issues a syscall; with irqfd the injection happens synchronously in whatever context called eventfd_signal() — which may be a vhost kernel worker, or a hardware interrupt handler in vfio-pci. The insight to take: the alt block is the honest part of the picture. irqfd_wakeup() runs with interrupts disabled and the eventfd’s wait-queue lock held, so it can only take the fast path when the GSI route resolves without sleeping; when it cannot, KVM falls back to schedule_work() and the injection is deferred to a workqueue — still without userspace, but no longer synchronous. Reading only the fast path and concluding “irqfd is always inline” is the common overstatement.

Why hanging on the wait queue is the right design

It is worth pausing on the vfs_poll() trick, because the same technique appears on the other side of the pipe and the symmetry is not a coincidence. KVM does not read the eventfd; it registers a wait-queue entry with a custom wake function on the eventfd’s internal wait queue, using the poll machinery as the installation mechanism rather than as a polling mechanism:

init_waitqueue_func_entry(&irqfd->wait, irqfd_wakeup);
init_poll_funcptr(&irqfd->pt, irqfd_ptable_queue_proc);
...
events = vfs_poll(fd_file(f), &irqfd->pt);
if (events & EPOLLIN)
	schedule_work(&irqfd->inject);   /* an event was already pending */

vfs_poll() calls the file’s ->poll() handler, which calls back into irqfd_ptable_queue_proc(), which does add_wait_queue_priority(wqh, &irqfd->wait). The _priority variant matters: it puts KVM’s entry at the head of the wait queue, ahead of ordinary epoll waiters, so KVM’s injection runs before any userspace poller is even considered for wake-up. And the return value of vfs_poll() is used for a genuine correctness fix — if the eventfd already had a pending count when the irqfd was registered, that event would otherwise be lost forever, so KVM injects it immediately (“Check if there was an event already pending on the eventfd before we registered, and trigger it as if we didn’t miss it”).

drivers/vhost/vhost.c (v6.12) uses the identical technique on the kick eventfd, in vhost_poll_start():

mask = vfs_poll(file, &poll->table);
if (mask)
	vhost_poll_wakeup(&poll->wait, 0, 0, poll_to_key(mask));

Same vfs_poll-to-install pattern, same already-pending recovery. So the two ends of a virtio fast path are built out of the same primitive pointed in opposite directions: vhost hangs a wake function on the kick eventfd that KVM signals, and KVM hangs a wake function on the call eventfd that vhost signals. Neither side ever calls read(), and neither side ever runs a polling loop.

resamplefd: Level-Triggered Interrupts

Edge interrupts are a pulse; level-triggered interrupts assert a line that stays high until the device deasserts it, and the guest must acknowledge (EOI) before the same line can fire again. A plain irqfd cannot model this — it only knows “fire.” KVM_CAP_IRQFD_RESAMPLE adds it. When KVM_IRQFD_FLAG_RESAMPLE is set, userspace passes a second eventfd in resamplefd. “When operating in resample mode, posting of an interrupt through kvm_irqfd.fd asserts the specified gsi in the irqchip. When the irqchip is resampled, such as from an EOI, the gsi is de-asserted and the user is notified via kvm_irqfd.resamplefd. It is the user’s responsibility to re-queue the interrupt if the device making use of it still requires service” (api.rst §4.75).

Mechanically, resampling irqfds for the same GSI share a kvm_kernel_irqfd_resampler carrying an IRQ ack notifier. On assert, irqfd_inject raises the line (to 1, and leaves it) using the dedicated KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID. When the guest EOIs the interrupt, KVM’s ack notifier fires irqfd_resampler_ack, which deasserts the GSI once and then signals every resamplefd sharing the GSI via irqfd_resampler_notify → eventfd_signal(irqfd->resamplefd). The deassert-once-then-notify-all dance is deliberate: the comment in the source warns that “we can’t do multiple de-asserts or we risk racing with incoming re-asserts.” This is exactly what a passed-through PCI device needs — its INTx line is level-triggered, so VFIO uses a resampling irqfd to learn (via the resamplefd) when the guest has serviced the interrupt and the physical line can be unmasked.

stateDiagram-v2
    direction TB

    IDLE: <b>Line low, armed</b><br/>GSI de-asserted in the irqchip<br/>host device IRQ unmasked<br/><i>nothing pending</i>

    ASSERTED: <b>Line high, guest not yet served</b><br/>kvm_set_irq&#40;RESAMPLE_IRQ_SOURCE_ID, gsi, <b>1</b>&#41;<br/>and it STAYS at 1<br/><i>host device IRQ masked by vfio_intx_handler</i>

    INSERVICE: <b>Guest servicing</b><br/>vector in the LAPIC ISR<br/>guest driver reading device registers<br/><i>the physical line is still asserted</i>

    RESAMPLING: <b>EOI seen</b><br/>irq ack notifier fires<br/>irqfd_resampler_ack&#40;&#41;

    [*] --> IDLE
    IDLE --> ASSERTED: device asserts INTx;<br/>host handler masks the line and<br/>eventfd_signal&#40;trigger&#41;<br/>=> irqfd_inject&#40;&#41;

    ASSERTED --> INSERVICE: vCPU accepts the vector

    INSERVICE --> RESAMPLING: guest writes EOI

    RESAMPLING --> IDLE: <b>de-assert exactly ONCE</b><br/>kvm_set_irq&#40;..., gsi, 0&#41;<br/>then irqfd_resampler_notify&#40;&#41;<br/>signals EVERY resamplefd on this GSI

    note right of RESAMPLING
        Order is load-bearing. The source
        warns that de-asserting more than
        once "risks racing with incoming
        re-asserts" — so one de-assert for
        the shared GSI, then N notifications,
        one per irqfd sharing it.
    end note

    note left of ASSERTED
        A plain (non-resample) irqfd would
        instead PULSE here: set 1, then
        immediately set 0. Correct for an
        edge interrupt, wrong for a level
        one — the guest would EOI a line
        that was already low and the device
        would never be re-serviced.
    end note

The level-triggered interrupt lifecycle under a resampling irqfd. What it shows: the GSI stays asserted across the entire window from injection to guest EOI, and the transition back to idle is driven by an acknowledgement notifier inside KVM rather than by anything the host device does. The insight to take: the resamplefd is not a second interrupt channel — it is a completion notification flowing the other way. The device backend does not learn “the guest handled it” from the guest; it learns it from KVM, which is the only party that can see the EOI. Combined with the caveat from the API documentation that “closing the resamplefd is not sufficient to disable the irqfd,” this makes resample mode a strictly two-fd, explicitly-torn-down arrangement.

The mechanics behind that diagram: resampling irqfds that target the same GSI are gathered into a shared struct kvm_kernel_irqfd_resampler, which owns one kvm_irq_ack_notifier registered with the irqchip. On assert, irqfd_inject() takes the else branch and raises the line to 1 and leaves it there, using a dedicated source id, KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID, so that resample-sourced assertions are tracked separately from ordinary KVM_USERSPACE_IRQ_SOURCE_ID ones. On EOI, irqfd_resampler_ack() de-asserts once and then walks the list calling eventfd_signal(irqfd->resamplefd) for each member. The API contract then puts the ball back in userspace’s court: “It is the user’s responsibility to re-queue the interrupt if the device making use of it still requires service.”

There is a permission asymmetry worth knowing: kvm_arch_irqfd_allowed() in arch/x86/kvm/irq.c (v6.12) permits a plain irqfd whenever irqchip_in_kernel(kvm) — which includes the split-irqchip configuration where the local APIC is in-kernel but the IOAPIC/PIC are in userspace — but requires the full in-kernel irqchip (irqchip_kernel(kvm)) for a resampling one. The reason is structural: resampling needs the in-kernel irqchip to observe the EOI, and in split-irqchip mode the IOAPIC that would see it lives in QEMU. So on a split-irqchip VM, KVM_IRQFD_FLAG_RESAMPLE fails and INTx passthrough must fall back to userspace handling.

How This Powers vhost, vDPA, and VFIO

The reason these two ioctls matter is that the same eventfd can be handed to two subsystems. For vhost-net, QEMU opens /dev/vhost-net, creates two eventfds, and wires each to both KVM and vhost: the “kick” eventfd is registered with VHOST_SET_VRING_KICK on vhost and as an ioeventfd on KVM (guest notify-write → eventfd → vhost worker wakes); the “call” eventfd is registered with VHOST_SET_VRING_CALL on vhost and as an irqfd on KVM (vhost completion → eventfd → guest interrupt). As the Red Hat deep-dive puts it, “qemu allocates one eventfd and registers it to both vhost and KVM” (Red Hat, Deep dive into virtio-networking and vhost-net). The per-packet loop is then entirely in-kernel — see vhost (In-Kernel virtio Backend) and virtio Notifications and Virtqueue Kicks for the device-side view.

The vhost side of the wiring is visible in drivers/vhost/vhost.c (v6.12). VHOST_SET_VRING_KICK takes a file descriptor, eventfd_fget()s it into vq->kick, and arranges for vhost_poll_start(&vq->poll, vq->kick) — hanging vhost’s wake function on the eventfd, exactly as KVM does on the other one. VHOST_SET_VRING_CALL takes the second descriptor and eventfd_ctx_fdget()s it into vq->call_ctx.ctx, whose only use is one three-line function:

/* This actually signals the guest, using eventfd. */
void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
{
	/* Signal the Guest tell them we used something up. */
	if (vq->call_ctx.ctx && vhost_notify(dev, vq))
		eventfd_signal(vq->call_ctx.ctx);
}

Note the vhost_notify(dev, vq) guard. Signalling is conditional on virtio’s notification-suppression rules — VIRTIO_RING_F_EVENT_IDX and the used-ring NO_NOTIFY flag — so a busy guest that is already polling its used ring suppresses interrupts entirely and the irqfd is never signalled at all. The fastest interrupt is still the one you do not send; irqfd makes the ones you do send cheap. virtio Notifications and Virtqueue Kicks covers the suppression protocol.

sequenceDiagram
    autonumber
    participant NET as Physical NIC / tap
    participant VW as vhost-net kernel worker<br/>(vhost_worker task)
    participant KE as kick eventfd
    participant CE as call eventfd
    participant KVM as KVM
    participant G as Guest virtio-net driver
    participant Q as QEMU<br/>(userspace VMM)

    Note over Q: SETUP ONLY — happens once, at device realize
    Q->>KE: ioctl(VHOST_SET_VRING_KICK, fd)<br/>vhost_poll_start(): vfs_poll + wake fn
    Q->>KVM: ioctl(KVM_IOEVENTFD, {addr, len:0, fd})<br/>=> entry on KVM_MMIO_BUS + KVM_FAST_MMIO_BUS
    Q->>CE: ioctl(VHOST_SET_VRING_CALL, fd)<br/>=> vq->call_ctx.ctx
    Q->>KVM: ioctl(KVM_IRQFD, {fd, gsi})<br/>=> vfs_poll + add_wait_queue_priority
    Note over Q: <b>QEMU now leaves the datapath entirely.</b>

    Note over NET,G: STEADY STATE — one received packet, zero userspace involvement
    NET->>VW: packet arrives on tap, and the<br/>worker copies it into the guest RX virtqueue
    VW->>VW: vhost_add_used_and_signal_n()
    VW->>CE: vhost_signal(): if vhost_notify() says so,<br/>eventfd_signal(call_ctx)
    CE->>KVM: wait queue wakes irqfd_wakeup() INLINE
    KVM->>G: kvm_arch_set_irq_inatomic() => vIRQ
    G->>G: guest NAPI poll drains the RX ring
    G->>G: refills descriptors, writes the notify register
    G->>KVM: VM exit, fast-MMIO hit on the ioeventfd
    KVM->>KE: eventfd_signal(kick)
    KVM->>G: VMRESUME immediately
    KE->>VW: vhost worker wakes, refills from tap
    Note over Q: QEMU still blocked in ioctl(KVM_RUN).<br/>It has not run once in this whole loop.

A full virtio-net receive cycle across the two eventfds. What it shows: the top block is one-time setup in which QEMU hands the same two file descriptors to two different kernel subsystems, and the bottom block is the steady state in which those subsystems talk to each other directly. The insight to take: QEMU’s role is reduced from “the device” to “the thing that introduced two kernel components to each other.” The four syscalls per packet Tsirkin’s commit message cites are the four that vanish between the two blocks — kick-read, kick-handle, call-write, and interrupt-inject all become in-kernel eventfd signals. Note also that the loop is closed on both sides: the guest’s refill in step 9 rides the ioeventfd back to the same worker that used the irqfd, so the two mechanisms are not independent optimisations but the two halves of one ring.

For VFIO passthrough, real hardware raises real host interrupts. VFIO exposes a per-IRQ “trigger” eventfd; the host kernel’s MSI/MSI-X handler for the assigned device signals that eventfd, and because it is registered as a KVM irqfd, the guest interrupt is injected without VFIO ever touching userspace. VFIO Framework owns the device-assignment side of this — the container/group/device model, the DMA mapping, and why level-triggered INTx needs masking before the eventfd is signalled at all. What belongs here is the other end of the wire: how a VFIO eventfd finds its way to a KVM irqfd, and what happens when the hardware can cut both of them out.

The IRQ-bypass rendezvous: two subsystems, one token

The irqfd path participates in a small, self-contained matchmaking service, virt/lib/irqbypass.c, whose header comment states its purpose exactly: “Various virtualization hardware acceleration techniques allow bypassing or offloading interrupts received from devices around the host kernel. Posted Interrupts on Intel VT-d systems can allow interrupts to be received directly by a virtual machine. ARM IRQ Forwarding allows forwarded physical interrupts to be directly deactivated by the guest. This manager allows interrupt producers and consumers to find each other to enable this sort of bypass.”

It is two global lists and a mutex. Producers and consumers register independently, in either order, and are connected when their tokens match. The token is the crux, and both sides use the same value: the struct eventfd_ctx * itself.

On the KVM side, at the end of kvm_irqfd_assign() (virt/kvm/eventfd.c, v6.12):

#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
	if (kvm_arch_has_irq_bypass()) {
		irqfd->consumer.token = (void *)irqfd->eventfd;
		irqfd->consumer.add_producer = kvm_arch_irq_bypass_add_producer;
		irqfd->consumer.del_producer = kvm_arch_irq_bypass_del_producer;
		irqfd->consumer.stop = kvm_arch_irq_bypass_stop;
		irqfd->consumer.start = kvm_arch_irq_bypass_start;
		ret = irq_bypass_register_consumer(&irqfd->consumer);
		if (ret)
			pr_info("irq bypass consumer (token %p) registration fails: %d\n",
				irqfd->consumer.token, ret);
	}
#endif

And on the VFIO side, in vfio_msi_set_vector_signal() (drivers/vfio/pci/vfio_pci_intrs.c, v6.12), immediately after taking the host interrupt:

	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
	vfio_pci_memory_unlock_and_restore(vdev, cmd);
	if (ret)
		goto out_put_eventfd_ctx;
 
	ctx->producer.token = trigger;      /* the same eventfd_ctx * */
	ctx->producer.irq = irq;            /* the HOST irq number */
	ret = irq_bypass_register_producer(&ctx->producer);
	if (unlikely(ret)) {
		dev_info(&pdev->dev,
		"irq bypass producer (token %p) registration fails: %d\n",
		ctx->producer.token, ret);
		ctx->producer.token = NULL;
	}

Three things to take from this. First, the token is trigger, the eventfd_ctx * VFIO holds a reference on — bit-identical to irqfd->eventfd on the KVM side, because it is the same object, obtained from the same file descriptor QEMU passed to both ioctls. That is the entire matching mechanism: no names, no IDs, just pointer equality on a shared kernel object. Second, the producer carries irq — the host IRQ number — which is what the arch code needs in order to reprogram the interrupt remapping table entry. Third, failure is non-fatal: VFIO logs at dev_info level, clears the token, and carries on. The device keeps working via the ordinary software irqfd path; you simply lose the hardware acceleration, silently, unless you are reading dmesg.

flowchart TD
    subgraph SETUP["Registration — order does not matter"]
      Q1["QEMU: ioctl&#40;VFIO_DEVICE_SET_IRQS,<br/>DATA_EVENTFD, fd&#41;"] --> P["vfio_msi_set_vector_signal&#40;&#41;<br/>request_irq&#40;vfio_msihandler&#41;<br/><b>producer.token = eventfd_ctx *</b><br/>producer.irq = host IRQ<br/>irq_bypass_register_producer&#40;&#41;"]
      Q2["QEMU: ioctl&#40;KVM_IRQFD,<br/>{same fd, gsi}&#41;"] --> C["kvm_irqfd_assign&#40;&#41;<br/><b>consumer.token = irqfd-&gt;eventfd</b><br/>irq_bypass_register_consumer&#40;&#41;"]
    end

    P --> M{"irqbypass.c: scan the other list<br/><b>tmp-&gt;token == token?</b>"}
    C --> M
    M -->|"no match yet"| PARK["sit on the list<br/>and wait for the other half"]
    M -->|"match"| CONN["__connect&#40;prod, cons&#41;<br/>prod.stop&#40;&#41;, cons.stop&#40;&#41;<br/>cons.add_producer&#40;&#41;<br/>cons.start&#40;&#41;, prod.start&#40;&#41;"]

    CONN --> ARCH["x86: kvm_arch_irq_bypass_add_producer&#40;&#41;<br/>irqfd-&gt;producer = prod<br/>kvm_arch_start_assignment&#40;&#41;<br/>pi_update_irte&#40;kvm, prod-&gt;irq, irqfd-&gt;gsi, <b>1</b>&#41;"]

    ARCH --> GATE{"vmx_pi_update_irte&#40;&#41; checks"}
    GATE -->|"routing entry is not<br/>KVM_IRQ_ROUTING_MSI"| FALL["irq_set_vcpu_affinity&#40;host_irq, NULL&#41;<br/><b>stay in remapped mode</b><br/>software irqfd path is used"]
    GATE -->|"not a single-vCPU destination<br/>&#40;multicast / broadcast /<br/>multi-CPU lowest-priority&#41;"| FALL
    GATE -->|"vector not postable<br/>&#40;NMI, SMI, INIT, ExtINT&#41;"| FALL
    GATE -->|"all checks pass"| POST["<b>program the IRTE in posted mode</b><br/>target = the vCPU's posted-interrupt<br/>descriptor, not a host CPU vector"]

    POST --> WIN(["Device MSI write is consumed by VT-d,<br/>ORed into the guest's PIR,<br/>delivered by the notification vector.<br/><b>No host IRQ handler. No eventfd.<br/>No irqfd. No VM exit.</b>"])

    style POST fill:#e6f4ea
    style WIN fill:#e6f4ea
    style FALL fill:#fff6e5

How a passed-through device’s MSI vector gets promoted from “software irqfd” to “no software at all.” What it shows: the rendezvous is a pointer-equality match on a shared eventfd_ctx, after which the x86 arch hook reprograms the IOMMU’s interrupt remapping table entry so the device’s MSI write targets a vCPU’s posted-interrupt descriptor directly. The insight to take: the irqfd is not merely a fast path — it is the registration point for a faster path that renders it unused. When posted-interrupt mode engages, irqfd_wakeup() is never called for that vector, because vfio_msihandler() is never called either; the interrupt never becomes a host interrupt at all. The amber boxes matter just as much: the promotion is conditional and best-effort, and any of four conditions silently demotes you back to the software path, where everything still works but the exit is back.

The gate conditions in the diagram are read from vmx_pi_update_irte() in arch/x86/kvm/vmx/posted_intr.c (v6.12), whose comment states the restrictions plainly: “VT-d PI cannot support posting multicast/broadcast interrupts to a vCPU, we still use interrupt remapping for these kind of interrupts. For lowest-priority interrupts, we only support those with single CPU as the destination, e.g. user configures the interrupts via /proc/irq or uses irqbalance to make the interrupts single-CPU. […] In addition, we can only inject generic interrupts using the PI mechanism, refuse to route others through it.” There is a nice loop back to IRQ Affinity and irqbalance hidden in that comment: whether posted-interrupt delivery engages for a given vector can depend on whether the guest has spread that interrupt across multiple vCPUs.

The whole feature is gated at the top by one predicate in arch/x86/kvm/x86.c (v6.12):

bool kvm_arch_has_irq_bypass(void)
{
	return enable_apicv && irq_remapping_cap(IRQ_POSTING_CAP);
}

Both halves are required: APIC virtualization must be enabled (see APIC Virtualization (APICv and AVIC)) and the IOMMU must advertise interrupt-posting capability. Without APICv there is no virtual-APIC page for the posted interrupt to land in; without VT-d posting the IOMMU cannot address a posted-interrupt descriptor. If either is missing, kvm_irqfd_assign() skips consumer registration entirely and every VM in the system uses the software path. On the machine this note was edited on, /proc/interrupts carries the counters that would show this working — PIN (“Posted-interrupt notification event”), NPI (“Nested posted-interrupt event”) and PIW (“Posted-interrupt wakeup event”) — all reading 0, which is what you expect on a host with no running VMs and no assigned devices. Those three lines are the cheapest way to tell whether posted-interrupt delivery is actually happening on a hypervisor host. See Posted Interrupts for the descriptor format and the notification-vector mechanics.

Failure Modes and Subtleties

The trickiest real bugs are lifetime races, which is why eventfd.c is heavily commented as “race-free decouple logic.” Teardown is a two-phase dance: closing the watched eventfd sends EPOLLHUP, and irqfd_wakeup then deactivates the irqfd and queues irqfd_shutdown on a dedicated cleanup workqueue (irqfd_cleanup_wq), which first synchronize_srcu_expediteds, unhooks from the wait queue (eventfd_ctx_remove_wait_queue), flushes any in-flight inject work, and only then frees. A naive “close fd, free struct” would race the wakeup callback. A practical gotcha: the API explicitly notes that “closing the resamplefd is not sufficient to disable the irqfd” — you must KVM_IRQFD with KVM_IRQFD_FLAG_DEASSIGN (matching fd and gsi) to remove it.

For ioeventfd, the classic symptom of a misregistration is spurious exits to userspace: if the guest’s notify address or write width does not exactly match the registered (addr, len, datamatch), ioeventfd_in_range returns false, ioeventfd_write returns -EOPNOTSUPP, and the write falls through to normal MMIO emulation — a slow userspace exit on every kick, silently destroying throughput. Diagnose by checking that the guest driver’s notify offset/width matches what the VMM registered (a frequent bug when virtio-pci “notify multiplier” geometry is computed wrong). A second subtlety is the -EBUSY/-EEXIST returns: an eventfd may back at most one irqfd (kvm_irqfd_assign scans for and rejects a duplicate eventfd with -EBUSY), and overlapping ioeventfds collide with -EEXIST.

Previously-flagged claim, now resolved

An earlier revision of this note flagged the division of labour between the VFIO MSI-X eventfd handler and the IRQ-bypass producer registration as unverified. It has now been read directly. drivers/vfio/pci/vfio_pci_intrs.c at v6.12 calls irq_bypass_register_producer(&ctx->producer) in vfio_msi_set_vector_signal(), immediately after request_irq(irq, vfio_msihandler, 0, ctx->name, trigger), with producer.token = trigger (the eventfd_ctx *) and producer.irq = irq (the host IRQ number); it is torn down with irq_bypass_unregister_producer(&ctx->producer) on the disable path. The KVM side sets irqfd->consumer.token = (void *)irqfd->eventfd, the same pointer. The flag is retired; the mechanism is documented under The IRQ-bypass rendezvous above.

The error returns, and what each one actually means

Both ioctls fail in ways whose numeric codes are more informative than they look. Collected from virt/kvm/eventfd.c and arch/x86/kvm/irq.c at v6.12:

ioctl	Return	Trigger	What it usually means in practice
KVM_IRQFD	-EAGAIN	kvm_arch_intc_initialized(kvm) is false	The VM has no in-kernel interrupt controller yet. Either KVM_CREATE_IRQCHIP has not been called, or the VMM is configured for a fully userspace irqchip. The classic silent-slow-path cause
KVM_IRQFD	-EINVAL	kvm_arch_irqfd_allowed() said no	With KVM_IRQFD_FLAG_RESAMPLE on a split irqchip: resample needs the full in-kernel irqchip so KVM can observe the EOI
KVM_IRQFD	-EBUSY	an irqfd already exists for this eventfd	One eventfd backs at most one irqfd. Usually a VMM bug: reusing a descriptor across two GSIs
KVM_IRQFD	-ENOMEM / -EBADF	allocation, or fdget on a non-eventfd	The fd passed was not created by eventfd(2)
KVM_IOEVENTFD	-EINVAL	len not in {0,1,2,4,8}; addr + len overflow; unknown flag bits; DATAMATCH with len == 0	A malformed registration — caught at setup, so at least it is loud
KVM_IOEVENTFD	-EEXIST	ioeventfd_check_collision()	Two registrations claim the same write. Note the asymmetry: same address + same length + different datamatch is legal; add a zero-length entry at that address and everything collides
KVM_IOEVENTFD	-ENOENT	deassign of a registration that does not exist	The (addr, len, datamatch, flags) tuple must match the assignment exactly, including PIO vs MMIO
KVM_IOEVENTFD	-ENOSPC	kvm_io_bus_register_dev()	NR_IOBUS_DEVS exhausted — but see the Production Notes: ioeventfds are explicitly excluded from that limit

The first row deserves emphasis because it is the failure that costs the most performance for the least visible reason. -EAGAIN from KVM_IRQFD is not an error the guest can see; a VMM that treats it as “retry later” or logs it at debug level will run the entire VM with every completion interrupt going through a userspace ioctl, at full correctness and a fraction of the throughput.

Alternatives and When to Choose Them

The alternative to ioeventfd is plain MMIO emulation: every notify write VM-exits all the way to the VMM, which decodes it and kicks its own backend. That is the only option when the backend lives in userspace and you have not wired vhost/ioeventfd — it costs a full userspace round-trip per kick. The alternative to irqfd is the KVM_IRQ_LINE/KVM_SET_IRQ/KVM_SIGNAL_MSI ioctls: the VMM, having decided to raise an interrupt, makes an explicit ioctl into KVM. This is fine for low-rate, VMM-originated interrupts but requires the VMM to be scheduled and to make a syscall per interrupt; irqfd exists precisely so a non-VMM actor (kernel thread, hardware) can raise the interrupt without involving the VMM at all. The rule of thumb: use the explicit ioctls for control-plane and rare interrupts; use irqfd/ioeventfd for any high-rate datapath, which is every performance-sensitive virtio or passthrough device.

Laid out as a grid, with the guest→host and host→guest options each ranked from slowest to fastest:

Direction	Mechanism	Who must run per event	Exit class	When it is the right choice
Guest → host	Full MMIO/PIO emulation	KVM and the userspace VMM thread	Heavy	Registers with real side effects or return values — configuration space, status reads, anything the guest expects an answer from
	ioeventfd with datamatch	KVM only	Light	Several queues share one notify register: legacy virtio-pci, modern PIO notify, virtio-ccw
	ioeventfd, zero length (fast MMIO)	KVM only, no instruction decode	Light (cheapest)	Modern virtio-pci, one notify address per queue. The default in QEMU 9.x
	Guest-side notification suppression (EVENT_IDX)	nobody — the guest skips the write	None	Backend is actively polling; virtio negotiates this dynamically (virtio Notifications and Virtqueue Kicks)
Host → guest	KVM_IRQ_LINE / KVM_SET_IRQ / KVM_SIGNAL_MSI	the VMM thread, one syscall per interrupt	—	Control-plane interrupts the VMM itself originates, and low-rate emulated devices
	irqfd, workqueue fallback	KVM workqueue (schedule_work)	—	Automatic — used when the GSI route cannot be resolved atomically
	irqfd, atomic fast path	the signaller’s own context, inline	—	The normal case for MSI routes with a resolvable destination
	irqfd + IRQ bypass → posted interrupts	nobody — VT-d writes the guest’s PIR directly	None	Assigned devices with MSI/MSI-X, APICv enabled, single-vCPU destination
	Host-side suppression (vhost_notify() returns false)	nobody	None	The guest is already polling its used ring

Two patterns are worth reading off the table. First, both directions terminate in the same place: the fastest option is not a faster mechanism but no event at all, reached either by the guest suppressing its kick or the backend suppressing its interrupt. Every mechanism above those two rows is a way of making an event that must happen cheaper. Second, the escalation is not something you choose once — irqfd silently falls back from posted interrupts to the atomic path to the workqueue path depending on hardware capability and routing, all under the same registration, which is exactly why the observable symptom of a misconfiguration is “slower than expected” rather than “broken.”

Production Notes

The ioeventfd=on default, verified

QEMU enables ioeventfd for virtio devices by default, and the default is per-device-class rather than on the shared PCI proxy. For virtio-blk, from hw/virtio/virtio-blk-pci.c (v9.1.0):

static Property virtio_blk_pci_properties[] = {
    DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),   /* default: ON */
    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
                       DEV_NVECTORS_UNSPECIFIED),
    DEFINE_PROP_END_OF_LIST(),
};

The trailing true is the default value, so -device virtio-blk-pci,ioeventfd=off is the opt-out. The flag’s own header comment states the rationale — “Performance improves when virtqueue kick processing is decoupled from the vcpu thread using ioeventfd for some devices” (include/hw/virtio/virtio-pci.h, v9.1.0) — which is a slightly different framing from the one this note has used so far and worth holding alongside it: the win is not only “fewer exits” but “the vCPU thread does not do the backend’s work,” so a kick no longer steals vCPU time from the guest.

There is one place QEMU forcibly disables it regardless of the property, in virtio_pci_realize():

/* fd-based ioevents can't be synchronized in record/replay */
if (replay_mode != REPLAY_MODE_NONE) {
    proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
}

Deterministic record/replay cannot reproduce an eventfd wake-up at a deterministic instruction boundary, so the optimisation is switched off wholesale. This is a good reminder that ioeventfd changes when the backend observes the kick relative to the guest’s instruction stream, not merely how fast.

Firecracker and Cloud Hypervisor lean on ioeventfd/irqfd even harder — their minimal virtio-only device model is built around the assumption that the kick is an ioeventfd and the completion is an irqfd, which is part of why they boot fast and stay lean.

Uncertain

Verify: the specific claim that Firecracker and Cloud Hypervisor require (rather than merely prefer) ioeventfd/irqfd, and that this is a material contributor to their boot time. Reason: this is a plausible and widely-repeated characterisation, but neither project’s source was fetched during this task; the KVM-side and QEMU-side statements in this note are verified, this one is not. To resolve: read src/vmm/src/devices/virtio/ in firecracker-microvm/firecracker and vmm/src/device_manager.rs in cloud-hypervisor/cloud-hypervisor at a pinned tag and confirm whether a non-ioeventfd fallback path exists at all. uncertain

Scaling: how many ioeventfds is too many?

The number of ioeventfds scales with virtqueues — modern virtio-pci registers one per queue, and a multi-queue virtio-net device on a 32-vCPU guest can easily reach 64 or more per device. The in-kernel bus is a sorted array searched with bsearch(), so a lookup is O(log n) in the number of entries on that bus.

The often-repeated claim that this bumps into NR_IOBUS_DEVS is wrong, and it is worth correcting explicitly because the constant is easy to find and easy to misread. NR_IOBUS_DEVS is 1000 (include/linux/kvm_host.h, v6.12), but the check in kvm_io_bus_register_dev() subtracts the ioeventfd count before comparing, with a comment saying so:

/* exclude ioeventfd which is limited by maximum fd */
if (bus->dev_count - bus->ioeventfd_count > NR_IOBUS_DEVS - 1)
	return -ENOSPC;

So the 1000-device ceiling applies to emulated devices on the bus, and ioeventfds are deliberately exempt — their real ceiling is the VMM process’s open-file-descriptor limit (RLIMIT_NOFILE), since each one holds an eventfd. The practical consequence for operators is that a VM with hundreds of virtqueues needs its VMM’s nofile limit raised, not a KVM tunable; and the practical consequence for the datapath is that lookup cost grows only logarithmically, except within a datamatch group where it grows linearly — one more reason the modern per-queue-address layout is preferable to legacy datamatch multiplexing.

The mistake that costs the most

The single most common production mistake is leaving an in-kernel irqchip off (or a fully-userspace irqchip configured) and then wondering why irqfd setup fails with -EAGAIN and the whole vhost fast path silently degrades to userspace emulation. The subtler variant is split irqchip (KVM_IRQCHIP_SPLIT, QEMU’s -machine kernel_irqchip=split), which is the default for many modern configurations because it is required for interrupt remapping with an emulated IOMMU. Split irqchip passes irqchip_in_kernel() and so permits ordinary irqfds — but it fails irqchip_kernel(), so resampling irqfds are rejected with -EINVAL. A guest with an assigned legacy-INTx PCI device on a split-irqchip machine therefore loses the resample fast path specifically, while every MSI-X device on the same machine is unaffected. The symptom is a single device performing badly on an otherwise healthy host, which is exactly the shape of problem that gets misdiagnosed as a driver issue.

See Also

• Interrupt Injection and the Virtual APIC — what happens after an irqfd asserts a GSI: how the bit lands in the LAPIC IRR/ISR and gets injected
• vhost (In-Kernel virtio Backend) — registers these two eventfds with both vhost and KVM to escape the userspace datapath
• virtio Notifications and Virtqueue Kicks — the device-side view of the kick (ioeventfd) and the completion interrupt (irqfd), with notification suppression
• Posted Interrupts — the hardware fastest path the irqfd registers as an IRQ-bypass consumer to reach
• APIC Virtualization (APICv and AVIC) — when the interrupt can be delivered to a running vCPU with no exit
• VFIO Framework — passthrough devices whose hardware MSI/MSI-X eventfds become irqfds
• MMIO and Port IO Emulation — the slow path that ioeventfd short-circuits
• The KVM ioctl API — the VM-level ioctl surface these belong to
• Two-Dimensional Paging (EPT and NPT) — why the notify page traps at all, and what an “EPT misconfiguration” exit is
• The Network Receive Path — what the guest does after the irqfd fires: the NAPI poll loop the injected interrupt schedules
• Threaded Interrupt Handlers — the host-side counterpart on bare metal: vfio_msihandler() is a three-line hard-IRQ handler for exactly the reasons that note explains
• Linux Virtualization MOC — parent map (§8 Interrupt Virtualization)