vault

IRQ Affinity and irqbalance

14 min read

IRQ Affinity and irqbalance

IRQ affinity is the policy that decides which CPU services which interrupt. Every interrupt line in Linux carries a CPU mask — the set of processors its handler is allowed to run on — exposed to userspace as /proc/irq/<N>/smp_affinity (a hex bitmask) and /proc/irq/<N>/smp_affinity_list (a CPU list) (Documentation/core-api/irq/irq-affinity.rst). Steering interrupts matters for two opposite reasons: throughput, where you want a busy NIC’s interrupts spread across many CPUs so no single core is the bottleneck and cache lines stay warm — the job of the userspace irqbalance daemon, which periodically redistributes IRQs (irqbalance.1); and latency / determinism, where you want interrupts off a CPU entirely so a real-time or HPC thread runs uninterrupted — the job of static pinning plus CPU isolation. Layered on top, modern multiqueue devices use managed IRQs: the kernel itself spreads one hardware queue per CPU and marks those lines so that neither userspace nor irqbalance may move them (the IRQD_AFFINITY_MANAGED flag).

This note is pinned to Linux 6.12 LTS (released 2024-11-17), verified against the raw v6.12 source blobs in sources. The irqbalance details are from the project’s master man page and README — flagged below where they could drift.

Mental Model

An interrupt line is not bound to one CPU by nature; it has a mask of permitted CPUs, and when the line fires the interrupt controller picks one CPU from that mask to deliver to. Tuning affinity is editing that mask. Three actors edit it, in a strict precedence:

  1. The kernel, for managed IRQs — non-negotiable, owns the mask, ignores everyone else.
  2. irqbalance, the userspace daemon — re-edits masks of unmanaged IRQs every few seconds for throughput.
  3. You, via echo … > /proc/irq/N/smp_affinity — static pinning, but only on unmanaged IRQs, and irqbalance will fight you unless you ban the IRQ from it.
flowchart TB
  subgraph KERNEL["Kernel side (per IRQ line)"]
    DESC["irq_desc.irq_data.affinity<br/>(the CPU mask)"]
    MANAGED{"IRQD_AFFINITY_MANAGED?"}
    MANAGED -->|"yes (NVMe / blk-mq / NIC queues)"| KMGMT["Kernel spreads 1 queue/CPU,<br/>locks the mask<br/>(userspace writes rejected)"]
    MANAGED -->|"no (legacy / shared lines)"| OPEN["Mask is writable"]
  end
  OPEN --> WHO{"Who steers it?"}
  WHO -->|"userspace daemon"| IRQB["irqbalance:<br/>every ~10s, read /proc/interrupts,<br/>spread hot IRQs across CPUs"]
  WHO -->|"admin"| PIN["Static pin:<br/>echo mask > smp_affinity<br/>+ ban from irqbalance"]
  IRQB --> DESC
  PIN --> DESC
  KMGMT --> DESC
  DESC --> CTRL["irq_chip->irq_set_affinity()<br/>programs APIC/GIC routing"]

The affinity decision hierarchy. What it shows: every IRQ line has one CPU mask; whether the kernel, irqbalance, or the admin gets to set it depends first on the managed flag and then on banning. The insight to take: managed IRQs are the kernel’s exclusive domain (it spreads one queue per CPU and refuses userspace edits), while everything else is a tug-of-war between irqbalance and manual pinning that you resolve by banning an IRQ from irqbalance when you want to own it. Affinity is a mask, not a single CPU — the controller still chooses within it.

The /proc Interface

The kernel exposes one directory per IRQ under /proc/irq/. The two files that matter for steering are smp_affinity and smp_affinity_list, described as: “/proc/irq/IRQ#/smp_affinity and /proc/irq/IRQ#/smp_affinity_list specify which target CPUs are permitted for a given IRQ source” (irq-affinity.rst). They are two views of the same mask — smp_affinity is a hex bitmask, smp_affinity_list is a human-readable range.

The documentation’s worked example uses IRQ 44 (an eth1 line). Initially it is allowed on all CPUs:

[root@moon 44]# cat smp_affinity
ffffffff

Restrict it to CPUs 0–3 by writing the bitmask 0x0f:

[root@moon 44]# echo 0f > smp_affinity
[root@moon 44]# cat smp_affinity
0000000f

After which “IRQ44 was delivered only to the first four processors (0-3)” — verifiable by watching that IRQ’s per-CPU counters in /proc/interrupts. Move it to CPUs 4–7 with 0xf0:

[root@moon 44]# echo f0 > smp_affinity

For machines with hundreds of CPUs the bitmask is unwieldy, so the list form is preferred:

[root@moon 44]# echo 1024-1031 > smp_affinity_list
[root@moon 44]# cat smp_affinity_list
1024-1031

Two rules from the doc bound what you can do: “It’s not allowed to turn off all CPUs” (an empty mask is rejected — an interrupt must land somewhere), and “if an IRQ controller does not support IRQ affinity then the value will not change from the default of all cpus.” A third file, /proc/irq/default_smp_affinity, “specifies default affinity mask that applies to all non-active IRQs” and defaults to 0xffffffff — the template a newly-activated IRQ inherits before anyone steers it.

What the Kernel Does With a Write — irq_set_affinity

Writing smp_affinity ends up in the kernel’s irq_set_affinity path (kernel/irq/manage.c). The call chain is irq_set_affinity → __irq_set_affinity → irq_set_affinity_locked → irq_do_set_affinity, taking the descriptor’s lock along the way. The real work is in irq_do_set_affinity:

  1. Online-CPU validation. The requested mask is intersected with cpu_online_mask; an affinity that names only offline CPUs is rejected (unless force). You cannot route an interrupt to a CPU that is not running.
  2. Housekeeping filtering for managed IRQs. When the IRQ is managed and CPU isolation (isolcpus/nohz_full) is configured, the kernel intersects the mask with the housekeeping CPUs, with the explicit comment: “This prevents the affinity setter from routing the interrupt to an isolated CPU to avoid that I/O submitted from a housekeeping CPU causes interrupts on an isolated one.” This is the direct hook between affinity and CPU isolation.
  3. Chip callback. Finally it calls chip->irq_set_affinity(data, mask, force), which is where the irq_chip driver actually reprograms the hardware routing — rewriting an I/O APIC redirection-table entry, an MSI message address on x86, or the GIC distributor’s target register on ARM. The generic layer never touches the controller; the chip driver does.

Whether a userspace write is permitted at all is gated earlier by irq_can_set_affinity_usr, which checks __irq_can_set_affinity(desc) (descriptor exists, balancing allowed, chip implements irq_set_affinity) and that the IRQ is not managed:

bool irq_can_set_affinity_usr(unsigned int irq)
{
        struct irq_desc *desc = irq_to_desc(irq);
        return __irq_can_set_affinity(desc) &&
                !irqd_affinity_is_managed(&desc->irq_data);
}

This is the line that makes writes to a managed IRQ’s smp_affinity fail — and it is why you sometimes see Cannot change IRQ affinity from tools that try.

Drivers, IRQF_NOBALANCING, and the IRQD_NO_BALANCING Flag

A driver that wants an IRQ pinned and never moved by automatic balancing requests it with the IRQF_NOBALANCING flag. At setup time the kernel records this on the descriptor (manage.c):

irq_settings_set_no_balancing(desc);
irqd_set(&desc->irq_data, IRQD_NO_BALANCING);

IRQD_NO_BALANCING “prevents the kernel from automatically migrating the interrupt across CPUs for load balancing purposes.” The classic user is the per-CPU timer IPI and similar lines that must stay where they are. It is important to keep two distinct concepts straight, because the brief conflates the symptoms and they behave differently:

  • IRQD_NO_BALANCING (from IRQF_NOBALANCING): blocks both the kernel’s internal balancing and makes the line ineligible for balancing — the affinity is frozen by the driver.
  • IRQD_AFFINITY_MANAGED (managed IRQs, below): blocks userspace changes, but the kernel’s own affinity mechanism still manages it. As the introducing commit states: “Interrupts marked with this flag are excluded from user space interrupt affinity changes. Contrary to the IRQ_NO_BALANCING flag, the kernel internal affinity mechanism is not blocked” (patchwork, hch 2016).

So “no-balancing” freezes the mask outright; “managed” hands the mask to the kernel and locks out userspace.

Managed IRQs — the Kernel Spreads One Queue Per CPU

Modern high-performance devices — NVMe SSDs, multiqueue NICs, anything on blk-mq — do not have one interrupt; they have many, ideally one hardware queue per CPU, so that a CPU submitting I/O receives the completion interrupt on itself, keeping the whole submit→complete path on one core’s caches. Hand-tuning that with smp_affinity would be hopeless and fragile, so the kernel does it automatically. When a driver calls pci_alloc_irq_vectors() with the PCI_IRQ_AFFINITY flag, “pci_alloc_irq_vectors() will spread the interrupts around the available CPUs” (msi-howto.rst).

The spreading algorithm is irq_create_affinity_masks → group_cpus_evenly (kernel/irq/affinity.c). It divides the available vectors among CPU groups evenly (NUMA-aware), copies each resulting CPU set into an irq_affinity_desc, and — crucially — marks the spread vectors as managed:

for (i = affd->pre_vectors; i < nvecs - affd->post_vectors; i++)
        masks[i].is_managed = 1;

Setting is_managed = 1 causes each line to carry IRQD_AFFINITY_MANAGED. From then on the kernel owns that mask: it spreads the queues at allocation, and on CPU hotplug it re-migrates them automatically — “If all housekeeping CPUs in the affinity mask are offline, the interrupt will be migrated by the CPU hotplug code once a housekeeping CPU which belongs to the affinity mask comes online” (manage.c). The original motivation, per the LWN coverage of the patch series, was to “allow spreading around MSI and MSI-X vectors so that they have per-cpu affinity if possible, or at least per-node,” taking the algorithm proven in blk-mq and generalizing it, demonstrated on the NVMe driver (LWN 693653). The payoff is that the queue→CPU mapping is consistent with the block layer’s view, so completions land on the submitting CPU.

The practical rule that falls out: irqbalance must not touch managed IRQs, and it cannot — the kernel rejects the userspace write via irq_can_set_affinity_usr. On an NVMe-heavy box, most of the high-traffic IRQs are managed and already optimally spread; irqbalance has nothing to do for them.

irqbalance — the Userspace Throughput Daemon

irqbalance is “a daemon to help balance the cpu load generated by interrupts across all of a systems cpus” (README.md). Its strategy is not naive round-robin: “irqbalance identifies the highest volume interrupt sources, and isolates each of them to a single unique cpu, so that load is spread as much as possible over an entire processor set, while minimizing cache miss rates for irq handlers.” In other words it pins each hot IRQ to its own CPU (so its handler’s working set stays cache-resident) rather than smearing one IRQ across many cores.

Mechanically it loops: every interval it samples /proc/interrupts to find per-IRQ load, builds a model of the cache/NUMA topology, decides a new placement, and writes the chosen masks to each smp_affinity. The sampling interval is the --interval/-t option — “irqbalance will sleep for <time> seconds between samples of the irq load on the system cpus. Defaults to 10” (irqbalance.1). Key options for an operator:

  • --banirq <N> / -i — “Add the specified IRQ to the set of banned IRQs. irqbalance will not affect the affinity of any IRQs on the banned list, allowing them to be specified manually.” This is how you reserve an IRQ for your own static pinning.
  • --banmod <module> — ban all IRQs belonging to a module — the bulk version of --banirq.
  • --policyscript <path> / -l — “the referenced script or directory will execute once for each discovered IRQ, with the sysfs device path and IRQ number passed as arguments,” letting you script per-IRQ policy (ban, hint, etc.).
  • IRQBALANCE_BANNED_CPULIST / IRQBALANCE_BANNED_CPUS (environment) — “Provides a cpulist [or mask] which irqbalance should ignore and never assign interrupts to.” This is the standard way to keep irqbalance from ever placing an interrupt on your isolated CPUs.
  • --oneshot / -o — run the balance once and exit, useful for a boot-time placement without a resident daemon.
  • --powerthresh / -p — move CPUs into powersave (no IRQs) when enough cores are idle, the thermal/power-saving angle.

Uncertain

Verify: the precise default value of --interval (documented as 10 s) and the exact set/spelling of options (--banmod, --policyscript, IRQBALANCE_BANNED_CPULIST). Reason: these are quoted from the irqbalance project’s master branch man page, not from a pinned irqbalance release, and distributions ship varying versions; the kernel side (v6.12) is verified, but irqbalance versions independently of the kernel. To resolve: check man irqbalance on the actual target distro/version. uncertain

Static Pinning vs irqbalance — When to Choose Which

The two strategies optimize for opposite goals, and the right choice is workload-dependent.

irqbalance (dynamic) — choose for general-purpose throughput. A web server, database, or mixed workload benefits from irqbalance automatically following the hot IRQs and spreading them; you do not want to hand-tune dozens of lines, and the cache-locality heuristic generally helps. This is the distro default on most general-purpose systems.

Static pinning (manual) — choose for latency and determinism. A real-time control loop, a low-latency trading engine, a DPDK/network-function dataplane, or any workload with a CPU that must never be interrupted needs the opposite: pin device IRQs to a small set of housekeeping CPUs, and ban irqbalance from undoing it. The recipe is: ban the IRQ from irqbalance (--banirq or IRQBALANCE_BANNED_CPULIST for the isolated cores), then echo <housekeeping-mask> > /proc/irq/<N>/smp_affinity. Without the ban, irqbalance will silently move the IRQ back on its next interval — a classic “my pinning keeps reverting” bug.

The two are not mutually exclusive: a common production layout runs irqbalance for the bulk of IRQs while banning the handful tied to isolated CPUs, so the housekeeping cores get balanced and the isolated cores stay quiet.

Interaction With CPU Isolation

The whole point of isolcpus / nohz_full (see CPU Isolation isolcpus and nohz_full) is to give a CPU to a single thread with no kernel noise — and an interrupt is noise. Keeping IRQs off isolated CPUs requires coordination on three fronts:

  1. Managed IRQs — handled by the kernel: irq_do_set_affinity filters managed-IRQ masks down to housekeeping CPUs, as quoted above, so the automatic per-CPU spread skips the isolated set.
  2. irqbalance — must be told via IRQBALANCE_BANNED_CPULIST (or the mask form) never to place an IRQ on the isolated cores.
  3. Static lines — any IRQ you pin by hand must be pointed at housekeeping CPUs only.

Get all three right and the isolated CPU shows near-zero counts across its /proc/interrupts rows; miss one and a single rogue device IRQ lands on the latency-critical core every so often and blows the tail latency. This is why low-latency tuning guides treat IRQ affinity and CPU isolation as one combined task, not two.

Failure Modes and Diagnosis

“My pinning keeps reverting.” irqbalance overwrote your manual smp_affinity on its next interval. Fix: --banirq the line or stop the daemon; verify with watch -n1 cat /proc/irq/<N>/smp_affinity.

“Cannot change IRQ affinity” on write. The IRQ is managed (IRQD_AFFINITY_MANAGED) — irq_can_set_affinity_usr rejects the userspace write by design. This is not a bug; the kernel owns that line. Check via the irq’s chip/flags; managed NVMe/NIC-queue IRQs are expected to refuse manual steering. (Reported repeatedly against irqbalance, e.g. on 6.12.x kernels, where the daemon logs a failure trying to move a managed line.)

Interrupt lands on an isolated CPU anyway. One of the three fronts above was missed — most often irqbalance was not given the banned cpulist, or a hand-pinned line points at an isolated core. Diagnose by reading /proc/interrupts and checking which CPU column is incrementing for the offending IRQ.

All interrupts pile on CPU 0. Either irqbalance is not running and nobody spread the unmanaged lines, or the controller does not support affinity (the mask is stuck at all-CPUs and the hardware always picks the lowest). Confirm with cat /proc/interrupts — a lopsided CPU 0 column is the tell.

Production Notes

The standard diagnostic is /proc/interrupts: rows are IRQs, columns are CPUs, cells are per-CPU counts; watching which column grows tells you where an interrupt is actually landing, independent of what smp_affinity claims. For NIC tuning specifically, vendors ship scripts (e.g. Intel’s set_irq_affinity) that pin each queue’s IRQ to a distinct CPU and disable irqbalance, because for a multiqueue NIC the queue→CPU mapping should match the RX/TX-queue-to-CPU mapping (RPS/XPS) — letting irqbalance reshuffle it breaks that alignment. On NVMe and other blk-mq storage the kernel already does the right thing via managed IRQs, so the guidance is usually leave them alone. The recurring real-world lesson is that affinity is a system property: the IRQ mask, the irqbalance ban list, the CPU-isolation set, and the device’s queue mapping must all agree, or one of them silently undoes the others.

See Also

Graph View

Backlinks