vault

PCI and PCIe Enumeration

17 min read

PCI and PCIe Enumeration

Peripheral Component Interconnect (PCI) and its serial successor PCI Express (PCIe) are self-describing buses: every device exposes a standardized configuration space that the kernel reads to learn the device’s identity (vendor and device IDs), its class, and how much memory and I/O address space it needs. Enumeration is the boot-time and hot-plug process of walking that bus topology — visiting every possible bus, device, and function, reading each one’s config space, allocating the address windows it requested through its Base Address Registers (BARs), and minting a struct pci_dev into the unified device model. Each discovered pci_dev is registered on the pci_bus_type, where the model’s matching machinery pairs it with a struct pci_driver and calls that driver’s .probe() to bring the hardware to life (drivers/pci/probe.c, v6.12; drivers/pci/pci-driver.c, v6.12). This note traces that walk end to end against the Linux 6.12 LTS source.

Mental Model

A PCI system is a tree of buses joined by bridges. The root is the host bridge (the PCI controller built into the chipset/SoC); below it hang devices, and any device that is a PCI-to-PCI bridge opens a new downstream bus, recursively. Every endpoint is addressed by a Bus / Device / Function (BDF) triple: 8 bits of bus (256 buses per domain), 5 bits of device/slot (32 devices per bus), 3 bits of function (8 functions per device). Enumeration is a depth-first walk of this tree: at each device the kernel reads config space, and at each bridge it descends.

flowchart TB
  HB["Host bridge (pci_scan_root_bus)<br/>creates bus 0"] --> B0
  subgraph B0["Bus 0 — pci_scan_child_bus"]
    D0["00:00.0 endpoint<br/>(pci_scan_slot)"]
    BR1["00:01.0 PCI-to-PCI bridge<br/>(header type 1)"]
    D2["00:1f.6 endpoint NIC<br/>(BDF = enp0s31f6)"]
  end
  BR1 -->|"program PRIMARY=0,<br/>SECONDARY=1, recurse<br/>pci_scan_bridge"| B1
  subgraph B1["Bus 1 — child bus"]
    D10["01:00.0 GPU endpoint"]
    BR2["01:01.0 bridge"]
  end
  BR2 -->|"SECONDARY=2, recurse"| B2["Bus 2 ..."]
  D2 -.->|"read config space"| CFG["Config space (256B / 4KiB):<br/>VENDOR_ID DEVICE_ID<br/>COMMAND STATUS CLASS<br/>HEADER_TYPE<br/>BAR0..BAR5<br/>CAPABILITY_LIST -> MSI/MSI-X/PCIe"]
  CFG -.->|"size BARs,<br/>create pci_dev,<br/>pci_device_add"| MODEL["Device model:<br/>device_add on pci_bus_type<br/>-> match -> .probe()"]

The PCI topology as a recursively-discovered tree. What it shows: the host bridge mints bus 0; pci_scan_child_bus scans each slot’s functions, and every PCI-to-PCI bridge it finds gets its SECONDARY bus number programmed and is recursed into, deepening the tree; for each endpoint the kernel reads config space, sizes the BARs, and registers a pci_dev that the model then binds to a driver. The insight to take: there is no central registry of “what is plugged in” — the kernel learns the entire hardware inventory by physically reading standardized registers at every BDF coordinate, and the tree’s shape is dictated by where bridges sit.

Configuration Space — The Self-Description

Every PCI function has a configuration space: a small register file with a kernel-readable, standardized layout defined in include/uapi/linux/pci_regs.h. Conventional PCI defines 256 bytes (PCI_CFG_SPACE_SIZE 256); PCIe extends this to 4096 bytes (PCI_CFG_SPACE_EXP_SIZE 4096), the extra 3840 bytes holding extended capabilities. The first 64 bytes are the standard header; the rest is capability lists. The header’s first fields are what enumeration keys on:

  • PCI_VENDOR_ID 0x00 (16 bits) and PCI_DEVICE_ID 0x02 (16 bits) — the manufacturer-assigned identity. A read of all-ones (0xFFFF) at offset 0 means no device is present at that BDF, which is exactly how the scan detects empty slots.
  • PCI_COMMAND 0x04 — the control register. Its bits gate whether the device responds at all: PCI_COMMAND_IO 0x1 enables I/O-space decoding, PCI_COMMAND_MEMORY 0x2 enables memory-space decoding, and PCI_COMMAND_MASTER 0x4 enables bus mastering (the device’s ability to initiate DMA). At enumeration these are typically off; the driver turns them on.
  • PCI_STATUS 0x06, PCI_CLASS_REVISION 0x08, PCI_CLASS_DEVICE 0x0a — status flags and the device’s class code (e.g. “mass storage / NVMe”, “network / Ethernet”), used for class-based driver matching.
  • PCI_HEADER_TYPE 0x0e (8 bits) — the layout discriminator. The low seven bits (PCI_HEADER_TYPE_MASK 0x7f) select the layout: PCI_HEADER_TYPE_NORMAL 0 is a regular endpoint (six BARs), PCI_HEADER_TYPE_BRIDGE 1 is a PCI-to-PCI bridge (two BARs plus bus-number and window registers), PCI_HEADER_TYPE_CARDBUS 2 is a CardBus bridge. The top bit, PCI_HEADER_TYPE_MFD 0x80 (0x80), is the multi-function flag: if set on function 0, the device has more than one function and the scan must probe functions 1–7.
  • PCI_BASE_ADDRESS_0 0x10 through PCI_BASE_ADDRESS_5 0x24 — the six BARs (more below).
  • PCI_CAPABILITY_LIST 0x34 — a pointer to the head of a singly-linked capabilities list; walking it yields PCI_CAP_ID_MSI 0x05 (Message Signaled Interrupts), PCI_CAP_ID_MSIX 0x11 (MSI-X), and PCI_CAP_ID_EXP 0x10 (the PCI Express capability).
  • For a type-1 bridge, PCI_PRIMARY_BUS 0x18, PCI_SECONDARY_BUS 0x19, and PCI_SUBORDINATE_BUS 0x1a hold the three bus numbers that define the bridge’s downstream range.

The kernel reads these with pci_read_config_byte/word/dword (and writes with the pci_write_* counterparts), which dispatch through the host bridge’s config-access ops.

Two ways to reach config space — port I/O vs ECAM

How a config read physically happens depends on the host bridge. The legacy x86 mechanism is the port-I/O pair 0xCF8/0xCFC: write a BDF+offset address to the CONFIG_ADDRESS port 0xCF8, then read/write the CONFIG_DATA port 0xCFC. This only reaches the first 256 bytes — it cannot address extended config space.

The modern mechanism is ECAM, the Enhanced Configuration Access Mechanism (also called MMCONFIG / memory-mapped configuration). The host bridge maps the entire config space of every function into a contiguous physical memory region; a config access is just a normal memory load/store to a computed address. The kernel learns the region’s base from firmware — on ACPI systems from the MCFG ACPI table: “The MCFG table must describe the ECAM space of non-hot pluggable host bridges,” and “A host bridge consumes ECAM memory address space and converts memory accesses into PCI configuration accesses” (PCI/acpi-info, v6.12). The address of a function’s 4 KiB config block is computed by bit-packing the BDF into the offset from the ECAM base:

addr = ECAM_base + (bus << 20) + (device << 15) + (function << 12) + register_offset

Walking the shifts symbol by symbol: bus << 20 reserves bits 20–27 (8 bits → 256 buses), device << 15 reserves bits 15–19 (5 bits → 32 devices), function << 12 reserves bits 12–14 (3 bits → 8 functions), and bits 0–11 (12 bits → 4096) index the register within that function’s config block (OSDev PCI Express). ECAM is what makes the 4 KiB extended space reachable, and on ARM/PCIe-native systems it is described by Device Tree rather than MCFG.

Uncertain

Verify: the exact ECAM address formula and that bits 20–27/15–19/12–14 are the bus/device/function fields. Reason: the formula here comes from the PCIe base-spec convention as documented by OSDev and corroborated by the kernel’s MCFG handling, but it was not read off a primary specification PDF (PCIe spec is paywalled) at this tag. To resolve: cross-check against the PCI Firmware Specification §4.1.2 or arch/x86/pci/mmconfig*.c. uncertain

The Enumeration Walk — Step by Step

The walk begins at a host bridge. pci_scan_root_bus “initiates enumeration by creating a root bus” via pci_create_root_bus, then calls pci_scan_child_bus to discover everything below it; afterward it “updates the subordinate bus number based on discovered devices” (probe.c, v6.12).

pci_scan_child_bus (via pci_scan_child_bus_extend) “performs the actual device discovery by iterating through device slots (devfn 0–255) and calling pci_scan_slot.” It loops over all 256 devfn values on the bus. pci_scan_slot scans one slot: it calls pci_scan_single_device for function 0, and — only if function 0 reports the multi-function bit — for functions 1–7. (PCIe optimization: only_one_child short-circuits this for links that can structurally hold just one device, and next_fn handles Alternative Routing-ID Interpretation, ARI, which lets a device expose more than 8 functions.)

pci_scan_single_device → pci_scan_device does the presence test: it reads the vendor/device ID with pci_bus_read_dev_vendor_id (which includes Configuration Request Retry Status, RRS, retry logic for devices still initializing). If the read returns no valid vendor, the slot is empty and the scan moves on. Otherwise it allocates the device structure with pci_alloc_dev — which “allocates a zeroed pci_dev structure and initializes list heads and locks” — and calls pci_setup_device to fill it in.

pci_setup_device “reads the header type via pci_hdr_type, class via pci_class, and subsystem IDs.” For a normal endpoint it calls pci_read_bases(dev, 6, PCI_ROM_ADDRESS) to enumerate the six BARs; for a bridge it calls pci_read_bridge_windows to read the I/O, memory, and prefetchable-memory windows (probe.c, v6.12).

Finally pci_device_add “initializes the embedded device struct, sets DMA parameters, calls pci_init_capabilities to probe capabilities, and registers the device with the kernel device model via device_add.” That device_add call is the bridge into the unified device model: it places the pci_dev’s embedded struct device onto the pci_bus_type and triggers matching (see struct bus_type and Bus Registration and Device-Driver Matching).

Recursing into bridges and assigning bus numbers

Back in pci_scan_child_bus_extend, after scanning the current bus’s endpoints, the code “identifies PCI-to-PCI bridges and recursively invokes pci_scan_bridge_extend to enumerate subordinate buses.” Here is where the three bus-number fields earn their keep. Each bridge owns, in its config space, the PCI_PRIMARY_BUS register block: primary = the bus on the upstream side, secondary = the first bus number on the downstream side, subordinate = the last (highest) bus number reachable through this bridge. The kernel programs all three with a single pci_write_config_dword(dev, PCI_PRIMARY_BUS, buses) (probe.c, v6.12).

The classic algorithm is depth-first: when the scanner reaches a bridge it assigns the next free bus number as the bridge’s secondary, writes primary/secondary/subordinate (initially setting subordinate to 0xFF so all downstream config cycles are routed through during discovery), recurses to scan that secondary bus and everything below it, and on return writes back the true subordinate — the highest bus number it actually found. This is why subordinate numbers nest: a bridge’s [secondary, subordinate] range strictly contains the ranges of every bridge beneath it, and config-cycle routing works by each bridge forwarding only addresses whose bus falls in its [secondary, subordinate] window. pci_scan_bridge_extend runs in two passes — “First pass processes BIOS-configured bridges; second pass assigns numbers to unconfigured ones” — and can read fixed bus numbers from the Enhanced Allocation (EA) capability via pci_ea_fixed_busnrs.

BARs — Sizing and Resource Assignment

A BAR (Base Address Register) is how a device advertises a region of address space it needs decoded to it. A device does not pick its own base address; instead each BAR reports two things: what kind of region (memory or I/O, 32- or 64-bit, prefetchable or not) and how big. The kernel sizes a BAR with a clever write-and-read-back trick implemented in __pci_read_base: it “reads and sizes a single BAR by writing all-ones to the address register, reading back the size mask, then restoring the original value” (probe.c, v6.12). When you write 0xFFFFFFFF to a BAR, the device leaves the address bits it does not implement as zero, so the bits that read back as 1 above the type bits form a mask; the region size is the value of the lowest set bit of that mask. pci_size “computes BAR size from the mask by isolating the lowest set bit via size & ~(size-1)” — i.e. a BAR whose top settable bit is bit 20 decodes a 1 MiB region.

The low bits of a BAR are type flags, not address: PCI_BASE_ADDRESS_SPACE_IO 0x01 (bit 0 set) marks an I/O-space BAR versus PCI_BASE_ADDRESS_SPACE_MEMORY 0x00 for a memory BAR; among memory BARs, PCI_BASE_ADDRESS_MEM_TYPE_64 0x04 marks a 64-bit BAR (which consumes two consecutive BAR slots — the next BAR holds the upper 32 bits of the base, which is why __pci_read_base “returns 1 if the BAR is 64-bit, 0 if 32-bit” so the caller can skip the consumed slot), and PCI_BASE_ADDRESS_MEM_PREFETCH 0x08 marks the region as prefetchable (no side effects on read, so the CPU/bridge may prefetch and combine writes). The kernel records each sized BAR as a struct resource in the device’s resource[] array, converting bus addresses to CPU physical addresses via pcibios_bus_to_resource. Actual base addresses are assigned later (by firmware, or by the kernel’s resource allocator if firmware left a BAR unprogrammed) so that all regions across the whole tree fit without overlap inside the host bridge’s address windows.

The pci_dev / pci_driver Model and Matching

Each enumerated function is a struct pci_dev (include/linux/pci.h, v6.12). Its key fields: the identity (unsigned short vendor, device, subsystem_vendor, subsystem_device; unsigned int class); the topology (struct pci_bus *bus, unsigned int devfn); the sized regions (struct resource resource[DEVICE_COUNT_RESOURCE]); interrupt state (unsigned int irq, the msi_enabled/msix_enabled flags); the bound struct pci_driver *driver; and crucially struct device dev — the embedded generic device that makes a pci_dev a first-class node in the device model. The macro #define to_pci_dev(n) container_of(n, struct pci_dev, dev) recovers the pci_dev from a struct device *.

A driver is a struct pci_driver with const char *name, a const struct pci_device_id *id_table (the list of IDs it claims), and the callbacks probe, remove, suspend, resume, shutdown, plus an embedded struct device_driver driver. The match table entries are struct pci_device_id with fields vendor, device, subvendor, subdevice, class, class_mask, driver_data; any field set to PCI_ANY_ID is a wildcard. The convenience macros build common entries: PCI_DEVICE(vend, dev) matches a specific vendor/device with subsystem fields wildcarded; PCI_DEVICE_CLASS(class, mask) matches by class with all ID fields wildcarded; PCI_VDEVICE(vend, dev) uses the PCI_VENDOR_ID_##vend symbolic constant; PCI_DEVICE_DATA(vend, dev, data) attaches a driver_data payload the probe can retrieve.

Matching is the pci_bus_type’s job (pci-driver.c, v6.12):

const struct bus_type pci_bus_type = {
	.match		= pci_bus_match,
	.probe		= pci_device_probe,
	.remove		= pci_device_remove,
	.uevent		= pci_uevent,
	/* ... */
};

When the model needs to test a device against a driver it calls .match = pci_bus_match, which calls pci_match_device. That function “Look[s] at the dynamic ids first, before the static ones” (drivers can gain IDs at runtime via sysfs new_id), then iterates drv->id_table through pci_match_id, comparing vendor/device/subvendor/subdevice/class via pci_match_one_device. On a match the model calls .probe = pci_device_probe → __pci_device_probe → pci_call_probe → local_pci_probe, which does the load-bearing line:

pci_dev->driver = pci_drv;
rc = pci_drv->probe(pci_dev, ddi->id);

— it records the bound driver on the pci_dev and then invokes the driver’s own .probe(), passing the matching pci_device_id so the probe can branch on driver_data. Drivers register with pci_register_driver(drv) (a macro over __pci_register_driver, which “initializes the dynamic ID list and calls the core driver_register function”); a whole driver collapses to one module_pci_driver(my_driver) macro that synthesizes the module init/exit (pci.h, v6.12). This is the generic bind flow specialized for PCI — the only PCI-specific part is the match function.

What a Driver’s .probe() Does

Enumeration leaves a device present but inert: address windows are sized but decoding may be off and bus mastering is off. The driver’s .probe() activates it (Documentation/PCI/pci.rst):

static int my_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
	int err;
 
	err = pci_enable_device(pdev);          /* (1) */
	if (err) return err;
	err = pci_request_regions(pdev, "mydrv"); /* (2) */
	if (err) goto disable;
	pci_set_master(pdev);                   /* (3) */
	dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); /* (4) */
	err = pci_alloc_irq_vectors(pdev, 1, nvecs, PCI_IRQ_MSIX | PCI_IRQ_MSI); /* (5) */
	regs = pci_iomap(pdev, 0, 0);           /* (6) */
	/* ... register subsystem device ... */
}

(1) pci_enable_device — per the source it “Ask[s] low-level code to enable I/O and memory,” calling pci_enable_device_flags(dev, IORESOURCE_MEM | IORESOURCE_IO), which sets the PCI_COMMAND_IO/PCI_COMMAND_MEMORY bits so the device begins decoding its BARs, and brings it to power state D0 (pci.c, v6.12). (2) pci_request_regions reserves the BAR resources so two drivers cannot claim the same window. (3) pci_set_master sets PCI_COMMAND_MASTER, enabling bus mastering — without it the device cannot DMA. (4) dma_set_mask_and_coherent declares the device’s DMA addressing reach (see The DMA API and, for address translation/isolation, The IOMMU). (5) pci_alloc_irq_vectors allocates MSI-X, falling back to MSI, falling back to legacy INTx (the flags PCI_IRQ_MSIX | PCI_IRQ_MSI permit both); pci_irq_vector(pdev, n) then yields the Linux IRQ number to pass to request_irq. (6) pci_iomap/pci_ioremap_bar maps a memory BAR into kernel virtual address space — the mechanism behind Memory-Mapped IO and ioremap — after which readl/writel touch the device’s registers.

Failure Modes and Common Misunderstandings

“My device enumerates (lspci shows it) but my driver’s probe never runs.” The device is in the model, so enumeration succeeded; the failure is in matching. The id_table lacks the device’s vendor/device (or its class_mask is wrong), or the driver module is not loaded. Check /sys/bus/pci/devices/0000:00:1f.6/ for the device and /sys/bus/pci/drivers/<name>/ for the driver; force a match with echo "vendor device" > /sys/bus/pci/drivers/<name>/new_id.

“lspci shows the device but BARs read as <unassigned>.” Firmware left a BAR unprogrammed and the kernel’s resource allocator could not fit it (often a large 64-bit BAR above 4 GiB on a system without the “above-4G decoding” / resizable-BAR firmware option). The kernel logs BAR N: no space for ... / BAR N: failed to assign. Enabling above-4G decoding or 64-bit-aware firmware resolves it; this is the classic GPU-passthrough headache.

“Reads of config space return all 0xFF after a fault.” A device that has gone offline (surprise removal, an Advanced Error Reporting fatal error) returns 0xFFFFFFFF for all config reads — the same pattern as an empty slot. Code must distinguish “absent” from “errored,” which is why pci_dev_is_present exists and why drivers should not treat an all-ones read as valid data.

Multi-function confusion. Forgetting the PCI_HEADER_TYPE_MFD 0x80 check means functions 1–7 are never scanned; conversely, probing functions 1–7 on a single-function device wastes config cycles and can confuse buggy hardware. The kernel keys strictly on the bit in function 0’s header type.

Alternatives and When to Choose Them

PCI/PCIe enumeration is the discovery path for self-describing buses. The siblings in the device model differ by how the device is discovered, not how it is bound:

  • USB (USB Device Enumeration) is also self-describing and hot-pluggable, but the host controller walks a physical topology of hubs and reads USB descriptors rather than a flat BDF config space.
  • Platform devices (Platform Devices and Drivers) are not discoverable: SoC-integrated peripherals have no config space, so firmware must describe them via Device Tree or ACPI. There is no scan; the device is instantiated from a firmware node.
  • PCIe is the convergence point: even on ARM servers and embedded systems, PCIe slots enumerate by the same config-space walk, with ECAM described by Device Tree instead of the x86 MCFG table.

Choose to think in PCI terms whenever the device is on a real or emulated PCIe link (NICs, NVMe, GPUs, virtio devices in a VM); think in Device-Tree/ACPI terms for on-die SoC blocks.

Production Notes

In virtual machines, virtio devices appear as ordinary PCI devices and enumerate through this exact path — which is why lspci inside a guest shows Virtio network device at a normal BDF and why predictable interface names like enp0s3 work in VMs (the BDF-derived path naming in udev Rules and Predictable Device Naming is computed from precisely the bus/device/function this enumeration assigns). Hot-plug reuses the machinery: when a device is added to a powered slot, the PCIe hot-plug controller fires an interrupt, and the kernel runs pci_scan_slot/pci_device_add on just that slot rather than re-walking the whole tree. SR-IOV (Single Root I/O Virtualization) makes one physical function spawn many lightweight virtual functions, each enumerating as its own pci_dev with its own BDF — the reason a single NIC can present dozens of pci_devs. And VFIO, the foundation of device passthrough to VMs, hands a whole pci_dev (config space, BARs, interrupts) to userspace behind the IOMMU’s protection — built directly on the pci_dev abstraction this enumeration produces.

See Also

Graph View

Backlinks