vault

Uevents and the Kernel-Userspace Netlink Channel

14 min read

Uevents and the Kernel-Userspace Netlink Channel

A uevent (“userspace event”) is the kernel’s push notification that a device appeared, changed, or disappeared. Where sysfs is a pull interface (userspace reads files when it wants to), hotplug needs the opposite: the instant a USB stick is inserted the kernel must tell userspace. It does so by calling kobject_uevent(kobj, action) with an action from the kobject_action enum (KOBJ_ADD, KOBJ_REMOVE, KOBJ_CHANGE, KOBJ_MOVE, KOBJ_ONLINE, KOBJ_OFFLINE, KOBJ_BIND, KOBJ_UNBIND), building an environment of KEY=value strings (ACTION, DEVPATH, SUBSYSTEM, MODALIAS, …), and broadcasting that environment over a netlink socket of family NETLINK_KOBJECT_UEVENT (kobject_uevent.c, v6.12). The userspace device manager udev (systemd-udevd) listens on that socket and acts — creating /dev nodes, loading modules, naming network interfaces. Pinned to the 6.12 Long-Term-Support (LTS) kernel (released 2024-11-17). NETLINK_KOBJECT_UEVENT is netlink protocol number 15 (netlink.h).

Mental Model

Think of the uevent path as a one-way loudspeaker bolted onto the kobject graph. Every change to that graph — a new struct device registered, a driver binding, a disk going offline — can be announced. The announcement is not the data itself; it is a small message saying “this thing, at this sysfs path, in this subsystem, just did this.” Userspace receives the announcement and, if it cares, reads the actual details back out of /sys (or acts on the variables carried in the message). The message travels over netlink, the kernel’s socket-based messaging family, as a broadcast to multicast group 1 — so every listener (udev, and anything else that subscribes) gets every event without the kernel tracking subscribers. The key design point: the kernel never blocks on userspace. It fires the broadcast and moves on; if no one is listening, the event is simply dropped.

flowchart LR
  CHG["device added /<br/>driver bound /<br/>disk offlined"] --> KU["kobject_uevent(kobj,<br/>KOBJ_ADD)"]
  KU --> ENV["build env:<br/>ACTION DEVPATH<br/>SUBSYSTEM SEQNUM"]
  ENV -->|"bus->uevent /<br/>class->dev_uevent /<br/>type->uevent"| MORE["add MODALIAS<br/>DEVTYPE DRIVER ..."]
  MORE --> NL["netlink_broadcast()<br/>NETLINK_KOBJECT_UEVENT<br/>(group 1)"]
  NL --> UD["systemd-udevd<br/>(listening socket)"]
  UD -->|"apply rules"| ACT["mknod /dev node<br/>load module<br/>name iface"]
  KU -.->|"legacy, early boot only"| HELP["fork uevent_helper<br/>(/sbin/hotplug)"]

The uevent pipeline from a graph change to udev’s action. What it shows: kobject_uevent builds a default environment, lets the subsystem hooks (bus->uevent, class->dev_uevent, type->uevent) enrich it with MODALIAS/DEVTYPE/DRIVER, then netlink_broadcasts it to group 1, where systemd-udevd receives it and applies rules. The insight: the modern path is the netlink broadcast (top); the dashed uevent_helper fork (bottom) is the legacy mechanism, effectively dead except for an early-boot corner. The subsystem hooks are where MODALIAS — the string that lets udev autoload the right driver — is injected.

The kobject_action Enum: What Can Be Announced

The vocabulary of events is fixed in include/linux/kobject.h, and the header is emphatic that it is not extensible casually:

/* Do not add new actions here without checking with the driver-core
 * maintainers. Action strings are not meant to express subsystem
 * or device specific properties. ... you want to send a
 * kobject_uevent_env(kobj, KOBJ_CHANGE, env) with additional event
 * specific variables added to the event environment. */
enum kobject_action {
	KOBJ_ADD,
	KOBJ_REMOVE,
	KOBJ_CHANGE,
	KOBJ_MOVE,
	KOBJ_ONLINE,
	KOBJ_OFFLINE,
	KOBJ_BIND,
	KOBJ_UNBIND,
};

Each enum value maps to a lowercase string the wire protocol carries, via a parallel table in lib/kobject_uevent.c (the comment “the strings here must match the enum” enforces index alignment):

static const char *kobject_actions[] = {
	[KOBJ_ADD]	= "add",     [KOBJ_REMOVE]  = "remove",
	[KOBJ_CHANGE]	= "change",  [KOBJ_MOVE]    = "move",
	[KOBJ_ONLINE]	= "online",  [KOBJ_OFFLINE] = "offline",
	[KOBJ_BIND]	= "bind",    [KOBJ_UNBIND]  = "unbind",
};

What each means in device-model terms: add/remove fire when a device is registered/unregistered (device_add/device_del) — the bread-and-butter hotplug events. change is the catch-all “something about me changed, re-read me” event; the header steers subsystems toward KOBJ_CHANGE plus custom environment variables rather than inventing new actions. move fires when a kobject is renamed/reparented (kobject_move). online/offline are for logical onlining of hot-pluggable resources (CPUs, memory blocks) via the online sysfs attribute. bind/unbind fire when a driver claims or releases a device — distinct from add/remove, because a device can exist (added) yet have no driver (unbound). The KOBJ_UNBIND case is special-cased in the code (below) to strip MODALIAS from the environment, since after unbinding the device’s module-alias is no longer the relevant signal.

Uncertain

Verify: the exact kernel release that introduced KOBJ_BIND/KOBJ_UNBIND (commonly cited as the 4.x series). Reason: these actions are present and load-bearing in the v6.12 enum that was read directly, but the introducing version was not checked against a primary commit/changelog during this task. To resolve: git log -S KOBJ_BIND include/linux/kobject.h against the mainline tree, or check the commit that added them. The behavior described here is verified against v6.12; only the introduction date is unpinned. uncertain

Mechanical Walk-through: kobject_uevent_env

The real work is in kobject_uevent_env() in lib/kobject_uevent.c. (kobject_uevent(kobj, action) is just kobject_uevent_env(kobj, action, NULL).) Trace it:

1. Find the owning kset and its uevent_ops. A uevent is filtered and named by the kset the kobject belongs to, so the function walks up parent pointers to the nearest kset. A kobject with no kset cannot send a uevent at all — it returns -EINVAL:

	top_kobj = kobj;
	while (!top_kobj->kset && top_kobj->parent)
		top_kobj = top_kobj->parent;
	if (!top_kobj->kset)
		return -EINVAL;             /* no kset -> no uevent */
	kset = top_kobj->kset;
	uevent_ops = kset->uevent_ops;

2. Honor suppression and the filter. If kobj->uevent_suppress is set the event is silently dropped (drivers set this to batch-register many children and emit one event). Then the kset’s filter callback runs; if it returns false, drop. For all devices the kset is devices_kset with device_uevent_ops, whose filter (dev_uevent_filter in drivers/base/core.c) returns true only for objects that are real devices with a bus or a class — which is why bare kobjects and busless/classless devices stay quiet:

	if (kobj->uevent_suppress)        return 0;   /* batched/suppressed */
	if (uevent_ops && uevent_ops->filter)
		if (!uevent_ops->filter(kobj)) return 0;  /* filtered out */

3. Determine the subsystem name. The kset’s name callback supplies SUBSYSTEM=. For devices, dev_uevent_name returns the bus name if the device is on a bus, else the class name — so a NIC reports SUBSYSTEM=pci or SUBSYSTEM=net depending on which side you look at (the device appears under both).

4. Build the default environment. Allocate a struct kobj_uevent_env (a fixed buffer: UEVENT_NUM_ENVP = 64 pointers into a UEVENT_BUFFER_SIZE = 2048-byte buffer — kobject.h), compute the device’s sysfs path, and add the three mandatory variables:

	add_uevent_var(env, "ACTION=%s", action_string);     /* e.g. add */
	add_uevent_var(env, "DEVPATH=%s", devpath);           /* e.g. /devices/pci0000:00/.../net/eth0 */
	add_uevent_var(env, "SUBSYSTEM=%s", subsystem);       /* e.g. net */

DEVPATH is the sysfs path relative to /sys; udev prepends /sys to read the device back. add_uevent_var appends a NUL-terminated KEY=value into the buffer and records a pointer in envp[], failing with a warning if it would overflow the 64-pointer / 2048-byte limits.

5. Let the subsystem hooks enrich the environment. This is where MODALIAS and friends come from. The kset’s uevent callback runs; for devices that is dev_uevent, which adds device-node variables and then calls the bus, class, and device-type hooks in turn (core.c):

	if (MAJOR(dev->devt)) {                            /* has a /dev node? */
		add_uevent_var(env, "MAJOR=%u", MAJOR(dev->devt));
		add_uevent_var(env, "MINOR=%u", MINOR(dev->devt));
		add_uevent_var(env, "DEVNAME=%s", name);   /* -> /dev node name */
	}
	if (dev->type && dev->type->name)
		add_uevent_var(env, "DEVTYPE=%s", dev->type->name);
	if (dev->driver)
		add_uevent_var(env, "DRIVER=%s", dev->driver->name);
	if (dev->bus && dev->bus->uevent)
		dev->bus->uevent(dev, env);                /* <-- adds MODALIAS */
	if (dev->class && dev->class->dev_uevent)
		dev->class->dev_uevent(dev, env);
	if (dev->type && dev->type->uevent)
		dev->type->uevent(dev, env);

The bus’s uevent hook is what injects MODALIAS. For example platform_uevent in drivers/base/platform.c does add_uevent_var(env, "MODALIAS=%s%s", PLATFORM_MODULE_PREFIX, ...); the PCI and USB buses build their own ID-encoded MODALIAS strings. That string is precisely the key udev (or the kernel’s autoloader) feeds to modprobe to load the matching driver — the same string a driver’s module advertises via MODULE_DEVICE_TABLE. This is the linchpin of cold/hotplug autoloading; see Device-Driver Matching for how the MODALIAS string is constructed and matched.

6. Stamp a sequence number and broadcast. A monotonically increasing SEQNUM is added (so userspace can detect lost or reordered events), then the message goes out over netlink:

	add_uevent_var(env, "SEQNUM=%llu", atomic64_inc_return(&uevent_seqnum));
	retval = kobject_uevent_net_broadcast(kobj, env, action_string, devpath);

For KOBJ_ADD the code also sets kobj->state_add_uevent_sent = 1 so the core can guarantee a matching remove later; for KOBJ_UNBIND it calls zap_modalias_env(env) to strip MODALIAS before sending.

kobject_uevent_net_broadcast ultimately builds a socket buffer in alloc_uevent_skb (kobject_uevent.c). The on-wire layout is a header line followed by the NUL-separated environment:

	/* header: "action@devpath\0" */
	scratch = skb_put(skb, len);
	sprintf(scratch, "%s@%s", action_string, devpath);
	/* then the raw env buffer: KEY=value\0KEY=value\0... */
	skb_put_data(skb, env->buf, env->buflen);
	...
	parms->dst_group = 1;       /* multicast group 1 */
	netlink_broadcast(uevent_sock, ..., 0, 1, GFP_KERNEL);

So a uevent on the wire is literally add@/devices/.../net/eth0\0ACTION=add\0DEVPATH=/devices/.../net/eth0\0SUBSYSTEM=net\0...\0. The first token before @ is the action, the rest of that line is DEVPATH, and everything after the first NUL is the KEY=value environment, each pair NUL-terminated. This is exactly what udevadm monitor --kernel --property prints. The broadcast goes to multicast group 1; any process that opens a NETLINK_KOBJECT_UEVENT socket and binds to group 1 receives every event.

Per-Network-Namespace Sockets

The uevent socket is created per network namespace, in uevent_net_init registered as a pernet_operations (kobject_uevent.c):

	ue_sk->sk = netlink_kernel_create(net, NETLINK_KOBJECT_UEVENT, &cfg);
	...
	/* Restrict uevents to initial user namespace. */
	if (sock_net(ue_sk->sk)->user_ns == &init_user_ns)
		list_add_tail(&ue_sk->list, &uevent_sock_list);

Each network namespace gets its own uevent socket, but only sockets in the initial user namespace are added to the global broadcast list. This is why a network-namespaced or containerized process generally does not see the host’s device uevents unless explicitly arranged — a deliberate isolation boundary, and a frequent source of “udev sees nothing inside my container” confusion.

The /sys/…/uevent File: Read the Env, Write to Re-trigger

Every device has a uevent attribute (/sys/devices/.../uevent), itself declared as static DEVICE_ATTR_RW(uevent) (core.c) — a nice closure with Device Attributes and sysfs Files. It does two things:

Reading it runs the same uevent ops and prints the environment that would be sent, one KEY=value per line — uevent_show re-invokes kset->uevent_ops->uevent(&dev->kobj, env) and emits each pair with sysfs_emit_at. So cat /sys/class/net/eth0/uevent shows you DEVTYPE, INTERFACE, IFINDEX, etc. without sniffing netlink.

Writing to it synthesizes a uevent. uevent_store calls kobject_synth_uevent(&dev->kobj, buf, count):

static ssize_t uevent_store(struct device *dev, struct device_attribute *attr,
			    const char *buf, size_t count)
{
	int rc = kobject_synth_uevent(&dev->kobj, buf, count);
	if (rc) { dev_err(dev, "uevent: failed to send synthetic uevent: %d\n", rc); return rc; }
	return count;
}

kobject_synth_uevent parses the written string with kobject_action_type — so echo add > uevent re-emits a KOBJ_ADD, echo change > uevent a KOBJ_CHANGE. It even accepts a trailing UUID and KEY=value extras (parsed by kobject_action_args into SYNTH_UUID= and SYNTH_ARG_*= variables) so a tool can correlate a synthetic event with its own request. This is the kernel mechanism behind udevadm trigger — replaying device events at coldplug, whose default action is change (udevadm(8)). This resolves the open question flagged in sysfs and the Kernel Object Hierarchy: udevadm trigger works by writing the action string to each device’s uevent file, driving kobject_synth_uevent.

The Legacy uevent_helper Path

Before the netlink broadcast existed, the kernel delivered hotplug events by forking a userspace binary once per event — the path in /proc/sys/kernel/hotplug, historically /sbin/hotplug. That path still exists, guarded by CONFIG_UEVENT_HELPER, at the tail of kobject_uevent_env (kobject_uevent.c):

#ifdef CONFIG_UEVENT_HELPER
	if (uevent_helper[0] && !kobj_usermode_filter(kobj)) {
		add_uevent_var(env, "HOME=/");
		add_uevent_var(env, "PATH=/sbin:/bin:/usr/sbin:/usr/bin");
		init_uevent_argv(env, subsystem);
		info = call_usermodehelper_setup(env->argv[0], env->argv, env->envp, ...);
		call_usermodehelper_exec(info, UMH_NO_WAIT);   /* fork the helper */
	}
#endif

The comment in the source says it bluntly: this is “usually only enabled during early boot.” Forking a process per event was catastrophic during the boot storm of thousands of devices (a fork-per-uevent meltdown), so modern distributions ship uevent_helper empty (uevent_helper[0] is \0, so the block is skipped) and rely entirely on the netlink broadcast, where a single long-running udevd consumes events from a socket. Many kernels disable CONFIG_UEVENT_HELPER entirely. Do not write to /sys/kernel/uevent_helper on a running system — pointing it at a binary makes the kernel fork that binary on every device event.

Failure Modes and Misunderstandings

  • “My device exists in /sys but udev never reacted.” The kobject may lack a kset, or the device has neither a bus nor a class (so dev_uevent_filter drops it), or uevent_suppress was set during registration, or the listener started after the event. Diagnose with udevadm monitor (live) and cat /sys/.../uevent (what would be sent) and replay with udevadm trigger.
  • Confusing KERNEL events with UDEV events. udevadm monitor shows two streams: --kernel is the raw uevent straight off the netlink socket; --udev is the event after udev finished processing rules. They differ in timing and in the variables udev adds (ID_*, DEVLINKS). The kernel only ever sends the first; the second is udev’s own internal multicast. Reading the kernel one as if udev had run is a common mistake.
  • No events inside a container/netns. As shown above, the uevent socket is per-network-namespace and only the initial-user-namespace socket is on the broadcast list. A container with its own netns/userns sees nothing by default; this is intended isolation, not a bug. (Network Namespaces, Linux Containers and Isolation MOC.)
  • Relying on action proliferation. Subsystems sometimes want a new action verb; the header forbids it without driver-core sign-off and points to KOBJ_CHANGE + custom env vars instead. Code that switches on novel action strings will not see them.
  • Lost events and SEQNUM gaps. Netlink broadcast can drop messages under memory pressure (netlink_broadcast is best-effort). SEQNUM lets a listener detect a gap; a robust manager re-reads /sys to reconcile rather than assuming it saw every event. This is why udev periodically re-triggers.
  • MODALIAS missing on unbind. Expecting MODALIAS on a KOBJ_UNBIND event fails — zap_modalias_env strips it, by design.

Alternatives and When to Choose Them

  • Polling /sys directly — for one-off scripts that just want current state, reading the relevant attribute (or uevent file) is simpler than opening a netlink socket. Use uevents only when you need to react to changes.
  • inotify / fanotify on /dev or /sys — watches filesystem changes, but sysfs does not generate inotify events reliably for attribute changes, and /dev only changes after udev already acted. The uevent socket is the authoritative source for device change notification; inotify on /dev is a downstream, lossier proxy.
  • Direct NETLINK_KOBJECT_UEVENT socket (e.g. via libudev’s monitor, or raw) — when you are writing a device manager or want events without depending on udev. mdev (BusyBox) and eudev are alternative consumers of the very same broadcast. The kernel side is identical regardless of consumer.
  • udevadm monitor — the diagnostic front-end to this channel; use it to see the raw events rather than to consume them programmatically.

Production Notes

This channel is load-bearing for every modern Linux system. Hot-plugging a USB device emits add uevents up the device tree; systemd-udevd receives them, matches udev rules, and creates /dev/sd* nodes and by-id/by-path symlinks — with devtmpfs having already created the bare node at the KOBJ_ADD, so udev’s job is naming, permissions, and triggers rather than mknod. Predictable network interface names (enp3s0) are computed by udev from the SUBSYSTEM/DEVPATH/ID_NET_NAME_* it derives off the uevent. Module autoloading is the MODALIAS-in-the-uevent → modprobe loop. During boot, udevadm trigger writes change/add to every device’s uevent file (the synthetic path above) to “coldplug” devices that were enumerated before udevd started — without it, devices present at boot would have no rules applied. Operators debugging “device not coming up” almost always reach for udevadm monitor (watch the live netlink stream) and udevadm info /sys/... (dump what udev knows), both of which sit directly on the mechanism in this note.

See Also

Graph View

Backlinks