sysfs is the pseudo-filesystem mounted at /sys that exposes the kernel’s unified device model — its devices, drivers, buses, and classes, and the links between them — to userspace as a directory tree. Per its own documentation, “sysfs is a RAM-based filesystem initially based on ramfs … [that] provides a means to export kernel data structures, their attributes, and the linkages between them to userspace,” and it is “tied inherently to the kobject infrastructure” (sysfs.rst, v6.12). The organizing primitive is the kobject (kernel object): a small embeddable struct carrying a name, a parent pointer, a reference count, and a type. The mapping is mechanical and total — every registered kobject becomes a directory, and each of its attributes becomes a file inside that directory. Underneath, sysfs does not implement its own VFS plumbing; it is built on kernfs, a generic in-memory pseudo-filesystem library whose only other in-tree clients are cgroupfs and resctrl (debugfs and tracefs, despite the family resemblance, are not kernfs — see The tracefs Filesystem). The convention that makes /sys usable — one value per file — is what distinguishes it from its older, undisciplined sibling procfs and the proc Filesystem.
This note pins its source citations to Linux 6.12, a maintained long-term-support (LTS) series released 2024-11-17; mainline has since moved on to the 7.x series. Every struct, function, and documentation quote below was read from raw.githubusercontent.com at tag v6.12 on 2026-09-04. Live /sys output quoted throughout comes from a Fedora 44 machine running 7.1.8-200.fc44.x86_64, and is labelled as such wherever it appears — the layout and conventions are unchanged between the two, which is itself a demonstration of the stability promise discussed at the end.
The device model is an object graph; sysfs is that graph made visible
This note and Device-Driver Matching are two halves of one subject. The kernel’s unified device model is a reference-counted in-memory object graph whose nodes are all the same primitive — a struct kobject embedded inside a larger struct. sysfs is the projection of that graph into a filesystem: every kobject is a directory, every exported field is a file, and every relationship the graph encodes is a symlink. The matching note owns the rule that adds an edge between a device and a driver; this note owns the shape of the graph, the primitive it is built from, and how it is rendered. If you have ever watched /sys/bus/pci/devices/<addr>/driver appear the moment a module loaded, you have watched an edge being created in one note and rendered by the other.
Mental Model
Think of sysfs as a filesystem projection of an in-kernel object graph. The kernel already maintains a tree of kobjects (a device’s kobject has a parent kobject, which has a parent, up to a subsystem root). sysfs walks that graph and renders each node as a directory and each node’s exported fields as files. The kobject was introduced by Patrick Mochel in the 2.5/2.6 development era as the backbone of the new driver model — the copyright headers in include/linux/kobject.h (2002–2003, Mochel) and include/linux/sysfs.h (2001–2002, Mochel) date the design. The struct kobject first appeared in the 2.5.45 development kernel and, per LWN’s “The Zen of Kobjects” (2003), “was initially meant as a simple way of unifying kernel code which manages reference counted objects” before “mission creep” made it central to the device model and sysfs (LWN 51437). sysfs exists because the kernel needed a structured, ABI-stable way to expose this model — explicitly as a corrective to procfs’s free-form sprawl.
flowchart TB
subgraph KERN["In-kernel object graph"]
KSET["kset<br/>(a set of kobjects,<br/>e.g. /sys/bus/pci)"]
K1["kobject<br/>(a device)"]
KT["kobj_type / ktype<br/>(release + sysfs_ops + default_groups)"]
A1["attribute<br/>(name, mode)"]
KSET --> K1
K1 -->|"->ktype"| KT
KT -->|"default_groups"| A1
end
K1 -->|"->sd (kernfs_node)"| KN["kernfs_node<br/>(the in-memory tree node)"]
subgraph KERNFS["kernfs (generic pseudo-fs library)"]
KN
end
KN -->|"mounted as /sys"| FS["sysfs"]
FS -->|"VFS: read()/write()"| OPS["sysfs_ops.show / .store"]
OPS -->|"dispatch by ktype"| SHOW["attribute show()/store()"]
FS -.->|"kobject_uevent over netlink"| UDEV["udevd (userspace)"]
The kobject-to-sysfs projection. What it shows: a kobject belongs to a kset, has a ktype (which supplies the sysfs_ops and default attribute groups), and holds a pointer sd to a kernfs_node — the actual in-memory tree node that kernfs (a generic library, shared with cgroupfs and resctrl) manages; sysfs is the mount of that tree, and file I/O is dispatched through sysfs_ops.show/.store to per-attribute callbacks. The insight: sysfs itself is thin — the real data structures are the kobject graph and the kernfs tree, and /sys is just a window onto them, with kobject_uevent notifying userspace’s device manager udevd of changes over a netlink socket. Note the two chains that meet at the kobject: the object graph above it and the kernfs tree below it.
The five structs in that picture are worth drawing precisely, because the relationships among them are a mix of embedding (one struct physically contains another), pointing (a reference to a shared descriptor), and back-pointing (kernfs holds a void *priv back at the kobject). Getting those three apart is most of understanding sysfs.
classDiagram
direction LR
class kobject {
+const char* name
+list_head entry
+kobject* parent
+kset* kset
+kobj_type* ktype
+kernfs_node* sd
+kref kref
+state_initialized : 1
+state_in_sysfs : 1
+state_add_uevent_sent : 1
}
class kset {
+list_head list
+spinlock_t list_lock
+kobject kobj
+kset_uevent_ops* uevent_ops
}
class kobj_type {
+release(kobject*) void
+sysfs_ops* sysfs_ops
+attribute_group** default_groups
+child_ns_type(kobject*) kobj_ns_type_operations*
+namespace(kobject*) const void*
+get_ownership(kobject*, kuid_t*, kgid_t*) void
}
class sysfs_ops {
+show(kobject*, attribute*, char*) ssize_t
+store(kobject*, attribute*, const char*, size_t) ssize_t
}
class attribute_group {
+const char* name
+is_visible(kobject*, attribute*, int) umode_t
+attribute** attrs
+bin_attribute** bin_attrs
}
class attribute {
+const char* name
+umode_t mode
}
class kernfs_node {
+atomic_t count
+atomic_t active
+kernfs_node* parent
+const char* name
+rb_node rb
+unsigned int hash
+unsigned short flags
+umode_t mode
+u64 id
+void* priv
}
class kernfs_elem_dir
class kernfs_elem_symlink
class kernfs_elem_attr
kset *-- kobject : embeds its own kobject<br/>(so a kset IS a directory)
kobject --> kset : kset (membership)
kobject --> kobject : parent (builds the /sys tree)
kobject --> kobj_type : ktype (shared descriptor)
kobj_type --> sysfs_ops : sysfs_ops
kobj_type --> attribute_group : default_groups[]
attribute_group --> attribute : attrs[]
kobject --> kernfs_node : sd
kernfs_node --> kobject : priv (back-pointer)
kernfs_node --> kernfs_node : parent
kernfs_node ..> kernfs_elem_dir : union, if KERNFS_DIR
kernfs_node ..> kernfs_elem_symlink : union, if KERNFS_LINK
kernfs_node ..> kernfs_elem_attr : union, if KERNFS_FILE
The five structs behind every directory and file in /sys, with embedding shown as a solid diamond and pointers as arrows. What it shows: a ksetembeds a kobject (so a kset is itself a directory and can be a parent), while a kobject only points at its kset, its ktype, and its kernfs node. The ktype is a shared descriptor — thousands of kobjects of the same kind point at one kobj_type — which is where the release() destructor, the sysfs_ops dispatch pair, and the default attribute groups live. The kernfs_node is a tagged union: exactly one of dir, symlink, or attr is valid, selected by flags. The insight to take: there are two parent chains, and they are kept in sync but are not the same chain. kobject->parent is the object hierarchy; kernfs_node->parent is the filesystem hierarchy. kobject->sd and kernfs_node->priv are the two ends of the bridge between them, and every sysfs read has to cross that bridge in both directions — down from a dentry to the kernfs node, then sideways through priv to the kobject, then out through ktype->sysfs_ops.
The kobject: Names, Parents, Reference Counts, and Types
Walk the load-bearing fields. name is the directory name in sysfs. parent points at the kobject whose directory will contain this one — this is what builds the /sys tree, because “that directory is created as a subdirectory of the kobject’s parent, expressing internal object hierarchies to userspace” (sysfs.rst). kset is the set this kobject belongs to (more below). ktype points at the kobject’s type descriptor, which supplies the operations and the default attributes. sd (“sysfs dentry,” historically) is the pointer into kernfs — the actual tree node. kref is the embedded reference counter.
The kobject is almost never used standalone; it is embedded in a larger structure. The canonical pattern, from the kobject documentation, is that struct device (and dozens of others) contain a struct kobject member, and code recovers the container from a kobject pointer via container_of() (kobject.rst):
container_of(ptr, type, member) does pointer arithmetic: given the address of the kobj field and the knowledge that it sits at a known offset inside struct my_object, it returns the address of the enclosing object. This is the kernel’s idiom for “object-oriented” composition without C++ — the kobject provides reference counting and sysfs presence; the embedding struct provides the actual data. The docs are emphatic: “No structure should EVER have more than one kobject embedded within it” (kobject.rst), because the single embedded kref is the lifetime of the whole object.
Reference counting governs lifetime.kobject_init() sets the count to one; kobject_get() increments and kobject_put() decrements; when the count hits zero, the kobject’s ktype->release() method runs to free the enclosing structure. Crucially, after kobject_add() you must neverkfree() the object directly — only kobject_put(), because “other portions of the kernel can get a reference on any kobject that is registered in the system” (e.g. a userspace process holding a sysfs file open), so “a structure protected by a kobject cannot be freed before its reference count goes to zero” (kobject.rst, v6.12). Every kobject must provide a release() method, and providing an empty one to silence the complaint is explicitly called out as wrong.
Verified defect: the documented "warning" for a missing release() is not actually a warning
Documentation/core-api/kobject.rst states plainly: “Note that the kernel will warn you if you forget to provide a release() method. Do not try to get rid of this warning by providing an ‘empty’ release function.” That is not what the code does at v6.12. kobject_cleanup() in lib/kobject.c emits it through pr_debug():
if (t && !t->release) pr_debug("'%s' (%p): does not have a release() function, it is broken and must be fixed. See Documentation/core-api/kobject.rst.\n", kobject_name(kobj), kobj);
pr_debug() compiles to nothing unless the file defines DEBUG or the kernel is built with CONFIG_DYNAMIC_DEBUGand the call site has been explicitly enabled at runtime. On a stock distribution kernel you get no output at all. The message has been pr_debug for a long time: fetching lib/kobject.c at tags v5.4, v5.10, v5.15, v6.1, v6.6, and v6.12 shows pr_debug at every one, so the documentation has been describing behaviour the code does not have for at least six years. Verified 2026-09-04 by reading all six tags. This matters practically: you cannot rely on the kernel telling you that you forgot a release(). The WARN(1, ...) calls that do exist in lib/kobject.c are for an empty name, an uninitialized object, and namespace-type violations — not for a missing release.
The one state bit that governs all of this is state_in_sysfs, set by kobject_add_internal() only after create_dir() succeeds, and consulted by kobject_cleanup() to decide whether it needs to remove the directory on the caller’s behalf (lib/kobject.c, v6.12). Reading kobject_add_internal() shows exactly what “adding” a kobject means:
parent = kobject_get(kobj->parent);/* join kset if set, use it as parent if we do not already have one */if (kobj->kset) { if (!parent) parent = kobject_get(&kobj->kset->kobj); kobj_kset_join(kobj); kobj->parent = parent;}error = create_dir(kobj);if (error) { ... if (error == -EEXIST) pr_err("%s failed for %s with -EEXIST, don't try to register things with the same name in the same directory.\n", ...);} else kobj->state_in_sysfs = 1;
Three rules fall straight out of this. A kset doubles as a default parent: if you set kobj->kset but no kobj->parent, the kset’s own embedded kobject becomes the parent, which is why a device registered into the pci kset lands under the pci directory without anyone saying so. Adding takes a reference on the parent, which is why a parent cannot be freed while children exist. And duplicate names are a hard error with a distinctive message — -EEXIST from kobject_add() is nearly always two subsystems racing to register the same name in the same directory.
create_dir() in turn is where the object becomes a filesystem:
static int create_dir(struct kobject *kobj){ const struct kobj_type *ktype = get_ktype(kobj); int error; error = sysfs_create_dir_ns(kobj, kobject_namespace(kobj)); if (error) return error; if (ktype) { error = sysfs_create_groups(kobj, ktype->default_groups); if (error) { sysfs_remove_dir(kobj); return error; } } /* @kobj->sd may be deleted by an ancestor going away. Hold an * extra reference so that it stays until @kobj is gone. */ sysfs_get(kobj->sd); ...}
The directory is made first, then every attribute in ktype->default_groups is materialised inside it, and the whole thing is rolled back if any file fails. This is the atomicity guarantee that makes it safe for udev to open() an attribute the moment it sees the add uevent: by the time the event goes out, either all the default files exist or none of them do.
stateDiagram-v2
[*] --> Uninitialized
Uninitialized --> Initialized : kobject_init(kobj, ktype)<br/>kref = 1, state_initialized = 1
Initialized --> InSysfs : kobject_add(kobj, parent, fmt, ...)<br/>→ kobject_add_internal() → create_dir()<br/>state_in_sysfs = 1, parent refcount++
Initialized --> Released : kobject_put() with no add<br/>(error path — legal and common)
InSysfs --> InSysfs : kobject_get() / kobject_put()<br/>kref++ / kref--
InSysfs --> Announced : kobject_uevent(kobj, KOBJ_ADD)<br/>state_add_uevent_sent = 1
Announced --> Announced : kobject_uevent(KOBJ_CHANGE / ONLINE /<br/>OFFLINE / BIND / UNBIND)
Announced --> Deleted : kobject_del()<br/>→ __kobject_del() removes the sysfs dir,<br/>KOBJ_REMOVE sent, parent refcount--
InSysfs --> Deleted : kobject_del()
Deleted --> Released : final kobject_put(), kref hits 0<br/>→ kobject_cleanup() → ktype->release()
Released --> [*] : container_of() + kfree()<br/>inside release()
note left of Released
kobject_cleanup() also handles the
case where the caller forgot kobject_del():
if state_in_sysfs is still 1 it calls
__kobject_del() itself, then release().
A missing release() is only reported
via pr_debug -- usually silently.
end note
note right of InSysfs
Directory now exists at
/sys/<parent path>/<name>
with every attribute from
ktype->default_groups.
end note
The kobject lifecycle, with the call that drives each transition. What it shows: initialisation, sysfs presence, uevent announcement, deletion, and release are five separate steps, not one — and the refcount lives across all of them. Deleting a kobject removes its directory but does not free it; freeing happens only when the last reference drops, which may be long afterwards if a userspace process still holds an attribute file open. The insight to take: the two most common kobject bugs are visible as illegal edges on this diagram. kfree() from InSysfs or Deleted is a use-after-free waiting to happen, because the graph shows other holders may still be there. And an empty release() turns the Released → [*] transition into a leak: the object is removed from every list and every reference is gone, but the memory is never returned.
ktype and sysfs_ops: How a File Read Reaches a Callback
release is the destructor just discussed. default_groups is a NULL-terminated array of attribute groups that are automatically created as files when the kobject is added. sysfs_ops is the indirection that connects a read()/write() on a sysfs file to the right callback:
When userspace reads a sysfs file, sysfs “calls the appropriate method for the type. The method then translates the generic struct kobject and struct attribute pointers to the appropriate pointer types, and calls the associated methods” (sysfs.rst). The driver model’s dev_attr_show is the textbook example — it container_of()s the generic struct attribute up to a struct device_attribute and the generic struct kobject up to a struct device, then calls the attribute’s own show:
#define to_dev_attr(_attr) container_of(_attr, struct device_attribute, attr)static ssize_t dev_attr_show(struct kobject *kobj, struct attribute *attr, char *buf){ struct device_attribute *dev_attr = to_dev_attr(attr); struct device *dev = kobj_to_dev(kobj); ssize_t ret = -EIO; if (dev_attr->show) ret = dev_attr->show(dev, dev_attr, buf); if (ret >= (ssize_t)PAGE_SIZE) printk("dev_attr_show: %pS returned bad count\n", dev_attr->show); return ret;}
This two-level dispatch — generic sysfs_ops per ktype, specific show/store per attribute — is what lets one filesystem (sysfs) serve heterogeneous object types (devices, drivers, buses, modules) without knowing anything about any of them.
Tracing one cat all the way down makes the layering concrete. The glue on the sysfs side is sysfs_kf_seq_show(), and it is worth reading in full because it is where the famous page-size contract is actually enforced (fs/sysfs/file.c, v6.12):
static int sysfs_kf_seq_show(struct seq_file *sf, void *v){ struct kernfs_open_file *of = sf->private; struct kobject *kobj = of->kn->parent->priv; /* (1) kernfs → kobject */ const struct sysfs_ops *ops = sysfs_file_ops(of->kn); /* (2) ktype's ops */ ssize_t count; char *buf; if (WARN_ON_ONCE(!ops->show)) return -EINVAL; /* acquire buffer and ensure that it's >= PAGE_SIZE and clear */ count = seq_get_buf(sf, &buf); if (count < PAGE_SIZE) { seq_commit(sf, -1); return 0; } memset(buf, 0, PAGE_SIZE); /* (3) always zeroed */ count = ops->show(kobj, of->kn->priv, buf); /* (4) into the ktype */ if (count < 0) return count; if (count >= (ssize_t)PAGE_SIZE) { printk("fill_read_buffer: %pS returned bad count\n", ops->show); count = PAGE_SIZE - 1; /* (5) "try to struggle along" */ } seq_commit(sf, count); return 0;}
Step (1) is the bridge crossing: of->kn is the file’s kernfs node, so of->kn->parent is the directory’s node, and its priv is the kobject. Step (2) reaches the ktype’s sysfs_ops. Note of->kn->priv at step (4) — the file node’s own priv is the struct attribute *. Step (3) zeroes the whole page before every show(), so a show() that writes nothing yields an empty file rather than stale bytes. Step (5) is the guard against a show() that overran: the kernel logs the offending function symbol (%pS) and truncates to PAGE_SIZE - 1 rather than corrupting anything.
sequenceDiagram
autonumber
participant U as userspace<br/>cat /sys/.../speed
participant VFS as VFS
participant KF as kernfs<br/>(fs/kernfs/file.c)
participant SY as sysfs glue<br/>(fs/sysfs/file.c)
participant KT as ktype's sysfs_ops<br/>(dev_attr_show)
participant DA as the attribute's own show()<br/>(e.g. speed_show)
U->>VFS: open("/sys/class/net/enp191s0/speed")
VFS->>KF: kernfs_fop_open() on the kernfs_node
KF->>KF: allocate kernfs_open_file, take the hashed<br/>open_file_mutex for this node
U->>VFS: read(fd, buf, 4096)
VFS->>KF: kernfs_fop_read_iter() → seq_read_iter()
KF->>SY: kernfs_ops.seq_show = sysfs_kf_seq_show()
SY->>SY: kobj = of.kn.parent.priv<br/>attr = of.kn.priv<br/>ops = kobj.ktype.sysfs_ops
SY->>SY: seq_get_buf() then memset(buf, 0, PAGE_SIZE)
SY->>KT: ops.show(kobj, attr, buf)
KT->>KT: container_of(attr) to device_attribute<br/>kobj_to_dev(kobj) to device
KT->>DA: dev_attr.show(dev, dev_attr, buf)
DA-->>KT: sysfs_emit(buf, "%d") plus newline, returns byte count
KT-->>SY: ssize_t count
SY->>SY: if count >= PAGE_SIZE, log %pS and clamp to PAGE_SIZE-1
SY-->>KF: seq_commit(sf, count)
KF-->>VFS: bytes copied to userspace
VFS-->>U: the text 1000 and a newline
One cat of a sysfs attribute, from syscall to driver callback. What it shows: four distinct layers, each doing one job — VFS routes the syscall, kernfs owns the node and its locking, the sysfs glue does the two priv lookups and enforces the page contract, and the ktype’s sysfs_ops performs the container_of() upcasts before reaching the attribute’s own function. The insight to take: sysfs itself contains almost no logic. Steps 7 and 8 are the entirety of what “sysfs” adds on top of kernfs — two pointer dereferences and a memset — and steps 10 and 11 are the entirety of what the driver model adds on top of sysfs. This is why the same machinery serves devices, drivers, buses, modules, and cgroups without modification: every layer is a thin, general adapter, and the type-specific knowledge is confined to one container_of() at the very bottom.
The write path is shorter still, because there is no buffering to manage:
kernfs has already copied the user’s bytes into a kernel buffer and NUL-terminated them before this runs, which is why store() implementations can safely use string helpers like sysfs_streq() and kstrtoint() directly on buf.
Attributes: One Value Per File
An attribute is a file. The base type, from include/linux/sysfs.h, is deliberately tiny:
A bare attribute carries no read/write logic; subsystems wrap it. The driver model’s struct device_attribute embeds an attribute plus a show/store pair:
The cardinal convention of sysfs is stated bluntly in the docs: “Attributes should be ASCII text files, preferably with only one value per file. … Mixing types, expressing multiple lines of data, and doing fancy formatting of data is heavily frowned upon. Doing these things may get you publicly humiliated and your code rewritten without notice” (sysfs.rst). This one-value-per-file rule is the entire reason /sys is scriptable: cat /sys/class/net/eth0/speed returns a number, not a paragraph you must parse. It is the precise discipline procfs lacks.
Is the Rule Actually Followed? A Survey, Repeated Sixteen Years Later
His finding, on a Linux 2.6.32 laptop: “of the 9254 files, 1189 are empty and 7168 have only one word… This leaves 897 (nearly 10%) which need further examination. They range from two words (487 cases) to 297 words (one case).” He noted that among nearly 900 offenders there were “less than 100 base names,” and after filtering repeated patterns, “closer to 62” distinct attributes — small enough to inspect by hand. His conclusion was blunt: “to answer our opening question, ‘no’, the one item per file rule is not enforced in any meaningful way.”
Running the identical command on the test machine (Fedora 44, kernel 7.1.8-200.fc44.x86_64, 2026-09-04) gives a direct sixteen-year comparison:
Measure
Brown, 2010 (Linux 2.6.32)
This machine, 2026 (Linux 7.1.8)
regular files under /sys
9,254
53,617 found; 31,517 readable as a non-root user
empty (0 words)
1,189 (12.8%)
3,223 (10.2%)
exactly one word
7,168 (77.5%)
26,380 (83.7%)
two or more words
897 (9.7%)
1,914 (6.1%)
exactly two words
487
411
largest file
297 words
189,931 words — /sys/kernel/btf/vmlinux
The one-value-per-file rule, measured in 2010 and re-measured in 2026 with the same command. What it shows:/sys grew by roughly 5.8× in file count while the proportion of multi-word files fell from about 10% to about 6%. The insight to take: the convention held up better than its author expected — enforcement is still purely social (review), yet compliance improved as the tree grew, which suggests the review pressure is real. Note the caveat on the row totals: this survey ran unprivileged, so 22,100 root-only-readable files are excluded from the word counts; the true multi-word proportion could differ. And the “largest file” row is not a violation at all — /sys/kernel/btf/vmlinux is a binary attribute holding the kernel’s BTF type information, where wc -w is measuring whitespace in a binary blob. The right lesson is that wc -w is a blunt instrument, exactly as Brown said.
The multi-word files that remain are dominated by a handful of base names. On this machine, excluding the BTF blobs, the top offenders by count are uevent (588 instances), desc (128), properties (92), name (69), .note.gnu.build-id (56), base_addr (48), resource (39), trans_table (38), time_in_state_ms (38), and scaling_available_governors (32). Most of these are deliberate and defensible:
uevent is by design a multi-line KEY=value environment dump — it is a serialisation of the whole uevent, not an attribute value. It is the single most common “violation” and nobody considers it one.
scaling_available_governors and friends are Brown’s “enumerated type” case: a list of possible values, paired with a separate file holding the current one (scaling_governor). That is arguably still one value per file — the value is “the set of options.”
queue/scheduler is the bracketed variant Brown described, and it is alive and unchanged in 2026: reading it on this machine returns [none] mq-deadline kyber bfq, listing every available I/O scheduler with the active one in brackets. One file, two pieces of information.
name and similar string attributes contain spaces legitimately — Brown’s examples were "Dell Inc." and "i8042 KBD port", and the same class of value is still there.
What has genuinely improved is the egregious end. Brown’s headline example was a PCI wireless device’s statistics attribute containing “a hex dump of 240 bytes of data, complete with ASCII decoding at the end of each line” — a text file pretending to be a hex editor. Attributes of that kind are now generally written as binary attributes (struct bin_attribute, declared with BIN_ATTR_*), which is the sanctioned escape hatch for anything that is not a text value.
show() and store(): The Single-Call, Page-Buffer Contract
sysfs’s read/write contract is unusual and must be understood to write a correct attribute. “sysfs allocates a buffer of size (PAGE_SIZE) and passes it to the method. sysfs will call the method exactly once for each read or write” (sysfs.rst). Consequences:
On read, show() must fill the entire value into the buffer in one shot; userspace partial reads and seeks are served from sysfs’s copy. Seeking back to zero re-arms show() (it is called again). The buffer is always PAGE_SIZE (4096 on x86).
On write, the whole buffer is passed to store() in the first write; sysfs appends a terminating NUL so helpers like sysfs_streq() are safe. Userspace should read-modify-write the whole value.
New show() implementations “should only use sysfs_emit() or sysfs_emit_at()” to format output — these are bounds-checked wrappers that prevent the PAGE_SIZE overruns that plagued hand-rolled snprintf code. A minimal example: return sysfs_emit(buf, "%s\n", dev->name); (sysfs.rst, v6.12).
The two emit helpers are three lines each and enforce their contract with a WARN() (fs/sysfs/file.c, v6.12):
int sysfs_emit(char *buf, const char *fmt, ...){ if (WARN(!buf || offset_in_page(buf), "invalid sysfs_emit: buf:%p\n", buf)) return 0; ... len = vscnprintf(buf, PAGE_SIZE, fmt, args); return len;}int sysfs_emit_at(char *buf, int at, const char *fmt, ...){ if (WARN(!buf || offset_in_page(buf) || at < 0 || at >= PAGE_SIZE, "invalid sysfs_emit_at: buf:%p at:%d\n", buf, at)) return 0; ... len = vscnprintf(buf + at, PAGE_SIZE - at, fmt, args); return len;}
offset_in_page(buf) must be zero — sysfs_emit() demands a page-aligned buffer, which is exactly what sysfs hands to show() and nothing else. That is the whole reason sysfs_emit_at() exists: a show() that builds a list by appending cannot advance the pointer and call sysfs_emit(buf + len, ...), because the advanced pointer is no longer page-aligned and would trip the WARN. It must call sysfs_emit_at(buf, len, ...) and let the helper do the arithmetic, which also lets it size the remaining space correctly as PAGE_SIZE - at. Both use vscnprintf, which returns the number of characters actually written (never the number that would have been written), so the return value is always a safe running offset. And both return 0 rather than a negative errno on misuse, so a botched call yields an empty file rather than a failed read.
Declaring Attributes: The __ATTR Macros and Groups
Hand-filling an attribute struct is verbose, so sysfs provides a family of macros in include/linux/sysfs.h. Each one takes a bare identifier and derives the filename (via __stringify), the callback names (via token pasting), and the permission bits:
Macro
Expands to .mode
Requires you to define
Notes
__ATTR(name, mode, show, store)
your mode, checked by VERIFY_OCTAL_PERMISSIONS()
the two functions, named explicitly
the general form; everything else is sugar over it
__ATTR_RO(name)
0444
name_show()
world-readable, no store
__ATTR_WO(name)
0200
name_store()
root-writable only, and unreadable even by root
__ATTR_RW(name)
0644
name_show() and name_store()
the common read/write knob
__ATTR_RO_MODE(name, mode)
your mode
name_show()
for read-only attributes that must not be world-readable (e.g. 0400)
__ATTR_RW_MODE(name, mode)
your mode
both
e.g. 0600
__ATTR_PREALLOC(name, mode, show, store)
SYSFS_PREALLOC | mode
the two functions
buffer allocated at open(), so write() cannot fail on allocation — used where a write must not OOM
The __ATTR macro family at v6.12. What it shows: which mode each macro hard-codes and which callbacks it expects to find by name. The insight to take: the name-derivation is why the callbacks must be called exactly <name>_show and <name>_store — the macro pastes the tokens, so a mismatch is a compile error rather than a silent wiring bug.
VERIFY_OCTAL_PERMISSIONS() deserves a look, because it is a small piece of API design that removes a whole bug class at compile time (include/linux/kernel.h, v6.12):
/* Permissions on a sysfs file: you didn't miss the 0 prefix did you? */#define VERIFY_OCTAL_PERMISSIONS(perms) \ (BUILD_BUG_ON_ZERO((perms) < 0) + \ BUILD_BUG_ON_ZERO((perms) > 0777) + \ /* USER_READABLE >= GROUP_READABLE >= OTHER_READABLE */ \ BUILD_BUG_ON_ZERO((((perms) >> 6) & 4) < (((perms) >> 3) & 4)) + \ BUILD_BUG_ON_ZERO((((perms) >> 3) & 4) < ((perms) & 4)) + \ /* USER_WRITABLE >= GROUP_WRITABLE */ \ BUILD_BUG_ON_ZERO((((perms) >> 6) & 2) < (((perms) >> 3) & 2)) + \ /* OTHER_WRITABLE? Generally considered a bad idea. */ \ BUILD_BUG_ON_ZERO((perms) & 2) + \ (perms))
BUILD_BUG_ON_ZERO(cond) evaluates to the integer 0 when cond is false and fails to compile when it is true, so the whole expression sums to perms in the good case and does not build otherwise. The five checks catch, in order: a negative mode; a mode written in decimal (644 is 0o1204, which exceeds 0777 — this is the typo the leading comment is asking about); a mode where group or other can read but the owner cannot; the same for write; and any world-writable bit at all, which is rejected outright with the terse justification “Generally considered a bad idea.” None of these can reach a running kernel.
The driver-model wrappers DEVICE_ATTR_RO/RW/WO, BUS_ATTR_*, DRIVER_ATTR_*, and CLASS_ATTR_* build on these, each substituting its own attribute struct type — that layer is Device Attributes and sysfs Files. The driver-model wrappers DEVICE_ATTR_RO/RW/WO, BUS_ATTR_*, and DRIVER_ATTR_* build on these. Rather than registering attributes one at a time, the modern idiom bundles them into a struct attribute_group (an attrs array plus an optional name that creates a subdirectory and an optional is_visible callback to hide attributes dynamically), and points the ktype’s default_groups at it so all the files appear and disappear atomically with the kobject (sysfs.h). For the simplest case, kobject_create_and_add("name", parent) plus sysfs_create_group() makes a directory with kobj_attribute files without defining a custom ktype at all (kobject.rst).
kset: Grouping kobjects into Subsystems
A kset is “a group of kobjects” that “want to be grouped together and operated on in the same manner” — it is “the basic container type for collections of kobjects” (kobject.rst). Structurally a kset embeds its own kobject, so a kset is itself a directory in sysfs, and the kobjects belonging to it appear beneath it:
struct kset { struct list_head list; /* all kobjects in this set */ spinlock_t list_lock; struct kobject kobj; /* the kset's own directory */ const struct kset_uevent_ops *uevent_ops;};
A kset does three distinct jobs, and it is worth separating them because they are often conflated.
It is a directory. Because the kset embeds a kobject, kset_create_and_add("devices", NULL, bus_kobj) makes /sys/bus/<name>/devices/ a real directory that other kobjects can hang under. bus_register() does exactly that twice — once for devices and once for drivers — for every bus in the system (drivers/base/bus.c, v6.12).
It is a default parent. As shown earlier, kobject_add_internal() uses the kset’s kobject as the parent when no explicit parent was given. This is the mechanism by which “register this device into the PCI bus” also means “put its directory under the PCI bus directory,” with no path handling anywhere.
whether an event for this member is sent at all — a kset can suppress events for kobjects userspace has no business seeing
name(kobj)
returns const char *
the value of SUBSYSTEM= in the event, which is how udev routes rules
uevent(kobj, env)
returns int
appends subsystem-specific variables — this is where MODALIAS=, DEVNAME=, and DEVTYPE= are added
The three kset_uevent_ops hooks. What it shows: filtering, naming, and enrichment are three separate decisions, all owned by the kset rather than by the individual kobject. The insight to take: this is why uevents carry consistent subsystem metadata even though every device in the kernel is a different type of thing. A kobject does not know what a SUBSYSTEM= is; its kset does, and answers on its behalf for every member. It is also the reason a kobject with no kset above it emits no uevent — there is nobody to answer these three questions.
/sys/bus/pci is a kset; each PCI device’s kobject is a member, and bus_kset is in turn the kset holding all the bus kobjects. The same pattern gives the tree its named roots: kernel_kobj, mm_kobj, hypervisor_kobj, power_kobj, and firmware_kobj are exported globals (kobject.h, v6.12) that subsystems pass as a parent to plant their subtree at /sys/kernel/, /sys/kernel/mm/, /sys/hypervisor/, /sys/power/, and /sys/firmware/ respectively.
kernfs: The In-Memory Tree Underneath
sysfs does not implement directory locking, inode management, or the dentry plumbing itself. It is built on kernfs, described in include/linux/kernfs.h as a “pseudo filesystem decoupled from vfs locking.” kernfs was factored out of sysfs (historically sysfs’s internals) into a reusable library so that other pseudo-filesystems could share it — most importantly cgroupfs (the control-group filesystem), which is by far its heaviest user and the reason for most of its scalability work. The complete client list is enumerated below, and it is shorter than most people assume. Each node in the tree is a struct kernfs_node:
struct kernfs_node { atomic_t count; atomic_t active; struct kernfs_node *parent; const char *name; struct rb_node rb; /* siblings in a red-black tree */ unsigned short flags; /* KERNFS_DIR / KERNFS_FILE / KERNFS_LINK */ umode_t mode; union { struct kernfs_elem_dir dir; struct kernfs_elem_symlink symlink; struct kernfs_elem_attr attr; }; u64 id; void *priv; /* back-pointer, e.g. to the kobject */ struct rcu_head rcu;};
Each kernfs_node is a directory, a regular file (attribute), or a symlink, discriminated by flags (KERNFS_DIR, KERNFS_FILE, KERNFS_LINK), and the union that follows carries exactly the payload for that one type. Siblings are kept in a red-black tree (rb) keyed by a 31-bit hash of the namespace tag plus the name — kernfs_name_hash() truncates to 31 bits explicitly “so it fits in an off_t”, because that hash doubles as the directory offset returned by readdir() (fs/kernfs/dir.c, v6.12). The priv field is the back-pointer sysfs uses to store “a pointer to the kobject that implements a directory” (sysfs.rst, v6.12) — that is the kobject->sd ↔ kernfs_node->priv linkage that ties the two halves together.
kernfs holds the whole tree in RAM (there is no backing device) and manages its own locking. The per-file open lock is not a single global mutex but a hash table of mutexes sized from the CPU count, and the header documents the sizing empirically (include/linux/kernfs.h, v6.12):
/* * At the moment size of hash table of locks is being set based on * the number of CPUs as follows: * * NR_CPU NR_KERNFS_LOCK_BITS NR_KERNFS_LOCKS * 1 1 2 * 2-3 2 4 * 4-7 4 16 * 8-15 6 64 * 16-31 8 256 * 32 and more 10 1024 * * The above relation between NR_CPU and number of locks is based * on some internal experimentation which involved booting qemu * with different values of smp, performing some sysfs operations * on all CPUs and observing how increase in number of locks impacts * completion time of these sysfs operations on each CPU. */#ifdef CONFIG_SMP#define NR_KERNFS_LOCK_BITS (2 * (ilog2(NR_CPUS < 32 ? NR_CPUS : 32)))#else#define NR_KERNFS_LOCK_BITS 1#endif#define NR_KERNFS_LOCKS (1 << NR_KERNFS_LOCK_BITS)
Walking the formula: ilog2(x) is the floor of the base-2 logarithm, so ilog2(16) = 4; the count is clamped at 32 CPUs; and the result is doubled, giving 2 * 4 = 8 bits and therefore 2^8 = 256 mutexes on a 16–31 CPU machine, capping at 1,024 locks for any machine with 32 or more. A comment beside the struct explains why hashing is safe here at all: “Hashed mutexes are safe to use here because operations using these don’t rely on global exclusion.” The reason this exists is cgroups — thousands of concurrent open()s on cgroup control files across many CPUs made a single open_file_mutex a measurable bottleneck.
kernfs also exposes a richer per-file operations vector than sysfs uses. struct kernfs_ops carries seq_show/seq_start/seq_next/seq_stop, raw read/write, poll, mmap, llseek, plus two policy knobs sysfs leaves at their defaults: atomic_write_len (writes up to that size are delivered to write() in one call; larger ones are rejected with -E2BIG) and prealloc (allocate the buffer at open() rather than per operation). cgroupfs uses both. In short: kernfs is the generic “in-memory hierarchy of nodes with per-file read/write callbacks” engine, and each client layers its own policy on top.
Who Actually Uses kernfs — and Who Does Not
This is a place where a plausible-sounding claim is easy to make and wrong. A whole-tree identifier search for kernfs_create_root at v6.12 returns exactly three call sites (Elixir cross-reference, v6.12), which is the definitive list of filesystems built on kernfs:
The complete set of kernfs clients at v6.12. What it shows: three filesystems, all of them mounted under /sys, all of them exporting a hierarchy of kernel objects with per-file callbacks. resctrl — the Intel Resource Director Technology / AMD Platform QoS control interface — is the one people forget; it is present on the test machine as /sys/fs/resctrl. The insight to take: kernfs is not “the pseudo-filesystem library”; it is a specific library for object hierarchies with attribute files, and only three subsystems have that shape. Everything else in the kernel that looks like a pseudo-filesystem uses something else.
debugfs and tracefs are not kernfs clients. This is worth stating flatly because the resemblance is strong and the assumption is common — both live under /sys/kernel/, both are RAM-backed, both present directories of small files. But fs/tracefs/inode.c at v6.12 contains zero references to any kernfs_* symbol. It is built directly on the VFS’s libfs helpers: simple_fill_super(), get_tree_single(), simple_lookup(), simple_statfs(), kill_litter_super(), simple_recursive_removal(). fs/debugfs/inode.c is the same. They are libfs filesystems, not kernfs filesystems, and they descend from a different lineage — debugfs was written as a deliberately unconstrained scratch space, tracefs was split out of debugfs so that the tracing interface could be mounted without exposing all of debugfs. The tracefs/debugfs split and everything built on it is covered in depth in The tracefs Filesystem.
The distinction is not academic. It determines what a filesystem gets for free and what it must implement itself:
flowchart TB
VFS["VFS<br/>(dentry cache, inode ops, syscall entry)"]
subgraph KFS["kernfs — object-hierarchy engine"]
K["fs/kernfs/<br/>· kernfs_node rb-tree per directory<br/>· 31-bit ns+name hash doubles as readdir offset<br/>· hashed open_file_mutex table (up to 1024)<br/>· per-root rw_semaphore<br/>· kernfs_ops: seq_show/read/write/poll/mmap<br/>· atomic_write_len, prealloc<br/>· namespace tagging (KERNFS_NS)<br/>· active-reference draining for safe removal"]
end
subgraph LFS["libfs — generic simple-filesystem helpers"]
L["fs/libfs.c<br/>· simple_fill_super / simple_lookup<br/>· simple_statfs / kill_litter_super<br/>· plain dcache-resident trees<br/>· no attribute abstraction"]
end
VFS --> K
VFS --> L
K --> SYSFS["sysfs<br/>/sys<br/>kobject graph"]
K --> CG["cgroupfs<br/>/sys/fs/cgroup<br/>cgroup hierarchy"]
K --> RES["resctrl<br/>/sys/fs/resctrl<br/>RDT/QoS groups"]
L --> DBG["debugfs<br/>/sys/kernel/debug<br/>NO ABI promise"]
L --> TRC["tracefs<br/>/sys/kernel/tracing<br/>split out of debugfs"]
L --> OTH["ramfs, pipefs, and<br/>other simple pseudo-fs"]
SYSFS -.->|"one value per file<br/>+ Documentation/ABI"| ABI["stable userspace ABI"]
DBG -.->|"explicitly none"| NOABI["may change or vanish<br/>at any release"]
kernfs and libfs as two different foundations, with their real clients at v6.12. What it shows: three filesystems sit on kernfs and get an attribute abstraction, namespace tagging, scalable open-file locking, and safe removal semantics; the libfs filesystems get a dcache-resident tree and implement everything else themselves. The insight to take: the choice of foundation tracks the stability promise, not the storage medium. kernfs’s machinery — atomic group creation, active-reference draining, per-node namespace tags — exists to make an interface safe to expose as a permanent ABI. debugfs and tracefs deliberately do not want that machinery because they deliberately do not make that promise.
One Graph, Several Views: The Shape of /sys
This is the single fact that makes /sys comprehensible, and it is worth stating before any of the detail: there is exactly one tree, and it lives under /sys/devices. Everything else at the top level is a flat directory of symlinks pointing into it. The kernel’s own rules document is unambiguous — “There is only one valid place in sysfs where hierarchy can be examined and this is below: /sys/devices” (sysfs-rules.rst, v6.12).
Here is the real /sys of a live machine (Fedora 44, kernel 7.1.8-200.fc44.x86_64, read 2026-09-04), annotated. Only twelve directories exist at the top level, and eight of them are views or namespaces rather than topology:
/sys/├── devices/ THE TREE. Every struct device is a real directory here,│ │ nested by physical/logical parentage. 40 entries on│ │ this box: pci0000:00, platform, virtual, cpu, system,│ │ amd_iommu_0, LNXSYSTM:00 (the ACPI root), ...│ └── pci0000:00/ ← PCI domain 0000, root of the PCI tree│ └── 0000:00:02.1/ ← a root port (parent bridge)│ └── 0000:bf:00.0/ ← the Realtek NIC. 51 entries:│ ├── vendor device class ← identity, one value per file│ ├── modalias ← the autoload key (see the matching note)│ ├── driver_override ← userspace veto over matching│ ├── config resource0 ← BINARY attributes, not text│ ├── subsystem -> ../../../../bus/pci│ ├── driver -> ../../../../bus/pci/drivers/r8169│ ├── iommu_group -> ../../../../kernel/iommu_groups/16│ ├── firmware_node -> ../../../LNXSYSTM:00/.../device:05│ └── net/│ └── enp191s0/ ← a CHILD device, class "net"│ ├── speed mtu address carrier│ └── statistics/ ← a named attribute_group = subdirectory├── bus/ VIEW BY BUS TYPE. 48 entries: pci, usb, i2c, platform,│ └── pci/ acpi, spi, virtio, auxiliary, ...│ ├── devices/ → flat list of symlinks into /sys/devices│ │ └── 0000:bf:00.0 -> ../../../devices/pci0000:00/0000:00:02.1/0000:bf:00.0│ └── drivers/ → one directory per registered driver│ └── r8169/│ ├── 0000:bf:00.0 -> ../../../../devices/pci0000:00/.../0000:bf:00.0│ ├── bind unbind ← write a device name here│ ├── new_id remove_id ← teach the driver a new ID│ ├── module -> ../../../../module/r8169│ └── uevent├── class/ VIEW BY FUNCTION. 83 entries: net, block, tty, drm, hwmon,│ └── net/ input, thermal, nvme, ... — grouping cuts ACROSS buses│ └── enp191s0 -> ../../devices/pci0000:00/0000:00:02.1/0000:bf:00.0/net/enp191s0├── block/ VIEW BY BLOCK DEVICE (a legacy sibling of class/block)│ └── dm-0 -> ../devices/virtual/block/dm-0├── dev/ VIEW BY DEVICE NUMBER — the reverse index for stat(2)│ ├── char/10:130 -> ../../devices/platform/sp5100-tco/misc/watchdog│ └── block/252:0 -> ../../devices/virtual/block/dm-0├── module/ 289 entries — one per loaded module; parameters/ and state├── firmware/ ACPI tables, DMI/SMBIOS, EFI variables├── fs/ per-filesystem hierarchies: cgroup, btrfs, ext4, bpf,│ pstore, selinux, resctrl, fuse, tmpfs├── kernel/ runtime knobs: mm/, slab/, iommu_groups/, uevent_seqnum,│ debug/ (debugfs mount point), tracing/ (tracefs mount point)├── power/ system-wide power management state└── hypervisor/ hypervisor-specific data (empty on bare metal)
The complete top level of a live /sys, with one device followed all the way down and every symlink resolved. What it shows:devices/ holds real directories with real nesting; bus/, class/, block/, and dev/ hold nothing but flat lists of symlinks, each pointing back into devices/ by a relative path. The insight to take: the same physical NIC is reachable by four different paths that all resolve to the same two inodes — and which of the two depends on whether you asked about the PCI function or the network interface. /sys/bus/pci/devices/0000:bf:00.0 lands on the PCI device; /sys/class/net/enp191s0 lands on its child, one level deeper. This is why readlink -f before comparing paths is mandatory and why the rules document insists “all elements of a devpath must be real directories. Symlinks pointing to /sys/devices must always be resolved to their real target.”
flowchart LR
subgraph TREE["/sys/devices — the one real tree"]
direction TB
ROOT["pci0000:00"] --> PORT["0000:00:02.1<br/>(root port)"]
PORT --> NIC["0000:bf:00.0<br/>struct pci_dev<br/>(a real directory)"]
NIC --> IF["net/enp191s0<br/>struct net_device<br/>(a real directory, child)"]
end
subgraph VIEWS["flat symlink views"]
direction TB
BD["/sys/bus/pci/devices/0000:bf:00.0"]
BDR["/sys/bus/pci/drivers/r8169/0000:bf:00.0"]
CL["/sys/class/net/enp191s0"]
DV["/sys/dev/char/M:m<br/>/sys/dev/block/M:m"]
BL["/sys/block/dm-0"]
end
BD -.->|"symlink"| NIC
BDR -.->|"symlink — this one only<br/>exists while BOUND"| NIC
CL -.->|"symlink"| IF
DV -.->|"symlink, keyed by<br/>major:minor from stat(2)"| IF
BL -.->|"symlink"| IF
NIC -->|"subsystem →"| BUSDIR["/sys/bus/pci"]
NIC -->|"driver →"| DRVDIR["/sys/bus/pci/drivers/r8169"]
NIC -->|"iommu_group →"| IOG["/sys/kernel/iommu_groups/16"]
NIC -->|"firmware_node →"| ACPI["/sys/devices/LNXSYSTM:00/.../device:05"]
The one tree and the five views onto it, plus the back-links that point the other way. What it shows: every view is a flat directory of symlinks; the device itself carries back-links (subsystem, driver, iommu_group, firmware_node) that re-enter the views and other subtrees. The insight to take: the driver symlink is the one that comes and goes. It is created by driver_bound() when a match succeeds and removed on unbind — so readlink /sys/.../driver is the canonical “is this device bound, and to what?” query, and its absence is exactly the Unbound state described in Device-Driver Matching. Every other link in this picture is stable for the life of the device.
Why class and bus Are Different Views of the Same Thing
bus/ groups by how the device is attached; class/ groups by what the device does. A USB Ethernet dongle and a PCIe NIC appear in different bus/ directories and the same class/net directory; a PCIe NVMe drive appears under bus/pci and under class/nvmeandclass/block. Neither view is more real than the other and neither is the tree — they are two indexes over one set of nodes.
The kernel’s own rules document is blunt about how far that abstraction goes: “devices are only ‘devices’ … There is no such thing like class-, bus-, physical devices, interfaces, and such that you can rely on in userspace. Everything is just simply a ‘device’. Class-, bus-, physical, … types are just kernel implementation details which should not be expected by applications that look for devices in sysfs” (sysfs-rules.rst, v6.12). It then enumerates the only four properties a program may rely on — devpath, kernel name, subsystem, and driver — and tells you exactly how to obtain each:
Property
Example
How to read it
Rule
devpath
/devices/pci0000:00/0000:00:02.1/0000:bf:00.0
the real path under /sys, with /sys stripped
identical to DEVPATH= in the uevent; must be fully symlink-resolved
kernel name
0000:bf:00.0, sda, enp191s0
last element of the devpath
may contain spaces and !
subsystem
pci, net, block
read the subsystem symlink, take only the last element
a plain string, never a path
driver
r8169
read the driver symlink, take only the last element
absent link means no driver — never inherit it from a parent
The four stable device properties per sysfs-rules.rst. What it shows: the complete set of things a userspace program is permitted to depend on, and the exact retrieval method for each. The insight to take: three of the four are obtained by resolving a symlink and discarding everything but the basename. That is not incidental — it is the mechanism by which the kernel keeps the freedom to move devices around in the tree while userspace keeps working. The document is explicit that “the kernel is free to insert devices into the chain,” so you must “walk up the chain until you find the device that matches the expected subsystem” rather than counting ../ levels.
Two further rules from that document deserve highlighting because they are routinely violated. “Never depend on the ‘device’-link”: the device symlink inside a class-device directory is described as “a workaround for the old layout, where class devices are not created in /sys/devices/,” and “Accessing /sys/class/net/eth0/device is a bug in the application.” And “sysfs is always at /sys”: “Parsing /proc/mounts is a waste of time. Other mount points are a system configuration bug you should not try to solve.”
Uncertain
Verify: sysfs-rules.rst describes a planned consolidation — “It is planned to merge all three classification directories into one place at /sys/subsystem, following the layout of the bus directories” — and instructs applications to prefer /sys/subsystem if it exists. It does not exist on the test machine (kernel 7.1.8, 2026-09-04): ls /sys returns twelve entries and subsystem is not among them. The plan therefore appears never to have been carried out, and the document has been advising against a directory that has not materialised for many years. Reason: the absence is verified directly, but the history of why the merge was abandoned was not traced to a primary source (the relevant discussion would be on lore.kernel.org, which is behind an Anubis proof-of-work challenge and unreachable from this environment). To resolve: find the mailing-list thread that shelved /sys/subsystem, or a commit reverting the work. The practical guidance is unaffected — scan all three of /sys/bus, /sys/class, /sys/block, exactly as the document says to do when /sys/subsystem is missing. uncertain
The Top-Level Directories in Detail
The first-level directories of /sys encode the relationships among kernel structures (sysfs.rst, v6.12):
devices/ — “a filesystem representation of the device tree. It maps directly to the internal kernel device tree, which is a hierarchy of struct device.” This is the canonical location; everything else is symlinks into it.
bus/ — one directory per bus type (pci, usb, i2c, …); each contains devices/ (symlinks to the devices on that bus, pointing back into devices/) and drivers/ (one directory per loaded driver for that bus).
class/ — devices grouped by functional type (net/, block/, tty/, …) regardless of which bus they sit on; symlinks back into devices/.
block/ — symlinks to block devices.
dev/ — char/ and block/ subdirectories of <major>:<minor> symlinks, “a quick way to lookup the sysfs interface for a device from the result of a stat(2).”
module/ — parameters and state for loaded modules (/sys/module/<name>/parameters/<param> is how you read a module parameter at runtime).
firmware/, kernel/, power/, hypervisor/ — firmware tables, kernel runtime knobs, power-management state, and hypervisor data; the kernel exposes the kernel_kobj, power_kobj, firmware_kobj, hypervisor_kobj, and mm_kobj globals (kobject.h) as the chaining points for these subtrees.
The same physical device thus appears multiple times: once under devices/ (the real node) and as symlinks under bus/, class/, block/, and dev/. The docs stress this is an ABI: “the sysfs directory structure and the attributes in each directory define an ABI between the kernel and user space,” and all new attributes “must be documented in Documentation/ABI” (sysfs.rst).
uevents and udev: Pushing Changes to Userspace
sysfs is a pull interface — userspace reads files. But hotplug needs push: when a USB stick is inserted, userspace must be told a new device appeared. That is the job of uevents. When a kobject is added, removed, or changed, the kernel calls kobject_uevent(kobj, action) with an action from the kobject_action enum (KOBJ_ADD, KOBJ_REMOVE, KOBJ_CHANGE, KOBJ_MOVE, KOBJ_ONLINE, KOBJ_OFFLINE, KOBJ_BIND, KOBJ_UNBIND — kobject.h). The implementation in lib/kobject_uevent.c builds an environment of KEY=value strings — add_uevent_var(env, "ACTION=%s", ...), "DEVPATH=%s" (the device’s sysfs path), "SUBSYSTEM=%s" — and broadcasts it as a netlink message over the NETLINK_KOBJECT_UEVENT socket via netlink_broadcast(). Userspace’s device manager, udevd (systemd-udevd), listens on that netlink socket, receives each uevent, applies its rule database (/etc/udev/rules.d/, /usr/lib/udev/rules.d/), and acts — creating /dev nodes, setting permissions, loading firmware, running scripts, assigning persistent network names. So the full loop is: kernel changes the kobject graph → sysfs reflects it as files → kobject_uevent notifies udev over netlink → udev consumes the event and reads back the details from /sys. You can synthesize events for testing by writing to a kobject’s uevent attribute (kobject_synth_uevent), which is how udevadm trigger re-runs rules.
sequenceDiagram
autonumber
participant DRV as kernel subsystem
participant KOBJ as kobject core<br/>(lib/kobject_uevent.c)
participant KSET as the kobject's kset<br/>(uevent_ops)
participant BUS as bus/class ->uevent()
participant NL as netlink socket<br/>NETLINK_KOBJECT_UEVENT
participant UD as systemd-udevd
participant SYS as /sys
DRV->>KOBJ: kobject_uevent(kobj, KOBJ_ADD)
KOBJ->>KOBJ: walk up parents to find the owning kset<br/>(no kset means no uevent is sent at all)
KOBJ->>KSET: uevent_ops.filter(kobj) - drop this event?
KSET-->>KOBJ: keep
KOBJ->>KSET: uevent_ops.name(kobj) - the SUBSYSTEM= value
KOBJ->>KOBJ: add_uevent_var ACTION, DEVPATH, SUBSYSTEM, SEQNUM
KOBJ->>BUS: uevent_ops.uevent(kobj, env) - subsystem adds its own vars
BUS-->>KOBJ: MODALIAS, DEVNAME, DEVTYPE, driver-specific keys
KOBJ->>NL: netlink_broadcast() the NUL-separated KEY=value block
NL->>UD: recvmsg on the multicast group
UD->>UD: match against rules in /usr/lib/udev/rules.d and /etc/udev/rules.d
UD->>SYS: read back attributes named by the rules<br/>(ATTR{...} matches re-open files under /sys)
UD->>UD: mknod in /dev, set owner and mode, add symlinks,<br/>rename the interface, run programs
A uevent from kobject_uevent() to a /dev node. What it shows: the event is assembled in three stages — the core adds ACTION, DEVPATH, SUBSYSTEM and a global SEQNUM; the kset may filter it or rename its subsystem; the bus or class ->uevent() callback appends whatever it needs, including the MODALIAS that drives module autoloading. The insight to take: step 2 is the one that surprises people. kobject_uevent() walks up the parent chain looking for a kset, and a kobject with no kset anywhere above it emits no event at all — this is the usual reason a hand-rolled kobject shows up in /sys but udev never notices it. Step 12 is the other half of the design: the event carries only a handful of variables, and udev reads everything else back out of /sys by path. The push channel is deliberately thin because the pull channel already exists.
Synthetic uevents: udevadm trigger, Verified From Both Sides
A previous revision of this note flagged the udevadm trigger mechanism as unverified. It is now confirmed against both halves of the interface.
Kernel side. Writing to a kobject’s uevent attribute calls kobject_synth_uevent() (lib/kobject_uevent.c, v6.12). It parses the buffer as an action name (add, remove, change, move, online, offline, bind, unbind) optionally followed by arguments. With no arguments it injects SYNTH_UUID=0; with arguments, the first argument must be a valid UUID string (uuid_is_valid(), UUID_STRING_LEN = 36 characters), which becomes SYNTH_UUID=, and anything after it is parsed as space-separated KEY=VALUE pairs appended to the environment. A malformed write produces a distinctive kernel log line: synth uevent: <devpath>: unknown uevent action string or … incorrect uevent action arguments.
Userspace side.udevadm trigger does exactly this write. The systemd implementation calls sd_device_trigger_with_uuid() and, on failure, logs "Failed to write '%s' to '%s/uevent'" — naming the file it writes (systemd src/udev/udevadm-trigger.c, v257). The UUID form is used “if the user explicitly asked for it, or if --settle has been specified, so that we can recognize our own uevent”, and the code falls back to the plain sd_device_trigger() form on -EINVAL because “this might be caused by an old kernel which doesn’t know the UUID logic (pre-4.13)”. The same source documents the errno contract it observes: -ENOENT when a device has no uevent file or is already gone (ignored), -ENODEV from “some buggy device drivers e.g. /sys/devices/vio” (warned about), -EROFS when /sys is read-only (fatal), and -EACCES/-EPERM for an unprivileged caller — deliberately not fatal, because “some device can be owned by a user, e.g., network devices configured in a network namespace.”
So the loop closes: udevadm trigger writes change into /sys/.../uevent, kobject_synth_uevent() builds an environment with SYNTH_UUID set, kobject_uevent_env() broadcasts it over netlink, and udevd receives an event indistinguishable from a real hotplug except for that UUID. This is what makes udevadm trigger --settle deterministic: it can recognise its own events coming back.
The Stability Promise: Documentation/ABI
Everything above describes a mechanism. This section describes the contract, and it is the reason /sys paths are safe to hard-code into scripts, monitoring agents, and container runtimes while almost nothing else about kernel internals is.
The claim is made in sysfs.rst itself, in four sentences (sysfs.rst, v6.12):
The sysfs directory structure and the attributes in each directory define an ABI between the kernel and user space. As for any ABI, it is important that this ABI is stable and properly documented. All new sysfs attributes must be documented in Documentation/ABI. See also Documentation/ABI/README for more information.
That last pointer is the substance. Documentation/ABI/README defines four tiers of stability — not three — expressed as four subdirectories, and states what each one promises (ABI/README, v6.12):
Tier
Files at v6.12
The promise, quoted
What a userspace author should do
stable/
47
“Userspace programs are free to use these interfaces with no restrictions, and backward compatibility for them will be guaranteed for at least 2 years. Most interfaces (like syscalls) are expected to never change and always be available.”
depend on it freely
testing/
537
“felt to be stable, as the main development of this interface has been completed. The interface can be changed to add new features, but the current interface will not break by doing this, unless grave errors or security problems are found”
depend on it, but add your project’s name to the Users: field so maintainers can warn you
obsolete/
21
“still remaining in the kernel, but are marked to be removed at some later point in time. The description of the interface will document the reason why it is obsolete and when it can be expected to be removed”
migrate off it
removed/
15
“a list of the old interfaces that have been removed from the kernel”
historical record only
The four ABI stability tiers with their file counts at tag v6.12 (counted via the GitHub contents API, 2026-09-04). What it shows: the promise is graded, and the grading is visible in the source tree as a directory a file lives in. The insight to take: the distribution is the interesting part. testing/ holds 537 files against stable/’s 47 — more than ten to one. Almost every sysfs interface anyone actually uses (sysfs-bus-pci, sysfs-block, sysfs-class-net) lives in testing/, which promises only that existing behaviour will not break, not that it is finished. “sysfs is a stable ABI” is therefore a useful approximation, not a literal reading of the tree — the literal reading is “sysfs is a documented ABI with a graded and mostly-provisional stability commitment.” That is still vastly more than any other kernel-internals interface offers.
Every file in these directories carries a fixed set of fields — What:, Date:, KernelVersion:, Contact:, Description:, and optionally Users: — which makes the whole tree machine-readable and gives every documented path a date of introduction. This is how you answer “when did this attribute appear?” without a git archaeology session: the PCI driver_override file is dated April 2014, new_idDecember 2003, in Documentation/ABI/testing/sysfs-bus-pci.
Movement between tiers is one-directional and rule-bound.testing → stable when the developers declare it finished. stable → obsolete “as long as the proper notification is given.” obsolete → removed only “as long as the documented amount of time has gone by.” And critically: interfaces in testing “cannot be removed from the kernel tree without going through the obsolete state first.” There is no path that deletes a documented interface without first announcing its death in the tree.
stateDiagram-v2
[*] --> testing : a new attribute is merged<br/>with its ABI file<br/>(537 files at v6.12)
[*] --> stable : the author declares it finished<br/>from the start<br/>(47 files at v6.12)
testing --> stable : developers consider the<br/>interface complete
stable --> obsolete : proper notification given;<br/>the file records WHY and WHEN<br/>(21 files at v6.12)
testing --> obsolete : must pass through here -<br/>cannot be removed directly
obsolete --> removed : the documented waiting<br/>period has elapsed<br/>(15 files at v6.12)
removed --> [*] : historical record;<br/>the path no longer exists
note right of testing
"must be aware of changes that can occur
before these interfaces move to be marked
stable" - add your name to Users:
end note
note right of stable
"backward compatibility will be guaranteed
for at least 2 years"
end note
The ABI tier state machine defined by Documentation/ABI/README, with v6.12 file counts. What it shows: the legal transitions and, by their absence, the illegal ones — there is no testing → removed edge and no edge back out of removed. The insight to take: the value of this scheme is not that interfaces never change; it is that every change is announced in the tree before it happens, in a file with a Contact: address. That is a weaker guarantee than “never breaks” and a far more practical one, and it is why lspci, lsblk, udev, systemd, and every container runtime can parse exact /sys paths and still work across a decade of kernels.
Two things are explicitly not ABI, and the README names them: Kconfig (“Userspace should not rely on the presence or absence of any particular Kconfig symbol, in /proc/config.gz, in the copy of .config commonly installed to /boot, or in any invocation of the kernel build process”) and kernel-internal symbols (“Do not rely on the presence, absence, location, or type of any kernel symbol, either in System.map files or the kernel binary itself”). The second points at Documentation/process/stable-api-nonsense.rst, whose whole argument is that the kernel deliberately has no stable internal API — which is precisely why the sysfs boundary needs an explicit, documented, tiered promise. The stability lives at the userspace edge, and nowhere behind it.
sysfs-rules.rst adds one more clause that is easy to miss: error codes are part of the contract too, but weakly. “Avoid dependency on specific error codes wherever possible… Error codes will not be changed without good reason, and should a change to error codes result in user-space breakage, it will be fixed, or the offending change will be reverted. Userspace applications can, however, expect the format and contents of the attribute files to remain consistent in the absence of a version attribute change.” So the value format is a firm promise; the errno is a soft one.
Failure Modes and Misunderstandings
Putting more than one value in a file. The single most common sysfs mistake; it breaks the scripting contract and gets rejected in review. If you need multiple values, you need multiple files (or, rarely, an array of homogeneous values, or a binary attribute). The rule is convention enforced by code review, not by the kernel — see the survey above, which found roughly 6% of readable /sys files carrying more than one word in 2026, down from about 10% in 2010 (LWN 378884).
Exceeding PAGE_SIZE in show(). Because the buffer is exactly one page and the method is called once, output longer than 4,096 bytes on x86 cannot be delivered. sysfs_kf_seq_show() detects it, logs fill_read_buffer: <symbol> returned bad count naming the offending function, and clamps to PAGE_SIZE - 1 — “try to struggle along,” as the comment puts it. sysfs_emit() prevents it at the source; hand-rolled sprintf does not.
Calling sysfs_emit(buf + len, ...) to append. Trips the page-alignment WARN and returns 0, silently producing an empty or truncated file. Use sysfs_emit_at(buf, len, ...).
kfree() after kobject_add(). Frees an object other code may still reference via an open sysfs file → use-after-free. Always kobject_put().
Missing or empty release(). An empty release leaks the embedding structure, and — contrary to the documentation — the kernel will not tell you the release is missing on a stock build, because the message is a pr_debug(). Every ktype needs a real release that container_of()s and kfree()s.
Two kobjects in one struct. Explicitly forbidden: “No structure should EVER have more than one kobject embedded within it.” Two krefs means two independent lifetimes for one allocation, and whichever hits zero first frees memory the other still owns.
Assuming the object is “present” because the directory exists. sysfs reference-counts the kobject, which may outlive the physical device (a removed disk whose sysfs node is still open). A show() may need to check liveness.
Building a devpath out of symlinks.sysfs-rules.rst: “using or exposing symlink values as elements in a devpath string is a bug in the application.” Always readlink -f (or realpath()) to a /sys/devices/... path first, then compare.
Counting ../ levels to find a parent. “The kernel is free to insert devices into the chain.” Walk up until the subsystem link matches what you want.
Inheriting a property from a parent device. “Never copy any property of the parent-device into a child-device. Parent device properties may change dynamically without any notice to the child device.” If eth0 has no driver link, it has no driver — full stop.
Confusing sysfs with procfs, debugfs, or cgroupfs. New per-device knobs go in sysfs (documented ABI, one value per file); throwaway debug interfaces go in debugfs (explicitly no ABI promise); legacy free-form data lives in procfs; resource control lives in cgroupfs.
Alternatives and When to Choose Them
Interface
Foundation
Stability promise
Shape of the data
Reach for it when
sysfs (/sys)
kernfs
documented in Documentation/ABI, four graded tiers
one value per text file, hierarchical
exporting a per-object attribute or knob that userspace tooling will depend on
it is not yours to choose — this is ftrace’s own interface, split out of debugfs so tracing can be mounted without exposing all of debugfs
cgroupfs (/sys/fs/cgroup)
kernfs
documented, versioned (v1/v2)
one value per file, hierarchical, writable
resource control — a sibling kernfs client, not an alternative
resctrl (/sys/fs/resctrl)
kernfs
documented
one value per file
CPU cache/bandwidth allocation on x86; the third kernfs client
netlink
sockets
per-family
typed, structured messages
high-rate or push-style configuration and events; sysfs uevents already ride a netlink family
ioctl
character device
per-driver
arbitrary structs
complex, transactional operations that do not decompose into independent values
configfs (/sys/kernel/config)
its own
documented
one value per file, userspace creates the objects
the inverse of sysfs: sysfs shows kernel objects that already exist, configfs lets userspace mkdir new ones into being
Kernel↔userspace interfaces compared by foundation, promise, and shape. What it shows: the three axes that actually distinguish them — what library they are built on, what they promise, and whether the data decomposes into independent scalar values. The insight to take: the choice is almost always made by the promise you are willing to make, not by convenience. If you would be unhappy to be held to this interface in five years, it belongs in debugfs. If you would be happy, it belongs in sysfs with an entry in Documentation/ABI/testing. And the configfs row is the one people forget: sysfs is strictly a view of objects the kernel created, so if userspace needs to create kernel objects by making directories, sysfs is structurally the wrong tool.
Production Notes
/sys is the substrate of modern Linux device management. udevd turns its uevents into /dev nodes and persistent network-interface names; lsblk, lspci, and nvme list read device topology from /sys/class/block, /sys/bus/pci, and friends; container runtimes and orchestrators read /sys for device and (via the kernfs-shared cgroupfs) resource information; power-management and thermal daemons write tunables under /sys/devices/system/cpu/.../cpufreq/ and /sys/class/thermal/. Reading a CPU’s current frequency (cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq) or a network link’s speed (cat /sys/class/net/enp191s0/speed, which returned 1000 on the test machine) is the everyday face of the show/store machinery described above.
Scale. The test machine (a 32-core AMD workstation, kernel 7.1.8) carries 53,617 regular files under /sys, spread across 40 top-level entries in /sys/devices, 48 buses, 83 classes, and 289 loaded modules. Every one of those files is a kernfs_node plus an attribute struct held in RAM for the life of the object; sysfs has no backing store and no eviction. This is normally negligible, but it is the reason systems that create very large numbers of devices (thousands of loop devices, dense NVMe fabrics, high-count SR-IOV virtual functions) see measurable memory and boot-time cost from sysfs alone, and it is why the open_file_mutex hash table exists.
/sys is a security boundary. It is mounted nosuid,nodev,noexec by default (sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) on the test machine) and carries FS_USERNS_MOUNT in its file_system_type, meaning an unprivileged user namespace may mount it — which is why sysfs_init_fs_context() checks kobj_ns_current_may_mount(KOBJ_NS_TYPE_NET) and tags the superblock with the calling network namespace (fs/sysfs/mount.c, v6.12). That namespace tag is what makes /sys/class/net inside a container show only that container’s interfaces. The chain is worth following once, because it is the only namespace-awareness sysfs has and its narrowness explains a lot of container behaviour:
flowchart TB
NC["struct class net_class = {<br/>.name = 'net',<br/>.ns_type = &net_ns_type_operations,<br/>.namespace = net_namespace,<br/>...}<br/>(net/core/net-sysfs.c)"]
NC --> CCNT["class_child_ns_type(kobj)<br/>returns class->ns_type<br/>(drivers/base/class.c)"]
CCNT --> CD["create_dir() sees kobj_child_ns_ops() != NULL<br/>→ sysfs_enable_ns(kobj->sd)<br/>(lib/kobject.c)"]
CD --> FLAG["/sys/class/net's kernfs_node<br/>gains the KERNFS_NS flag"]
FLAG --> TAG["every child node stores a<br/>namespace tag in kernfs_node.ns<br/>(the struct net *, via class->namespace())"]
MNT["sysfs_init_fs_context():<br/>kfc->ns_tag = kobj_ns_grab_current(KOBJ_NS_TYPE_NET)<br/>→ the superblock remembers the mounter's netns"]
TAG --> LOOK{"kernfs lookup / readdir<br/>on a KERNFS_NS directory"}
MNT --> LOOK
LOOK -->|"node.ns == superblock's ns_tag"| SHOW["visible"]
LOOK -->|"node.ns != superblock's ns_tag"| HIDE["invisible — as if it<br/>does not exist"]
SHOW --> RESULT["/sys/class/net inside a container<br/>lists only that netns's interfaces"]
HIDE --> RESULT
How /sys/class/net becomes namespace-aware, from class declaration to readdir filtering. What it shows: the mechanism is entirely opt-in per class, driven by one field (.ns_type) in one struct, and it works by tagging child kernfs nodes and comparing that tag against a tag the superblock captured at mount time. The insight to take: exactly one namespace type is implemented — KOBJ_NS_TYPE_NET — and exactly one class uses it. Everything else in /sys is global./sys/devices, /sys/bus, /sys/block, /sys/class/block, and every device attribute are the host’s, unfiltered, regardless of which namespaces the reader is in. This is why container runtimes bind-mount /sys read-only and explicitly mask paths rather than trusting sysfs to isolate anything.
There is one more consequence of that diagram that catches people out, and it is directly demonstrable. The tag is captured on the superblock at mount time, not read from the current task’s namespace at lookup time. So entering a new network namespace is not by itself enough — you must also mount a fresh sysfs, or you keep looking through the old superblock’s tag. On the test machine (2026-09-04):
$ ls /sys/class/net | wc -l # on the host25$ unshare -rn sh -c 'ls /sys/class/net | wc -l'25 # new netns, but /sys is the INHERITED mount # → still tagged with the original namespace$ unshare -rn --mount sh -c 'mount -t sysfs sysfs /sys && ls /sys/class/net'lo # new netns AND a fresh sysfs mount # → the superblock captured the NEW tag
The first unshare has a genuinely empty network namespace — ip link inside it shows only lo — and yet /sys/class/net still lists all 25 host interfaces, because sysfs_init_fs_context() recorded kobj_ns_grab_current(KOBJ_NS_TYPE_NET) when /sys was mounted at boot. Remounting sysfs inside the namespace is what makes the view match reality. This is precisely why every container runtime mounts a fresh sysfs in the container’s mount namespace rather than bind-mounting the host’s, and why a hand-rolled unshare --net debugging session shows a confusing /sys unless you remember the --mount and the remount.
The ABI promise is what makes tooling possible. Because lspci, udev, systemd, and every orchestrator parse exact sysfs paths and single-value formats, kernel developers cannot rename or restructure existing attributes without breaking userspace — which is why the documentation insists every new attribute be registered in Documentation/ABI before merge, and why an attribute added in 2003 (new_id) still has the same path and the same write format in 2026.
See Also
Device-Driver Matching — the companion note. The device model is the object graph this note renders; matching is the operation that creates the driver symlink you see appear under /sys
The Linux Device Model — struct device, struct bus_type, struct class and how they compose into the graph sysfs projects
Device Attributes and sysfs Files — the driver-model layer above this one: DEVICE_ATTR_*, attribute groups, is_visible, binary attributes, and a full worked example