The We Do Not Break Userspace ABI Promise

Of all the rules that govern Linux kernel development, one stands above the rest and is enforced with a vehemence that has become legend: once a system call ships in a released kernel, it must keep working — unchanged, forever — for every binary that already uses it. A change that makes a previously-working userspace program misbehave is, by definition, a kernel bug to be reverted, no matter how “correct” the change is in the abstract. Linus Torvalds states it as the first rule of kernel maintenance: “If a change results in user programs breaking, it’s a bug in the kernel. We never EVER blame the user programs” (Torvalds, LKML, 2012-12-23). The reasoning is teleological — the kernel exists to serve userspace, so breaking userspace is the worst possible failure. This single rule is why the syscall surface is one of the most stable interfaces in all of computing, and it shapes the concrete engineering patterns by which Linux extends itself: numbers are never reused, flags are added but never changed, and new behavior arrives as new syscalls rather than altered semantics of old ones.

This note covers the promise itself — its statement, its enforcement, the patterns it forces, and real reverted regressions. The conceptual machinery of what is being frozen (the binary contract) is the sibling The Kernel ABI vs API Distinction; the procedure for adding a new syscall under these constraints is Adding a New System Call to Linux.

The Promise, Stated Precisely

The userspace-facing ABI — syscall numbers, their argument registers, the layout of every structure they read and write, the errno values they return — is treated as immutable once released. The kernel’s own documentation puts syscalls at the most-frozen end of its stability spectrum: they “are expected to never change and always be available” (abi-stable, v6.12). The companion stable-api-nonsense.rst confirms: “The kernel to userspace interface … That interface is very stable over time, and will not break” (v6.12). The standing proof is that binaries linked against kernels older than 0.9 still execute on modern releases — three decades of binary compatibility across a kernel that has otherwise been rewritten many times over.

Crucially, “breaking userspace” is defined behaviorally, not by the letter of any spec. If a program relied on a behavior — even an undocumented quirk, even a bug — and a “fix” changes that behavior such that the program stops working, that is a regression. The intent of the original code does not matter; only whether a real, existing program broke. This is why the rule is so strict in practice: it is not “don’t change documented behavior,” it is “don’t change observable behavior that anything depends on.”

The Canonical Rant — Verbatim

The definitive statement came in December 2012 on the Linux Kernel Mailing List, when a media-subsystem maintainer (Mauro Carvalho Chehab) defended a commit that changed a syscall’s error return — replacing -EINVAL with -ENOENT — which broke PulseAudio and several KDE applications (thread, 2012-12-23). Torvalds’s reply is reproduced widely; the load-bearing passages (felipec.wordpress.com; linuxreviews):

“If a change results in user programs breaking, it’s a bug in the kernel. We never EVER blame the user programs.”

“WE DO NOT BREAK USERSPACE!”

Uncertain

Verify: the exact wording of the longer passages from the 2012-12-23 email beyond the two short quotes above (e.g. “Seriously. How hard is this rule to understand?” and the lines about the kernel serving userspace). Reason: both the original archive (lkml.org/lkml/2012/12/23/75) and the canonical archive (lore.kernel.org) returned an Anubis anti-bot “Access Denied” page on fetch in this environment, so the two short quotes used above are corroborated from secondary reproductions (felipec’s blog and LinuxReviews, which quote the same lines identically) and the thread title is confirmed via search; the longer passages are not independently verified here. To resolve: retrieve the raw email from lore.kernel.org/marc.info from an unblocked client and confirm verbatim. uncertain

The substance behind the slogan: the error-code change was more correct by some reading of the API, but PulseAudio had been built and shipped against the old code and depended on -EINVAL. Because a real, deployed binary broke, the change was a regression and had to be reverted — correctness of the new behavior was irrelevant.

flowchart TD
  CH["a kernel change ships<br/>(refactor, 'fix', cleanup)"]
  CH --> Q{"did any existing<br/>userspace binary<br/>stop working?"}
  Q -->|"no"| OK["fine — change stands"]
  Q -->|"yes"| BUG["it is a KERNEL BUG<br/>(a regression)"]
  BUG --> REV["revert it — the<br/>'correctness' of the change<br/>does not matter"]
  REV --> ALT["want the new behavior?<br/>add a NEW syscall / flag,<br/>never change the old one"]

The decision procedure behind “we do not break userspace.” What it shows: the only test that matters is whether a real existing binary broke; if so, the change is reverted regardless of how technically correct it was, and the desired new behavior must be reintroduced through a new, opt-in interface. The insight: this turns “correctness” upside down relative to most software — for the kernel ABI, compatibility outranks correctness, because the kernel’s purpose is to keep userspace running.

The Consequences — Engineering Patterns Forced by the Promise

Because old behavior can never change, the kernel can only ever add. Four concrete patterns fall out.

1. Syscall numbers are append-only and never reused

A syscall number is a permanent ABI coordinate: a binary contains the literal integer in rax, so reassigning that integer to a different call would silently break every binary that used it. Numbers are therefore only ever appended. The adding-syscalls.rst guidance is explicit that “a new system call forms part of the API of the kernel, and has to be supported indefinitely” (v6.12). Even retired syscalls keep their slots — the number is left wired to a stub returning -ENOSYS rather than recycled. See System Call Numbers and the ABI.

2. New behavior arrives as a new syscall, never changed semantics

When the kernel wants to change what a syscall does, it does not edit the old one; it ships a sibling. syscalls(2) documents whole families of these “version-2” calls created precisely because the original could not be extended without breaking it (syscalls(2)):

Original	Successor	Why a new call was needed
dup2	dup3 (2.6.27)	dup3 adds a flags arg (e.g. O_CLOEXEC) that dup2 could not grow
pipe	pipe2 (2.6.27)	pipe2 adds a flags arg for O_CLOEXEC/O_NONBLOCK atomically
eventfd	eventfd2 (2.6.27)	added flags for close-on-exec
renameat	renameat2 (3.15)	added flags (RENAME_NOREPLACE, RENAME_EXCHANGE, RENAME_WHITEOUT)
clone	clone3 (5.3)	replaced the cramped fixed-register arg list with an extensible struct

The version numbers are confirmed against the manual pages directly: dup3 and pipe2 both appeared in “Linux 2.6.27, glibc 2.9” (dup(2); pipe(2)) — a single release that added a whole batch of *2/*3 variants precisely to retrofit the O_CLOEXEC close-on-exec flag, which the originals had no room for. The table is a real enumeration; the lesson is in the pattern, not the rows. renameat2 is the clean illustration: “a renameat2() call with a zero flags argument is equivalent to renameat()” (renameat2(2)). The old call’s behavior is preserved bit-for-bit; the new behavior is strictly opt-in through a brand-new number. See Adding a New System Call to Linux.

3. The flags-argument pattern — design for extension up front

The recurrence of “version-2” calls taught a lesson now baked into the syscall-design guidance: include a flags argument from the very first version, even if no bits are defined yet, so the call can grow without a successor. The kernel must reject unknown bits so that programs cannot accidentally depend on undefined behavior (adding-syscalls, v6.12):

/* validate flags before doing anything */
if (flags & ~(THING_FLAG1 | THING_FLAG2 | THING_FLAG3))
        return -EINVAL;

Line-by-line:

• flags & ~(...) masks off every known flag bit, leaving only bits the kernel does not recognize.
• If any unknown bit is set, the call returns -EINVAL immediately. This is the linchpin of forward compatibility: by rejecting undefined bits today, the kernel keeps them free to define tomorrow with a guaranteed meaning, and guarantees that an old kernel will cleanly refuse a flag it doesn’t understand rather than silently ignore it. The guidance is explicit that “if no flags values are used yet, check that the flags argument is zero” (v6.12).

4. The versioned-struct (size-field) pattern — extensible argument blocks

For calls with many arguments, the modern idiom passes a pointer to a struct that carries its own size field, so the struct can gain fields over time without a new syscall. The kernel reconciles version skew in both directions (adding-syscalls, v6.12):

• Newer userspace, older kernel (struct larger than the kernel expects): “the kernel code should check that any memory beyond the size of the structure that it expects is zero” — i.e. the caller must not be requesting a feature this kernel lacks.
• Older userspace, newer kernel (struct smaller than the kernel’s): “the kernel code can zero-extend a smaller instance of the structure” — the missing fields default to zero-behavior, which must mean “as before.”

This is exactly how clone3 (its struct clone_args), openat2 (struct open_how), and the sched_setattr/sched_getattr (struct sched_attr) families work. The helper copy_struct_from_user() implements the size-reconciliation logic in one place. The design rule that makes it work: every new field’s zero value must reproduce the old behavior, so that a struct from an old binary (whose new fields are implicitly zero) behaves identically to the pre-extension call.

Enforcement — How the Promise Is Kept in Practice

The rule is not self-executing; it is enforced socially and procedurally:

• Maintainer and Linus veto. A patch shown to cause a regression is reverted, frequently with a public rebuke. The 2012 media thread is the archetype, but the pattern repeats: a “Reported-by: a real user” plus “this used to work” is sufficient grounds to revert, and the burden is on the change author to prove nothing broke.
• The stable-tree discipline. The stable/LTS trees are governed by Documentation/process/stable-kernel-rules.rst, which is deliberately conservative about what may be backported: a patch “must be obviously correct and tested,” “cannot be bigger than 100 lines, with context,” must fix a real problem (“an oops, a hang, data corruption, a real security issue, a hardware quirk, a build error … or some ‘oh, that’s not good’ issue”), and must reject speculation — “No ‘This could be a problem…’ type of things like a ‘theoretical race condition’” (stable-kernel-rules, v6.12). The same rules require that when a fix is backported to one stable series it be present in all newer supported series, “to prevent regressions users might encounter during updates.” The net effect reinforces the promise: the stable trees take small, proven fixes and refuse risky churn, so an LTS kernel a user upgrades within a series almost never changes observable behavior.
• Bisection and Fixes: tags. git bisect makes it cheap to pin a regression to a single commit; the Fixes: tag and reported-regression tracking make regressions visible.
• The -rc cycle. Roughly seven release candidates per release exist largely to surface regressions before they reach users; a regression found during -rc is the normal, expected catch point.

There is a pragmatic escape valve: if a behavior breaks userspace but nobody actually depends on it, the change can stand — the rule is about real breakage, not theoretical. Torvalds has occasionally allowed a behavior change to remain after it became clear no real program relied on the old behavior. The test is empirical: does a real, deployed program break?

Real Reverted Regressions

• The 2012 media ENOENT/EINVAL change (LKML 2012-12-23) — a media-subsystem commit changed a syscall’s error return from -EINVAL to -ENOENT; PulseAudio and KDE apps that special-cased -EINVAL broke. Reverted; produced the canonical rant.
• /proc and /sys format changes — userspace tools (ps, top, monitoring agents) parse the text of these files, making their format part of the ABI. Changes that reorder or rename fields have been reverted when tools broke; this is why new /proc fields are appended, not inserted.

Uncertain

Verify: specific named, dated examples of post-release reverts beyond the 2012 media case (e.g. particular /proc field reverts, the ext4/overlayfs behavioral reverts sometimes cited). Reason: these are recalled from kernel lore and LWN coverage but were not pinned to a primary commit/thread during this research. To resolve: search git log for Revert commits with regression rationale and corroborate against LWN articles. uncertain

Common Misunderstandings

• “The promise covers internal kernel interfaces too.” No — the opposite. In-kernel APIs and ABIs are deliberately unstable; only the userspace-facing interface is frozen. See The Kernel ABI vs API Distinction and stable-api-nonsense.rst.
• “A documented behavior can be changed if the docs allow it.” No. The test is whether a real program breaks, not what the manual says. Undocumented and even buggy behaviors are protected if something depends on them.
• “New syscalls mean the old ones get cleaned up eventually.” No — dup2 is not going away because dup3 exists. The old numbers and behaviors persist indefinitely; the surface only grows.
• “This makes Linux’s syscall list bloated and inconsistent.” Yes, deliberately so. The dup2/dup3, pipe/pipe2, eventfd/eventfd2 proliferation is the visible cost of the promise — accepted as the price of never breaking a binary.

Why This Trade-Off, and the Alternatives

The promise prioritizes compatibility over cleanliness and over abstract correctness. The alternative — a kernel free to change syscall semantics to “fix” them — is what many userspace projects and some OSes do, and it produces churn that pushes maintenance cost onto every downstream consumer at every upgrade. Linux instead absorbs the cost internally: it accepts a growing, sometimes ugly syscall surface and a permanent maintenance obligation, in exchange for the property that no userspace binary ever needs to be touched because the kernel changed. The payoff is enormous in aggregate — distributions can upgrade kernels under a frozen application base, and decades-old proprietary binaries keep running. The freedom the kernel does keep is purely internal (see The Kernel ABI vs API Distinction): it can refactor any internal interface at will precisely because it refuses to expose those internals as a stable contract. The two rules are complementary halves of one design — freeze the outside, churn the inside.

See Also

• The Kernel ABI vs API Distinction — what is frozen (the binary contract) vs the unstable in-kernel interfaces
• Adding a New System Call to Linux — the procedure for extending the ABI under these constraints
• System Call Numbers and the ABI — the append-only, never-reused number space
• SYSCALL_DEFINE and Syscall Handler Wrappers — the macro that pins a syscall’s ABI in place
• The errno Convention and Negative Return Values — error codes are part of the frozen contract (the 2012 regression was an errno change)
• The Compat Syscall Layer for 32-bit Binaries — preserving the ABI for 32-bit binaries on 64-bit kernels
• Linux System Call Interface MOC — the parent map