vault

Adding a New System Call to Linux

13 min read

Adding a New System Call to Linux

Adding a system call is the most consequential thing you can do to the kernel’s userspace contract: a syscall, once shipped in a release, “forms part of the API of the kernel, and has to be supported indefinitely” (adding-syscalls.rst). Because of the “we do not break userspace” rule, a syscall’s number, signature, and semantics are frozen the moment a stable kernel goes out the door — you can never reuse the number, never change the argument layout, and never break a program that already calls it. The official process, documented in Documentation/process/adding-syscalls.rst, is therefore as much about forethought — is a syscall even the right interface, and is its signature future-proof? — as about the mechanics of writing SYSCALL_DEFINE, allocating a number in every architecture’s table, wiring up unistd.h, handling 32-bit compatibility, testing it, and shipping a man page. This note walks the whole pipeline as of Linux 6.12 LTS (the doc is stable across the 6.x series, but the number-allocation step changed mechanically in 6.11, noted below).

This note is the kernel-developer counterpart to The syscall() Generic Wrapper Function (which is how userspace reaches a syscall that has no libc wrapper yet) and builds directly on SYSCALL_DEFINE and Syscall Handler Wrappers (the macro that defines the entry point) and The System Call Table (where the number lands).

Mental Model

A syscall is not a function you add in one place; it is a number wired to a handler, replicated across every architecture the kernel supports, with a userspace contract attached. Adding one means threading a single new identity through several independent tables and headers, plus a body of forward-compatibility discipline so that the signature you pick today still works on the kernels and userspaces of a decade from now.

flowchart TB
  Q{"Is a syscall even<br/>the right interface?"}
  ALT["Alternatives:<br/>new fs/device · sysfs/proc<br/>· fcntl/prctl command"]
  API["Design signature:<br/>flags arg · size-versioned struct<br/>· fd handles · loff_t offsets<br/>· capability check"]
  IMPL["Generic implementation:<br/>SYSCALL_DEFINEn · prototype in<br/>syscalls.h · CONFIG option<br/>· COND_SYSCALL stub"]
  NUM["Allocate the number:<br/>generic scripts/syscall.tbl<br/>+ x86 syscall_64.tbl / _32.tbl"]
  COMPAT["32-bit compat layer<br/>(only if struct/64-bit/ptr-to-ptr<br/>layout differs): COMPAT_SYSCALL_DEFINEn"]
  TEST["selftest in<br/>tools/testing/selftests/<br/>(calls via syscall())"]
  MAN["man page to<br/>linux-man@vger.kernel.org"]
  Q -->|"yes"| API
  Q -->|"maybe not"| ALT
  API --> IMPL --> NUM --> COMPAT --> TEST --> MAN

The end-to-end pipeline for adding a syscall. What it shows: the work splits into a design phase (is this a syscall at all, and is its signature extensible?) and a wiring phase (define it, number it in every arch table, add compat shims, test, document). The insight: the irreversible decisions are at the top — once the number and signature ship, they are permanent — so the doc front-loads the “should you even?” and “is it future-proof?” questions before any code.

Step 0 — Is a Syscall Even the Right Interface?

The doc opens by insisting you consider alternatives first, because every syscall is permanent overhead on the ABI (adding-syscalls.rst):

  • A filesystem-like object or device. If your operations can be modeled as objects you read/write, create a new filesystem or device instead. This is more easily encapsulated in a kernel module (a syscall must be built into the core kernel), and if the object yields a file descriptor, userspace gets poll/select/epoll notification for free. The downside the doc flags: operations that don’t map to read/write end up as ioctl(2) requests, “which can lead to a somewhat opaque API.”
  • sysfs or /proc. For merely exposing runtime system information, a node in sysfs or procfs is more appropriate — but the doc warns these “require that the relevant filesystem is mounted, which might not always be the case (e.g. in a namespaced/sandboxed/chrooted environment),” and explicitly forbids using debugfs as a production userspace API.
  • An fcntl(2) command for operations specific to a file descriptor, or a prctl(2) command for operations specific to a task/process. Both are multiplexing syscalls, “best reserved for near-analogs of existing” commands or simple flag get/set — not a dumping ground for unrelated functionality.

Only if none of these fit is a fresh syscall justified. And because it is permanent, the doc says: “it’s a very good idea to explicitly discuss the interface on the kernel mailing list,” and to CC linux-api@vger.kernel.org on the proposal.

Step 1 — Design the Signature for Extension

This is the part that most distinguishes a good syscall from a regret. The doc points at the table’s own scar tissue — the pairs of syscalls that exist because the first one wasn’t extensible: eventfd/eventfd2, dup2/dup3, inotify_init/inotify_init1, pipe/pipe2, renameat/renameat2. Each …2/…1 exists only because the original forgot a flags argument. The forward-compatibility discipline:

Take a flags argument, and reject unknown bits. For simple syscalls, a flags parameter is the preferred extension point. You must validate it so that a program built against a newer kernel cannot silently get old behaviour on an older one (adding-syscalls.rst):

if (flags & ~(THING_FLAG1 | THING_FLAG2 | THING_FLAG3))
    return -EINVAL;

If no flags are defined yet, check the argument is zero. The point is that an unknown flag must fail loudly with EINVAL, never be ignored — otherwise the meaning of a flag bit is ambiguous across kernel versions.

For many arguments, use a size-versioned struct. Encapsulate the parameters in a structure passed by pointer, with a size field as the first member:

struct xyzzy_params {
    u32 size; /* userspace sets p->size = sizeof(struct xyzzy_params) */
    u32 param_1;
    u64 param_2;
    u64 param_3;
};

As long as every later-added field defaults to “zero == old behaviour,” this handles version mismatch in both directions: a newer userspace calling an older kernel works because the kernel checks that any memory beyond the struct size it knows is zero (i.e. the new param_4 == 0); an older userspace calling a newer kernel works because the kernel zero-extends the smaller struct it receives. The doc cites perf_event_open(2) and its perf_copy_attr() function (kernel/events/core.c) as the reference implementation of this pattern.

Other signature rules, each citing a real failure the kernel learned from:

  • Use file descriptors as object handles. “Don’t invent a new type of userspace object handle when the kernel already has mechanisms and well-defined semantics for using file descriptors.”
  • Provide an O_CLOEXEC-equivalent flag for any syscall returning a new fd, so userspace can close the race window between getting the fd and setting FD_CLOEXEC — a window where a fork()+execve() in another thread could leak the descriptor. (But don’t reuse the literal O_CLOEXEC value, which is architecture-specific and lives in a crowded O_* numbering space.)
  • Consider an *at() variant if there’s a filename argument: xyzzyat(dfd, path, …, flags) is more flexible, and with AT_EMPTY_PATH it gives an fxyzzy()-on-an-fd operation for free.
  • Use loff_t for file offsets so 64-bit offsets work even on 32-bit architectures.
  • Govern privileged operations with a capable() check against a specific capability bit, “avoid adding new uses of the already overly-general CAP_SYS_ADMIN” (the doc is blunt that lumping things under CAP_SYS_ADMIN defeats the purpose of capabilities — see Linux Security MOC). If the syscall manipulates another process, gate it with ptrace_may_access().
  • Put explicitly-64-bit parameters on odd-numbered argument slots (parameter 1, 3, 5). Some non-x86 architectures pass a 64-bit value as a contiguous pair of 32-bit registers, and odd-numbered placement lets that pair align cleanly. (This is the kernel mirror of the userspace 64-bit-argument caveat in The syscall() Generic Wrapper Function.)

Step 2 — Write the Generic Implementation

The entry point is added with the SYSCALL_DEFINEn() macro, not a hand-written function, because the macro also emits metadata used by other tooling (see SYSCALL_DEFINE and Syscall Handler Wrappers for what it expands to and why x86 decodes struct pt_regs on the fly):

SYSCALL_DEFINE3(xyzzy, int, arg1, char __user *, arg2, unsigned int, flags)
{
    /* ... */
}

The n (here 3) is the argument count, followed by (type, name) pairs. Alongside it you add:

  • A prototype in include/linux/syscalls.h, asmlinkage-marked to match the call convention: asmlinkage long sys_xyzzy(...);
  • A CONFIG option (typically in init/Kconfig) so the feature is optional — and verify the kernel “still builds with the new CONFIG option turned off.”
  • A fallback stub in kernel/sys_ni.c via COND_SYSCALL(xyzzy);, which provides a -ENOSYS-returning implementation when the feature is compiled out. This is what makes syscall(__NR_xyzzy) return ENOSYS rather than crashing on a kernel built without the feature.

A hard rule from the doc’s final section: never call sys_xyzzy() from inside the kernel. Since v4.17 on x86-64, the syscall calling convention decodes struct pt_regs on the fly in a wrapper, so only the parameters a syscall actually uses are passed; calling the sys_ entry directly would feed it garbage and “may cause serious trouble down the call chain.” Shared logic goes in a ksys_xyzzy() helper that both the syscall stub and other kernel code call.

Step 3 — Allocate the Number in Every Architecture Table

Uncertain

Verify: the precise generic-table mechanism name at 6.12. Reason: the prose of adding-syscalls.rst at the v6.12 tag still instructs editing include/uapi/asm-generic/unistd.h directly, but the kernel also ships scripts/syscall.tbl (confirmed present at v6.12), introduced around 6.11, which several architectures (arc, arm64, csky, hexagon, loongarch, nios2, openrisc, riscv) now consume to generate their unistd.h — so the doc text lags the actual mechanism for those arches. To resolve: read scripts/syscall.tbl, scripts/syscallhdr.sh/syscalltbl.sh, and a consuming arch’s Makefile.syscalls at the v6.12 tag. uncertain

There is no single global syscall number — each architecture has its own. You must wire the new call into each table you support.

The generic table (consumed by the architectures that share one — arc, arm64, csky, hexagon, loongarch, nios2, openrisc, riscv, and others). As of 6.11+ this is scripts/syscall.tbl, whose format is <number> <abi> <name> <native-entry> [<compat-entry>] — confirmed at v6.12, e.g. its first line is 0 common io_setup sys_io_setup compat_sys_io_setup (scripts/syscall.tbl). You add a line like 468 common xyzzy sys_xyzzy. (Historically, and as the older doc text still describes, this was done by hand-editing include/uapi/asm-generic/unistd.h with __SYSCALL(__NR_xyzzy, sys_xyzzy) and bumping __NR_syscalls; that header is now generated from the table for those arches.)

The x86 tables, which x86 maintains separately. For a syscall that’s not “special,” add a common entry (covering both x86_64 and x32) to arch/x86/entry/syscalls/syscall_64.tbl — whose format header reads <number> <abi> <name> <entry point> [<compat entry point> [noreturn]], with the abi being common, 64, or x32 (syscall_64.tbl):

468   common   xyzzy     sys_xyzzy

and an i386 entry to arch/x86/entry/syscalls/syscall_32.tbl:

468   i386     xyzzy     sys_xyzzy

The doc warns twice that “these numbers are liable to be changed if there are conflicts in the relevant merge window” — two patches in flight may both grab 468, and the maintainer renumbers one. This is why you never depend on a pending number and why, once merged and released, the number is sacred (see System Call Numbers and the ABI).

Step 4 — Compatibility Layer (Only Sometimes)

For most syscalls, the same 64-bit implementation serves 32-bit userspace transparently — even an explicit pointer argument is handled automatically. A compat_sys_xyzzy() is needed only when 32-bit and 64-bit layouts genuinely differ, specifically when an argument is (adding-syscalls.rst):

  • a pointer to a pointer,
  • a pointer to a struct containing a pointer (e.g. struct iovec __user *),
  • a pointer to a varying-sized integral type (time_t, off_t, long, …),
  • a pointer to a struct containing a varying-sized integral type,

or when an argument is explicitly 64-bit even on 32-bit (e.g. loff_t, __u64), because a 32-bit app splits that into two 32-bit halves the compat layer must reassemble. Notably, a pointer to an explicit 64-bit type does not need compat — the doc gives splice(2)’s loff_t __user * as the counter-example.

When needed, you write COMPAT_SYSCALL_DEFINEn(xyzzy, …), add an asmlinkage prototype to include/linux/compat.h, and (if a struct differs) define a struct compat_xyzzy_args there using compat_ types (compat_uptr_t, compat_long_t, …) for the variable-width fields, keeping fixed-width fields like u64 as-is. The compat stub typically converts to 64-bit and calls the shared ksys_/inner helper. You then point the 32-bit table column at the compat entry (on x86, __ia32_compat_sys_xyzzy in syscall_32.tbl; for x32, decide whether the layout matches the 32-bit or 64-bit version — if a pointer-to-pointer is involved, x32 is ILP32 so it follows the 32-bit layout via __x32_compat_sys_xyzzy). See The Compat Syscall Layer for 32-bit Binaries.

Step 5 — Syscalls That “Return Elsewhere,” Testing, and the Man Page

Special control-flow syscalls. A few syscalls don’t return to the next instruction with the same stack/registers/address space: rt_sigreturn returns to a different location, fork/vfork/clone change the memory space, execve/execveat change the whole program image. These need arch-specific assembly entry points that save/restore extra registers — on x86_64 a stub_xyzzy in arch/x86/entry/entry_64.S referenced from the table, with a stub32_xyzzy counterpart in entry_64_compat.S for 32-bit. (For user-mode Linux, a #define stub_xyzzy sys_xyzzy in arch/x86/um/sys_call_table_64.c keeps the UML build working since it simulates registers.) The doc also flags the audit subsystem as a special case: if your syscall is analogous to open/exec/socketcall, audit’s arch-specific classifiers need updating.

Testing. Add a self-test under tools/testing/selftests/. Because there is by definition no libc wrapper for a brand-new syscall, the test must invoke it via [[The syscall() Generic Wrapper Function|syscall()]] — this is the canonical reason that generic wrapper exists. Verify it across ABIs: compiled as x86_64 (-m64), x86_32 (-m32), and x32 (-mx32). For deeper coverage, contribute to the Linux Test Project, or xfstests for filesystem changes.

The man page. “All new system calls should come with a complete man page,” ideally in groff markup (plain text accepted), CC’ed to linux-man@vger.kernel.org (adding-syscalls.rst; man-pages patch guide).

Patch structure. Split the series into distinct commits: (1) core implementation — CONFIG, SYSCALL_DEFINEn, prototype, generic number, fallback stub; (2) the x86 wiring; (3) the selftest; (4) the man-page draft — and CC linux-api@vger.kernel.org on the whole proposal.

Common Misunderstandings and Failure Modes

  • Forgetting the flags argument. The single most common regret, immortalized by the …2 syscall pairs. If there is any chance the syscall will grow, take a flags (or size-versioned struct) from day one.
  • Ignoring unknown flag bits. Accepting (rather than EINVAL-rejecting) unknown bits makes a flag’s meaning version-dependent and breaks forward compatibility — the exact thing flags were supposed to provide.
  • Assuming a pending number is stable. A number can be renumbered during the merge window; only a released number is frozen. Conversely, never reuse a retired number (see System Call Numbers and the ABI).
  • Wiring only x86 and forgetting the generic table (or vice versa), leaving the syscall returning ENOSYS on architectures you thought you supported.
  • Calling sys_xyzzy() from kernel code. Forbidden on x86-64 since v4.17 due to the pt_regs decoding convention; use a ksys_ helper.
  • Shipping a syscall that should have been sysfs/fcntl/prctl. The doc’s whole first section exists because permanent ABI surface is expensive; reviewers will push back hard if an alternative fits.

See Also

Graph View

Backlinks