vault

SYSCALL_DEFINE and Syscall Handler Wrappers

15 min read

SYSCALL_DEFINE and Syscall Handler Wrappers

When a kernel developer writes a system call, they do not write a function named sys_open directly — they write SYSCALL_DEFINE3(open, const char __user *, filename, int, flags, umode_t, mode). That single macro invocation expands into a small family of functions: the real handler that does the work (__do_sys_open), a thin sign-extending shim around it (__se_sys_open), and an architecture-specific entry stub that decodes the syscall’s arguments out of the saved register frame (__x64_sys_open on x86-64, taking a single struct pt_regs *). This layering is not bureaucracy: each layer exists to close a specific class of bug — argument-register leakage, missing sign-extension of 32-bit arguments into 64-bit registers, and the calling-convention mismatch between “the C function the author wrote” and “what the CPU actually handed the kernel.” This note traces SYSCALL_DEFINE0 through SYSCALL_DEFINE6 and COMPAT_SYSCALL_DEFINE as they expand in Linux 6.12 LTS, on both the generic path and the x86-64 pt_regs-based path, and explains the security history that motivated the wrappers.

This note is part of the Linux System Call Interface MOC. It assumes you understand where the handler sits in the bigger picture — that a syscall number indexes The System Call Table to reach one of these stubs, and that the register frame the stub reads from is set up by Per-Architecture Syscall Entry Assembly and consumed via The pt_regs Register Frame. The bookkeeping that runs around the handler (seccomp, audit, ptrace, signals) lives in The Generic Syscall Entry and Exit Layer; this note is strictly about the handler and its wrappers.

Mental Model: One Macro, Three Layers

The cleanest way to hold SYSCALL_DEFINE in your head is as a factory that, from one declaration, manufactures three nested functions, each peeling off one concern:

flowchart TB
  TABLE["sys_call_table[__NR_read]<br/>(one function pointer)"]
  STUB["__x64_sys_read(const struct pt_regs *regs)<br/>ARCH STUB: decode args from regs->di, regs->si, ...<br/>clear unused arg registers"]
  SE["__se_sys_read(long fd, long buf, long count)<br/>SIGN-EXTEND shim: all args widened to long,<br/>then cast back to declared types"]
  DO["__do_sys_read(unsigned int fd, char __user *buf, size_t count)<br/>THE REAL HANDLER: copy_from_user, do I/O, return count"]
  TABLE -->|"call through pointer"| STUB
  STUB -->|"__se_sys_read(regs->di, regs->si, regs->dx)"| SE
  SE -->|"__do_sys_read((unsigned int)fd, (char __user *)buf, (size_t)count)"| DO

The three-layer expansion of SYSCALL_DEFINE3(read, ...) on x86-64. What it shows: the syscall table holds a pointer to the arch stub, not to the author’s function. The stub pulls each argument out of the saved pt_regs register frame and calls the sign-extension shim, which widens every argument to a full long and then casts it back to the declared C type before finally calling the real handler. The insight: the author writes only the innermost function (__do_sys_read); the macro fabricates the two outer layers to make the C-function world and the raw-register world meet safely. The arch stub is what the table actually points at, which is why the table is full of __x64_sys_* symbols, not sys_*.

The Generic Expansion (Architectures Without a pt_regs Wrapper)

Start with the simplest, architecture-independent form, defined in include/linux/syscalls.h. The numbered macros are trivial dispatchers onto a common SYSCALL_DEFINEx (per include/linux/syscalls.h at v6.12):

#define SYSCALL_DEFINE1(name, ...) SYSCALL_DEFINEx(1, _##name, __VA_ARGS__)
#define SYSCALL_DEFINE2(name, ...) SYSCALL_DEFINEx(2, _##name, __VA_ARGS__)
/* ... through SYSCALL_DEFINE6 ... */
#define SYSCALL_DEFINE_MAXARGS	6

The suffix number is the argument count, and the maximum is six — SYSCALL_DEFINE_MAXARGS is 6. That is not an accident of taste: it is the number of argument registers the most constrained supported calling convention provides. A syscall that conceptually needs seven arguments (like the old mmap) must instead pack them into a struct and pass a pointer, which is exactly why mmap historically went through a mmap_arg_struct. The arguments are written as alternating type, name pairs — SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count) — because the macro machinery (__MAP) walks the variadic list two tokens at a time.

SYSCALL_DEFINEx expands to two things stacked together (per include/linux/syscalls.h):

#define SYSCALL_DEFINEx(x, sname, ...)				\
	SYSCALL_METADATA(sname, x, __VA_ARGS__)			\
	__SYSCALL_DEFINEx(x, sname, __VA_ARGS__)

SYSCALL_METADATA emits the struct syscall_metadata and the sys_enter/sys_exit ftrace tracepoint plumbing — the type-name strings, argument count, and event structures that CONFIG_FTRACE_SYSCALLS uses so that tools can print a syscall’s arguments symbolically (this is the kernel-side hook behind [[Syscall Tracepoints sys_enter and sys_exit]]). When CONFIG_FTRACE_SYSCALLS is off, SYSCALL_METADATA expands to nothing. The interesting part is __SYSCALL_DEFINEx, the generic version of which reads (per include/linux/syscalls.h at v6.12):

#define __SYSCALL_DEFINEx(x, name, ...)					\
	__diag_push();							\
	__diag_ignore(GCC, 8, "-Wattribute-alias",			\
		      "Type aliasing is used to sanitize syscall arguments");\
	asmlinkage long sys##name(__MAP(x,__SC_DECL,__VA_ARGS__))	\
		__attribute__((alias(__stringify(__se_sys##name))));	\
	ALLOW_ERROR_INJECTION(sys##name, ERRNO);			\
	static inline long __do_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__));\
	asmlinkage long __se_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__));	\
	asmlinkage long __se_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__))	\
	{								\
		long ret = __do_sys##name(__MAP(x,__SC_CAST,__VA_ARGS__));\
		__MAP(x,__SC_TEST,__VA_ARGS__);				\
		__PROTECT(x, ret,__MAP(x,__SC_ARGS,__VA_ARGS__));	\
		return ret;						\
	}								\
	__diag_pop();							\
	static inline long __do_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__))

Reading it line by line:

  • asmlinkage long sys##name(...) __attribute__((alias(...))) declares the public symbol — e.g. sys_read — and makes it a GCC alias for __se_sys_read. asmlinkage (which on most architectures means __attribute__((regparm(0)))) tells the compiler that arguments arrive on the stack in the classic convention rather than in registers, matching what the assembly entry path expects. The __diag_ignore around it suppresses GCC’s -Wattribute-alias warning, because the alias deliberately points a function of declared types at a function of all-long types; the comment in the header is explicit: “Type aliasing is used to sanitize syscall arguments.”
  • static inline long __do_sys##name(...declared types...) — forward declaration of the real handler, with the exact types the author wrote (unsigned int fd, char __user *buf, size_t count).
  • asmlinkage long __se_sys##name(...all long...) — the sign-extension shim. Note __SC_LONG: it rewrites every argument’s type to long (or long long for 64-bit types on 32-bit arches, via __builtin_choose_expr). Its body calls __do_sys##name with __SC_CAST, which is (__force t) a — i.e. each long argument is force-cast back to the declared type. The shim is what makes every argument occupy a full machine word before the real handler sees it.
  • __SC_TEST expands to a BUILD_BUG_ON_ZERO per argument that fails the build if any argument type is wider than long (other than explicit long long), catching ABI mistakes at compile time.
  • __PROTECT / asmlinkage_protect is a now-mostly-vestigial barrier (a no-op on x86) historically used to stop the compiler from clobbering argument slots on the stack before the assembly stub could read them.
  • The macro ends mid-declaration: static inline long __do_sys##name(...) with no body and no semicolon, so the code the developer writes after SYSCALL_DEFINE3(...) — the { ... } block — becomes the body of __do_sys_read. That is the syntactic trick: the macro opens a function signature and the surrounding source closes it.

SYSCALL_DEFINE0 is special-cased because a zero-argument syscall needs no sign-extension shim at all (per include/linux/syscalls.h):

#define SYSCALL_DEFINE0(sname)					\
	SYSCALL_METADATA(_##sname, 0);				\
	asmlinkage long sys_##sname(void);			\
	ALLOW_ERROR_INJECTION(sys_##sname, ERRNO);		\
	asmlinkage long sys_##sname(void)

Here sys_getpid is the handler directly; there is no __se_ layer because there is nothing to sign-extend.

Why the Sign-Extension Shim Exists — and the CVE That Forced the Discipline

The __se_ layer answers a concrete security question: what is in the high bits of an argument register? On x86-64 a syscall argument that is logically a 32-bit int arrives in a 64-bit register (rdi, rsi, …). The C ABI says the upper 32 bits of that register are undefined when passing a 32-bit value — callers are allowed to leave garbage there. If a handler declared int fd and the compiler treated the 64-bit register as already holding a clean int, two things could go wrong: a comparison or array index could use the garbage upper bits, and worse, a 32-bit value that should be sign-extended (a negative int becoming a negative long) might not be, so -1 arrives as 0x00000000FFFFFFFF instead of 0xFFFFFFFFFFFFFFFF.

The canonical disaster is CVE-2007-4573 (nelhage 2010). On the x86-64 32-bit-compat entry path, the normal int 0x80 route did movl %eax,%eax to zero-extend the syscall number into the full rax, but the ptrace-traced path (ia32_tracesys) used a LOAD_ARGS macro that loaded full 64-bit values. A tracer could ptrace(PTRACE_POKEUSER) a 64-bit value into rax after the syscall-number bounds check, so the table lookup indexed far past sys_call_table[] into attacker-mapped memory — a direct, reliable root exploit. The fix (LOAD_ARGS32) switched to 32-bit moves (movl ... %r11d) that zero-extend properly. The lesson the kernel internalized: never trust the upper bits of a register that crossed the syscall boundary; explicitly normalize them. The __se_ shim is the systematic, every-syscall expression of that lesson — every argument is widened to a full long in a controlled way before the handler runs.

Uncertain

CVE-2007-4573 predates the SYSCALL_DEFINE wrapper machinery (the wrappers were reworked substantially around 2009 and again with the x86 pt_regs conversion in 2018). The historical claim here is that this class of register-width bug is what motivated the sanitizing wrapper discipline, which the in-tree header comment (“used to sanitize syscall arguments”) and the LWN syscall-wrapper coverage support. The exact causal chain from this specific CVE to the specific __se_/pt_regs design is a reconstruction, not a single cited statement. Reason: no single primary source ties them in one sentence. To resolve: read the commit messages on the x86 syscall_wrapper.h introduction (commit fa697140f9a2 and follow-ups). uncertain

The x86-64 pt_regs Wrapper: A Fourth Layer for Register Sanitization

On x86-64 (and several other arches) the kernel sets CONFIG_ARCH_HAS_SYSCALL_WRAPPER, which suppresses the generic sys_* prototypes and pulls in arch/x86/include/asm/syscall_wrapper.h. The x86 version redefines __SYSCALL_DEFINEx so the table-facing symbol takes a single const struct pt_regs * argument rather than the syscall’s logical arguments. Per arch/x86/include/asm/syscall_wrapper.h at v6.12, the macro stack is:

#define __SYSCALL_DEFINEx(x, name, ...)					\
	static long __se_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__));	\
	static inline long __do_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__));\
	__X64_SYS_STUBx(x, name, __VA_ARGS__)				\
	__IA32_SYS_STUBx(x, name, __VA_ARGS__)				\
	static long __se_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__))	\
	{								\
		long ret = __do_sys##name(__MAP(x,__SC_CAST,__VA_ARGS__));\
		__MAP(x,__SC_TEST,__VA_ARGS__);				\
		__PROTECT(x, ret,__MAP(x,__SC_ARGS,__VA_ARGS__));	\
		return ret;						\
	}								\
	static inline long __do_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__))

The __se_ shim and __do_ handler are exactly as before; the new piece is __X64_SYS_STUBx, which generates __x64_sys_##name:

#define __SYS_STUBx(abi, name, ...)					\
	long __##abi##_##name(const struct pt_regs *regs);		\
	ALLOW_ERROR_INJECTION(__##abi##_##name, ERRNO);			\
	long __##abi##_##name(const struct pt_regs *regs)		\
	{								\
		return __se_##name(__VA_ARGS__);			\
	}
 
#define SC_X86_64_REGS_TO_ARGS(x, ...)					\
	__MAP(x,__SC_ARGS						\
		,,regs->di,,regs->si,,regs->dx				\
		,,regs->r10,,regs->r8,,regs->r9)

So __x64_sys_read(const struct pt_regs *regs) calls __se_sys_read(regs->di, regs->si, regs->dx) — it manually pulls the first three argument fields out of the saved register frame and discards the rest. This is the key behavior: because the stub explicitly names only as many regs-> fields as the syscall has arguments, the unused argument registers are never propagated into the call chain. The header comment shows the generated assembly for a 4-argument syscall, including xor %r9d,%r9d and xor %r8d,%r8d to clear the unused registers, and states the rationale plainly: “This approach avoids leaking random user-provided register content down the call chain.” Before the 2018 pt_regs conversion, every sys_* was called with the actual argument registers, so a syscall that ignored its later arguments still received whatever userspace left in r8/r9 — a Spectre-era information-flow concern (see Speculation Barriers and Spectre Hardening at the Syscall Boundary). The pt_regs stub turns the register frame into a single trusted source and reads from it deliberately.

The same header generates parallel stubs for the other ABIs the kernel must serve from one SYSCALL_DEFINE: __ia32_sys_* (32-bit native or a “common” compat call, decoding EBX, ECX, EDX, ESI, EDI, EBP via the SYSCALL_PT_ARGS mapping borrowed from s390x) and, for X32, __x64_compat_sys_*. do_syscall_64() then dispatches the table entry — regs->ax = x64_sys_call(regs, unr) — passing the whole pt_regs to whichever __x64_sys_* stub the table holds (per arch/x86/entry/common.c at v6.12). The dispatch is bounds-checked with array_index_nospec() as a Spectre-v1 mitigation, the direct descendant of the CVE-2007-4573 lesson hardened against speculative execution.

SYSCALL_DEFINE0 on x86 is likewise redefined to produce a pt_regs-taking stub (__do_sys_##sname(const struct pt_regs *__unused)) so that the naming stays congruent and the COND_SYSCALL machinery — which provides weak sys_ni_syscall fallbacks for syscalls a given config does not build — keeps working uniformly.

COMPAT_SYSCALL_DEFINE: The 32-bit Translation Handlers

A 64-bit kernel running a 32-bit process cannot use the native handler directly: a 32-bit struct stat, a 32-bit time_t, a 32-bit pointer, and a 32-bit long all have different sizes and layouts than the kernel’s native versions. COMPAT_SYSCALL_DEFINE defines a separate handler, __do_compat_sys_##name, that takes the compat-ABI types and translates them. On x86 (per arch/x86/include/asm/syscall_wrapper.h):

#define COMPAT_SYSCALL_DEFINEx(x, name, ...)					\
	static long __se_compat_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__));	\
	static inline long __do_compat_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__));\
	__IA32_COMPAT_SYS_STUBx(x, name, __VA_ARGS__)				\
	__X32_COMPAT_SYS_STUBx(x, name, __VA_ARGS__)				\
	static long __se_compat_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__))	\
	{									\
		return __do_compat_sys##name(__MAP(x,__SC_DELOUSE,__VA_ARGS__));\
	}									\
	static inline long __do_compat_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__))

This generates __ia32_compat_sys_* (for int 0x80 / 32-bit native compat) and __x64_compat_sys_* (for X32), each decoding registers in the appropriate order, calling __se_compat_sys_*, which calls the author’s __do_compat_sys_*. The compat shim uses __SC_DELOUSE rather than a plain cast: it masks pointer-typed arguments to 32 bits (“de-louses” them) so a 32-bit process cannot smuggle high bits into a pointer. The compat-table for 32-bit syscalls (ia32_sys_call_table in older trees, now folded into the generated ia32_sys_call()) points at these __ia32_compat_sys_* stubs for any syscall that has a compat handler, and at the regular __ia32_sys_* stub otherwise. Deeper coverage of the translation work lives in The Compat Syscall Layer for 32-bit Binaries; this note covers only how COMPAT_SYSCALL_DEFINE expands.

How the Handler Gets Its Arguments — End to End

Putting the layers in motion for read(2) on x86-64:

  1. Userspace sets rax = 0 (__NR_read), rdi = fd, rsi = buf, rdx = count, executes syscall. The CPU traps; the entry assembly saves all registers into a pt_regs frame on the kernel stack (see Per-Architecture Syscall Entry Assembly and The pt_regs Register Frame).
  2. do_syscall_64(regs, nr) runs the generic entry work (seccomp/ptrace/audit — see The Generic Syscall Entry and Exit Layer), then calls x64_sys_call(regs, 0), which indexes the table and invokes __x64_sys_read(regs).
  3. __x64_sys_read reads regs->di, regs->si, regs->dx, and calls __se_sys_read((long)fd, (long)buf, (long)count) — three full longs.
  4. __se_sys_read casts each back to the declared type — (unsigned int)fd, (char __user *)buf, (size_t)count — and calls __do_sys_read.
  5. __do_sys_read is the body the author wrote: it validates and copy_to_users the data (see copy_to_user and copy_from_user), returns the byte count as a long.
  6. The return value propagates back out as regs->ax, which the exit path leaves in place for sysret to deliver to userspace (see Returning to Userspace and exit_to_user_mode).

The author wrote one function. The macro built the other three so that the messy realities of register layout, argument width, and ABI variation never reach that one function.

Common Misunderstandings

  • “The syscall table points at sys_read.” On any arch with CONFIG_ARCH_HAS_SYSCALL_WRAPPER (x86-64, arm64, s390, …) the table points at the pt_regs stub — __x64_sys_read — not at sys_read. The bare sys_* symbols may not even exist as call targets. Tools that grep the table for sys_ names will be confused on modern x86-64.
  • “asmlinkage matters for performance.” It is about calling convention, not speed: it forces stack-based argument passing to match the assembly entry stub’s expectations. With the pt_regs wrapper the table-facing stub takes a single pointer anyway, so the classic asmlinkage concern largely evaporates on x86-64.
  • “Six arguments is a C limitation.” No — it is the ABI’s argument-register count. The kernel could in principle pass more on the stack, but it deliberately caps syscalls at six register arguments (SYSCALL_DEFINE_MAXARGS) for uniformity and to avoid stack-marshalling in the hot path; structs-by-pointer handle the rest.
  • “__do_sys_* is static inline, so it is inlined into the stub.” It is declared static inline, but the __se_/__x64_ layers take its address and call through it; the compiler usually does inline __do_sys_* into __se_sys_* and sometimes the whole chain into the stub. The inline keyword is a hint plus a way to avoid “defined but unused” warnings, not a guarantee.

Alternatives and History

Before the modern macros (pre-2009), syscalls were often declared as plain asmlinkage long sys_foo(...) by hand, which made the sign-extension and register-leak problems each author’s responsibility — and they got them wrong, repeatedly. The SYSCALL_DEFINE family centralized the discipline; the LWN coverage of the syscall-wrapper work and its follow-up document the motivation. The 2018 x86 pt_regs conversion (David Sterba / Dominik Brodowski) was the next step, driven by Spectre-era register-leak concerns. The trajectory is consistent: every revision pushes more of the “don’t trust userspace’s registers” burden out of human hands and into a macro that gets it right every time.

See Also

Graph View

Backlinks