vault

System Calls and the Userspace Kernel Boundary

13 min read

System Calls and the Userspace Kernel Boundary

A system call is the single controlled doorway between unprivileged user space (ring 3) and the privileged kernel (ring 0) — “the fundamental interface between an application and the Linux kernel” (syscalls(2)). Because the hardware forbids a ring-3 program from executing privileged instructions or touching kernel memory directly (see User Mode and Kernel Mode), a process that needs the kernel to do something for it — open a file, map memory, send a packet, fork — cannot simply call a kernel function. It must instead trap: load a syscall number and a few register arguments, execute one special instruction, and let the CPU jump it into a kernel-chosen entry point while raising the privilege level atomically. This boundary is, at once, the system’s stability contract (the kernel promises never to break the syscall interface), its security perimeter (every privileged request is mediated and policy-checked here), and its observability tap (instrument the boundary and you see everything a program asks the kernel for). This note is the conceptual frame for the entire syscall subsystem: why the boundary must be a trap and not a call, what crosses it in each direction, and why the kernel can never trust what arrives.

Mental Model

Picture the kernel behind a wall with exactly one door. The wall is the hardware privilege boundary from User Mode and Kernel Mode: kernel code and data sit in supervisor-protected memory at CPL 0, and the application lives outside at CPL 3. There is no window — a program cannot peek at kernel memory — and no side entrance — a program cannot jmp into kernel code, because the destination pages are supervisor-only and would page-fault, and even if they didn’t, the program would still be at CPL 3 and the first privileged instruction would #GP. The only way in is the door, and the door has three peculiar properties: (1) it opens only when you knock with the special syscall instruction; (2) the moment you step through, you are not where you pointed — you land at a fixed spot the kernel chose in advance; and (3) a guard inspects everything you carry. Those three properties are what make a system call a controlled transfer rather than an arbitrary jump.

flowchart LR
  subgraph US["Userspace — ring 3 (CPL=3)"]
    APP["application"]
    WRAP["libc wrapper<br/>sets rax=nr,<br/>rdi/rsi/rdx/r10/r8/r9=args"]
    APP --> WRAP
  end
  WRAP -->|"syscall instruction (knock)"| DOOR{{"the boundary"}}
  DOOR -->|"BLOCKED: direct jmp/call<br/>to kernel addr faults<br/>(page fault or GP fault)"| X["fault"]
  DOOR -->|"TRAP: CPU raises CPL 3 to 0,<br/>jumps to MSR_LSTAR entry"| ENTRY["kernel entry<br/>(kernel chose this address)"]
  subgraph KS["Kernel — ring 0 (CPL=0)"]
    ENTRY --> CHK["validate: number in range,<br/>user pointers via access_ok,<br/>credentials / seccomp"]
    CHK --> WORK["do the privileged work"]
    WORK --> RET["return rax = value or -errno"]
  end
  RET -->|"sysret: CPL 0 to 3"| WRAP

The system-call boundary as a one-door wall. What it shows: a direct jump into the kernel is impossible (it faults), so the program must knock with syscall; the trap is what atomically raises privilege and — critically — lands execution at a kernel-defined address, not wherever the program pointed. Everything crossing inward is then validated before the kernel acts. The insight to take: “controlled” means the kernel, not the caller, decides where a privileged crossing goes and what is trusted — userspace supplies a number and arguments, never an address.

Why a Process Must Trap, Not Call

The most common misconception is that a syscall is “just a function call into the kernel.” It cannot be, for two independent hardware reasons, and understanding both is the crux of this note.

Reason one: kernel pages are unreachable from ring 3. The kernel’s code and data live in the upper half of the virtual address space (on x86-64, from 0xffff800000000000 up, Documentation/arch/x86/x86_64/mm.rst, v6.12), in pages whose page-table entries carry the supervisor (U/S) bit. As covered in User Mode and Kernel Mode, the Memory Management Unit faults any ring-3 access to such a page. So a literal call 0xffffffff81234567 from user code does not transfer control into the kernel — it raises a page fault before fetching the first instruction, and the program receives SIGSEGV. You cannot call code you cannot even read.

Reason two: privilege does not follow a jump. Suppose, hypothetically, the kernel’s pages were readable from ring 3. A call or jmp into them would still execute at CPL 3, because the privilege level lives in the CS selector and an ordinary control transfer within the same segment does not change it. The very first privileged instruction the kernel code tried — say a MOV to CR3 to switch page tables, or a CLI to mask interrupts — would raise #GP(0) (“This instruction can be executed only when the current privilege level is 0”, felixcloutier, MOV CRn). Privileged work requires CPL 0, and nothing a ring-3 jump can do raises CPL.

The syscall instruction is engineered to solve exactly these two problems at once. It is a deliberate hardware trap: per the architecture reference, SYSCALL “invokes an OS system-call handler at privilege level 0 … by loading RIP from the IA32_LSTAR MSR” and sets CPL := 0 (felixcloutier, SYSCALL) — so it atomically raises the CPL from 3 to 0 and loads RIP from a model-specific register the kernel wrote at boot. So the program does not, and cannot, choose where it lands; it lands at entry_SYSCALL_64, the kernel’s gatekeeper (entry_64.S, v6.12). This “you become privileged only by jumping where the kernel told you to” coupling is what makes the crossing controlled: there is exactly one inbound address per architecture, fully under kernel control, with a single well-defined dispatch point behind it. The detailed mechanics of that instruction — the GS swap, the stack switch, the pt_regs save — belong to The Syscall Instruction and Trap Mechanism; here the point is simply why the trap is necessary at all.

What Crosses the Boundary

The boundary is narrow on purpose. Only a few, well-typed things cross it, and the kernel treats everything inbound as hostile until proven otherwise.

Inbound, in registers: a number and up to six arguments. Userspace places the syscall number in rax and arguments in the fixed sequence rdi, rsi, rdx, r10, r8, r9 (syscall(2)). The number selects which service; it is sign-extended from 32 bits and dispatched through the kernel’s syscall table after a bounds check (if (likely(unr < NR_syscalls)), arch/x86/entry/common.c, v6.12), with array_index_nospec() to prevent the bounds check from being speculatively bypassed (a Spectre-v1 hardening). An out-of-range number is not a crash — it dispatches to sys_ni_syscall, returning -ENOSYS. The number→handler mapping is the subject of The System Call Table and System Call Numbers and the ABI; the register convention is The x86-64 Syscall Calling Convention.

Inbound, the dangerous part: user pointers. Many arguments are not values but pointers into user memory — the buffer for read(), the path string for open(), the struct stat for stat(). The kernel runs at CPL 0, where the supervisor bit does not stop it, so a naive dereference of a user-supplied pointer is a catastrophe waiting to happen: a malicious program could pass a kernel address and trick the privileged kernel into reading or writing kernel memory on its behalf, defeating the entire isolation model. The kernel therefore never dereferences a user pointer directly. It first confirms the address lies in the user range — access_ok(), which on most architectures reduces to addr <= TASK_SIZE_MAX and an overflow check (asm-generic/access_ok.h, v6.12) — and then copies the data with fault-tolerant accessors (copy_from_user/copy_to_user) that return an error rather than oopsing on an unmapped or bad address. This discipline is the entire subject of The access_ok Check and User Pointer Validation and copy_to_user and copy_from_user; the conceptual takeaway here is that the boundary’s inbound traffic is untrusted by construction, and validating it is the price of running at CPL 0 on behalf of CPL 3.

Outbound: a single return value. On the way back, the kernel places the result in rax. The convention is that a return in the small negative range [-4095, -1] is an error code; the libc wrapper detects it, negates it into errno, and returns -1 to the caller (syscalls(2); the detail lives in The errno Convention and Negative Return Values). Any data the syscall produced was written back into user memory through copy_to_user, not returned in registers. The transition home is a sysret/iret that lowers CPL from 0 to 3.

What does not cross. The kernel’s stack, its internal pointers, and its memory layout never cross outward; userspace never sees kernel addresses (and KASLR plus information-leak hardening work to keep it that way). And userspace never supplies an address to jump to inside the kernel — only a number. This asymmetry is deliberate: the more the boundary hides, the smaller the attack surface.

The Boundary as Three Contracts

Why is this one interface treated as sacred? Because three different system-wide guarantees all key off it.

The stability contract. The set of syscall numbers and their argument conventions is an append-only, frozen ABI. New functionality arrives as new syscalls or new flags, never as changed semantics for an existing number — which is why a binary compiled decades ago still runs on a current kernel. This is Linus Torvalds’s “we do not break userspace” rule, the cultural and engineering anchor of the whole interface; its depth lives in The We Do Not Break Userspace ABI Promise. The practical upshot: because the boundary is the only contract userspace depends on, freezing it is enough to keep userspace working across kernel evolution.

The security perimeter. Every privileged operation funnels through this one door, so it is the natural and only place to enforce policy. Inside each handler, at CPL 0, the kernel checks the calling process’s credentials(7) — its effective UID/GID and capabilities, “used by the kernel to determine the permissions that the process will have when accessing shared resources.” Before the handler even runs, optional gatekeepers can intercept the call: seccomp runs a BPF program on every syscall to allow/deny/kill/trap it (the foundation of container sandboxing, see seccomp and Syscall Filtering), and ptrace lets a tracer stop the process at syscall entry/exit (see ptrace and Syscall Tracing). The generic entry layer wires these in: syscall_enter_from_user_mode() does “ptrace, seccomp, audit, and syscall tracing work” before dispatch (core-api/entry docs). Because there is exactly one perimeter, confining a process by the syscalls it may make is a complete sandbox.

The observability tap. For the same reason — one door, all traffic — the boundary is where you watch a process. strace attaches via ptrace and prints every syscall and its arguments; syscall tracepoints (sys_enter/sys_exit) feed perf and eBPF tools. Instrument the boundary once and you have a complete record of what a program asked the kernel for, without touching the program. This is why “what syscalls is it making?” is the first question in so much production debugging (see How strace Works and Linux Tracing and Observability MOC).

A Worked Crossing — open() Front to Back

Tie it together with a concrete call. An application calls open("/etc/hosts", O_RDONLY):

  1. Userspace marshals. The glibc open wrapper places the syscall number for openat in rax, the directory-fd and the pointer to the path string and the flags in rdi, rsi, rdx, and executes syscall (syscall(2)). The path string itself stays in user memory; only its address goes in a register.
  2. The trap. The CPU raises CPL 3→0 and jumps to entry_SYSCALL_64. The number is sign-extended and bounds-checked; dispatch reaches the openat handler (common.c, v6.12).
  3. Validation. Before the handler can read the path, it must pull the string across the boundary safely — strncpy_from_user (built on the access_ok discipline) confirms the pointer is in the user range and copies the bytes, returning -EFAULT rather than faulting if the program lied about the address.
  4. Policy. The handler checks the process credentials against the file’s permissions, plus any seccomp filter that may already have allowed the call to reach here.
  5. Work and return. The kernel resolves the path, allocates a file descriptor, and returns its number in rax — or a negative error (-ENOENT, -EACCES). sysret drops back to CPL 3; the libc wrapper turns a negative result into errno + -1, or hands the fd back to the application.

Every step is a manifestation of one principle: the kernel does the privileged work for the program, on a request it does not trust, and never lets the program reach in directly.

Common Misunderstandings

  • “A syscall is a function call into the kernel.” It is a hardware trap. A literal call into kernel addresses faults (kernel pages are supervisor-only) and, even if it didn’t, would run at CPL 3 and #GP on the first privileged instruction. The trap is what raises privilege and redirects to a kernel-chosen entry — see The Syscall Instruction and Trap Mechanism.
  • “The kernel reads my buffer like any pointer.” No — at CPL 0 the supervisor bit doesn’t protect the kernel from a hostile pointer, so it validates via access_ok and copies with copy_*_user. A raw dereference of a user pointer in kernel code is a security bug.
  • “Userspace tells the kernel where to jump.” Userspace supplies a number, never an address. The entry address is fixed in MSR_LSTAR by the kernel; this is the whole reason the crossing is controlled.
  • “Calling a libc function like read() is the syscall.” read() is a thin userspace wrapper that marshals registers and executes the trap, then translates the result to errno. The wrapper is not the boundary; the syscall instruction inside it is. See libc Syscall Wrappers and errno Translation.

See Also

Graph View

Backlinks