vault

libc Syscall Wrappers and errno Translation

13 min read

libc Syscall Wrappers and errno Translation

When a C program calls open("/nope", O_RDONLY), it is not executing a syscall instruction directly — it calls a C-library wrapper function, a thin stub that loads the syscall number and arguments into the architecture’s registers, executes the syscall instruction, and then translates the kernel’s raw result into the POSIX convention every C programmer expects. The translation is mechanical: the kernel returns a small negative number in RAX (see The errno Convention and Negative Return Values); the wrapper tests whether that value falls in the reserved band [-4095, -1], and if so it negates the value into the thread-local errno variable and returns -1 to the caller; otherwise it returns the value unchanged. This note traces that wrapper, the exact range check (cmpq $-4095, %rax in assembly, (unsigned long)val > -4096UL in C), the per-thread storage of errno via thread-local storage (TLS) and __errno_location, and the cases — like the vDSO — where there is no trap and no errno touch at all.

Mental Model

There are three distinct layers between your open() call and the kernel, and conflating them is the source of most confusion. At the top is the POSIX API: “on error, return -1 and set the global errno.” In the middle is the libc wrapper, the only code that knows both worlds — it speaks the raw kernel protocol below and presents the POSIX convention above. At the bottom is the kernel ABI: “result or -errno in RAX, that’s it.” The wrapper’s entire job is impedance matching between the kernel’s single-register negative-return scheme and the C library’s two-channel “return value plus errno” scheme. errno is invented and owned entirely by libc; the kernel has never heard of it.

flowchart TB
  APP["Application:<br/>fd = open(path, flags)<br/>if (fd == -1) perror(...)"] --> WRAP
  subgraph WRAP["libc wrapper (open)"]
    LOAD["load nr into RAX,<br/>args into RDI/RSI/RDX/R10/R8/R9"]
    INSN["syscall instruction"]
    CHECK{"RAX in [-4095, -1]?<br/>(cmpq $-4095,%rax; jae)"}
    SETE["errno = -RAX<br/>(write via %fs: TLS)<br/>return -1"]
    PASS["return RAX unchanged"]
    LOAD --> INSN --> CHECK
    CHECK -->|"yes (error)"| SETE
    CHECK -->|"no (success)"| PASS
  end
  INSN -.->|trap| K["kernel handler<br/>returns -ENOENT (= -2)"]
  K -.->|"RAX = -2"| CHECK
  SETE --> APP
  PASS --> APP

The wrapper sits between the POSIX-shaped caller and the register-shaped kernel. What it shows: the kernel returns -2 (-ENOENT) in RAX; the wrapper’s range test catches it, negates it to errno = 2, and returns -1; on success the raw value passes through untouched. The insight to take: errno is set by libc, in userspace, after the trap returns — never by the kernel. The negation (errno = -RAX) and the -1 return are the two halves of the translation that turn the kernel’s one-register protocol into POSIX’s two-channel one.

The Range Check — Where libc Decides “Error”

The boundary that defines an error is the same MAX_ERRNO = 4095 reservation the kernel uses (walked symbol-by-symbol in The errno Convention and Negative Return Values); libc just phrases the test for its own side. In glibc’s hand-written x86-64 assembly stubs, sysdeps/unix/sysv/linux/x86_64/sysdep.h defines the PSEUDO macro that builds a syscall entry point:

#  define PSEUDO(name, syscall_name, args) \
  .text;                                                              \
  ENTRY (name)                                                        \
    DO_CALL (syscall_name, args, 0, 0);   /* movl $__NR_xxx,%eax; syscall */ \
    cmpq $-4095, %rax;                     /* is RAX >= -4095 (unsigned)? */ \
    jae SYSCALL_ERROR_LABEL                /* if so, jump to error handler */

DO_CALL expands to loading the syscall number into %eax and executing syscall. The two lines that matter are cmpq $-4095, %rax followed by jae (jump if above-or-equal). jae is an unsigned conditional jump, so the comparison treats %rax as unsigned: it branches to the error handler exactly when (unsigned long)rax >= (unsigned long)-4095, i.e. when rax is one of -4095 .. -1 as a signed value. This is the same band as the kernel’s IS_ERR_VALUE. The glibc source even carries the historical comment explaining the magic number: “Linus said he will make sure the no syscall returns a value in -1 .. -4095 as a valid result so we can safely test with -4095.”

The C-language path (used by the inline INLINE_SYSCALL macro for most syscalls) phrases the identical test arithmetically. From sysdeps/unix/sysv/linux/sysdep.h:

#define INTERNAL_SYSCALL_ERROR_P(val) \
  ((unsigned long int) (val) > -4096UL)
 
#define INLINE_SYSCALL(name, nr, args...)                       \
  ({                                                            \
    long int sc_ret = INTERNAL_SYSCALL (name, nr, args);        \
    __glibc_unlikely (INTERNAL_SYSCALL_ERROR_P (sc_ret))        \
    ? SYSCALL_ERROR_LABEL (INTERNAL_SYSCALL_ERRNO (sc_ret))     \
    : sc_ret;                                                   \
  })
 
#define INTERNAL_SYSCALL_ERRNO(val)  (-(val))

INTERNAL_SYSCALL_ERROR_P(val) is (unsigned long)val > -4096UL. Note the > against -4096 rather than >= against -4095 — these are the same set: “strictly greater than 2⁶⁴ − 4096” is “at least 2⁶⁴ − 4095,” which is the band [-4095, -1]. INLINE_SYSCALL runs the raw INTERNAL_SYSCALL (the inline syscall instruction), tests the result with the __glibc_unlikely branch hint, and on error calls SYSCALL_ERROR_LABEL with INTERNAL_SYSCALL_ERRNO(sc_ret) = -(sc_ret) — the negation that turns -2 into 2. On success it yields sc_ret unchanged. musl is even terser; src/internal/syscall_ret.c:

long __syscall_ret(unsigned long r)
{
	if (r > -4096UL) {        // same band test
		errno = -r;       // negate into errno
		return -1;        // POSIX failure return
	}
	return r;                 // success: pass through
}

Every libc-wrapped syscall in musl runs its raw result through __syscall_ret; the syscall() macro itself is literally #define syscall(...) __syscall_ret(__syscall(__VA_ARGS__)) (src/internal/syscall.h). The two libcs differ in style but implement byte-for-byte the same protocol.

Setting errno — and Why It Is Per-Thread

Once the wrapper knows it has an error, it writes the negated value into errno. The subtlety is that errno is thread-local: setting it in one thread must not be visible in another (errno(3): “errno is thread-local; setting it in one thread does not affect its value in any other thread”). If errno were a single global, two threads each issuing a failing syscall would race and clobber each other’s error indication. The fix is thread-local storage (TLS): each thread has its own errno slot, reached through the thread pointer.

In glibc’s x86-64 assembly error handler, from the same sysdep.h, the write goes directly through the %fs segment register, which on x86-64 points at the thread control block:

#  define SYSCALL_SET_ERRNO                     \
  movq SYSCALL_ERROR_ERRNO@GOTTPOFF(%rip), %rcx;\  /* TLS offset of errno */
  neg %eax;                                     \  /* errno = -RAX */
  movl %eax, %fs:(%rcx);                           /* store into this thread's slot */
 
#  define SYSCALL_ERROR_HANDLER \
0:                              \
  SYSCALL_SET_ERRNO;            \
  or $-1, %RAX_LP;              \  /* RAX = -1 : POSIX failure return */
  ret;

SYSCALL_ERROR_ERRNO@GOTTPOFF(%rip) resolves to the TLS offset of the errno (actually __libc_errno) variable; neg %eax negates the kernel’s -errno to a positive errno; movl %eax, %fs:(%rcx) stores it at that offset relative to %fs, i.e. in the calling thread’s own TLS block. Then or $-1, %RAX_LP forces the return value to -1 and the stub returns. (There is also an RTLD_PRIVATE_ERRNO variant the dynamic linker uses before TLS is set up, writing to a private rtld_errno.) musl takes a different but equivalent route — it stores errno inside the thread’s pthread structure, reached via the thread pointer. From src/errno/__errno_location.c:

int *__errno_location(void)
{
	return &__pthread_self()->errno_val;
}

__pthread_self() returns a pointer to the current thread’s control structure (via the thread pointer register), and errno_val is a field within it; so each thread’s errno lives in its own thread struct. The C-visible errno is a macro that dereferences this function — from musl’s include/errno.h: #define errno (*__errno_location()). glibc uses the identical __errno_location() indirection (the standard requires it, so errno cannot be a plain global), backed by its TLS variable. This is why portable code must #include <errno.h> and use the errno macro rather than declaring extern int errno — the macro hides a function call that finds the right per-thread slot.

Mechanical Walk-through — open() on a Missing File

Trace the whole pipeline with one concrete call: int fd = open("/does/not/exist", O_RDONLY); followed by if (fd == -1) perror("open");.

  1. The application calls glibc’s open wrapper. The wrapper (built from the PSEUDO/INLINE_SYSCALL machinery above) loads __NR_openat into RAX and the arguments into RDI, RSI, RDX, R10 per the x86-64 calling convention, then executes syscall.
  2. The CPU traps to ring 0; the kernel’s openat handler tries to resolve the path, fails at the missing component, and returns -ENOENT (signed -2). Via do_syscall_64, regs->ax = -2 (see The errno Convention and Negative Return Values).
  3. sysret returns to userspace with RAX = -2. The wrapper runs its range check: cmpq $-4095, %rax — is -2 in [-4095, -1]? Yes. jae branches to SYSCALL_ERROR_HANDLER.
  4. The handler computes neg %eax → 2, stores 2 into this thread’s errno slot via %fs:, sets RAX = -1, and returns.
  5. The application sees fd == -1, enters the if, and calls perror("open"). perror reads errno (= 2), maps it through strerror to the message string, and prints open: No such file or directory. The mapping from 2 to that string comes from the same name table seen in the kernel’s errno-base.h (ENOENT 2).

The whole translation — negate, set TLS errno, return -1 — happened entirely in userspace, after the kernel had already finished. The kernel’s only contribution was the -2 in RAX.

The Raw syscall() Wrapper and the Inline Stubs

Not every syscall has a hand-written stub. glibc and musl provide a generic path, the [[The syscall() Generic Wrapper Function|syscall() function]], for calling any syscall by number — syscall(SYS_openat, AT_FDCWD, path, flags, mode). Per syscall(2), syscall() “saves CPU registers before making the system call, restores the registers upon return from the system call, and stores any error returned by the system call in errno” — it runs the same range-check-and-set-errno logic as a dedicated wrapper. The inline INTERNAL_SYSCALL macros shown earlier are how glibc emits the syscall instruction inline for the common arities; each internal_syscallN pins arguments to the right registers with GCC’s register ... asm("rdi") constructs and lists "memory", "cc", "r11", "cx" as clobbers (the kernel destroys RCX and R11 across syscall). The difference between a dedicated wrapper and syscall() is mostly that the dedicated one knows the exact argument types and any compat fix-ups, whereas syscall() passes raw longs and the caller must handle architecture quirks itself (syscall(2) gives the ARM-EABI readahead example).

The vDSO Shortcut — No Trap, No errno

A few “syscalls” never reach this wrapper logic at all. For hot, side-effect-free reads like clock_gettime, libc first tries a function exported by the vDSO — a small shared object the kernel maps into every process. __vdso_clock_gettime reads kernel-maintained timekeeping data straight from a shared page and returns without executing a syscall instruction and without trapping (vdso(7)). On the vDSO fast path there is no kernel transition, so there is no RAX negative-return to check and errno is not touched on success; libc only falls back to the real trapping syscall (and its errno translation) when the vDSO path declines. The mechanism of how the vDSO reads time without trapping is out of scope here — see How the vDSO Reads Kernel Time Without Trapping and Raw Syscalls vs vsyscall vs vDSO Performance. The point for this note is only that the errno-translation wrapper is bypassed entirely on that path.

Failure Modes and Common Misunderstandings

The classic bug is reading errno when the call succeeded. The C standard and errno(3) are explicit: a successful call may leave errno nonzero (a function is allowed to clobber it internally), and errno is never reset to zero by a successful call. So errno is meaningful only after a call has signalled failure through its return value. Code that does open(...); if (errno) ... is wrong; it must test the return (fd == -1) first.

A second trap is the -1 is a valid result family — getpriority(2), some ioctls — where -1 can be a legitimate success. The portable idiom (errno(3)) is to set errno = 0 before the call and then test errno afterward, since the return value alone cannot disambiguate. A third is errno clobbering across library calls: any intervening libc call (even printf) may set errno, so the value must be saved immediately if it is needed later — int saved = errno; right after the failing call.

A subtler issue is thread-safety assumptions in old code. Pre-threads Unix declared extern int errno; as a global. On modern Linux that declaration is wrong and will not even compile against <errno.h> (which #defines errno to a function call). Code ported from ancient sources sometimes still tries it; the fix is always to use the header’s macro. Finally, on the raw-instruction path (Go, custom runtimes, hand-written assembly that does not go through libc), errno is not set at all — the negative return is the only error signal, and the runtime must do its own band check. The Go runtime does exactly this; see System Calls and the Scheduler.

Alternatives and When They Apply

Most application code should use the dedicated wrappers (open, read, write) — they handle argument types, compat translation, and cancellation points correctly. Reach for the generic [[The syscall() Generic Wrapper Function|syscall()]] only when a syscall has no wrapper — a brand-new syscall whose glibc stub has not shipped yet, or one glibc historically declined to wrap until a later release (gettid and getrandom are commonly cited examples of syscalls that for years had to be called via syscall()) — accepting that you handle architecture-specific argument marshalling yourself. Bypassing libc entirely (issuing the syscall instruction directly) is what Go and some sandboxes/static binaries do; it avoids libc’s TLS setup and cancellation machinery but means you forfeit errno translation, vDSO acceleration unless you replicate it, and any compat fix-ups — you are then implementing the kernel ABI by hand. The trade-off is control and minimal dependencies versus reimplementing correctness that libc already got right.

See Also

Graph View

Backlinks