vault

LSM Hooks and the security_ Call Sites

14 min read

LSM Hooks and the security_ Call Sites

The visible surface of the LSM framework is a few hundred functions named security_*() — security_inode_permission(), security_bprm_check(), security_file_open(), security_socket_create(), security_task_kill() — sprinkled through the kernel at every point where it is about to do something a security policy might want to veto. Each is a thin wrapper: the kernel proper calls security_file_open(file) from its file-open path, and the wrapper fans the call out to every registered security module’s file_open callback, aggregates their verdicts, and returns a single int (zero to allow, a negative errno to deny). At Linux 6.12 this fan-out is implemented with static calls — a per-hook unrolled loop over a table of static_call() trampolines that the framework patches at boot — which replaced the older linked-list (security_hook_heads / hlist) traversal to cut the indirect-call cost on a path that runs on essentially every syscall. The entire hook catalogue is declared once, in include/linux/lsm_hook_defs.h, via a single macro LSM_HOOK(...) that is #included repeatedly with different definitions to generate the prototypes, the dispatch tables, and the per-hook defaults from one source of truth.

This note is the mechanical companion to The Linux Security Module Framework (which covers history, registration, and security blobs). Here the focus is narrow and concrete: what a security_*() call site looks like, how the LSM_HOOK macro generates everything from one list, how the static-call dispatch reaches each module in CONFIG_LSM order, and how return values aggregate so the first module to deny wins. All code is quoted from the v6.12 LTS tree (released 2024-11-17).

Uncertain

Verify: the static-call dispatch (replacing the security_hook_heads hlist) landed in a specific kernel version — commonly attributed to Linux 6.0 via KP Singh’s “Reduce overhead of LSMs with static calls” series. Reason: the v6.12 source unambiguously uses static calls (verified directly), but the exact introduction release was not pinned to a primary commit during this research — gh search commits returned no match and a candidate commit URL 404’d. To resolve: locate the merge commit (e.g. git log --oneline -- security/security.c around v6.0-rc1, or the patch series on the linux-security-module list) and confirm the release. The v6.12 mechanism below is verified; only the introduction version is uncertain. uncertain

Mental Model — One List, Three Faces

The crucial trick to internalise is that there is exactly one authoritative list of LSM hooks, and it lives in include/linux/lsm_hook_defs.h. Each line is a LSM_HOOK(RET, DEFAULT, NAME, args...) invocation. That file is never compiled directly; instead, other headers #define LSM_HOOK(...) to mean different things and then #include the list, so the same ~250 lines generate the function-pointer types, the dispatch tables, the trampoline declarations, and the call wrappers — all guaranteed consistent because they come from one source. This is the classic C “X-macro” pattern.

flowchart TB
  DEFS["lsm_hook_defs.h<br/>LSM_HOOK(int, 0, file_open, struct file *file)<br/>...one line per hook..."]
  DEFS -->|"#define LSM_HOOK as a fn-pointer field"| UNION["union security_list_options<br/>(int (*file_open)(struct file *), ...)"]
  DEFS -->|"#define LSM_HOOK as a static-call decl"| SC["per-hook static_call + static_key<br/>(LSM_STATIC_CALL, SECURITY_HOOK_ACTIVE_KEY)"]
  DEFS -->|"#define LSM_HOOK as a table field"| TAB["struct lsm_static_calls_table<br/>scall[MAX_LSM_COUNT] per hook"]
  UNION --> HL["struct security_hook_list<br/>(what each module registers)"]
  SC --> DISP["call_int_hook(file_open, file)<br/>in security_file_open()"]
  TAB --> DISP
  DISP -->|"unrolled loop, CONFIG_LSM order"| MODS["each enabled module's callback"]

One declaration list, multiple expansions. What it shows: lsm_hook_defs.h is #included several times with different LSM_HOOK definitions to emit the callback-pointer union, the per-hook static-call/key pair, and the dispatch table — then security_file_open() ties them together via call_int_hook(file_open, …). The insight: you never hand-maintain parallel lists of hook types, tables, and prototypes; changing one line in lsm_hook_defs.h ripples consistently through all of them, which is why the framework can carry hundreds of hooks without drift.

The Hook Declaration — LSM_HOOK(RET, DEFAULT, NAME, args...)

Every hook is one line in include/linux/lsm_hook_defs.h. The file’s own header states the convention: “The macro LSM_HOOK is used to define the data structures required by the LSM framework using the pattern: LSM_HOOK(<return_type>, <default_value>, <hook_name>, args...).” Representative lines, verbatim from v6.12:

LSM_HOOK(int, 0, capable, const struct cred *cred, struct user_namespace *ns,
	 int cap, unsigned int opts)
LSM_HOOK(int, 0, bprm_check_security, struct linux_binprm *bprm)
LSM_HOOK(int, 0, inode_permission, struct inode *inode, int mask)
LSM_HOOK(int, 0, inode_create, struct inode *dir, struct dentry *dentry, umode_t mode)
LSM_HOOK(int, 0, file_permission, struct file *file, int mask)
LSM_HOOK(int, 0, file_open, struct file *file)
LSM_HOOK(int, 0, mmap_file, struct file *file, unsigned long reqprot,
	 unsigned long prot, unsigned long flags)
LSM_HOOK(int, 0, task_kill, struct task_struct *p, struct kernel_siginfo *info,
	 int sig, const struct cred *cred)
LSM_HOOK(int, 0, ptrace_access_check, struct task_struct *child, unsigned int mode)
LSM_HOOK(int, 0, socket_create, int family, int type, int protocol, int kern)
LSM_HOOK(int, 0, socket_connect, struct socket *sock, struct sockaddr *address, int addrlen)
LSM_HOOK(int, 0, sb_mount, const char *dev_name, const struct path *path,
	 const char *type, unsigned long flags, void *data)
LSM_HOOK(int, 0, bpf, int cmd, union bpf_attr *attr, unsigned int size)

Reading one closely: LSM_HOOK(int, 0, inode_permission, struct inode *inode, int mask) declares a hook named inode_permission whose callbacks return int, whose default return (when no module overrides it) is 0, and whose argument list is (struct inode *inode, int mask). The default-0 convention means fail-open in the absence of policy: with no module objecting, the framework returns 0 and the kernel proceeds — consistent with “LSM can only restrict.” The default is not always 0, though. Some hooks default to a negative errno so that a default means “deny / not supported”: inode_getsecurity defaults to -EOPNOTSUPP, task_prctl to -ENOSYS, getprocattr to -EINVAL, and a few defaults are 1 (affirmative) like xfrm_state_pol_flow_match. Void hooks — cleanup/notification callbacks such as bprm_committed_creds, file_release, inode_free_security — use LSM_RET_VOID and have no verdict to aggregate; they exist so a module can react to an event, not gate it.

How one line becomes three data structures

The same file is read multiple times. As a callback-pointer union (include/linux/lsm_hooks.h), each line becomes a function-pointer field:

union security_list_options {
	#define LSM_HOOK(RET, DEFAULT, NAME, ...) RET (*NAME)(__VA_ARGS__);
	#include "lsm_hook_defs.h"
	#undef LSM_HOOK
	void *lsm_func_addr;
};

so the union has a member int (*inode_permission)(struct inode *, int) among hundreds of others. A module’s registration entry pairs that pointer with bookkeeping:

struct security_hook_list {
	struct lsm_static_call *scalls;
	union security_list_options hook;
	const struct lsm_id *lsmid;
} __randomize_layout;

When SELinux registers, it supplies a security_hook_list array — one entry per hook it implements — each with hook.inode_permission = selinux_inode_permission, lsmid identifying SELinux, and scalls pointing into the framework’s static-call table slots reserved for that hook. The framework copies these pointers into the table during initialize_lsm().

The Dispatch Table — Static Calls, Not a Linked List

The table all those pointers land in is generated, again, from the same hook list. From include/linux/lsm_hooks.h:

struct lsm_static_calls_table {
	#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
		struct lsm_static_call NAME[MAX_LSM_COUNT];
	#include <linux/lsm_hook_defs.h>
	#undef LSM_HOOK
} __packed __randomize_layout;
 
struct lsm_static_call {
	struct static_call_key *key;
	void *trampoline;
	struct security_hook_list *hl;
	struct static_key_false *active;
} __randomize_layout;

So for every hook there is an array of MAX_LSM_COUNT lsm_static_call slots — one per LSM that could possibly be enabled. MAX_LSM_COUNT is not a fixed number: from include/linux/lsm_count.h it is computed at build time as the count of enabled CONFIG_SECURITY_* options (capability, SELinux, Smack, AppArmor, TOMOYO, Yama, LoadPin, lockdown, SafeSetID, BPF-LSM, Landlock, IMA, EVM, IPE — up to 14 at v6.12), via a macro trick where each enabled config expands to 1, and the commas are counted. A kernel that builds only SELinux + capability has a MAX_LSM_COUNT of 2 and tiny tables; a distro kernel enabling everything has the full set. This per-build sizing is what lets the dispatch loop be unrolled (see below) without wasting slots.

Each slot holds a static_call_key *key and a trampoline. A static call is the kernel’s mechanism for an indirect call whose target is patched directly into the call instruction at runtime — so once the framework writes a module’s callback address into a slot, invoking it costs a direct call, not an indirect jump through a function pointer. The active field is a static_key_false (a code-patched branch): until a module populates a slot, that slot’s branch is patched out entirely, so empty slots cost nothing. The global instance is declared __ro_after_init so it is immutable once boot finishes — a hardening measure against an attacker overwriting a hook pointer:

struct lsm_static_calls_table static_calls_table __ro_after_init ...;

A code comment in security.c captures the layout intent: the callbacks are “filled backwards (from last to first)” so the dispatch can jump “directly to the first used static call, and execute all of them after” — the table is densely packed at one end so an enabled-LSM count smaller than MAX_LSM_COUNT still dispatches without checking empty middle slots.

The call macros — call_int_hook and call_void_hook

The wrapper at a call site invokes the table through one of two macros (from security/security.c):

#define call_int_hook(HOOK, ...)					\
({									\
	__label__ OUT;							\
	int RC = LSM_RET_DEFAULT(HOOK);					\
									\
	LSM_LOOP_UNROLL(__CALL_STATIC_INT, RC, HOOK, OUT, __VA_ARGS__);	\
OUT:									\
	RC;								\
})
 
#define call_void_hook(HOOK, ...)					\
	do {								\
		LSM_LOOP_UNROLL(__CALL_STATIC_VOID, HOOK, __VA_ARGS__);	\
	} while (0)

LSM_RET_DEFAULT(HOOK) expands to HOOK##_default, the per-hook default declared from the DEFAULT argument of LSM_HOOK — 0 for file_open, -EOPNOTSUPP for inode_getsecurity, and so on. LSM_LOOP_UNROLL(M, ...) literally unrolls M (the per-slot macro) MAX_LSM_COUNT times via the kernel’s UNROLL() helper — there is no runtime loop counter; the compiler emits up to MAX_LSM_COUNT inline copies. The per-slot integer macro is the heart of aggregation:

#define __CALL_STATIC_INT(NUM, R, HOOK, LABEL, ...)			\
do {									\
	if (static_branch_unlikely(&SECURITY_HOOK_ACTIVE_KEY(HOOK, NUM))) { \
		R = static_call(LSM_STATIC_CALL(HOOK, NUM))(__VA_ARGS__);\
		if (R != LSM_RET_DEFAULT(HOOK))				\
			goto LABEL;					\
	}								\
} while (0);

Walk it: for slot NUM, if that slot is active (static_branch_unlikely — a branch the kernel patches in only when a module populated the slot, so inactive slots are essentially free), call the slot’s callback via static_call(...), store its result in R, and if that result differs from the hook’s default, jump to the OUT label immediately. That goto LABEL is the entire aggregation policy: the loop short-circuits on the first module that returns a non-default value, and since for permission hooks the default is 0 (allow), the first module to return a non-zero (denial) wins and no later module is consulted. The void variant is simpler — no return to aggregate, so it just calls every active slot:

#define __CALL_STATIC_VOID(NUM, HOOK, ...)				\
do {									\
	if (static_branch_unlikely(&SECURITY_HOOK_ACTIVE_KEY(HOOK, NUM))) { \
		static_call(LSM_STATIC_CALL(HOOK, NUM))(__VA_ARGS__);	\
	}								\
} while (0);

A Call Site End to End — security_file_open

The thin wrappers the kernel proper actually calls are defined in security/security.c. Three real ones, verbatim:

int security_inode_permission(struct inode *inode, int mask)
{
	if (unlikely(IS_PRIVATE(inode)))
		return 0;
	return call_int_hook(inode_permission, inode, mask);
}
 
int security_bprm_check(struct linux_binprm *bprm)
{
	return call_int_hook(bprm_check_security, bprm);
}
 
int security_file_open(struct file *file)
{
	int ret;
 
	ret = call_int_hook(file_open, file);
	if (ret)
		return ret;
	return fsnotify_open_perm(file);
}

These illustrate three patterns. security_inode_permission shows a fast-path bypass: private inodes (anonymous/pseudo filesystem nodes) skip LSM entirely with a bare return 0, because there is no meaningful security context to check and it would be hot. security_bprm_check is the pure pass-through — it does nothing but dispatch; this is the hook every execve() hits, where SELinux decides whether the new program may run in the calling domain and computes the domain transition (Mandatory vs Discretionary Access Control). security_file_open shows the wrapper doing kernel work after the hooks agree: only if call_int_hook(file_open, ...) returns 0 (all modules allowed) does it call fsnotify_open_perm() — the security verdict gates a non-security side effect.

The full path for a file open: VFS’s do_dentry_open() calls security_file_open(file); the wrapper expands call_int_hook(file_open, file); the unrolled loop walks each active slot in CONFIG_LSM order — say SELinux first, then AppArmor, then a BPF-LSM program; the first one returning non-zero (e.g. SELinux returns -EACCES because the process’s type may not read this file’s type) short-circuits via goto OUT, the wrapper returns -EACCES, and do_dentry_open() fails the open with that errno. If every module returns 0, RC stays 0, fsnotify runs, and the open proceeds.

When the Framework Is Compiled Out

When CONFIG_SECURITY is not set, none of this machinery exists — and the kernel must still compile. include/linux/security.h provides inline stubs so the call sites are free:

static inline int security_inode_permission(struct inode *inode, int mask)
{
	return 0;
}
static inline int security_bprm_check(struct linux_binprm *bprm)
{
	return 0;
}

Every security_*() becomes a static inline returning the allow value, which the compiler inlines and dead-code-eliminates — so a kernel built without LSM pays zero runtime cost at the call sites. This is why kernel code can sprinkle security_*() calls liberally without an #ifdef at each one: the header makes them vanish when the framework is absent.

Failure Modes and Common Misunderstandings

“Hooks run in registration order / discovery order.” They run in CONFIG_LSM (or lsm=) order, fixed at boot by ordered_lsm_init() (LSM Stacking and Module Ordering). The order is observable at /sys/kernel/security/lsm, which the admin guide says “reflects the order in which checks are made” (admin guide). Two stacked MACs can therefore reach different verdicts depending on which runs first only when their defaults differ — for the common allow-by-default hooks, order is irrelevant to the outcome (any denial wins) but does decide which module’s denial errno surfaces.

“All modules always run for every hook.” No — the short-circuit goto OUT on the first non-default return means later modules are not consulted once one denies. A module that needs to observe every event must use a void hook (which never short-circuits) or accept that it may not be reached on a denial.

“It’s still a linked list of hooks (security_hook_heads).” That was true before the static-call conversion (see the uncertainty flag above for the exact version). At v6.12 the security_hook_heads/hlist traversal is gone; dispatch is the unrolled static-call loop. Documentation, blog posts, and older books describing the hlist_for_each_entry walk over security_hook_heads are describing pre-conversion kernels. The motivation for the change was performance: the LSM dispatch runs on essentially every syscall, and an indirect call through a function pointer incurs retpoline/branch-prediction cost on modern CPUs that a patched static call avoids.

“Adding a hook means editing many files.” Only one: a new LSM_HOOK(...) line in lsm_hook_defs.h generates the union member, the table slot, the static-call/key pair, and the default; you then add the security_<name>() wrapper in security.c and call it from the relevant kernel path. The X-macro design is precisely what keeps this a small change.

Production Notes

The static-call rework matters in production because LSM is on the hottest paths in the kernel — every open, read permission recheck, execve, socket, connect, mmap, and signal passes through a security_*() wrapper. On a SELinux-enforcing distro under heavy syscall load, the difference between an indirect-call dispatch and a patched static call is measurable. The __ro_after_init on static_calls_table is a deliberate exploit-mitigation: a classic attack against function-pointer dispatch tables is to overwrite a hook pointer to bypass or hijack policy; making the table read-only after boot closes that. When auditing why an operation was denied, the chain is: the kernel path → its security_*() wrapper → call_int_hook(<hook>, ...) → the first module returning non-zero. The audit subsystem logs that denial tagged with the deciding module, so “which LSM said no” is recoverable from the log even though the dispatch short-circuited.

See Also

Graph View