vault

Kernel Keyrings

15 min read

Kernel Keyrings

The kernel key retention service is the Linux kernel’s in-process secret store: a facility for caching cryptographic keys, authentication tokens, cross-domain user mappings, and similar sensitive blobs inside the kernel, where filesystems and other kernel services can find them but unprivileged userspace generally cannot read them (core.rst, v6.12). Each cached secret is a struct key with a unique 32-bit serial number (key_serial_t), a type, a description used for matching, an owner UID/GID, a permissions mask, an optional expiry, and a payload. Keys are organized into keyrings — a special key type whose payload is a list of links to other keys — and every process is implicitly subscribed to a small set of special keyrings (thread, process, session, plus per-UID user and user-session keyrings) so that “find me the key for this filesystem mount” can be answered relative to the calling task. Userspace reaches the service through three system calls — add_key(2), request_key(2), and keyctl(2) — while the kernel uses an in-tree request_key() C interface.

This note covers why the kernel needs to hold secrets, the anatomy of a struct key and its serial-number handle, the three syscalls, and the special per-process keyrings (KEY_SPEC_*). Its sibling Keyring Types and Key Management covers the key types (user, logon, keyring, big_key, asymmetric, encrypted, trusted), the system keyrings that anchor module and kexec signature verification (.builtin_trusted_keys, .secondary_trusted_keys, .machine, .platform, .blacklist), the possession-and-permissions model, and the /sbin/request-key upcall. Read this one first.

Why the Kernel Holds Secrets

A kernel that performs cryptography on a user’s behalf needs somewhere to keep the keys, and that “somewhere” cannot simply be a userspace process: the kernel runs in a context where the original requester may have exited, where multiple processes share a resource, or where the secret must survive an execve() that discards the address space. The key retention service was built for exactly this gap. The canonical consumers (core.rst) are:

  • Disk and file encryption. dm-crypt volume keys and the per-file-encryption keys for fscrypt and eCryptfs are loaded into a keyring so the block layer / filesystem can transparently encrypt and decrypt without prompting userspace on every I/O. The logon key type exists precisely so that such a secret can be written from userspace once but only read back from kernel space — userspace can never extract it again. fscrypt, for example, looks up its master keys in a filesystem keyring; the master key payload is unreadable from userspace.
  • Network filesystem credentials. AFS and NFS use the service to cache Kerberos tickets and rxgk/rxkad tokens; the classic worked example throughout the documentation is “an AFS filesystem might want to define a Kerberos 5 ticket key type.” When the VFS needs to authenticate to a server during a pathwalk, it issues a request_key() against the requesting process’s keyrings.
  • The kernel’s own trust anchors. The keys used to verify kernel module signatures and kexec image signatures live in system keyrings (.builtin_trusted_keys and friends) managed by the same service — see Keyring Types and Key Management, Kernel Module Signing, and Secure Boot and the Kernel Trust Chain. IMA and EVM likewise keep their appraisal certificates on .ima / .evm keyrings.

The service is gated by CONFIG_KEYS (“Enable access key retention support”) (Kconfig, v6.12). With it off, the whole subsystem compiles out and the helper macros become no-ops.

Anatomy of a Key

Each retained secret is a struct key (key.h, v6.12). The documentation enumerates its attributes (core.rst): a serial number, a type, a description (for matching in a search), access-control information, an expiry time, a payload, and a state.

The serial number is the userspace handle. Per the docs, “each key is issued a serial number of type key_serial_t that is unique for the lifetime of that key. All serial numbers are positive non-zero 32-bit integers.” key_serial_t is literally typedef int32_t key_serial_t in key.h. Because they are positive, the kernel reserves negative values to name the special per-process keyrings symbolically (the KEY_SPEC_* constants below) — a keyctl() caller passes either a real positive serial or a negative special ID, and the kernel resolves the latter to a real key for the calling task. The serial is allocated from a per-namespace red-black tree (serial_node in struct key).

The type is a registered struct key_type. Types must be registered by an in-kernel service (a filesystem, the crypto layer) before keys of that type can exist; “userspace programs cannot define new types directly.” Removing a type invalidates every key of that type.

The payload is the actual secret. The union key_payload in key.h is either a single RCU-protected pointer (rcu_data0) or an array void *data[4] — small payloads can even be stored inline, large ones are allocated and pointed to. Reading the payload back is mediated by the type’s read() method, which is what lets logon keys refuse to disclose their payload to userspace.

A key moves through a small state machine (core.rst):

  • Uninstantiated — exists but has no data yet (a key being constructed via an upcall).
  • Instantiated (positive) — the normal, fully-formed state.
  • Negative — a short-lived “this lookup failed” marker that throttles repeated upcalls; re-requesting returns the stored error.
  • Expired — its lifetime elapsed; accesses fail with EKEYEXPIRED.
  • Revoked — placed there by userspace; accesses fail with EKEYREVOKED and the key can no longer be found.
  • Dead — its type was unregistered.

The last three states are subject to garbage collection. The in-kernel enum key_state actually has only two members — KEY_IS_UNINSTANTIATED and KEY_IS_POSITIVE — with the negative case encoded as a stored negative error code in key->state (key_is_negative() tests key_read_state(key) < 0); revocation and death are tracked by flags bits (KEY_FLAG_REVOKED, KEY_FLAG_DEAD). Revoked/expired keys are collected only after key_gc_delay, which is 5 * 60 = 300 seconds by default (gc.c, v6.12, exposed at /proc/sys/kernel/keys/gc_delay); dead keys are unlinked and freed as soon as possible.

Mental Model

flowchart TB
  TASK["Task (struct cred)"]
  TASK --> TK["Thread keyring<br/>KEY_SPEC_THREAD_KEYRING (-1)"]
  TASK --> PK["Process keyring<br/>KEY_SPEC_PROCESS_KEYRING (-2)"]
  TASK --> SK["Session keyring<br/>KEY_SPEC_SESSION_KEYRING (-3)"]
  SK --> US["_uid_ses.UID (user-session)<br/>KEY_SPEC_USER_SESSION_KEYRING (-5)"]
  US --> UK["_uid.UID (user)<br/>KEY_SPEC_USER_KEYRING (-4)"]
  TK -. links .-> K1["key: serial 0x1a2b<br/>type=logon desc=fscrypt:...<br/>payload (kernel-only)"]
  UK -. links .-> K2["key: serial 0x004d2<br/>type=user desc=krb5tgt:..."]
  SEARCH["request_key() / KEYCTL_SEARCH"] -->|"thread → process → session"| TK
  SEARCH --> PK
  SEARCH --> SK

The per-process keyring hierarchy and how a search traverses it. What it shows: a task reaches keys indirectly through its three subscribed keyrings; the session keyring’s default link reaches the user-session keyring, which links the user keyring, so a search can fall through to per-UID storage. The insight to take: the negative KEY_SPEC_* numbers are not keys — they are per-task shortcuts the kernel resolves to whichever real keyring currently fills that role, which is why two processes passing KEY_SPEC_SESSION_KEYRING may touch entirely different keyrings.

The Special Per-Process Keyrings

The reason the service is useful for “find the credential for this operation” is that every process is anchored to a set of keyrings whose lifetime and inheritance rules mirror the process model itself. There are five special keyrings reachable by the negative KEY_SPEC_* IDs defined in keyctl.h (keyctl.h, v6.12):

#define KEY_SPEC_THREAD_KEYRING        -1   /* thread-specific keyring */
#define KEY_SPEC_PROCESS_KEYRING       -2   /* process-specific keyring */
#define KEY_SPEC_SESSION_KEYRING       -3   /* session-specific keyring */
#define KEY_SPEC_USER_KEYRING          -4   /* UID-specific keyring */
#define KEY_SPEC_USER_SESSION_KEYRING  -5   /* UID-session keyring */
#define KEY_SPEC_GROUP_KEYRING         -6   /* GID-specific keyring (not implemented) */
#define KEY_SPEC_REQKEY_AUTH_KEY       -7   /* assumed request_key auth key */
#define KEY_SPEC_REQUESTOR_KEYRING     -8   /* request_key() dest keyring */

The three process keyrings differ chiefly in their inheritance across clone/fork/execve (core.rst):

  • Thread keyring (-1) is the most ephemeral: it is discarded from the child on any clone, fork, vfork, or execve. It is created lazily — only when a thread first needs one. Its ownership follows the thread’s real UID/GID.
  • Process keyring (-2) is replaced with an empty one in the child on clone/fork/vfork unless CLONE_THREAD is set (in which case it is shared among threads of the process). execve() discards the old process keyring and makes a fresh one.
  • Session keyring (-3) is the persistent one: it survives clone, fork, vfork, and execve — even when execve runs a set-UID or set-GID binary. A process can replace its session keyring with prctl(PR_JOIN_SESSION_KEYRING, ...) or KEYCTL_JOIN_SESSION_KEYRING, optionally joining a named one.

Beyond the per-process trio, every UID resident on the system owns two keyrings (core.rst): a user keyring (-4) and a user-session keyring (-5). The default user-session keyring is initialized with a link to the user keyring, so a search of the session ring can fall through to per-UID storage. In /proc/keys these appear as _uid.<UID> and _uid_ses.<UID>; the kernel marks them with the KEY_FLAG_UID_KEYRING flag. When a process changes its real UID and previously had no session key, it is subscribed to the default session keyring for the new UID. If a process touches its session key when it has none, it is subscribed to its UID’s default.

KEY_SPEC_GROUP_KEYRING (-6) is reserved but not implemented (per keyrings(7)). The last two IDs are internal plumbing for the upcall machinery: KEY_SPEC_REQKEY_AUTH_KEY (-7) names the authorization key a request_key handler has assumed, and KEY_SPEC_REQUESTOR_KEYRING (-8) names the original requester’s destination keyring during construction — both covered under the upcall in Keyring Types and Key Management.

Two keyrings not counted against the user’s key quota are the process-specific and thread-specific keyrings (core.rst). Each UID has two quotas — total key/keyring count and total description-plus-payload bytes — exposed via /proc/key-users and tunable through /proc/sys/kernel/keys/{maxkeys,maxbytes,root_maxkeys,root_maxbytes}. The keyrings(7) man page documents the defaults as 200 keys / 20,000 bytes for non-root and 1,000,000 / 25,000,000 for root, but these are policy and version-dependent.

Uncertain

Verify: the per-UID quota default values (200 keys / 20,000 bytes non-root; 1,000,000 / 25,000,000 root). Reason: these are stated in keyrings(7) (a man page, secondary to the kernel for runtime defaults) and the v6.12 key_user_lookup/init code was not directly inspected for the constants. To resolve: read the initial maxkeys/maxbytes assignments in security/keys/key.c / sysctl registration for v6.12.

The Three System Calls

Userspace manipulates keys through three syscalls; keyctl(2) is a multiplexer with a function number as its first argument (core.rst, keyctl.h).

add_key(2) creates (or updates) a key and links it into a nominated keyring in one shot:

key_serial_t add_key(const char *type, const char *desc,
                     const void *payload, size_t plen,
                     key_serial_t keyring);

If a key of the same type and description already exists in the target keyring, add_key tries to update it (returning EEXIST if the type has no update method, requiring write permission). Otherwise it creates a new key, instantiates it from payload (which may be NULL/zero-length if the type allows), and links it — requiring write permission on the keyring. The new key is granted all user permissions and no group/other permissions. A new keyring is made by passing type = "keyring", the ring name as the description, and a NULL payload.

request_key(2) is add_key’s lookup counterpart — it searches before optionally constructing:

key_serial_t request_key(const char *type, const char *description,
                         const char *callout_info, key_serial_t dest_keyring);

It searches the process’s keyrings in order — thread, then process, then session — for a matching key. If none is found and callout_info is non-NULL, the kernel launches an upcall to /sbin/request-key to construct the key; if found (or constructed), it can be linked into dest_keyring. The full search algorithm and the upcall live in Keyring Types and Key Management.

keyctl(2) carries everything else. The command IDs from keyctl.h (v6.12) range from KEYCTL_GET_KEYRING_ID (0) up to KEYCTL_WATCH_KEY (32). The most-used:

  • KEYCTL_GET_KEYRING_ID — resolve a special (negative) ID to a real serial, creating the keyring if create is non-zero.
  • KEYCTL_JOIN_SESSION_KEYRING — replace the session keyring (anonymous or named).
  • KEYCTL_UPDATE, KEYCTL_READ — replace or read back a payload (each subject to write / read permission).
  • KEYCTL_REVOKE, KEYCTL_INVALIDATE — mark a key revoked (future use returns EKEYREVOKED) or invalidate it and wake the garbage collector for immediate removal.
  • KEYCTL_CHOWN, KEYCTL_SETPERM — change owner/group or the permissions mask (owner or CAP_SYS_ADMIN).
  • KEYCTL_DESCRIBE — return <type>;<uid>;<gid>;<perm>;<description> (requires view permission).
  • KEYCTL_LINK, KEYCTL_UNLINK, KEYCTL_MOVE, KEYCTL_CLEAR, KEYCTL_SEARCH — keyring graph operations.
  • KEYCTL_SET_TIMEOUT — set an expiry N seconds in the future (0 clears it).
  • KEYCTL_GET_PERSISTENT — fetch a UID’s persistent keyring (below).
  • KEYCTL_RESTRICT_KEYRING — attach a link-time restriction (e.g. “only accept keys signed by X”) — the mechanism that makes the trusted-key keyrings tamper-resistant.
  • KEYCTL_CAPABILITIES — query which features the running kernel supports (the KEYCTL_CAPS0_* / KEYCTL_CAPS1_* bitmasks: persistent keyrings, Diffie-Hellman, public-key ops, big_key, restrict-keyring, move, namespaced keyring names, notifications).
  • KEYCTL_WATCH_KEY — subscribe to key/keyring change notifications via a watch_queue pipe (only if CONFIG_KEY_NOTIFICATIONS=y).

Worked Example

$ keyctl add user mypass:db "s3cr3t" @s          # add a "user" key to the session ring (@s = -3)
723615803
$ keyctl show                                     # dump the session keyring tree
Session Keyring
 -3 --alswrv   1000  1000  keyring: _ses
 ...
723615803 --alswrv 1000 1000   \_ user: mypass:db
$ keyctl print 723615803                          # read the payload back (needs read perm)
s3cr3t
$ keyctl timeout 723615803 60                     # KEYCTL_SET_TIMEOUT: expire in 60s
$ keyctl revoke 723615803                          # KEYCTL_REVOKE
$ keyctl print 723615803
keyctl_read_alloc: Key has been revoked

Line 1 maps to add_key("user", "mypass:db", "s3cr3t", 6, KEY_SPEC_SESSION_KEYRING); the shortcuts @s @p @t @u @us @g correspond to session/process/thread/user/user-session/group. Line on keyctl show walks the keyring with repeated KEYCTL_READ (a keyring’s payload is its array of linked serials) plus KEYCTL_DESCRIBE. keyctl print is KEYCTL_READ; after KEYCTL_REVOKE it fails because a revoked key can no longer be found or read.

Failure Modes and Common Misunderstandings

  • “My key vanished after the program exited.” A key persists only as long as something anchors it — a link from a live keyring counts as a reference (keyrings(7)). A key created in the thread keyring is gone after execve; one in the session keyring survives until the session ends. If you need cross-process persistence for a UID, use the persistent keyring or the user keyring, not the thread keyring.
  • “I added the key but the kernel service can’t find it.” Searches require search permission at every level, and a search only recurses into nested keyrings that themselves grant search permission (core.rst). A key with only read but not search is readable by serial but invisible to request_key.
  • Negative-key throttling. After a failed upcall, the kernel installs a negative key for a short time; re-requesting returns the cached error (ENOKEY or the rejected errno) until it expires. This is deliberate, to avoid hammering /sbin/request-key for an unobtainable key — but it surprises people who fix the underlying problem and immediately retry within the negative window.
  • Confusing the special ID with a real serial. KEY_SPEC_SESSION_KEYRING (-3) is not a key; it is a per-task indirection. Logging it as if it were a stable serial across processes is wrong. Use KEYCTL_GET_KEYRING_ID to obtain the real positive serial when you need to refer to the same keyring later.
  • EDQUOT on add. Exceeding the per-UID key-count or byte quota returns EDQUOT. Long-lived daemons that churn keys without unlinking the old ones eventually hit this.

Persistent Keyrings

A normal user keyring exists only while the UID record exists in the kernel (i.e. while some process or login session for that UID is around). The persistent keyring (CONFIG_PERSISTENT_KEYRINGS) fills the gap for things like Kerberos credential caches: it is a per-UID keyring that “stays around after all processes of that UID have exited” — though not across a reboot (Kconfig, v6.12). It is fetched with keyctl(KEYCTL_GET_PERSISTENT, uid, dest_keyring), accessible by the owning UID or an administratively-privileged process (the active LSM rules on which). Crucially it has an inactivity timeout: persistent_keyring_expiry = 3 * 24 * 3600 — three days of non-use, with the timer reset on each access (persistent.c, v6.12). After the timeout it is reaped and recreated on demand.

Alternatives and When to Choose Them

  • A userspace agent (ssh-agent, gpg-agent). Fine when only userspace consumes the secret. Choose kernel keyrings when a kernel subsystem (dm-crypt, fscrypt, a network FS) must use the secret without a round trip to userspace on every operation, or when the secret must be unreadable by the very process that loaded it (the logon type).
  • Environment variables / config files. Trivial but leak through /proc/<pid>/environ, core dumps, and ps; keyrings are permission-checked and can be made kernel-only.
  • The TPM / a hardware security module directly. Strongest, but awkward to consume in the I/O path. Keyrings layer on top: a trusted key is sealed by the TPM yet cached in a keyring for fast in-kernel use (see Keyring Types and Key Management).
  • libsecret / D-Bus keyring (GNOME Keyring, KWallet). A userspace desktop secret store — unrelated to the kernel facility despite the shared word “keyring.” Do not conflate them.

Production Notes

The retention service is most visible in three places. First, systemd / PAM: pam_keyinit establishes a session keyring at login so per-session credentials are scoped correctly, and systemd’s credential machinery and LoadCredentialEncrypted= lean on the kernel’s encryption-key plumbing. Second, Kerberos: MIT Kerberos can use a KEYRING: credential cache type that stores the TGT in the kernel keyring (often the persistent keyring) rather than a file in /tmp, removing the file-based ccache as an attack surface. Third, filesystem encryption: fscrypt provisions master keys into a per-superblock keyring and stores per-file keys as logon-style secrets so userspace cannot read them back, and cryptsetup can pass volume keys via the keyring rather than command-line arguments. Operationally, /proc/keys and keyctl show are the first diagnostic stops; a leaked or unexpectedly-readable key in /proc/keys (wrong permission mask) is a real finding, and KEYCTL_WATCH_KEY lets a monitor observe instantiate/revoke/link events in near real time.

See Also

Graph View

Backlinks