vault

Specialized Maps (prog-array, sockmap, cgroup-storage)

17 min read

Specialized Maps (prog-array, sockmap, cgroup-storage)

Most BPF maps are generic key/value stores: you put bytes in, you get bytes out. A second family of map types looks like a map on the outside but is really a typed handle table for kernel objects — its “values” are not data the program reads, they are references to live kernel things: other BPF programs, sockets, network devices, CPUs, AF_XDP sockets, or per-cgroup storage regions. You cannot meaningfully bpf_map_lookup_elem() one of these and read the value as bytes; instead these maps are consumed by dedicated helpers (bpf_tail_call, bpf_redirect_map, bpf_sk_redirect_map, bpf_get_local_storage, bpf_get_stackid) that interpret the slot as a kernel-object reference and act on it. This note covers the major members as they exist in the 6.12 LTS kernel (released 2024-11-17): BPF_MAP_TYPE_PROG_ARRAY (the substrate for tail calls), SOCKMAP/SOCKHASH (socket redirection and sk_msg/sk_skb L7 steering), CGROUP_STORAGE/PERCPU_CGROUP_STORAGE and their modern replacement CGRP_STORAGE, the XDP redirect-target maps DEVMAP/CPUMAP/XSKMAP, and STACK_TRACE (stack-trace ID storage) (kernel BPF map docs).

The unifying idea is worth stating plainly because it is what makes these one note rather than a catalog: these maps hold kernel-object references, and the kernel manages their lifetime via reference counting. Inserting a fd into one of these maps takes a reference on the underlying object; removing the entry (or freeing the map) drops it, typically after an RCU grace period so in-flight users finish safely. The generic map machinery (in BPF Maps) gives you the table; the object-specific map_ops give you the reference semantics. We go deep on the prog-array mechanism — whose get_ptr/put_ptr is the cleanest illustration — and then treat the redirect-target and storage maps in terms of that same pattern, cross-linking the consumers (BPF Tail Calls, XDP Express Data Path, AF_XDP Zero-Copy Sockets) rather than re-explaining the dispatch they drive.

Mental Model

flowchart TB
  subgraph MAP["A specialized map = a table of kernel-object references"]
    direction LR
    PA["PROG_ARRAY<br/>slot -> struct bpf_prog*"]
    SM["SOCKMAP / SOCKHASH<br/>slot -> struct sock*"]
    DM["DEVMAP / CPUMAP / XSKMAP<br/>slot -> netdev / CPU / xsk"]
    CS["CGROUP_STORAGE / CGRP_STORAGE<br/>cgroup -> private storage region"]
    ST["STACK_TRACE<br/>stackid -> array of IPs"]
  end
  subgraph HELP["Consumed by dedicated helpers, not lookup-as-data"]
    H1["bpf_tail_call(ctx, &amp;prog_array, idx)"]
    H2["bpf_sk_redirect_map / bpf_msg_redirect_map"]
    H3["bpf_redirect_map(&amp;devmap, key, flags)"]
    H4["bpf_get_local_storage / bpf_cgrp_storage_get"]
    H5["bpf_get_stackid(ctx, &amp;stackmap, flags)"]
  end
  PA --> H1
  SM --> H2
  DM --> H3
  CS --> H4
  ST --> H5
  REF["Insert fd -> refcount++<br/>delete / free -> refcount-- (after RCU)"]
  MAP -. lifetime .-> REF

The object-handle map family. What it shows: five representative map types whose slots hold references to live kernel objects (programs, sockets, devices, CPUs, storage), each consumed by a purpose-built helper rather than by reading the value as data; inserting an fd bumps the object’s refcount and removing it drops the refcount after an RCU grace period. The insight to take: “it’s a map” is a packaging convenience (reuse the bpf() syscall, bpftool, map-in-maps) — the actual behavior is reference management plus a dedicated dispatch helper. Knowing which helper consumes a map type tells you everything about how it is used.

Prog-Array: a Table of BPF Program References (BPF_MAP_TYPE_PROG_ARRAY)

BPF_MAP_TYPE_PROG_ARRAY is the canonical object-handle map, and the deepest one to understand because tail calls — eBPF’s primary cross-program control flow — are built directly on it. It is an array whose values are file descriptors of other BPF programs. When a program executes bpf_tail_call(ctx, &prog_array, index), the kernel looks up the program at index and jumps to it without returning — the current program’s stack frame is reused, not nested. The dispatch semantics, the 33-deep limit, and the stack-reuse details belong to BPF Tail Calls; here we cover only how the map holds program references, which is the part owned by the map type.

The map’s value-handling is implemented by the fd-array family of map_ops in kernel/bpf/arraymap.c. The load-bearing function is prog_fd_array_get_ptr(), called when userspace updates a slot with a program fd (arraymap.c, prog_fd_array_get_ptr):

static void *prog_fd_array_get_ptr(struct bpf_map *map, struct file *map_file, int fd)
{
    struct bpf_prog *prog = bpf_prog_get(fd);   /* refcount++ on the program */
    if (IS_ERR(prog))
        return prog;
    if (!bpf_prog_map_compatible(map, prog)) {  /* type/JIT compatibility gate */
        bpf_prog_put(prog);                      /* refcount-- on failure */
        return ERR_PTR(-EINVAL);
    }
    return prog;
}

Three things happen here, and each is a general property of object-handle maps. (1) Reference acquisition: bpf_prog_get(fd) resolves the fd to a struct bpf_prog * and increments its refcount, so the program cannot be freed while it sits in the array. The matching prog_fd_array_put_ptr() calls bpf_prog_put(), and the comment is explicit that “bpf_prog is freed after one RCU or tasks-trace grace period” — the refcount drop is deferred so any CPU mid-tail-call into that program finishes safely (arraymap.c, prog_fd_array_put_ptr). (2) A compatibility gate: bpf_prog_map_compatible() rejects the insert unless the program is type-compatible with the map’s other entries. Practically, all programs reachable via one prog-array must share the same program type and JIT state — you cannot tail-call from an XDP program into a kprobe program. The first program inserted “locks in” the type for the whole array. (3) Refcount unwind on any failure path: if the compatibility check fails, bpf_prog_put() immediately undoes the get, so a rejected insert leaks nothing.

There is a further wrinkle specific to prog-arrays: because a JIT-compiled tail call is implemented as an indirect jump patched into native code, updating a slot must also re-patch every place that tail-calls through it. That is the job of the prog_array_map_poke_track/poke_run machinery (arraymap.c): the map keeps a list of “poke descriptors” (call sites in JITed programs that jump through this array), and a slot update walks that list and live-patches each indirect jump. This is invisible to the BPF programmer but is why prog-array updates are heavier than a plain array store. prog_array_map_seq_show_elem() is what lets bpftool map dump print the program id stored in a slot rather than raw bytes — confirming that the value really is a program handle, not data. Crucially, the fd-array ops provide no meaningful lookup_elem that returns the program to userspace; you can read back only the program id, never the struct bpf_prog *, because handing a raw kernel pointer to userspace would be unsafe.

See BPF Tail Calls for the dispatch side and BPF-to-BPF Function Calls for the alternative (ordinary function calls within one program, which do return).

Sockmap and Sockhash: Tables of Socket References (BPF_MAP_TYPE_SOCKMAP / SOCKHASH)

BPF_MAP_TYPE_SOCKMAP (introduced in 4.14) and BPF_MAP_TYPE_SOCKHASH (4.18) hold references to sockets — their values are socket descriptors, internally struct sock * (map_sockmap.rst). SOCKMAP is array-backed (integer key → socket); SOCKHASH is hash-backed (arbitrary key → socket), trading a hash computation for dense packing and non-integer keys. The point of holding sockets in a map is socket-level redirection and policy: a BPF program can splice data from one socket directly to another, in the kernel, bypassing the full network stack round-trip through userspace. This is the kernel substrate for sockmap-based L7 load balancing and service-mesh acceleration (e.g. Cilium’s socket-level redirect that short-circuits two local pods talking over TCP).

The mechanism has two cooperating program types attached to the map itself: a parser and a verdict program (map_sockmap.rst). When a socket is inserted into a sockmap, the kernel replaces that socket’s callbacks and attaches a struct sk_psock to it, and the socket inherits the map’s parser/verdict programs. The parser (stream_parser, attach type BPF_SK_SKB_STREAM_PARSER) decides how many bytes constitute a message — how much data must be queued before a verdict can be reached. The verdict program (stream_verdict/skb_verdict, attach types BPF_SK_SKB_STREAM_VERDICT/BPF_SK_SKB_VERDICT) then returns one of __SK_PASS, __SK_DROP, or __SK_REDIRECT. There is a parallel sk_msg path for egress: a BPF_PROG_TYPE_SK_MSG program (attach type BPF_SK_MSG_VERDICT) runs on sendmsg and can redirect the message to another socket.

The redirection itself is done by dedicated helpers that consume the map: bpf_sk_redirect_map(skb, map, key, flags) and bpf_msg_redirect_map(msg, map, key, flags) (and the _hash variants for SOCKHASH) redirect an skb or a message to the socket stored at key (map_sockmap.rst, helper descriptions). The BPF_F_INGRESS flag selects whether the data is delivered to the target socket’s ingress or egress path. Two finer-grained helpers shape message boundaries: bpf_msg_apply_bytes() tells the infrastructure how many bytes a verdict applies to, and bpf_msg_cork_bytes() holds a message back until enough bytes have arrived to reach a verdict (so a program can refuse to forward a half-message). bpf_msg_pull_data()/bpf_msg_push_data() let a SK_MSG program make a range of payload readable or insert metadata. A socket may live in multiple sockmaps, but it can inherit only one parser and one verdict program — adding it to a second map that would impose a second parser returns -EBUSY. As with prog-array, the value type is deliberately limited: it is __u32 or __u64 (the latter so userspace can read back a socket cookie), never the raw struct sock *, which “is neither safe nor useful” to expose (map_sockmap.rst, note).

XDP Redirect Targets: Devmap, Cpumap, Xskmap

Three map types exist almost entirely to be the target of bpf_redirect_map() from an XDP program. An XDP program runs at the earliest point in the receive path (in the NIC driver, before the network stack) and can redirect a raw frame to one of these targets; the map slot names where the frame goes. They share enqueue/send machinery and differ only in what kind of object the slots reference (map_devmap.rst).

BPF_MAP_TYPE_DEVMAP (4.14) and BPF_MAP_TYPE_DEVMAP_HASH (5.4) hold references to net devices. bpf_redirect_map(&devmap, key, flags) forwards the frame out the network interface referenced at key — i.e. XDP-based forwarding/routing entirely in the driver, never touching the stack. DEVMAP is array-indexed (key is typically the ifindex); DEVMAP_HASH packs devices densely behind a hash, useful when ifindexes are sparse (map_devmap.rst). The lower two bits of flags encode the XDP return code to use if the lookup fails (so a miss can fall back to XDP_PASS/XDP_DROP); the high bits can request BPF_F_BROADCAST (send to every device in the map) and BPF_F_EXCLUDE_INGRESS (omit the receiving interface), which together implement broadcast/multicast forwarding.

BPF_MAP_TYPE_CPUMAP (4.15) holds references to CPUs. Redirecting to a cpumap entry hands the raw xdp_frame to a dedicated kernel thread bound to that CPU, which then continues processing. This is software Receive Side Scaling: the CPU that the NIC delivered the packet to does minimal work (parse, hash, redirect) and the heavy processing happens on a different, less-loaded CPU. Since Linux 5.9 a cpumap entry can carry a second XDP program that runs on the remote CPU, so processing can be split across CPUs in two XDP stages (map_cpumap.rst). Compare Receive Side Scaling and Packet Steering for the hardware/RPS analogue.

BPF_MAP_TYPE_XSKMAP (4.18) holds references to AF_XDP sockets (XSKs). Redirecting to an xskmap entry delivers the raw frame to a userspace AF_XDP socket bound to a specific netdev queue, bypassing the entire network stack for high-performance userspace packet processing. The full zero-copy UMEM/fill-completion machinery is its own subject — see AF_XDP Zero-Copy Sockets for how the userspace side receives these frames. The xskmap is simply the kernel-side routing table from (redirect key) to xsk (map_xskmap.rst).

All three are the same idea: a redirect-target table, consumed by bpf_redirect_map, holding refcounted handles to devices/CPUs/sockets. The full XDP action model (XDP_REDIRECT, XDP_PASS, XDP_TX, XDP_DROP) lives in XDP Express Data Path.

Cgroup Storage: per-Cgroup Private State

BPF_MAP_TYPE_CGROUP_STORAGE and its per-CPU sibling BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE give cgroup-attached BPF programs a small, fixed-size private storage region keyed by the cgroup the program is attached to (map_cgroup_storage.rst). They exist only with CONFIG_CGROUP_BPF. The program does not look up a key explicitly; it calls bpf_get_local_storage(map, 0) and the kernel returns the storage belonging to the cgroup this invocation is running in — faster and simpler than a hash lookup, and with no need for the program to track which cgroups are live. The per-CPU variant gives each CPU its own region per storage (contention-free counters); the non-per-CPU variant shares one region across CPUs and leaves synchronization to the program (bpf_spin_lock or atomics). Storage is bound at attach time: even if the program is attached to a parent cgroup and fires in a child, the storage belongs to the parent.

The semantics changed in Linux 5.9 in a way worth knowing because it affects keying. Before 5.9, a CGROUP_STORAGE map could be used by only one program, and storage lifetime was strictly per-attachment (each attach created a fresh zeroed region, freed on detach). Since 5.9, storage can be shared by multiple programs: the kernel creates new storage only if the map has no entry for the (cgroup, attach-type) pair, otherwise it reuses the existing one, and storage is freed only when the map or the cgroup is freed (map_cgroup_storage.rst, “Semantics”). The key is either struct bpf_cgroup_storage_key (cgroup inode id + attach type, isolating storage per attach type) or — since 5.9 — a bare __u64 cgroup_inode_id (all attach types of that cgroup share one storage). Userspace can read/update via these keys but cannot create or delete entries; the kernel manages entry lifetime.

Uncertain

Verify: that BPF_MAP_TYPE_CGROUP_STORAGE and BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE are formally marked deprecated as of 6.12 (the UAPI enum aliases them to *_DEPRECATED names and a comment recommends CGRP_STORAGE), but whether they are slated for removal in any specific release was not confirmed. Reason: the v6.12 include/uapi/linux/bpf.h shows the _DEPRECATED aliasing and a “marked deprecated” comment, but no removal milestone was fetched. To resolve: check the kernel deprecation/feature-removal schedule and the commit that introduced the _DEPRECATED aliases. uncertain

The modern replacement is BPF_MAP_TYPE_CGRP_STORAGE (note: CGRP, not CGROUP), which first appears in the UAPI header at the v6.2 tag and is absent at v6.1 (verified by diffing the pinned v6.1 and v6.2 blobs), so it merged in Linux 6.2 (released 2023-02). The UAPI comment states the old CGROUP_STORAGE is deprecated because CGRP_STORAGE plus a local per-CPU kptr supersedes all of its functionality and more (bpf.h, map-type enum comments). CGRP_STORAGE is keyed by a cgroup fd (sizeof(int)), is available to all program types (not just cgroup-attached ones), supports BPF_F_NO_PREALLOC, and is accessed with bpf_cgrp_storage_get(map, cgroup, value, flags) / bpf_cgrp_storage_delete(map, cgroup) (map_cgrp_storage.rst). It belongs to the broader “local storage” family alongside SK_STORAGE, INODE_STORAGE, and TASK_STORAGE — storage keyed by a kernel object’s lifetime rather than by an explicit user-managed key.

Stack-Trace Map (BPF_MAP_TYPE_STACK_TRACE)

BPF_MAP_TYPE_STACK_TRACE stores captured call stacks, keyed by a 32-bit stack id. A BPF program calls bpf_get_stackid(ctx, &stackmap, flags); the kernel walks the current stack, hashes it, stores the array of instruction pointers in the map under a derived id, and returns that id (stackmap.c). The program then typically uses the stack id as a key into a separate counting map — this is exactly how flame-graph and profile-style tools aggregate “how many samples landed on this stack” without copying the full stack on every sample: identical stacks collapse to the same id. The map’s key_size must be 4 (the id) and max_entries is rounded up to a power of two for the internal bucket array.

The value layout depends on flags. By default each entry is an array of u64 instruction pointers. If the map is created with BPF_F_STACK_BUILD_ID, the kernel instead stores struct bpf_stack_build_id entries — each a build-id plus file offset rather than a raw runtime address (stackmap.c, stack_map_use_build_id). Build-id stacks are the key to symbolizing userspace stacks reliably: a raw address is meaningless once ASLR and shared-library load addresses are factored in, but (build-id, offset) can be resolved against the on-disk binary offline. At capture time bpf_get_stackid can request BPF_F_USER_STACK (walk the userspace stack instead of the kernel stack); the entry then records build-ids for the userspace frames where it can resolve the backing VMA, zeroing and flagging entries it could not parse. Stack-trace is “specialized” in a softer sense than the others — its values are data, not object handles — but it is grouped here because it is consumed by a dedicated helper (bpf_get_stackid/bpf_get_stack) and exists purely to back profiling tools, not as a general store.

Failure Modes and Common Misunderstandings

  • Reading an object-handle map as data. bpf_map_lookup_elem on a prog-array or sockmap does not give you the program/socket pointer in userspace — at best you get an id or a cookie. Treating the returned value as the kernel object is a category error; the kernel deliberately refuses to expose raw kernel pointers.
  • Type-mixing a prog-array. Inserting programs of different types (or incompatible JIT state) into one prog-array fails with -EINVAL from bpf_prog_map_compatible. All tail-call participants in one array must share a type. Symptom: a perfectly valid program fails to insert into the array.
  • Heavy prog-array updates. Because slot updates live-patch every JITed call site (the poke machinery), updating a prog-array is far costlier than a plain array store. Do it at setup, not in a hot loop.
  • Sockmap double-attach. Adding a socket to a second sockmap that would impose a second parser/verdict returns -EBUSY. A socket inherits exactly one of each.
  • Using the deprecated CGROUP_STORAGE on new code. It still works in 6.12 but is the legacy API with the awkward (cgroup, attach-type) keying and one-program-per-map history; prefer CGRP_STORAGE (6.2+) keyed by cgroup fd and usable from any program type.
  • Unsymbolizable userspace stacks. A plain STACK_TRACE map captures raw addresses; without BPF_F_STACK_BUILD_ID you cannot reliably symbolize userspace frames after the fact. Enable build-id mode for offline symbolization.

Alternatives and When to Choose Them

  • Need ordinary key/value state? Use a generic array map — these specialized types are the wrong tool unless you are literally redirecting to a kernel object, chaining programs, or storing per-cgroup/per-object state.
  • Per-object state without cgroups? The *_STORAGE family (SK_STORAGE, INODE_STORAGE, TASK_STORAGE, CGRP_STORAGE) attaches storage to a socket/inode/task/cgroup’s lifetime — cleaner than a hash keyed by an object id you must garbage-collect yourself.
  • Program chaining: tail call vs BPF-to-BPF call. A prog-array tail call jumps and does not return (good for splitting a large program past the complexity limit or dispatching by type); a BPF-to-BPF call is an ordinary function call that returns. Choose tail calls for dispatch/size, function calls for structure. See BPF Tail Calls.
  • Streaming events to userspace? That is the BPF Ring Buffer’s job, not any of these.

Production Notes

Prog-arrays underpin the dispatch tables in large XDP programs (e.g. Cilium and Katran split packet processing across many tail-called sub-programs to stay under the verifier’s complexity limit). Sockmap/sockhash power socket-level load balancing and service-mesh acceleration: Cilium’s socket-LB redirects two local endpoints’ TCP traffic socket-to-socket, skipping the loopback stack traversal, and uses sk_msg for L7 policy. DEVMAP/CPUMAP/XSKMAP are the backbone of XDP-based routers, software RSS, and AF_XDP fast paths (Katran, XDP load balancers, DPDK-style userspace stacks). STACK_TRACE with build-id mode is what bpftrace’s ustack/kstack and BCC’s profile/offcputime use to fold stacks into flame graphs cheaply. The common thread in production: you reach for one of these maps the moment your BPF program needs to act on a kernel object — jump to another program, send a packet somewhere, splice a socket, stash per-object state, or fingerprint a stack — rather than merely store and retrieve bytes.

See Also

Graph View

Backlinks