vault

cgroup BPF Programs

14 min read

cgroup BPF Programs

A cgroup BPF program is an eBPF program attached not to a network device or a kernel function, but to a control group — so it runs for every task in that cgroup (and, by default, its descendants) whenever the corresponding event occurs. This makes the control-group hierarchy a policy-enforcement surface: attach a program to a cgroup and you have hooked the network sockets, packets, device accesses, sysctl reads, and socket-option calls of an entire group of processes at once, without touching any individual program. The family spans packet filtering (CGROUP_SKB), socket lifecycle and address rewriting (CGROUP_SOCK, CGROUP_SOCK_ADDR), device access control (CGROUP_DEVICE, which in cgroup v2 replaces the old devices controller entirely), sysctl interception (CGROUP_SYSCTL), and socket-option interception (CGROUP_SOCKOPT) (cgroup-v2 admin guide, v6.12; enum bpf_prog_type, v6.12 bpf.h). Attachment is via BPF_PROG_ATTACH (or a BPF_LINK_CREATE link) referencing the cgroup’s directory file descriptor, with a hierarchical multi-program model governed by the BPF_F_ALLOW_MULTI and BPF_F_ALLOW_OVERRIDE flags. This is the cgroup specialization of the broader BPF Program Types taxonomy.

This is a §5 leaf of the Linux eBPF MOC. Cgroup BPF is what systemd uses for per-unit network/device policy and what Cilium uses for socket-based load balancing and host-firewalling; the orchestration context lives in cgroups Integration.

Mental Model — The Cgroup Tree as a Policy Surface

The unifying idea is that a cgroup is a named set of tasks arranged in a tree, and cgroup BPF lets you bolt a verdict program onto any node of that tree. Because cgroup v2 is a single unified hierarchy that every process lives in, “attach to this cgroup” is a precise, OS-native way to say “apply this policy to exactly these processes and their children.” A packet leaving a process is charged to that process’s cgroup; the kernel walks from that cgroup up to the root and runs the effective chain of attached programs. The program returns a verdict — allow or deny, or in the address-rewriting case a mutated socket address — and the kernel acts on it.

flowchart TD
  ROOT["root cgroup<br/>(prog A, MULTI)"]
  SVC["/system.slice<br/>(prog B, MULTI)"]
  UNIT["/system.slice/nginx.service<br/>(prog C, OVERRIDE)"]
  TASK["task: nginx worker<br/>connect() / send() / sysctl read"]
  ROOT --> SVC --> UNIT --> TASK
  TASK -. "event triggers" .-> CHAIN
  subgraph CHAIN["effective chain (bottom-up)"]
    direction TB
    EC["run C  -> then  B  -> then  A<br/>(child first; OVERRIDE at C<br/>shadows ancestor's single prog)"]
  end

A cgroup-BPF event resolving up the v2 hierarchy. What it shows: when an nginx worker performs a hooked operation, the kernel runs the programs found along its cgroup path, computed into a precomputed effective array; with BPF_F_ALLOW_MULTI the child’s programs run first and then the ancestors’ also run, while an OVERRIDE/NONE ancestor contributes at most one program that a child can shadow. The insight to take: policy is layered — a platform team can attach a baseline program at the root or a slice while individual units add their own, and the flag set on each attachment (MULTI vs OVERRIDE vs NONE) decides whether they stack or replace. The hierarchy is the access-control structure.

The Family of cgroup Program and Attach Types

Each cgroup hook is a distinct bpf_prog_type driven by one or more bpf_attach_type values. The names and SEC() mappings below are taken verbatim from the v6.12 libbpf program_types.rst table and the bpftool man page; the “since” versions are from the bpftool documentation’s own annotations (bpftool-cgroup, v6.12).

BPF_PROG_TYPE_CGROUP_SKB (SEC("cgroup_skb/ingress"), SEC("cgroup_skb/egress")) — per-cgroup packet filtering, attach types BPF_CGROUP_INET_INGRESS / BPF_CGROUP_INET_EGRESS (since 4.10). The program receives a struct __sk_buff and returns 1 to let the packet pass or 0 to drop it. This is the basis of per-cgroup network firewalling.

BPF_PROG_TYPE_CGROUP_SOCK (SEC("cgroup/sock_create"), post_bind4/6, sock_release) — fires at socket lifecycle events: socket creation (BPF_CGROUP_INET_SOCK_CREATE, since 4.10), after bind (BPF_CGROUP_INET4_POST_BIND/INET6_POST_BIND, since 4.17), and socket close (BPF_CGROUP_INET_SOCK_RELEASE, since 5.9). Used to constrain what sockets tasks in a cgroup may open and to tag/account them.

BPF_PROG_TYPE_CGROUP_SOCK_ADDR (SEC("cgroup/connect4"), connect6, bind4/6, sendmsg4/6, recvmsg4/6, getpeername4/6, getsockname4/6, and the *_unix variants) — the most powerful of the family: it runs at bind/connect/sendmsg/recvmsg and can rewrite the address. The program gets a struct bpf_sock_addr whose user_ip4/user_ip6/user_port fields are writable, so a connect4 program can transparently redirect a connection to 1.2.3.4:80 toward 127.0.0.1:8080. connect4/connect6/bind4/bind6 landed in 4.17, sendmsg4/6 in 4.18, recvmsg4/6 in 5.2, the getpeername/getsockname variants in 5.8, and the Unix-socket variants (connect_unix, …) in 6.7. This is the mechanism behind Cilium’s and kube-proxy-replacement socket load-balancing.

BPF_PROG_TYPE_CGROUP_DEVICE (SEC("cgroup/dev"), attach BPF_CGROUP_DEVICE, since 4.15) — device-access control. In cgroup v2 this is not optional alternative tooling; the v2 device controller has no interface files and is implemented entirely on top of cgroup BPF (cgroup-v2.rst, v6.12): “To control access to device files, a user may create bpf programs of type BPF_PROG_TYPE_CGROUP_DEVICE and attach them to cgroups with BPF_CGROUP_DEVICE flag. On an attempt to access a device file, corresponding BPF programs will be executed, and depending on the return value the attempt will succeed or fail with -EPERM.” The context is struct bpf_cgroup_dev_ctx, three __u32 fields: access_type — encoded as (BPF_DEVCG_ACC_* << 16) | BPF_DEVCG_DEV_*, packing the access kind (read/write/mknod) in the high half and the node type (block/char) in the low half — plus major and minor device numbers (v6.12 bpf.h). The program returns 1 to allow or 0 to deny.

BPF_PROG_TYPE_CGROUP_SYSCTL (SEC("cgroup/sysctl"), attach BPF_CGROUP_SYSCTL, since 5.2) — runs every time a task in the cgroup reads from or writes to a sysctl knob under /proc/sys. Context struct bpf_sysctl exposes write (0 = read, 1 = write) and a read-write file_pos; the program returns 0 to reject the access or 1 to proceed, and via helpers (bpf_sysctl_get_name, bpf_sysctl_get_current_value, bpf_sysctl_set_new_value) can inspect and even overwrite the value being set (prog_cgroup_sysctl.rst, v6.12).

BPF_PROG_TYPE_CGROUP_SOCKOPT (SEC("cgroup/getsockopt"), SEC("cgroup/setsockopt"), attach BPF_CGROUP_GETSOCKOPT/BPF_CGROUP_SETSOCKOPT, since 5.3) — intercepts the getsockopt(2)/setsockopt(2) syscalls. setsockopt runs before the kernel handler and can modify level/optname/optval/optlen (or set optlen = -1 to skip the kernel handler entirely), while getsockopt runs after and can rewrite what userspace sees, including custom socket options the kernel does not natively implement (prog_cgroup_sockopt.rst, v6.12).

Uncertain

Verify: the exact since kernel versions listed above (4.10 / 4.15 / 4.17 / 4.18 / 5.2 / 5.3 / 5.8 / 5.9 / 6.7). Reason: these come from the bpftool-cgroup man page’s own “(since X)” annotations at the v6.12 tag, which is a primary source but a documentation file that can lag the actual merge. To resolve: cross-check against git log for the introducing commit of each attach type. The current (v6.12) existence and behavior of every type above is verified against the v6.12 headers and docs; only the historical first-version labels carry this caveat. uncertain

Mechanical Walk-through — Attach, the Effective Chain, and Run

A cgroup program is attached by BPF_PROG_ATTACH with attr.target_fd set to a file descriptor for the cgroup directory (e.g. open("/sys/fs/cgroup/system.slice/nginx.service", O_DIRECTORY)), attr.attach_bpf_fd the program fd, attr.attach_type the specific hook, and attr.attach_flags the multi-prog flags. The modern path uses BPF_LINK_CREATE to get a bpf_link whose lifetime owns the attachment (it detaches on close), but the legacy BPF_PROG_ATTACH path is still widely used and is what bpftool cgroup attach drives.

The interesting machinery is how the kernel decides which programs run for a given task. It does not walk the cgroup tree at event time — that would be too slow on a packet-per-second hot path. Instead, whenever an attach/detach changes the tree, the kernel precomputes a flat effective program array per (cgroup, attach_type) and stores it in cgrp->bpf.effective[atype]. The hot path then runs that array directly. The computation is in compute_effective_progs() (kernel/bpf/cgroup.c, v6.12):

/* count number of effective programs by walking parents */
do {
    if (cnt == 0 || (p->bpf.flags[atype] & BPF_F_ALLOW_MULTI))
        cnt += prog_list_length(&p->bpf.progs[atype]);
    p = cgroup_parent(p);
} while (p);

Read this carefully: starting from the target cgroup and walking up to the root, it includes a cgroup’s own programs if it is the starting cgroup (cnt == 0) or if that ancestor was attached with BPF_F_ALLOW_MULTI. An ancestor attached without MULTI (i.e. NONE or OVERRIDE) contributes at most one program, and only if a closer cgroup did not already shadow it. The populated array therefore encodes the layering rule: child programs precede ancestor programs, and OVERRIDE/NONE ancestors yield to children. The array is swapped in atomically with rcu_replace_pointer so in-flight events on the hot path keep using the old array until a grace period passes (activate_effective_progs, cgroup.c v6.12).

Whether a new attach is even permitted is gated by hierarchy_allows_attach(), which walks up the parents: if any ancestor has a non-MULTI, non-OVERRIDE (i.e. NONE) single program attached, the descendant attach is refused; an ancestor with OVERRIDE allows the child to attach (and shadow), and an ancestor with MULTI always allows children (cgroup.c v6.12). This is the kernel enforcing the policy-delegation semantics structurally, not just by convention.

The Multi-Program Flags — BPF_F_ALLOW_MULTI vs OVERRIDE

The three attachment modes are defined exactly by the UAPI header’s own commentary (v6.12 bpf.h):

  • NONE (no flag, default) — only one program per cgroup per hook, and no further programs are allowed in the subtree. Attaching again replaces the existing program (flags must match). The strictest, least composable mode.
  • BPF_F_ALLOW_OVERRIDE (1<<0) — one program at this level, but a sub-cgroup may install its own program that yields the parent to the child (the child shadows the parent for tasks in the child).
  • BPF_F_ALLOW_MULTI (1<<1) — multiple programs may attach at this cgroup, and they stack with sub-cgroup programs: all eligible programs run, child-first then parent. New programs append to the end of the list; BPF_F_REPLACE (1<<2) with a replace_bpf_fd lets you swap a specific program in place.

The header gives a precise worked example that is worth internalizing. For the hierarchy cgrp1(MULTI: A,B) → cgrp2(OVERRIDE: C) → cgrp3(MULTI: D) → cgrp4(OVERRIDE: E) → cgrp5(NONE: F), an event in cgrp5 runs F, D, A, B in that order; detach F and it becomes E, D, A, B; detach F and D and it becomes E, A, B; detach F, E, and D and it becomes C, A, B. Two rules drive this: MULTI cgroups always contribute all their programs, while OVERRIDE/NONE cgroups contribute their single program only when no closer cgroup overrides it; and the order is strictly child-to-parent. Critically, “all eligible programs are executed regardless of return code from earlier programs” — there is no short-circuit; for an allow/deny hook the kernel combines verdicts (typically: a single 0 anywhere denies).

Configuration / Code — Attaching with bpftool and a connect4 Rewriter

Attach with bpftool (which drives BPF_PROG_ATTACH):

# Attach an egress packet filter to a systemd unit's cgroup, allowing multi-prog
$ bpftool prog load fw.o /sys/fs/bpf/fw type cgroup_skb
$ bpftool cgroup attach /sys/fs/cgroup/system.slice/nginx.service \
        egress pinned /sys/fs/bpf/fw multi
 
# List the effective chain for a cgroup (includes inherited programs)
$ bpftool cgroup show /sys/fs/cgroup/system.slice/nginx.service --effective
ID       AttachType      AttachFlags     Name
42       cgroup_inet_egress  multi       fw_egress
 
# Detach
$ bpftool cgroup detach /sys/fs/cgroup/system.slice/nginx.service \
        egress pinned /sys/fs/bpf/fw

Line by line: prog load … type cgroup_skb loads and pins the program; cgroup attach … egress … multi attaches it to the unit’s cgroup directory at the egress hook with BPF_F_ALLOW_MULTI; cgroup show … --effective prints the computed effective chain, including programs inherited from ancestor cgroups (without --effective it shows only programs attached directly to that cgroup) (bpftool-cgroup, v6.12).

A connect4 address-rewriter — the canonical CGROUP_SOCK_ADDR use, transparently redirecting connections to a backend:

SEC("cgroup/connect4")
int redirect_connect(struct bpf_sock_addr *ctx)
{
    /* Only rewrite TCP connections aimed at the virtual service IP 10.0.0.1:80 */
    if (ctx->user_family == AF_INET &&
        ctx->user_ip4 == bpf_htonl(0x0A000001) &&     /* 10.0.0.1 */
        ctx->user_port == bpf_htons(80)) {
        ctx->user_ip4 = bpf_htonl(0x7F000001);         /* -> 127.0.0.1 */
        ctx->user_port = bpf_htons(8080);              /* -> :8080      */
    }
    return 1;   /* 1 = allow the connect to proceed (with rewritten addr) */
}

Commentary: the context’s user_ip4/user_port are writable for the connect4 attach type — that writability is granted by the verifier specifically because expected_attach_type is BPF_CGROUP_INET4_CONNECT (the same program type at a read-only attach type could not modify them; see BPF Program Types on how attach type narrows the contract). Returning 1 lets the now-rewritten connect proceed; returning 0 would fail the syscall with -EPERM. The application thinks it connected to 10.0.0.1:80; the kernel actually connected it to 127.0.0.1:8080, and a matching getpeername4 program can lie back so the app still sees the virtual address.

Failure Modes and Common Misunderstandings

Forgetting --effective. bpftool cgroup show without the flag lists only directly-attached programs, so a program inherited from an ancestor cgroup appears to be “missing.” The effective chain is what actually runs; always check with --effective when debugging “my policy isn’t applied to this cgroup.”

NONE blocks the subtree. Attaching a single program with no flags (NONE) at a high cgroup forbids any attachment in the entire subtree below it — hierarchy_allows_attach will reject children. Platform programs that must coexist with workload programs should use MULTI.

Assuming short-circuit. Because all eligible programs run regardless of earlier return codes, you cannot rely on an early-deny program to stop later programs from executing side effects. Verdicts are combined, but side effects (map writes, etc.) all happen.

cgroup v1 vs v2 device controller confusion. The classic devices cgroup-v1 controller (the devices.allow/devices.deny interface files) and CGROUP_DEVICE BPF are different mechanisms. Under cgroup v2 there are no device interface files at all — device policy is only expressible as a CGROUP_DEVICE BPF program. Containers run under v2 (the default on modern distros) therefore have their device whitelists enforced by BPF, which is why runc/crun load a CGROUP_DEVICE program per container.

Attach-fd must be the cgroup directory. target_fd is a file descriptor for the cgroup’s directory in the cgroup filesystem, not a PID or a socket. Opening the wrong path silently attaches to the wrong scope.

Alternatives and When to Choose Them

Cgroup BPF is the right tool when the policy boundary is naturally “a set of processes” — a container, a systemd unit, a Kubernetes pod’s cgroup. For policy keyed on the device rather than the process set, use tc and cls_bpf (Traffic Control Hooks) (per-interface) or XDP (eXpress Data Path) (per-NIC, pre-stack) instead — those hook the data path regardless of which cgroup owns the traffic. For tracing what tasks in a cgroup do (rather than enforcing policy), use kprobe and uprobe BPF Programs or fentry fexit and BPF Trampolines with a cgroup-id filter in the program. Address rewriting at connect time (CGROUP_SOCK_ADDR) is uniquely a cgroup-BPF capability — there is no non-BPF equivalent that intercepts the syscall and mutates the destination so transparently. For device control specifically under cgroup v2, CGROUP_DEVICE is not an alternative, it is the only mechanism.

Production Notes

systemd builds its IP access controls and device controls on cgroup BPF. The systemd.resource-control(5) man page documents IPAddressAllow=/IPAddressDeny= (implemented by attaching an in-kernel CGROUP_SKB ingress/egress firewall program to the unit’s cgroup), the explicit escape hatches IPIngressFilterPath=/IPEgressFilterPath= and BPFProgram= (which let a unit attach its own pinned cgroup-BPF program — the page’s own example BPFProgram=bind6:/sys/fs/bpf/sock-addr-hook attaches a CGROUP_SOCK_ADDR bind6 hook), and DeviceAllow=/DevicePolicy=, whose v2 enforcement is CGROUP_DEVICE BPF as described above (systemd.resource-control(5)). These directives require cgroup-BPF kernel support. Container runtimes (runc, crun) do the same for the device whitelist under cgroup v2. Cilium uses CGROUP_SOCK_ADDR connect/sendmsg/recvmsg hooks for its socket-based load balancing (“kube-proxy replacement”): instead of DNAT’ing packets in the data path, it rewrites the destination at connect time inside the source pod’s cgroup, so the connection is established directly to the chosen backend with no per-packet translation (Cilium kube-proxy-free docs; see Cilium). The shared lesson across all three: because the hook is on the cgroup, the policy follows the workload through process forks and exec without per-process bookkeeping — the cgroup is the durable identity. For the orchestration view of how Kubernetes constructs these cgroups in the first place, see cgroups Integration.

See Also

Graph View