vault

BCC BPF Compiler Collection

12 min read

BCC BPF Compiler Collection

BCC (the BPF Compiler Collection, iovisor/bcc) is a toolkit for writing eBPF-based kernel-tracing programs in a high-level language — historically Python (or Lua) for the userspace front-end and C for the in-kernel program — together with a large library of ready-made production tools. Its defining original idea is that you embed the kernel-side eBPF program as a C source string inside your Python script, and BCC compiles that C at runtime, on the target machine, with an embedded Clang/LLVM, against that machine’s running kernel headers, then loads the resulting bytecode through the bpf() syscall (BCC reference guide). That runtime-compilation model is powerful but heavy — it drags a full compiler toolchain and matching kernel headers onto every box. The project has since added libbpf-tools, a parallel set of the same tools rewritten to compile once into portable CO-RE binaries with no runtime Clang and no kernel-headers dependency (libbpf-tools README). This note covers the BCC toolkit and programming model; the eBPF VM, verifier, and map internals it targets are owned by Linux eBPF MOC. Its one-liner sibling is bpftrace.

This note tracks BCC v0.36.1 (released 2026-02-10, per the GitHub releases API; v0.36.0 added “kernel support up to 6.18”), on the 6.12 / 6.18 LTS kernel era. BCC is userspace software released on its own cadence; version-sensitive facts are dated. The project’s README notes that “much of what BCC uses requires Linux 4.1 and above,” with eBPF first added in 3.15 (BCC README).

Mental Model — Two Programs in One File, Compiled at Runtime

The hardest thing to internalize about BCC is that a single .py file holds two programs in two languages targeting two execution environments. The userspace half is ordinary Python: it sets up, attaches probes, reads results, and prints. The kernel half is a C string — the eBPF program — that Clang compiles and the kernel runs. The Python BPF() constructor is the seam: you hand it the C string, it compiles and loads it, and hands back a handle through which the Python side attaches probes and pulls data out of maps.

flowchart TB
  PY["Python front-end<br/>(BPF(text=...), attach_kprobe,<br/>open_perf_buffer, print)"]
  CSTR["C BPF program<br/>(embedded string)"]
  CLANG["embedded Clang/LLVM<br/>(compiles AT RUNTIME)"]
  HDRS["running kernel's headers<br/>(/lib/modules/.../build)"]
  BC["eBPF bytecode"]
  KERN["kernel: verifier + JIT<br/>attached to kprobe/tracepoint"]
  MAP["BPF map / perf buffer"]
  CSTR --> CLANG
  HDRS --> CLANG
  CLANG --> BC
  BC --> KERN
  PY -->|BPF(text=...)| CLANG
  PY -->|attach_kprobe| KERN
  KERN --> MAP
  MAP -->|perf_buffer_poll| PY

The classic BCC architecture. What it shows: the C program string is fed, together with the running kernel’s headers, into an embedded Clang at program start; the resulting bytecode is verified, JITed, and attached; the Python front-end drives attachment and drains a perf buffer or map back to userspace. The insight: the compile step happens on the target machine at runtime, which is exactly why classic BCC needs Clang/LLVM and kernel headers installed everywhere it runs — and exactly the dependency CO-RE and libbpf-tools were built to eliminate.

The runtime-compile design solved a real problem of its era. Before CO-RE, a BPF program reading a kernel struct field needed the struct’s exact byte layout, which differs between kernel versions and configs. Compiling against the local kernel’s own headers at runtime guaranteed the layout matched — at the cost of carrying a compiler. Brendan Gregg’s early write-up frames BCC’s whole reason for existing: “eBPF provides amazing superpowers, [but] it’s hard to use via its assembly or C interface” — writing raw eBPF is “a brutal experience,” and BCC is “a front-end for eBPF, making it easier to write programs” (Gregg 2015).

The Programming Model — BPF(), Attach, Drain

Constructing the program. The entry point is the BPF object:

BPF({text=BPF_program | src_file=filename} [, cflags=[arg1, ...]])

You pass the C program either inline as text= or from a file as src_file=; cflags forwards compiler arguments like -D macros or -I include paths (reference guide). Construction is where the runtime Clang compile happens.

Attaching probes. Once compiled, the Python side binds C functions to events:

  • attach_kprobe(event="...", fn_name="...") — run the named C function on entry to a kernel function.
  • attach_kretprobe(event="...", fn_name="...") — on its return.
  • attach_uprobe(name="...", sym="...", fn_name="...") — on a userspace function.
  • attach_tracepoint(tp="subsys:event", fn_name="...") — on a static tracepoint.

There is also an implicit convention: a C function named kprobe__sys_clone is auto-attached to the sys_clone kprobe by its name prefix, and TRACEPOINT_PROBE(category, event) auto-attaches to a tracepoint — no explicit attach_* call needed.

Getting data out. Two paths. For streaming per-event data, BCC uses a perf buffer: the C side declares BPF_PERF_OUTPUT(events) and calls events.perf_submit(ctx, &data, sizeof(data)); the Python side registers a callback with table.open_perf_buffer(callback) and pumps it in a loop with b.perf_buffer_poll(). On kernels 5.8+ the newer ring buffer (BPF_RINGBUF_OUTPUT, ringbuf_output(), b.ring_buffer_poll()) is preferred — lower overhead, no per-CPU buffer waste. For aggregated data, the C side accumulates into a map (BPF_HASH, BPF_HISTOGRAM) and the Python side reads the whole map at the end — the in-kernel aggregation pattern, same as bpftrace.

Maps in the C program are declared with macros: BPF_HASH(name, key_type, leaf_type) (associative array), BPF_ARRAY(name, leaf_type, size), BPF_HISTOGRAM(name), BPF_STACK_TRACE(name, max_entries) (for stack-sampling via get_stackid()), and per-CPU variants BPF_PERCPU_HASH/BPF_PERCPU_ARRAY. Operations are method-style: map.lookup(&key), map.update(&key, &val), map.delete(&key), map.increment(key).

Debug helpers. bpf_trace_printk(fmt, ...) writes to the shared kernel trace pipe (limited: at most 3 args and one %s); the Python side reads it with b.trace_print(fmt="...") or b.trace_fields(). This is fine for hello-world but unsuitable for real tools, which use perf/ring buffers instead.

Licensing. GPL-only BPF helpers require a GPL-compatible program license; if you do not declare one, “BCC will automatically define the license of the program as GPL” via an implicit BPF_LICENSE.

Canonical Examples — From Hello World to a Real Histogram

These are verbatim from BCC’s Python developer tutorial (tutorial_bcc_python_developer.md), annotated.

Hello world — the whole program in one line:

from bcc import BPF
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

The C function is named kprobe__sys_clone, so BCC auto-attaches it to the sys_clone kprobe by name convention. It prints to the trace pipe; .trace_print() reads that pipe and echoes it. Every clone() (every new process/thread) prints a line. Note the \\n — it is escaped because it lives inside a Python string that becomes C source.

Per-event data via a perf buffer — the production pattern:

from bcc import BPF
 
prog = """
#include <linux/sched.h>
 
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
 
int hello(struct pt_regs *ctx) {
    struct data_t data = {};
    data.pid = bpf_get_current_pid_tgid();
    data.ts = bpf_ktime_get_ns();
    bpf_get_current_comm(&data.comm, sizeof(data.comm));
    events.perf_submit(ctx, &data, sizeof(data));
    return 0;
}
"""
 
b = BPF(text=prog)
b.attach_kprobe(event=b.get_syscall_fnname("clone"), fn_name="hello")
 
def print_event(cpu, data, size):
    event = b["events"].event(data)
    print("%-18.9f %-16s %-6d %s" % (event.ts, event.comm, event.pid, "Hello, perf_output!"))
 
b["events"].open_perf_buffer(print_event)
while 1:
    b.perf_buffer_poll()

The C side fills a typed struct data_t (pid from bpf_get_current_pid_tgid(), timestamp from bpf_ktime_get_ns(), name from bpf_get_current_comm()) and ships it via events.perf_submit(). Python attaches hello to the clone kprobe (get_syscall_fnname finds the version-correct symbol), registers print_event on the perf buffer, and perf_buffer_poll() blocks until events arrive and dispatches the callback. This is the skeleton nearly every real BCC tool follows.

In-kernel histogram — the aggregation pattern:

from bcc import BPF
from time import sleep
 
b = BPF(text="""
#include <uapi/linux/ptrace.h>
#include <linux/blkdev.h>
 
BPF_HISTOGRAM(dist);
 
int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
    dist.increment(bpf_log2l(req->__data_len / 1024));
    return 0;
}
""")
# ... attach, then:
b["dist"].print_log2_hist("kbytes")

BPF_HISTOGRAM(dist) declares a kernel map; on each block-I/O completion, bpf_log2l() computes the power-of-two bucket for the request size in KB and dist.increment() bumps it — all in the kernel, no per-event userspace work. At the end print_log2_hist() renders the map as an ASCII histogram. This is the direct ancestor of the biolatency tool, and exactly what bpftrace’s hist() does in one line.

The Tools — BCC’s Real Value

Most users never write a BCC program; they run its tools — dozens of focused, production-grade tracers shipped in the repo and packaged as bcc-tools (BCC README). The famous ones, by domain:

  • Process / syscalls: execsnoop (trace new process exec), opensnoop (trace open/openat), exitsnoop, killsnoop, syscount (count syscalls).
  • Storage / filesystem: biolatency (block-I/O latency histogram), biosnoop (per-I/O detail), biotop (top for disk I/O), ext4slower/xfsslower (slow FS ops), cachestat, filetop.
  • Networking: tcpconnect (trace active TCP connects), tcpaccept, tcplife, tcpretrans, tcptop.
  • CPU / scheduler: runqlat (run-queue latency histogram), offcputime (off-CPU stack analysis), profile (CPU profiler), cpudist, wakeuptime.
  • Generic / building blocks: funccount (count function calls), funclatency (function latency histogram), trace and argdist (ad-hoc probing without writing a program).

Each tool is a small, well-tested instance of the program model above. They are the reason BCC became the default observability layer on Linux fleets — execsnoop and biolatency in particular are staples of Netflix-era performance methodology.

The CO-RE Shift — libbpf-tools and the End of Runtime Clang

The runtime-compilation model has two costs that hurt at scale: every machine needs Clang/LLVM (hundreds of megabytes) and matching kernel headers installed, and compilation happens on each run — adding startup latency and memory, and failing entirely on minimal/immutable container images that lack headers. The answer is CO-RE (Compile Once – Run Everywhere): compile the BPF program once on a build machine into a portable object, and resolve struct-layout differences at load time using BPF Type Format (BTF) relocations applied by libbpf (BPF Portability and CO-RE). See CO-RE (Compile Once Run Everywhere) and CO-RE Relocations and Field Access for the relocation mechanics.

libbpf-tools/ is the BCC project’s reimplementation of its tools under this model (libbpf-tools README). Each tool is split into BPF C code, userspace C code, and a generated skeleton header. The tools are statically linked against libbpf (“all libbpf-based tools are linked statically against a version of libbpf”), so the shipped binary depends only on libc, libelf, and libz — no Clang, no Python, no kernel headers at runtime. The struct layouts come from a generated vmlinux.h (itself produced from a kernel’s built-in BTF), and the requirement on the target is that the running kernel was built with CONFIG_DEBUG_INFO_BTF=y so libbpf has BTF to relocate against. This is the modern default for shipping eBPF tools, and the larger Linux observability ecosystem (e.g. the BCC libbpf-tools binaries packaged by distros) has moved here. The trade-off: writing a libbpf-tool is more work than a quick BCC Python script — which is precisely why bpftrace (one-liners) and BCC Python (rapid prototyping) still coexist with libbpf-tools (shipping).

Failure Modes and Common Misunderstandings

“Failed to compile BPF module” / missing headers. Classic BCC compiles at runtime; if the linux-headers/kernel-devel package for the running kernel is absent, compilation fails before anything attaches. This is the single most common BCC deployment pain — and the entire motivation for libbpf-tools. On immutable or container hosts where you cannot install headers, prefer the libbpf-tools binaries.

High startup cost and memory. Each BCC Python tool launch spins up Clang/LLVM to compile the embedded C. On a constrained box this is visible latency and a real memory spike. Long-running BCC daemons pay it once; short-lived invocations pay it every time.

Perf-buffer event loss. If perf_buffer_poll() is not called often enough, or the per-CPU buffers are too small for the event rate, events are silently dropped (BCC reports lost-event counts). The fix is the same as bpftrace: aggregate in-kernel (histograms/counts) instead of streaming every event, or move to the ring buffer.

bpf_trace_printk limits. It is debug-only — 3 args max, one %s, and a single global pipe shared by all tracers. Real tools must use perf/ring buffers; using trace_print in production interleaves output from every tracer on the system.

Symbol/struct drift without CO-RE. Because classic BCC compiles against local headers, the same script can compile on one kernel and fail on another if a struct field was renamed. libbpf-tools/CO-RE mitigate this with BTF relocations and bpf_core_read()-style accessors that tolerate layout changes.

Alternatives and When to Choose Them

  • bpftrace — for one-liners and short scripts. If the investigation fits in a line or two and you want in-kernel aggregation now, bpftrace is faster to write than any BCC program. BCC wins when you need real userspace logic, rich data structures, packaging, or one of the existing tools. The full comparison is bpftrace vs BCC vs ftrace.
  • classic BCC (Python) vs libbpf-tools — Python BCC for rapid prototyping and interactive development (edit the C string, rerun); libbpf-tools for shipping (portable binary, no runtime deps, no headers). Same tools, two delivery models.
  • ftrace — when BPF is unavailable or you want zero dependencies and function-graph tracing. Lower ceiling, always present.

Production Notes

BCC is the backbone of the BPF Performance Tools toolset and ships as bcc-tools/python3-bcc in major distributions. In production the operational lesson is the dependency footprint: classic BCC’s runtime-compile model made it awkward on minimal container images and immutable hosts, and the industry’s response — visible in the BCC repo’s own libbpf-tools/ and in standalone CO-RE projects — was to move shipping tools to compiled-once CO-RE binaries gated on CONFIG_DEBUG_INFO_BTF=y. For a fleet, the practical guidance is: use BCC Python for exploration on a dev box where headers exist, and deploy the libbpf-tools (or equivalent CO-RE) binaries to production where you cannot assume Clang or headers. v0.36 (Jan–Feb 2026) continued this trajectory — its release notes cite refactoring tools to use fentry/fexit (BPF trampolines) “for better performance” over the older kprobe path (BCC releases).

Uncertain

Verify: the precise current split between classic-BCC and libbpf-tools coverage — i.e. which famous tools now have both a Python and a libbpf-tools version, and whether any are libbpf-tools-only. Reason: the libbpf-tools README describes the build/CO-RE model but does not enumerate a tool-by-tool parity table, and parity shifts every release. To resolve: diff tools/ against libbpf-tools/ in the v0.36.1 tag. uncertain

See Also

Graph View

Backlinks