vault

Low-Level vs High-Level Container Runtimes

16 min read

Low-Level vs High-Level Container Runtimes

The phrase “container runtime” hides a two-layer stack with a precise division of labor. A low-level (OCI) runtime — runc, crun, or youki — does exactly one small thing: it takes an Open Container Initiative (OCI) runtime bundle (a root filesystem plus a config.json) and turns it into a running Linux process by calling clone/unshare/setns, pivot_root, and execve, then exits, leaving a normal process behind (OCI runtime-spec bundle.md). A high-level runtime — containerd or CRI-O — sits above that and owns everything the low-level runtime deliberately refuses to do: pulling and unpacking images, managing layer storage (snapshots), wiring up networking, persisting container metadata and lifecycle state, streaming logs, and exposing the Container Runtime Interface (CRI) gRPC contract that the Kubernetes kubelet calls. When the kubelet asks the high-level runtime to start a container, the high-level runtime materializes a bundle on disk and delegates the actual container creation downward to a low-level runtime across the stable OCI boundary. The split exists because the OCI specifications are the contract between the two layers — they let any high-level runtime drive any low-level runtime, and let the same low-level runtime serve Docker, Podman, containerd, and CRI-O without modification.

Mental Model

The cleanest way to think about the stack is two contracts and three actors. Above the high-level runtime, the contract is the CRI — a gRPC API the kubelet speaks. Below the high-level runtime, the contract is the OCI runtime spec — a config.json schema plus a CLI verb set (create, start, kill, delete, state). The high-level runtime is the translator that sits between them: it consumes pod-level CRI calls from above and emits per-container OCI bundles and CLI invocations below.

flowchart TB
  KUBELET["kubelet<br/>(Kubernetes node agent)"]
  subgraph HIGH["HIGH-LEVEL runtime (containerd / CRI-O)"]
    CRISVC["CRI gRPC server<br/>RuntimeService + ImageService"]
    IMG["image pull + unpack<br/>(content store)"]
    SNAP["snapshot / storage<br/>(overlayfs lowers + upper)"]
    NET["CNI networking"]
    META["lifecycle + metadata + logs"]
  end
  subgraph BOUNDARY["the OCI boundary (a directory on disk)"]
    BUNDLE["OCI runtime bundle<br/>config.json + rootfs/"]
  end
  subgraph LOW["LOW-LEVEL (OCI) runtime"]
    RUNC["runc / crun / youki<br/>clone + setns + pivot_root + execve"]
  end
  PROC["container process<br/>(a normal Linux task)"]

  KUBELET -->|"CRI: gRPC over UNIX socket"| CRISVC
  CRISVC --> IMG --> SNAP
  CRISVC --> NET
  CRISVC --> META
  SNAP -->|"materializes"| BUNDLE
  META -->|"OCI CLI: create / start / kill / delete"| RUNC
  RUNC -->|"reads"| BUNDLE
  RUNC -->|"execve, then exits"| PROC

The container runtime stack as two contracts bracketing the high-level runtime. What it shows: the kubelet only ever speaks CRI (top); the low-level runtime only ever speaks OCI (bottom); the high-level runtime is the broker that turns pod-shaped CRI requests into container-shaped OCI bundles and CLI calls. The insight to take: the two boundaries are independently swappable — you can change the low-level runtime (runc→crun) without touching the kubelet, and change the high-level runtime (containerd→CRI-O) without touching the low-level runtime, precisely because each boundary is a published spec, not an internal API.

What a Low-Level Runtime Is (and Is Not)

A low-level runtime implements the OCI Runtime Specification, whose scope is narrow by design: it “specifies the configuration, execution environment, and lifecycle of a container” (OCI runtime-spec spec.md). Its only input is an OCI runtime bundle, defined as “a set of files organized in a certain way, and containing all the necessary data and metadata for any compliant runtime to perform all standard operations against it” (bundle.md). The bundle is just a directory containing two things: a config.json (which “MUST reside in the root of the bundle directory and MUST be named config.json”) and a root filesystem directory pointed to by the config’s root.path. Nothing more. There is no image, no registry, no network configuration, no log file — those concepts do not exist in the runtime spec. (The latest released runtime spec is v1.3.0, 2024-11-04, the fourth minor release of the v1 series, per the runtime-spec releases.)

The runtime spec also fixes the lifecycle: the operations a runtime MUST support are state, create, start, kill, and delete (runtime.md). The ordering is deliberate. create “MUST create a new container” — it sets up the namespaces, cgroups, mounts, and confinement from config.json but does not run the user program yet. start “MUST run the user-specified program” — only now does the entrypoint execve. kill sends a signal; delete “MUST delete the resources that were created during the create step.” This create/start split is not an accident: it gives the high-level runtime a window — after the namespaces exist but before the entrypoint runs — to attach Container Network Interface (CNI) networking, inject file descriptors, or set up stdio. The reference walk-through of these steps for the canonical low-level runtime lives in runc; this note is about why the layer boundary sits where it does, not the syscall choreography.

Uncertain

Verify: the precise 13-step lifecycle enumeration (prestart / createRuntime / createContainer / startContainer / poststart / poststop hook ordering). Reason: the step list was returned by an automated fetch summary of runtime.md, not transcribed character-for-character from the spec source, and hook ordering changed across spec versions (the deprecated prestart vs the newer createRuntime/createContainer hooks). To resolve: read the “Lifecycle” section of runtime.md at tag v1.3.0 directly. The five MUST-support operations (state/create/start/kill/delete) and the create-then-start split are confirmed from the primary doc. uncertain

The defining property of a low-level runtime is that it is not a daemon. runc create and runc start are one-shot CLI invocations: each runs, does its work, and exits. After runc start calls execve, there is no “runc” attached to the container — the process is a normal Linux task that merely happens to live in non-default namespaces with a seccomp filter installed. You can kill the runc binary and the container keeps running. This is the crux of the layering: because the low-level runtime exits, something else must stay alive to hold the container’s standard I/O, reap its exit status, and survive a restart of the high-level runtime. That “something else” is the shim (containerd) or the monitor (CRI-O), discussed below.

What a High-Level Runtime Owns

A high-level runtime is the long-lived component that the kubelet actually talks to, and it owns the entire set of responsibilities the OCI runtime spec excludes. Concretely:

  • Image pull and unpack. It fetches image manifests and layer blobs from a registry (the OCI Distribution Spec), verifies them by digest, and unpacks each layer’s tar changeset onto disk. (The image-side mechanics are owned by Container Image Layers and Copy-on-Write.)
  • Snapshot / storage management. It assembles the unpacked layers into the container’s root filesystem — in practice the read-only layers become the lower directories of an overlayfs mount with a fresh writable upper. This is the “rootfs” half of the bundle it later hands down.
  • Networking. It invokes CNI plugins to give the pod a network namespace with a real interface (a veth pair into a host bridge, typically), during the window the create/start split provides.
  • Lifecycle, metadata, and logs. It persists which containers exist and their state, multiplexes their stdout/stderr to log files the kubelet can read, and answers CRI status queries.
  • The CRI server. It exposes two gRPC services over a UNIX socket — RuntimeService (pod sandbox and container lifecycle, plus exec/attach/port-forward) and ImageService (image management) — which the kubelet calls exclusively (Kubernetes CRI docs). See Container Runtime Interface for the gRPC contract itself.

Crucially, CRI is pod-shaped while OCI is container-shaped. The kubelet’s first call for a new pod is RunPodSandbox, which asks the high-level runtime to create the pod’s shared infrastructure (its network and IPC namespaces, held by a pause container); subsequent CreateContainer/StartContainer calls add the workload containers into that sandbox. The high-level runtime is the layer that knows about pods at all — the low-level runtime has no concept of a pod, only of individual containers each described by their own config.json. Translating “a pod with three containers sharing a network namespace” into “three OCI bundles, two of which setns into the first’s netns” is exactly the high-level runtime’s job.

The Process Model: containerd → shim → runc

containerd does not invoke runc directly. Between them sits the runtime-v2 shim, a per-pod long-lived process. When containerd needs to start a pod sandbox, it fork/execs the shim binary with the start subcommand; the shim starts a listener, prints back a socket address, and the initial process exits, leaving a detached daemon (containerd runtime-v2.md). containerd then drives that shim over ttrpc (a lightweight gRPC variant for low-memory daemons). The shim, in turn, mounts the container’s root filesystem and fork/execs the actual runc binary to create and start the container, publishing TaskCreate/TaskStart/TaskExit events back up.

Why the indirection? Because runc exits, the shim is what stays alive to (1) hold the container’s stdio FIFOs/pipes, (2) be the subreaper that collects the container’s exit code, and (3) survive a containerd restart — the containers keep running, attached to their shims, while containerd is down, and containerd re-attaches to the still-running shims when it comes back up. The runtime-v2 protocol is explicit that “containerd does not know or care about whether the shim to container relationship is one-to-one, or one-to-many” — the grouping is entirely the shim’s decision. The default io.containerd.runc.v2 shim groups containers by the io.kubernetes.cri.sandbox-id label that the CRI plugin sets, so all containers in one Kubernetes pod are handled by a single shim (CRI and ShimV2, Alibaba Cloud). This is the modern “shimv2” model; it replaced the older arrangement where a Kata pod needed 2N+1 shims (a containerd-shim plus a kata-shim per container and for the sandbox) with one shim per pod.

sequenceDiagram
  participant K as kubelet
  participant C as containerd (daemon)
  participant S as shim (per pod, detached)
  participant R as runc (one-shot)
  participant P as container process
  K->>C: CRI CreateContainer / StartContainer (gRPC)
  C->>S: ttrpc Create / Start (one shim per pod)
  S->>R: fork/exec runc create
  R->>R: clone + setns + pivot_root, then exit
  S->>R: fork/exec runc start
  R->>P: execve(entrypoint), then exit
  Note over S,P: shim stays alive, holds stdio, reaps exit
  Note over C: containerd can restart;<br/>shim + container survive

The containerd process model, end to end. What it shows: containerd (daemon) → shim (per-pod, detached) → runc (one-shot CLI) → the container process. The insight to take: the shim is the durability layer the OCI spec’s “exit after exec” design requires — runc cannot be the thing that holds stdio or survives a daemon restart because runc is gone the moment the entrypoint runs; the shim fills that gap, and grouping it per-pod is what makes Kubernetes pod semantics efficient.

The Process Model: CRI-O → conmon → runc/crun

CRI-O reaches the same outcome with a different shape. CRI-O is a Kubernetes-only high-level runtime that implements only the CRI (cri-o.io). It has no long-lived shim daemon; instead, for each container it launches a small C monitor called conmon (“container monitor”). conmon is the process that “becomes the parent of the container process” and is responsible for holding the container’s pseudo-terminal (PTY) and pipes, forwarding logs, detecting the exit and writing the exit code to a file, and optionally enforcing exec timeouts (containers/conmon). The crio daemon fork/execs conmon, which in turn fork/execs the low-level runtime — runc or, increasingly on Red Hat platforms, crun — to create and start the container, then conmon stays resident as the container’s babysitter while the low-level runtime exits.

The role mapping is the clean way to remember it: conmon (CRI-O) plays the same role as the shim (containerd) — the durable per-container monitor that exists because the OCI low-level runtime is one-shot. The differences are matters of granularity and language: containerd’s shim is one Go process per pod driven over ttrpc; CRI-O’s conmon is one C process per container with no RPC surface (the crio daemon talks to conmon via file descriptors and signals, not a protocol). A newer Rust rewrite, conmon-rs, moves toward a per-pod monitor to reduce process count, mirroring containerd’s per-pod shim. Both designs satisfy the same invariant: a resident monitor outlives the one-shot OCI runtime and survives a restart of the high-level daemon.

Why the Split Exists — the OCI Boundary as a Contract

The layering is not incidental architecture; it is the direct consequence of standardizing the boundary. The OCI was founded in 2015 when Docker donated runc and the runtime spec, deliberately carving out the smallest possible “run a container from a bundle” interface so that the rest of the stack could be competed on. Three concrete benefits follow:

  1. Low-level runtimes are interchangeable. Because the bundle format and CLI verbs are fixed, swapping runc for crun is a one-line configuration change in containerd or CRI-O — no image, pod spec, or kubelet change. crun (latest 1.28, 2026-05-27) is a C reimplementation that is roughly 50× smaller than runc and faster to start; youki (latest v0.6.0, 2026-02-25, a CNCF sandbox project) is a Rust implementation that passes containerd’s end-to-end tests. All three consume the identical config.json. (runc itself is at v1.4.3, 2026-06-13.) The depth comparison of these runtimes lives in runc and crun.
  2. High-level runtimes are interchangeable. Because CRI is fixed above and OCI is fixed below, containerd and CRI-O are drop-in alternatives from the kubelet’s point of view. A cluster can run containerd on some nodes and CRI-O on others; pods schedule identically.
  3. The intermediate layers can be removed. The 2022 removal of dockershim in Kubernetes 1.24 collapsed the old kubelet → dockershim → Docker → containerd → shim → runc chain to kubelet → containerd → shim → runc, surfacing the OCI runtime as the obvious bottom of the stack (Dockershim Removal FAQ). Operationally nothing changed for workloads, precisely because the OCI contract had been holding the whole time and Docker was a redundant intermediary.

This is also why “sandboxed” runtimes plug in cleanly: gVisor’s runsc (a user-space kernel) and Kata’s kata-runtime (a per-pod lightweight VM) are low-level runtimes that consume the same bundle. From the high-level runtime’s perspective they are just another OCI CLI to exec; from the kubelet’s perspective the choice is made per-pod via RuntimeClass. The isolation strength differs wildly, but the contract is identical.

Failure Modes and Common Misunderstandings

“Docker is a container runtime.” Docker Engine is a high-level runtime (and more — a build tool, a CLI, an image store). Under the hood it uses containerd and runc. Conflating “Docker” with “the runtime” is what made the dockershim removal sound alarming; in reality Kubernetes had been bypassing Docker’s high-level features and using containerd→runc for years.

“runc is a daemon you can monitor.” It is not. ps will rarely show a runc process for a running container, because runc exited after execve. The resident process you will see is the shim (containerd-shim-runc-v2) or conmon. Looking for runc to confirm a container is alive is a category error.

Cgroup-driver mismatch across the boundary. The high-level runtime, the low-level runtime, and the kubelet must agree on the cgroup driver (systemd vs cgroupfs). A mismatch is silent: containers start, but their resource limits are written to a cgroup path nobody enforces, so limits “don’t apply” until an OOM at unexpected memory. The fix is to align --cgroup-driver end to end. (Detailed in runc.)

Stale shim or conmon after a crash. If a node hard-crashes, a shim or conmon may persist while its container is gone, or vice versa. The high-level runtime’s garbage collection reconciles this on restart by re-attaching to live shims and reaping dead ones; “phantom” containers visible only via crictl ps are the symptom of incomplete reconciliation.

Expecting the low-level runtime to do networking. runc gives a fresh container only a loopback interface. If a container has no external network, the bug is in the high-level runtime’s CNI invocation, not in runc — networking is above the OCI boundary by design.

Alternatives and When the Distinction Matters

For the low-level layer, the practical choice is runc (default, Go, reference), crun (C, low-memory, default on RHEL 9 / Fedora / OpenShift 4.18+), or youki (Rust, research/early-production). All are namespace+cgroup runtimes; the sandboxed runsc (gVisor) and kata-runtime trade performance for stronger isolation and are selected per-workload via RuntimeClass. The honest comparison of all five is in runc.

For the high-level layer, the choice is containerd (the dominant default across EKS/GKE/AKS/kubeadm/k3s) or CRI-O (Red Hat’s minimalist, Kubernetes-only runtime, the only supported runtime on OpenShift 4). containerd is general-purpose (it also backs Docker and has a non-CRI API); CRI-O ships only what Kubernetes needs. The deep treatments are in containerd and CRI-O.

The distinction matters most when you are debugging “why won’t my container start,” because the answer determines which component to inspect: an image-pull or storage failure is a high-level-runtime problem (crictl images, the snapshotter, the content store); a “container created but the entrypoint crashed with EACCES” is a low-level-runtime / bundle problem (the config.json’s capabilities or seccomp). Knowing the boundary tells you which logs to read — journalctl -u containerd/crio for the high level, /run/runc/<id>/state.json and the shim/conmon logs for the low level.

Production Notes

In production Kubernetes the visible bottom-of-stack components are containerd-shim-runc-v2 (containerd) or conmon (CRI-O) processes, one per pod or per container respectively — not runc. A useful operational tell: counting containerd-shim-runc-v2 processes gives you a pod count, while counting conmon processes on a CRI-O node gives you a container count. The per-pod shim model is materially cheaper at high pod density, which is part of why Kata and gVisor moved to shimv2 — the old 2N+1 shim explosion was a real memory cost on dense nodes.

The Red Hat ecosystem’s migration of its low-level default from runc to crun (OpenShift 4.18, RHEL 9) was operationally transparent — existing pod specs ran identically — which is the clearest demonstration that the OCI boundary holds in practice. The same neutrality means a managed-Kubernetes provider can swap the low-level runtime under you in a kubelet upgrade without a workload-visible change. When you pin behavior, pin it at a contract boundary (the pod spec, the image), not at an implementation (a specific runtime binary), because the implementations are designed to be swapped.

See Also

Graph View

Backlinks