vault

Container Runtime Interface

13 min read

Container Runtime Interface

The Container Runtime Interface (CRI) is the gRPC contract that decouples the kubelet from any specific container runtime. It is two gRPC services — RuntimeService (Pod sandbox and container lifecycle, plus exec/attach/port-forward) and ImageService (image pull/list/remove/status) — defined in protocol-buffers form in the kubernetes/cri-api repository, served by the runtime over a UNIX socket, and called by the kubelet exclusively. CRI was introduced in K8s 1.5 (December 2016) per the original Kubernetes blog announcement to end the era of in-tree, hard-wired runtime support, and it became the runtime interface in K8s 1.24 (May 2022) when the dockershim — the in-tree shim that adapted Docker Engine’s pre-CRI API to CRI — was removed. Today every conformant Kubernetes runtime (containerd, CRI-O, gVisor’s runsc via shim, Kata Containers via shim) is a CRI implementation; nothing else is even an option. The kubelet ships with CRI as its only runtime integration.

Mental Model

flowchart LR
    KUBELET[kubelet<br/>gRPC client]
    SOCK[(UNIX socket<br/>unix:///run/containerd/containerd.sock<br/>or /run/crio/crio.sock)]
    subgraph "CRI runtime (containerd / CRI-O / ...)"
        RUNTIMESVC[RuntimeService<br/>RunPodSandbox<br/>CreateContainer / StartContainer<br/>Exec / Attach / PortForward<br/>...]
        IMAGESVC[ImageService<br/>PullImage<br/>ListImages<br/>RemoveImage<br/>ImageStatus]
    end
    OCI[OCI runtime<br/>runc / crun / runsc]
    NS[Pod's network + IPC namespaces<br/>owned by the pause container]

    KUBELET -- "gRPC over UDS" --> SOCK
    SOCK --> RUNTIMESVC
    SOCK --> IMAGESVC
    RUNTIMESVC -- "spawn OCI bundle" --> OCI
    OCI --> NS

What this diagram shows. The kubelet is exclusively a gRPC client; the runtime is exclusively a gRPC server. The wire is a UNIX domain socket on the local filesystem (TCP is supported but almost never used — there’s no reason to). The runtime then internally calls down to an OCI low-level runtime (runc, crun, or runsc for sandboxed) to actually spawn the container’s processes inside Linux namespaces and cgroups. The insight to extract: CRI is just a wire format — the runtime is free to implement it any way it likes, and runtimes vary in their internal architecture (containerd has a CRI plugin, CRI-O is purpose-built CRI-only). What every runtime must agree on is the two gRPC services’ semantics.

Why CRI Exists — The Dockershim Saga

Before K8s 1.5, the kubelet had in-tree integrations: one Go package per runtime, each calling into the runtime’s native API directly. Docker was the original; rkt (CoreOS’s reaction to Docker) was the second. The integration code lived inside the kubelet binary and required deep kubelet expertise to modify. The 2016 blog post lists three pains: (1) every new runtime required modifying kubelet source, (2) the kubelet’s internal Pod-management interfaces weren’t stable so runtime integrations broke on every minor version, (3) the maintenance burden on the K8s maintainers was unsustainable. CRI was the fix: define a small, versioned, gRPC-shaped interface; require every runtime to implement that; let the kubelet code talk to a single, stable surface.

Docker Engine, however, was the original runtime and predated CRI. To support it during the transition, K8s shipped dockershim — an in-tree CRI server (so the kubelet still talked CRI to it) that translated CRI calls into Docker Engine’s REST API. dockershim was always documented as temporary. The deprecation timeline, well-summarised by the Kubernetes 2022-02-17 dockershim FAQ:

  • K8s 1.20 (December 2020): dockershim deprecated; kubelet logs a warning on every start.
  • K8s 1.21, 1.22, 1.23: continued deprecation warnings; community-maintained external dockershim replacements (Mirantis cri-dockerd) emerge.
  • K8s 1.24 (May 2022): dockershim removed from the kubelet entirely. Clusters using Docker Engine as their runtime had to migrate to containerd or CRI-O (both of which Docker Engine itself uses underneath, so the migration was usually invisible to images) or install cri-dockerd as a CRI shim if they really wanted to keep using dockerd.

The migration was loudly communicated and widely misunderstood. The crucial clarification the K8s team repeated dozens of times: “Docker images” and “Docker Engine” are not the same thing. The OCI image format that docker build produces is the same format every CRI runtime consumes; removing dockershim removes the daemon, not the image format. Every docker build-produced image continues to run on containerd or CRI-O without change. The deprecation only affected operators who specifically depended on dockerd being present on K8s nodes (for diagnostic tools like docker ps, or for the now-superseded approach of mounting /var/run/docker.sock into a Pod). For application engineers, the migration was effectively a no-op.

The Two gRPC Services

The CRI proto is defined in kubernetes/cri-api/pkg/apis/runtime/v1/api.proto. Since K8s 1.26 the kubelet requires the runtime to support the v1 CRI API; nodes whose runtime is too old (only v1alpha2) will fail to register. The two services:

RuntimeService

Manages PodSandboxes and containers:

service RuntimeService {
    // --- PodSandbox lifecycle (the per-Pod infrastructure container, == the pause container) ---
    rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandboxResponse) {}
    rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSandboxResponse) {}
    rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse) {}
    rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSandboxStatusResponse) {}
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {}
 
    // --- Container lifecycle (one or more per sandbox) ---
    rpc CreateContainer(CreateContainerRequest) returns (CreateContainerResponse) {}
    rpc StartContainer(StartContainerRequest) returns (StartContainerResponse) {}
    rpc StopContainer(StopContainerRequest) returns (StopContainerResponse) {}
    rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse) {}
    rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {}
    rpc ContainerStatus(ContainerStatusRequest) returns (ContainerStatusResponse) {}
    rpc UpdateContainerResources(...) returns (...) {}   // for in-place Pod resize
 
    // --- Streaming RPCs (the kubelet proxies these to kubectl exec/attach/port-forward) ---
    rpc Exec(ExecRequest) returns (ExecResponse) {}
    rpc Attach(AttachRequest) returns (AttachResponse) {}
    rpc PortForward(PortForwardRequest) returns (PortForwardResponse) {}
 
    // --- Status / events ---
    rpc Status(StatusRequest) returns (StatusResponse) {}
    rpc GetContainerEvents(GetEventsRequest) returns (stream ContainerEventResponse) {}  // evented PLEG
}

ImageService

service ImageService {
    rpc ListImages(ListImagesRequest) returns (ListImagesResponse) {}
    rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResponse) {}
    rpc PullImage(PullImageRequest) returns (PullImageResponse) {}
    rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResponse) {}
    rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResponse) {}
}

The runtime is free to serve both services on one socket (the common case — containerd and CRI-O both do) or on two separate sockets (the kubelet config exposes containerRuntimeEndpoint and imageServiceEndpoint independently for this reason). Two-socket setups are used when image management is delegated to a separate process — e.g. on some embedded/edge deployments where image storage lives on a peer node.

The PodSandbox Abstraction

The single most important CRI concept is the PodSandbox. A PodSandbox is an isolated environment that future containers will share — concretely, on Linux it is the set of Linux namespaces (network, IPC, sometimes PID) and the cgroup hierarchy that the Pod’s containers will join. The runtime models this as a stand-alone container — the Pause Container — whose entire job is to exist, hold the namespaces open via its PID, and do nothing else. (Its image is ~750 KB; the entry-point is a tight pause(2) loop that ignores all signals except SIGTERM. The Linux kernel keeps a namespace alive as long as some process is in it.)

When the kubelet starts a Pod, the call sequence is:

  1. RunPodSandbox(config) — runtime creates the pause container in the namespaces, calls the CNI plugin to wire up Pod networking inside that net namespace, returns a PodSandboxId.
  2. PullImage(image) for each container’s image (cached if already present).
  3. CreateContainer(podSandboxId, containerConfig, sandboxConfig) — runtime creates the OCI bundle but does not start it.
  4. StartContainer(containerId) — runtime invokes the OCI runtime (runc start) to begin the container’s processes inside the sandbox’s namespaces.
  5. Repeat 3–4 for each container in the Pod.

Container teardown reverses the order: StopContainer then RemoveContainer for each container, then StopPodSandbox (which un-wires the network via CNI and stops the pause container), then RemovePodSandbox. The sandbox-first ordering is what lets all the Pod’s containers share network and IPC — they all join the namespaces the pause container is already holding open.

The abstraction is deliberately generic. VM-based runtimes (Kata Containers, gVisor’s runsc, Firecracker via firecracker-containerd) implement a PodSandbox as a lightweight VM rather than a Linux namespace bundle. The CRI semantics are identical: same gRPC, same lifecycle, same Pod model. The kubelet doesn’t care whether the sandbox is a pause process plus net-ns or a Firecracker microVM — only that RunPodSandbox returns a sandbox ID that CreateContainer can attach to.

Configuration / Wire-Level View

The kubelet picks the runtime via two flags (or their config-file equivalents):

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
imageServiceEndpoint:     unix:///run/containerd/containerd.sock    # often the same socket

Defaults on the major runtimes:

RuntimeSocket
containerdunix:///run/containerd/containerd.sock
CRI-Ounix:///run/crio/crio.sock
cri-dockerd (Mirantis)unix:///var/run/cri-dockerd.sock

Debugging tools that speak CRI directly:

  • crictl — the kubectl-equivalent for the CRI socket; ships with K8s; reads /etc/crictl.yaml for the socket path. crictl ps, crictl pods, crictl logs <id>, crictl exec. This is the canonical “the kubelet says my container is wedged, what does the runtime see” tool.
  • ctr — containerd’s own CLI (not CRI-aware; talks to containerd’s lower-level namespaced API directly). Useful for inspecting images and snapshots.
  • crio CLI / runc: similar runtime-specific tools.

A worked CRI call from crictl shows what the kubelet itself sends:

$ sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock pods
POD ID         CREATED       STATE   NAME                        NAMESPACE    ATTEMPT  RUNTIME
e6b2...        2 hours ago   Ready   coredns-7db6d8ff4d-2sjx9    kube-system  0        runc
fd31...        2 hours ago   Ready   etcd-control-plane          kube-system  0        runc

Each row corresponds to one PodSandboxStatus return; the kubelet builds its .status for each Pod object from this exact data.

Failure Modes

CRI handshake errors at kubelet start. Kubelet logs Failed to connect to the container runtime or error: rpc error: code = Unimplemented on first call. Causes: (a) the runtime is not running (systemctl status containerd); (b) the socket path in containerRuntimeEndpoint doesn’t match the runtime’s actual socket; (c) on K8s 1.26+ the runtime is too old to speak v1 CRI and only supports v1alpha2 — the kubelet refuses to register the Node. Fix: upgrade the runtime; in particular containerd ≥ 1.6 and CRI-O ≥ 1.26 are the floors for K8s 1.26.

crictl works but the kubelet doesn’t. The kubelet’s gRPC client may be using a stale config — e.g. after switching from dockershim, the kubelet config still points at the old socket. journalctl -u kubelet shows the endpoint it’s actually trying.

Sandbox stuck in NotReady. crictl pods shows NotReady for a Pod’s sandbox. Usually a CNI plugin failure: the runtime called CNI to wire up the network and CNI returned an error (crictl logs --previous is unrelated — sandbox-level errors show in the runtime’s own log, e.g. journalctl -u containerd). Fix the CNI configuration before the kubelet will retry the sandbox.

Image pull failures. crictl pull <image> reproduces the same call the kubelet would make. Wrong registry credentials in ~/.config/containers/auth.json (CRI-O) or /etc/containerd/config.toml’s registry.configs.<host>.auth section (containerd) are the typical cause.

Streaming RPC failures (kubectl exec hangs). Exec/Attach/PortForward in CRI work in two stages: the kubelet calls Exec, the runtime returns a streaming URL, the API server proxies the client’s stdin/stdout/stderr to that URL. If the runtime’s streaming server is firewalled off from the API server (e.g. it binds to localhost only), kubectl exec hangs. Diagnose by checking the runtime’s streaming_server_address config.

v1alpha2 / v1 protobuf mismatch. Before K8s 1.26 the kubelet would fall back to v1alpha2 if v1 wasn’t available. From 1.26 onward this fallback was removed; mixed-version clusters where the kubelet is upgraded ahead of the runtime can refuse to register Nodes. Always upgrade the runtime first or in lockstep.

Alternatives and When to Choose Them

Within the CRI universe, the runtime options are siblings (see containerd and CRI-O for full comparisons). At a higher level, alternatives to “use CRI” are:

  • cri-dockerd (Mirantis): a community-maintained continuation of the dockershim, repackaged as an external CRI server that talks to Docker Engine underneath. Use if your operational tooling assumes dockerd is on every node and the migration cost is too high; otherwise prefer containerd.
  • Run-VM runtimes via CRI: Kata Containers, gVisor (runsc), and AWS Firecracker (via firecracker-containerd) are not CRI servers themselves — they plug in below containerd or CRI-O as alternative OCI runtimes (“runtime classes”). The kubelet still talks CRI; the runtime then dispatches based on the Pod’s runtimeClassName to runc-by-default or kata/runsc for VM-isolated Pods.
  • Don’t use Kubernetes: outside K8s, container management uses different stacks. Podman talks to OCI runtimes directly (no CRI). systemd-nspawn doesn’t use CRI. Docker Compose talks to Docker Engine. These are outside the K8s context.

For Kubernetes specifically, there is no alternative to CRI — it is the only kubelet→runtime interface that exists since 1.24. The choice is which CRI implementation, not whether to use CRI.

Production Notes

  • The Spotify / Shopify / Datadog dockershim migrations were public case studies. Each org swapped from dockershim to containerd in a rolling fashion: drain a node, replace the node’s image (or apt install containerd && systemctl restart kubelet), uncordon. The image layer stayed identical; the only operational change was that docker exec on the node no longer worked and was replaced by crictl exec for in-place debugging.
  • The CRI image storage is per-runtime. When migrating from Docker Engine to containerd, the image cache at /var/lib/docker/ is not read by containerd, which uses /var/lib/containerd/. Migration tooling re-pulls all images on first kubelet start — bandwidth-bound on slow networks, harmless on cloud nodes.
  • CRI v2 / “v2 shim” is unrelated. The “shim v2” terminology in containerd refers to the per-container shim protocol below CRI (the ttrpc-based protocol between containerd and runc), not to a “CRI v2” — there is no such thing yet. The kubelet’s CRI is on v1.
  • List streaming (CRIListStreaming, K8s 1.36 alpha). On nodes with many thousands of containers, ListPodSandbox / ListContainers / ListImages responses can exceed gRPC’s default 16 MiB message limit. K8s 1.36 introduces three new streaming RPCs — StreamPodSandboxes, StreamContainers, StreamImages — that the kubelet uses when the CRIListStreaming feature gate is on and the runtime advertises support (per the CRI overview, marked FEATURE STATE: Kubernetes v1.36 [alpha] disabled by default). These are separate new RPCs, not server-side streaming bolted onto the unary List*; the kubelet falls back to the unary calls when the gate is off or the runtime doesn’t implement them. Relevant to bare-metal hyperscale operators running hundreds of Pods per node.
  • Evented PLEG / GetContainerEvents — a cautionary tale of beta walked back to alpha. The kubelet historically polled the runtime via ListPodSandbox + ListContainers once per second per node (the Generic PLEG, Pod Lifecycle Event Generator) to detect container state changes. KEP-3386 — Kubelet Evented PLEG replaced that polling with the streaming GetContainerEvents RPC: the runtime pushes container lifecycle events to the kubelet as they happen. The feature went alpha in K8s 1.26, beta (still off-by-default) in K8s 1.27, and was then reverted to alpha in K8s 1.30 — and backported to v1.27.9, v1.28.6, and v1.29.1 — after a regression where static Pods failed to start and the kubelet’s pod-status cache fell out of sync (see containerd #9070). Stabilization work re-targeted beta for K8s 1.33 per the KEP. Runtime support floors per the Kubernetes docs: containerd ≥ 1.7 or CRI-O ≥ 1.26 (CRI-O additionally needs enable_pod_events = true). Operationally: leave the gate off in production unless you’re explicitly chasing PLEG-bound latency on dense nodes, because the failure mode (kubelet’s state cache silently drifts) is much worse than the polling overhead it replaces.

See Also

Graph View

Backlinks