Ephemeral Containers

Ephemeral containers are temporary containers injected into a running Pod for user-initiated debugging. They’re not for running application workloads — they exist to give an operator a shell inside a Pod that lacks one, to introspect a misbehaving container’s filesystem and processes, or to attach a debugger to a process that’s about to crashloop again. The Kubernetes documentation is emphatic about their purpose: “An ephemeral container is a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. You use ephemeral containers to inspect services rather than to build applications” (kubernetes.io — Ephemeral Containers). They were added to the API to solve a specific operational pain: in production, Pod images are increasingly minimal — distroless, scratch-based, or stripped of every binary not needed by the app — and kubectl exec can’t add binaries that don’t exist. Ephemeral containers are how you bring in bash, curl, tcpdump, or a language-specific debugger without rebuilding the Pod. The feature was KEP-277, proposed in 2017, graduating through alpha (1.16, 2019), beta (1.23, 2021), and finally stable (GA) in K8s 1.25 (v1.25 release notes, August 2022). The user-facing entry point is kubectl debug, which wraps the underlying ephemeralcontainers subresource call.

Mental Model

flowchart LR
    USER[Operator runs<br/>kubectl debug -it pod/x<br/>--image=busybox<br/>--target=app]
    APISERVER[(kube-apiserver)]
    SUBRES[ephemeralcontainers<br/>subresource<br/>PATCH /pods/x/ephemeralcontainers]
    POD[Pod x on Node N<br/>existing containers:<br/>app, sidecar]
    KUBELET[kubelet on Node N]
    NEW[debug container<br/>busybox image<br/>shares: net, IPC, volumes<br/>shares: PID namespace<br/>with target via --target=app]
    USER --> APISERVER
    APISERVER --> SUBRES
    SUBRES -- "watch event:<br/>ephemeralContainers&#91;&#93;<br/>modified" --> KUBELET
    KUBELET -- "create container<br/>in existing Pod<br/>sandbox" --> NEW
    NEW -.->|shareProcessNamespace<br/>can see app's PIDs| POD

What this diagram shows. Unlike spec.containers[] or spec.initContainers[], ephemeral containers can only be added via the ephemeralcontainers subresource (the K8s docs: “Ephemeral containers are created using a special ephemeralcontainers handler in the API rather than being added directly to pod.spec, so it’s not possible to add an ephemeral container using kubectl edit”). The user typically doesn’t call the subresource directly — kubectl debug does it. The kubelet, watching the Pod, sees a new entry appear in status.ephemeralContainerStatuses and creates the container inside the existing Pod sandbox: same network namespace, same IPC namespace, same volume mounts. When invoked with --target=<container>, kubectl also enables process-namespace sharing so the new container can see and signal processes in the target container — the key capability for debugging a distroless app whose PID 1 you need to strace. The insight to extract is that an ephemeral container is a process injected into a running Pod, not a new Pod being scheduled; it’s the closest thing K8s offers to “ssh into a running container” while still going through the API server (so the action is audited and authorized by RBAC).

Mechanical Walk-through

Why ephemeral and not just kubectl exec?

kubectl exec runs a command inside an existing container’s filesystem — so the command must already exist in the image. The trend toward minimal/distroless production images (Google distroless) removes sh, ls, ps, cat, curl, and every other shell utility from production images for security and size. On such an image, kubectl exec -- /bin/sh fails with “executable file not found.” Ephemeral containers solve this by running a different image (one that has the tools) inside the same Pod, with access to the target’s network namespace, volumes, and (optionally) PID namespace.

The ephemeralcontainers subresource

Ephemeral containers are exposed as a dedicated subresource on Pod (/api/v1/namespaces/{ns}/pods/{name}/ephemeralcontainers) — separate from /pods/{name} itself. This split exists because:

1. spec.containers[] and spec.initContainers[] are immutable after Pod creation. The Pod spec is created once and never edited; subsequent edits would conflict with the controller-managed reconciliation. Ephemeral containers can’t go into spec for the same reason.
2. Authorization is separable. A role can grant pods/ephemeralcontainers permission without granting full pods update — letting on-call engineers debug Pods they couldn’t otherwise modify.
3. The kubelet watches the Pod object and notices the appended spec.ephemeralContainers[] field (populated server-side from the subresource) and creates the container.

You cannot kubectl edit pod to add an ephemeral container; the API server rejects the modification. You must use the subresource, which kubectl debug does for you.

Restrictions

Ephemeral containers are intentionally a minimal-capability container:

Field / Capability	Allowed on ephemeral container?
image, command, args, env, volumeMounts	Yes
securityContext, tty, stdin	Yes
lifecycle (preStop/postStart hooks)	No
livenessProbe / readinessProbe / startupProbe	No
ports	No
resources (requests/limits)	No — Pod’s resource allocation is fixed at creation and not re-computed when ephemeral containers attach
Restart on failure	No — once the ephemeral container exits, it’s gone; the Pod’s lifecycle is unaffected
Remove the ephemeral container after debug session	No — once added, it cannot be removed from the Pod; only Pod deletion clears it

The K8s docs state these restrictions in their own words. On fields: “Ephemeral containers may not have ports, so fields such as ports, livenessProbe, readinessProbe are disallowed” and “Pod resource allocations are immutable, so setting resources is disallowed” (kubernetes.io — Ephemeral Containers, as of K8s v1.36 docs). On lifecycle: “Ephemeral containers differ from other containers in that they lack guarantees for resources or execution, and they will never be automatically restarted, so they are not appropriate for building applications.” And on mutability, the docs are explicit: “Like regular containers, you may not change or remove an ephemeral container after you have added it to a Pod.” That last sentence is the authoritative basis for the “no remove” rule, and it is especially important operationally: each kubectl debug call appends a new entry to spec.ephemeralContainers[] (with a unique name); they accumulate over a Pod’s lifetime. A Pod debugged 10 times has 10 ephemeral container entries in its spec, most of them long-exited. The complete allowed-field set for the type lives in the EphemeralContainer v1 core API reference; the disallowed fields above are enforced by API-server validation, which rejects the request rather than silently dropping them.

kubectl debug modes

kubectl debug has three primary modes (Debug Running Pods):

(1) Attach an ephemeral container to a running Pod. The default; uses the ephemeral-containers subresource.

kubectl debug -it mypod --image=busybox:1.36 --target=mycontainer

The --target=<container> flag enables process namespace sharing between the new ephemeral container and the named target container. Without --target, the ephemeral container can see the Pod’s network and volumes but not the target’s processes — useful for “is the service responding on port 8080?” but not for “what is PID 1 of the app doing?”

(2) Copy the Pod and modify it for debug. Creates a new Pod that’s a copy of the original, optionally with a different image or command.

kubectl debug mypod --copy-to=mypod-debug --image=ubuntu:24.04 --share-processes -- bash

Used when the original Pod is crashlooping too fast for an ephemeral container to be useful (since ephemeral containers can’t pre-empt or pause the target). The copy can have a different command (e.g., sleep infinity instead of the app’s entrypoint) so it stays alive long enough to inspect.

(3) Debug a Node. Runs a Pod on a specific node with hostNetwork, hostPID, and a host mount, for node-level debugging. Outside this note’s scope; see kubectl debug.

Process namespace sharing and --target

By default, each container in a Pod has its own PID namespace (PID 1 in container A is invisible to container B). The Pod-level shareProcessNamespace: true field shares PID namespaces across all containers in a Pod, but it’s a Pod-spec field set at Pod creation — you can’t toggle it on a running Pod.

The --target=<container> flag on kubectl debug is the runtime equivalent: it configures the ephemeral container to share its PID namespace with the named target container only. This is achieved at the container-runtime level (containerd/CRI-O exposes a per-container PID-namespace mode). After --target=app, the ephemeral container’s ps aux shows the target app’s processes; you can kill -USR1 <pid> to trigger heap dumps, strace -p <pid> to trace syscalls, gdb -p <pid> to attach a debugger, etc.

Stability and graduation history

KEP-277 timeline:

• K8s 1.16 (2019): Alpha behind EphemeralContainers feature gate; required setting the gate on both kube-apiserver and kubelet.
• K8s 1.23 (2021): Beta. kubectl debug became the official entry point; the API was reshaped to use the dedicated subresource (replacing an earlier design that polluted the Pod spec directly).
• K8s 1.25 (August 2022): Stable (GA) (release blog, merge PR). Feature gate is now always-enabled and deprecated.
• Conformance: Ephemeral-container creation is now part of K8s conformance tests; managed services (EKS, GKE, AKS) all support the feature on 1.25+.

Configuration / API Surface

The most common operator workflow:

# Step 1: A Pod is doing something wrong. We have only its name.
$ kubectl get pod payments-67f8c
NAME             READY   STATUS    RESTARTS   AGE
payments-67f8c   1/1     Running   0          18m
 
# Step 2: kubectl exec doesn't work because the image is distroless —
# /bin/sh doesn't exist.
$ kubectl exec -it payments-67f8c -- /bin/sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown
 
# Step 3: Inject a debug container. The --target=app flag shares the
# app container's PID namespace so we can see its processes.
$ kubectl debug -it payments-67f8c \
    --image=nicolaka/netshoot:latest \
    --target=app
Targeting container "app". If you don't see a command prompt, try pressing enter.
bash-5.2# ps aux | grep -v ps
PID   USER    TIME   COMMAND
    1 nonroot 0:42   /payments-server --port 8080
   17 root    0:00   bash
bash-5.2# tcpdump -ni any -c 10 'port 8080'
... captures live traffic to/from the app process on port 8080 ...
bash-5.2# strace -p 1 -e trace=network -f
... attaches to the app process and traces network syscalls ...
bash-5.2# exit
 
# Step 4: The ephemeral container exited but its entry remains in the Pod spec.
$ kubectl get pod payments-67f8c -o jsonpath='{.spec.ephemeralContainers[*].name}'
debugger-7xqwp
 
# Step 5: To "clean up", the Pod itself must be deleted — the ephemeral
# container entry cannot be removed in place.

Equivalent raw API call (for understanding what kubectl debug does under the hood):

# PATCH /api/v1/namespaces/default/pods/payments-67f8c/ephemeralcontainers
# Content-Type: application/strategic-merge-patch+json
{
  "spec": {
    "ephemeralContainers": [
      {
        "name": "debugger-7xqwp",            # auto-generated unique name
        "image": "nicolaka/netshoot:latest",
        "stdin": true,
        "tty": true,
        "targetContainerName": "app",        # the --target flag
        "command": ["bash"]
      }
    ]
  }
}

The targetContainerName field is the API-level form of --target; it triggers process-namespace sharing with the named container. There’s no ports, resources, livenessProbe, or lifecycle field — the API server rejects them.

Failure Modes

1. Ephemeral container fails on a target Pod that has already terminated. Ephemeral containers attach to a running Pod’s sandbox. If the target container crashlooped and the whole Pod is gone, there’s nothing to attach to. If the Pod’s main containers have all exited but the Pod still exists in Failed state, the ephemeral container can still be added (it’ll see the corpse). For genuine post-mortem on a Pod that’s gone, use the Kubernetes Events history and previous container logs (kubectl logs --previous).

2. --target flag missing. Without --target, the ephemeral container can see the Pod’s network namespace (so curl localhost:8080 works) and volume mounts, but cannot see the target’s processes (no ps, strace, or gdb against the app). Solution: always specify --target=<container-name> when you need process visibility.

3. Distroless target with no shell, ephemeral container with command: ["/bin/sh"], but --target is set so process namespace is shared. This is fine — the ephemeral container’s own image is the busybox/netshoot one that has /bin/sh. The target’s lack of shell is irrelevant; what matters is the ephemeral container’s image.

4. Ephemeral container entries accumulate. Each invocation appends an entry. A long-lived Pod debugged repeatedly bloats its spec. The spec field has no built-in pruning; the only cleanup is Pod deletion. Operationally this rarely matters — most debuggable Pods get replaced shortly after — but it can surprise audit tooling that flags “this Pod has 47 ephemeral containers.”

5. RBAC denies pods/ephemeralcontainers. The pods/ephemeralcontainers subresource is a separate authorization target from pods. A user with get/list/watch pods but without update pods/ephemeralcontainers cannot use kubectl debug. The error is a 403 “forbidden: cannot update resource ephemeralcontainers.” Solution: add the subresource to the role.

6. Pod resources don’t include the ephemeral container. The Pod’s requests/limits were set at creation; ephemeral containers do not contribute. If the node has 0 MiB free and the ephemeral container is memory-hungry (a JVM debug image is ~500 MiB), the ephemeral container competes for the node’s overhead memory and may OOM-kill peers. Mitigation: use lightweight debug images (busybox, distroless-debug, netshoot) and prefer the --copy-to pattern for memory-heavy diagnostics.

7. Trying to debug a static Pod. Per the K8s docs, ephemeral containers are not supported in Static Pods (kubelet-managed Pods like control-plane components on kubeadm clusters). The --copy-to mode also doesn’t help here because static Pods are file-driven, not API-driven. For control-plane debugging, you generally need node-level access (SSH and crictl).

8. TTY allocation failure. If the ephemeral container exits immediately (e.g., the image has no bash), kubectl debug -it hangs or returns a confusing TTY error. Verify the image has the expected shell, or set --command -- /bin/somebinary explicitly.

9. --target silently produces an isolated process namespace on an unsupporting runtime. --target (the API’s targetContainerName) requires the container runtime to support process-namespace targeting. The official docs warn that without runtime support, “the Ephemeral Container may not be started, or it may be started with an isolated process namespace so that ps does not reveal processes in other containers” (Debug Running Pods). The dangerous failure is the silent-degradation case: the debug container starts, you ps aux, and you see only the debug container’s own processes — not the app’s — even though you passed --target. Modern containerd and CRI-O support targeting, so this is rare on current clusters, but it has bitten users on older or non-standard runtimes (kubernetes/kubernetes#98362). Diagnose by checking whether the target app’s PID appears in the debug container’s ps; if not, the runtime did not honor targeting.

Alternatives and When to Choose Them

• kubectl exec. Choose first if the target image has a shell and the tools you need. Faster, doesn’t pollute the Pod spec, no RBAC subresource needed.

• kubectl logs --previous. Choose for post-mortem on a container that has already crashed. Logs are available even after the container is gone, as long as the Pod exists.

• kubectl debug --copy-to=.... Choose when the original Pod is unstable (crashlooping, OOMing) and won’t survive long enough for ephemeral-container attachment. The copy can use a different command (e.g., sleep infinity) to stay alive for inspection.

• kubectl debug node/<nodename>. Choose for node-level issues (kernel, kubelet, networking). Creates a host-privileged Pod with the node’s filesystem mounted; outside the per-Pod ephemeral-container model.

• Pre-baked debug image deployed as a sidecar. Choose for recurring debugging needs that justify shipping the debug tools in every Pod. Cost: per-Pod memory tax and increased attack surface. Most teams reject this in favor of ephemeral containers precisely because the latter doesn’t tax steady-state Pods.

• crictl directly on the node. Choose for deep debugging when even the API-server-mediated path doesn’t suffice (e.g., container runtime bug). Requires SSH to the node and is outside the K8s authorization model — generally a break-glass action.

• In-process diagnostics (Java JFR, Go pprof endpoints). Choose to expose runtime introspection from inside the app itself, accessible via kubectl port-forward. Doesn’t need ephemeral containers at all if the app exposes the right endpoints. Most languages have an equivalent (pprof in Go, JMX in Java, py-spy in Python).

Production Notes

• nicolaka/netshoot (github.com/nicolaka/netshoot) is the de facto standard “kitchen sink” debug image. Ships with tcpdump, dig, nslookup, curl, iperf, nmap, mtr, strace, tshark, nc, and dozens of others. Most operators have it aliased: alias kdebug='kubectl debug -it --image=nicolaka/netshoot:latest --target=app'.

• Distroless + ephemeral containers is the modern security-conscious pattern. Production Pods ship with the smallest possible image (no shell, no busybox, no package manager); operators inject debugging capability on demand. The combination achieves both small attack surface in steady state and full debuggability under incident. Google, Shopify, and many CNCF projects (e.g., Knative) ship distroless and rely on ephemeral containers for debug.

• Audit logging captures ephemeral-container creations. Because the subresource goes through kube-apiserver, the action appears in audit logs with the user identity, the Pod, the image used, and the command. This is a meaningful improvement over node-level docker exec (which bypasses K8s) — security teams can correlate debug sessions with incidents.

• Managed K8s nuances. EKS, GKE, and AKS all support ephemeral containers from 1.25+; GKE Autopilot has additional security constraints (which images are allowed) on top. AWS Fargate runs Pods in micro-VMs and historically had restrictions on shared-process-namespace debugging; check current EKS docs for your version.

• k8s.af failure stories include several where ephemeral containers would have helped but the team didn’t know about them — debugging cycles of “rebuild image with debug tools, redeploy, reproduce, repeat” wasted hours when a single kubectl debug would have answered the question. Awareness, not capability, is usually the bottleneck.

The disallowed-field set is stable on GA clusters. As of the K8s v1.36 documentation, the concept doc names exactly ports, livenessProbe, readinessProbe (grouped under “may not have ports”), and resources as disallowed, and the EphemeralContainer v1 core reference is the authoritative allow-list — the API server validates against it and rejects the request. (startupProbe and lifecycle are likewise not meaningful on a never-restarted, port-less container and are rejected by the same validation.) Since the feature reached GA in 1.25, this validation has been part of the stable API contract; the earlier alpha/beta churn in 1.16–1.24 is no longer relevant to any supported cluster (the oldest supported minors in 2026 are well past 1.25).

See Also

• Pod — the parent resource; ephemeral containers attach to a running Pod
• Init Containers — run-to-completion containers at Pod startup (different purpose)
• Sidecar Containers — long-running auxiliary containers (different purpose)
• kubectl — the CLI; kubectl debug is the entry point for ephemeral containers
• kubectl debug — sibling note with deeper coverage of debug modes and node debugging
• Kubernetes RBAC — the authorization model that gates the pods/ephemeralcontainers subresource
• Pod Lifecycle and Containers Status — how container statuses surface; ephemeral container status appears in status.ephemeralContainerStatuses[]
• Kubernetes Events — complementary diagnostic stream
• Kubernetes Audit Logging — how ephemeral-container creations are captured
• Static Pods — explicitly not supported by ephemeral containers
• Pod Probes — explicitly disallowed on ephemeral containers
• Kubernetes MOC — umbrella index