Pod Lifecycle and Containers Status

This note is the field-debugging companion to Pod Lifecycle. Where that note is the theory — the phase/condition/state machine every Pod traverses — this note is the practice: the concrete skill of pointing kubectl describe pod and kubectl get pod -o yaml at a misbehaving Pod and reading the symptoms correctly. The substrate of that skill is the status block of the Pod object: status.phase (one coarse word), status.conditions[] (milestone signals), and status.containerStatuses[] (the per-container detail where the actual diagnosis lives). Inside containerStatuses[] the load-bearing fields are state (current: waiting / running / terminated), lastState (the previous termination — what killed it before it restarted), restartCount, and the reason/exitCode codes (ImagePullBackOff, CrashLoopBackOff, OOMKilled, exit 137, exit 1). Almost every “why is my Pod broken” question reduces to reading two or three of these fields and recognizing a named pattern. This note is a diagnostic field guide. For the lifecycle theory read Pod Lifecycle first; for acting on a running broken Pod see kubectl debug.

Mental Model

The Pod status block is a three-layer hierarchy of increasing detail. phase answers “broadly, where is this Pod” in one word. conditions[] answers “which milestones has it passed.” containerStatuses[] answers “what is each container actually doing, and what happened to it last time.” Debugging means reading bottom-up: the phase is a hint, the container status is the answer.

flowchart TD
    DESC["kubectl describe pod / get pod -o yaml"]
    DESC --> PHASE["status.phase<br/>Pending | Running | Succeeded | Failed | Unknown<br/>(coarse — a hint only)"]
    DESC --> COND["status.conditions[]<br/>PodScheduled, Initialized,<br/>ContainersReady, Ready<br/>(which milestones passed)"]
    DESC --> CS["status.containerStatuses[]<br/>(THE diagnostic layer)"]
    CS --> STATE["state: waiting | running | terminated<br/>+ reason / exitCode"]
    CS --> LAST["lastState: terminated<br/>+ reason / exitCode / finishedAt<br/>(what killed the PREVIOUS run)"]
    CS --> RC["restartCount<br/>(how many times this has happened)"]
    STATE --> DX{recognize the<br/>named pattern}
    LAST --> DX
    DX --> A["ImagePullBackOff / ErrImagePull"]
    DX --> B["CrashLoopBackOff (+ read lastState)"]
    DX --> C["OOMKilled / exit 137"]
    DX --> D["CreateContainerConfigError"]
    DX --> E["ContainerCreating (stuck)"]

The debugging decision tree. The insight: phase and even state are not the diagnosis — lastState usually is. A Pod stuck in CrashLoopBackOff has state.waiting.reason=CrashLoopBackOff, which only says “it keeps dying.” Why it dies is in lastState.terminated — reason=OOMKilled, exitCode=137 is a completely different fix from reason=Error, exitCode=1. Reading state without lastState is the most common diagnostic mistake.

Mechanical Walk-through

phase vs conditions vs containerStatuses

A quick recap of Pod Lifecycle, because the distinction is the whole foundation:

• status.phase — Pending, Running, Succeeded, Failed, Unknown. Coarse. A Pod is Running whether all containers serve happily or one is crash-looping. Never trust the phase as a health signal.
• status.conditions[] — typed milestone flags: PodScheduled (scheduler placed it), Initialized (init containers done), ContainersReady (all containers report ready), Ready (the Pod-level signal Services route on). Each has status (True/False/Unknown) plus reason and message. A False condition with a reason is often the fastest pointer to the problem (e.g. PodScheduled=False, reason=Unschedulable).
• status.containerStatuses[] — one entry per container, the diagnostic substrate. Mirrored by status.initContainerStatuses[] for init containers.

kubectl describe pod renders all three in human form; kubectl get pod -o yaml gives the raw object.

Reading containerStatuses[].state and .lastState

Each container status carries two state objects. state is the current state; lastState is the previous terminated state (empty if the container has never restarted). Both are one of:

State	Key fields	Meaning
waiting	reason, message	Not yet running — creating, pulling, or backing off after a crash.
running	startedAt	Process is up. Not the same as ready — readiness is a separate probe signal.
terminated	exitCode, signal, reason, startedAt, finishedAt, containerID	Process has exited.

The diagnostic move: when state is waiting with reason=CrashLoopBackOff, the cause is in lastState.terminated — read its exitCode and reason.

kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState}' | jq

restartCount

containerStatuses[].restartCount is the count of times the kubelet has restarted this container. A high and climbing restartCount is the unambiguous signature of a crash loop. The kubectl get pod RESTARTS column is this field; kube_pod_container_status_restarts_total (see kube-state-metrics) is the metric you alert on with a rate() query.

The named diagnostic patterns

ImagePullBackOff / ErrImagePull

state.waiting.reason. The kubelet cannot pull the container image. ErrImagePull is the immediate failure; ImagePullBackOff is the kubelet backing off (exponential, capped ~5 min) before retrying. Causes: image doesn’t exist or tag is wrong, registry unreachable, missing/incorrect imagePullSecrets, rate-limited registry (Docker Hub anonymous pulls). Diagnosis: kubectl describe pod — the Events section (Kubernetes Events) carries the exact registry error (manifest unknown, unauthorized, dial tcp ... timeout).

CrashLoopBackOff

state.waiting.reason. The container starts, exits, and the kubelet backs off (exponential — roughly 10 s, 20 s, 40 s, …, capped at 5 min) before restarting it. CrashLoopBackOff itself says only “it keeps dying.” The cause is always in lastState.terminated plus the container’s own logs:

kubectl logs my-pod --previous   # the dying container's last words — almost always decisive

Common roots: the process exits non-zero on a config error, a missing dependency, an unhandled exception at startup, a failing migration; or it is killed externally (OOM, failing liveness probe — see Liveness Probe Misuse). The exit code in lastState.terminated.exitCode disambiguates.

OOMKilled — exit code 137

lastState.terminated.reason=OOMKilled, exitCode=137. The container exceeded its cgroup memory limit and the Linux OOM killer killed it. 137 = 128 + 9: the 128 offset signals “killed by signal,” 9 is SIGKILL. Fix: raise resources.limits.memory, or fix the leak. Classic cause: a JVM or Node process that ignores the cgroup limit and sizes its heap to the node’s total memory. See Resource Requests and Limits and QoS Classes.

CreateContainerConfigError

state.waiting.reason. The kubelet cannot assemble the container’s configuration — almost always a referenced ConfigMap or Secret that does not exist, or a key missing from one (an env.valueFrom.configMapKeyRef/secretKeyRef pointing at nothing). The image is fine; the spec references something absent. Fix: create the missing object or correct the reference. (Sibling: CreateContainerError — the runtime rejected container creation; InvalidImageName — a malformed image string.)

ContainerCreating — stuck

state.waiting.reason=ContainerCreating is normal and brief while the kubelet sets up the sandbox, mounts volumes, and wires the network. Stuck in it (minutes) means the setup is blocked — almost always volume or CNI: a PersistentVolume that won’t attach/mount, a Secret/ConfigMap volume referencing a missing object, or a CNI plugin failing to allocate a Pod IP. Diagnosis: the Events in kubectl describe pod — FailedMount, FailedAttachVolume, or CNI errors.

Exit codes — signal vs application

lastState.terminated.exitCode tells you how the process ended:

Exit code	Meaning
0	Clean success. With restartPolicy: Never/OnFailure, drives phase Succeeded.
1	Generic application error — an unhandled exception, an assertion, a deliberate exit(1). The app chose to die.
2	Misuse of a shell builtin (often a bad entrypoint script).
126 / 127	Command not executable / command not found — a broken command/args or ENTRYPOINT.
137	128 + 9 (SIGKILL) — OOMKilled, or a grace-period timeout escalating to SIGKILL.
139	128 + 11 (SIGSEGV) — a segfault; native crash.
143	128 + 15 (SIGTERM) — clean termination after K8s sent SIGTERM (normal during rollouts).

The rule of thumb: exit 1 is the app’s fault (read the logs); exit 128+N was a signal — something killed it, and which signal narrows the cause (OOMKilled → memory; SIGTERM → routine shutdown; SIGSEGV → native crash).

The init-container status block

Init containers run sequentially to completion before app containers start (Init Containers). Their statuses are a separate list: status.initContainerStatuses[], with the same state/lastState/restartCount shape. A Pod stuck Pending with Initialized=False is blocked on an init container — and you must read initContainerStatuses[], not containerStatuses[], to see why. A common miss: debugging the app container while an init container is the one in CrashLoopBackOff.

Configuration / API Surface

Reading a broken Pod end-to-end:

# Step 1 — the one-line overview.
kubectl get pod my-pod
# NAME      READY   STATUS             RESTARTS        AGE
# my-pod    0/1     CrashLoopBackOff   7 (40s ago)     14m   <- restarts climbing => crash loop
 
# Step 2 — the human-rendered status + Events. The first-line diagnostic.
kubectl describe pod my-pod
#   Look at: Containers -> State / Last State / Reason / Exit Code
#            Conditions table
#            Events section (the registry/volume/probe errors live here)
 
# Step 3 — the dying container's logs. Usually decisive for CrashLoopBackOff.
kubectl logs my-pod --previous
 
# Step 4 — raw status for precision / scripting.
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState}' | jq

A containerStatuses entry mid-crash-loop, annotated:

status:
  phase: Running                       # COARSE — "Running" despite the container being dead
  containerStatuses:
  - name: app
    ready: false                       # not serving — excluded from Service endpoints
    restartCount: 7                    # climbing => crash loop confirmed
    state:
      waiting:
        reason: CrashLoopBackOff        # "it keeps dying" — NOT the cause
        message: back-off 2m40s restarting failed container
    lastState:
      terminated:
        exitCode: 137                   # 128+9 => SIGKILL
        reason: OOMKilled               # <-- THE CAUSE: memory limit exceeded
        startedAt:  "2026-05-16T11:42:00Z"
        finishedAt: "2026-05-16T11:42:13Z"   # died 13s after starting
        containerID: containerd://9c1a...

Diagnosis from this block alone: the container is OOMKilled 13 seconds after each start — it allocates past its memory limit almost immediately. Fix: raise resources.limits.memory or fix the allocation. No log-reading even required.

Failure Modes

These are themselves the symptoms of failures; the table is the field guide.

1. CrashLoopBackOff — recognize: state.waiting.reason, climbing restartCount. Diagnose: lastState.terminated.exitCode + kubectl logs --previous. Exit 1 → app error in logs; 137 → OOM; 143 → it was told to stop (probe killing it, or a too-short grace period).

2. OOMKilled / exit 137 — recognize: lastState.terminated.reason=OOMKilled. Fix: raise the memory limit or fix the leak. Watch for runtimes (JVM, Node) sized to node memory rather than the cgroup.

3. ImagePullBackOff / ErrImagePull — recognize: state.waiting.reason. Diagnose: the Events section for the registry error. Fix: correct the tag, add imagePullSecrets, fix registry connectivity.

4. CreateContainerConfigError — recognize: state.waiting.reason. Cause: a referenced ConfigMap/Secret (or key) doesn’t exist. Fix: create it or correct the reference.

5. Stuck ContainerCreating — recognize: state.waiting.reason=ContainerCreating persisting minutes. Cause: volume (FailedMount/FailedAttachVolume) or CNI (no Pod IP). Diagnose: Events section.

6. Pod stuck Pending — phase=Pending, PodScheduled=False. The scheduler can’t place it: insufficient resources, unsatisfiable affinity/nodeSelector/taints, an unbound PVC. Diagnose: kubectl describe pod → the FailedScheduling Event with the per-node reasons.

7. Init container is the culprit — phase=Pending, Initialized=False. The fault is in initContainerStatuses[], not containerStatuses[]. Logs: kubectl logs my-pod -c <init-container>.

8. Running but Ready=False — the process is up but the readiness probe fails (or a readiness gate condition is unmet). The Pod gets no Service traffic. Diagnose: containerStatuses[].ready=false + the Unhealthy Events + the app’s readiness endpoint.

9. restartCount high but state=running — the container has crashed repeatedly but is currently up between backoffs. Don’t be fooled by the momentary running — restartCount and lastState reveal the instability.

10. Phase Unknown — the kubelet stopped reporting (node NotReady). The Pod’s real state is unknowable from the API; the node itself is the thing to investigate. See Node Pressure Conditions.

Alternatives and When to Choose Them

• kubectl describe pod vs kubectl get pod -o yaml. describe for the first pass — human-formatted, and it folds in the related Events. -o yaml (or -o jsonpath) for exact field values and scripting. Start with describe.
• kubectl logs --previous vs kubectl debug. logs --previous recovers the last words of a container that already died — the right tool for a clean crash that logs its error. kubectl debug is for a container that is up but wrong, or whose image has no shell/tools, or that crashes too fast to inspect — you attach an ephemeral container with real tooling.
• Reading status vs kube-state-metrics. Reading containerStatuses debugs one Pod now. kube-state-metrics exposes the same fields as fleet-wide Prometheus series (kube_pod_container_status_restarts_total, kube_pod_container_status_last_terminated_reason) so you can alert on crash loops rather than discover them by hand.
• Status fields vs Events. containerStatuses is the current truth (and the immediately-previous termination). Kubernetes Events is the time-ordered narrative — but Events expire after ~1 hour. Read both; they answer different questions.

Production Notes

• The first-line diagnostic is always: kubectl describe pod → read State / Last State / Exit Code → read the Events → kubectl logs --previous. This sequence resolves the large majority of Pod problems in well under a minute. Internalize it.
• lastState, not state, is usually the answer. state tells you the Pod is unhealthy; lastState.terminated tells you why. Operators who skip straight to lastState (and the exit code) debug far faster.
• Memorize the exit codes. 0 clean, 1 app error, 137 SIGKILL/OOM, 143 SIGTERM, 139 segfault, 126/127 bad command. The exit code alone often pre-classifies the bug before you open the logs.
• Init containers hide bugs in a separate status list. When a Pod won’t leave Pending with Initialized=False, debug initContainerStatuses[] and kubectl logs -c <init>. This is a frequent dead-end for people who only look at containerStatuses.
• Running is not Ready, and Ready is what routes traffic. A Running Pod with ready: false silently receives no Service traffic — a “deployment succeeded but the app is unreachable” mystery that is just an unmet readiness probe.
• k8s.af failure stories are full of misread status. OOMKills mistaken for app crashes (and “fixed” by restarting); CrashLoopBackOff chased in the app container when an init container was failing; ContainerCreating blamed on the image when a PVC wouldn’t attach. The cure in every case is reading the right field — lastState, initContainerStatuses, the Events — instead of guessing.

See Also

• Pod Lifecycle — the state-machine theory; this note is its field-debugging practice
• kubectl debug — the tool for acting on a running broken Pod (ephemeral containers, Pod copies)
• Kubernetes Events — the time-ordered narrative read alongside container statuses
• Pod Probes — liveness/readiness/startup; failing probes drive Unhealthy and restarts
• Liveness Probe Misuse — over-aggressive liveness probes that cause CrashLoopBackOff
• Resource Requests and Limits — memory limits whose breach yields OOMKilled
• QoS Classes — eviction ordering; relevant to Evicted Pods
• Init Containers — their statuses live in a separate initContainerStatuses[] list
• kube-state-metrics — exposes these status fields as fleet-wide Prometheus metrics
• Logging on Kubernetes — kubectl logs --previous and the durable-log story behind it
• Node Pressure Conditions — node-level causes behind phase Unknown and Evicted
• kubelet — the agent that populates containerStatuses and performs restarts
• kubectl — describe / get -o yaml / logs are the reader commands
• Kubernetes MOC — §14 Observability, umbrella index