vault

Pod Probes

18 min read

Pod Probes

Pod probes are the kubelet-driven health checks that determine, on a periodic schedule, whether each container in a Pod is alive, ready, or still starting. The kubelet is the actor — the API server is not involved in probe execution; probes are local node-level operations issued by the kubelet against the container on its own node. Three probe types exist, each driving distinct kubelet behaviour: livenessProbe decides whether to restart the container; readinessProbe decides whether the Pod should appear in Service / EndpointSlice endpoints; startupProbe decides whether the container has finished its initial startup and the other two probes should be allowed to run. The Kubernetes documentation gives the precise definitions (Probes): liveness “lets Kubernetes know when to restart a container,” readiness “lets Kubernetes know when a container is ready to start accepting traffic,” startup “verifies whether the application within a container is started.” Each probe can be implemented via one of four mechanisms — httpGet, tcpSocket, exec, or grpc (the last graduated to GA in K8s 1.27, release blog, KEP-2727). The bulk of K8s operational pain in real production is misconfigured probes: liveness probes that are aggressive about transient failures crashloop healthy Pods (the Liveness Probe Misuse anti-pattern); readiness probes that are tied to upstream dependencies cascade outage from one service to the next; startup probes that aren’t used on slow-starting JVM apps cause liveness probes to kill the app before it finishes booting. This note covers the mechanics, the configurable knobs, the failure modes, and the production wisdom for getting probes right.

Mental Model

stateDiagram-v2
    direction LR
    [*] --> Created : kubelet creates<br/>container

    Created --> Starting : container process<br/>begins
    state Starting {
        [*] --> SP_Running : startupProbe runs<br/>every periodSeconds<br/>(liveness/readiness suspended)
        SP_Running --> SP_Success : success
        SP_Running --> SP_Failure : failureThreshold<br/>consecutive failures
    }
    Starting --> Restart : startupProbe fails<br/>over failureThreshold
    Starting --> Live : startupProbe succeeds<br/>(or none defined)

    state Live {
        [*] --> Probing : livenessProbe and<br/>readinessProbe run<br/>concurrently
        Probing --> Ready : readinessProbe success<br/>→ Pod in EndpointSlice
        Probing --> Unready : readinessProbe failure<br/>→ Pod removed from<br/>EndpointSlice (kept alive)
        Probing --> Restart : livenessProbe failure<br/>→ kubelet restarts container
        Ready --> Probing : keep probing
        Unready --> Probing : keep probing
    }

    Restart --> Created : per Pod restartPolicy<br/>(Always/OnFailure/Never)

What this diagram shows. A container’s probe-driven state machine has two phases. In the Starting phase, only the startupProbe is evaluated; the livenessProbe and readinessProbe are dormant — the kubelet does not execute them. This is the entire purpose of startupProbe: it gives slow-booting containers (Java/Spring apps with multi-minute warm-up, large language models loading 70 GiB of weights) a long, lenient grace period without forcing you to compromise the post-startup livenessProbe’s aggressiveness. Once the startup probe succeeds (or if none is configured, immediately), the container enters the Live phase, where liveness and readiness run concurrently — independently — and drive two orthogonal outcomes. Liveness failure → restart (the container is broken; rebooting it might help). Readiness failure → remove from endpoints but keep running (the container is alive but can’t serve traffic right now; let it recover without restarting it). The insight to extract is that the three probes encode three orthogonal decisions: “should the container exist?” (liveness), “should traffic be sent to it?” (readiness), “is it still booting?” (startup). The single most common production mistake is conflating liveness with readiness — using liveness for “can the app talk to its database” turns a database hiccup into a cluster-wide restart storm.

Mechanical Walk-through

The four probe mechanisms

Every probe declaration must specify exactly one of four execution mechanisms:

  • httpGet — the kubelet (running on the same node as the Pod) issues an HTTP GET against the Pod’s IP on the specified port and path. Response codes in [200, 400) are success; anything else is failure. Optional httpHeaders allow customising the request (e.g., a Host: header to satisfy a virtual-hosted server, or an auth header). The port can be a number or a named port from the container’s ports[] list — the named-port form is recommended because renaming a port doesn’t break the probe.

  • tcpSocket — the kubelet opens a TCP connection to the Pod’s IP on the specified port. Successful connection = success; connection refused or timeout = failure. Useful for non-HTTP services (Redis, Postgres, message brokers) where a TCP handshake is a good proxy for “the process is listening.”

  • exec — the kubelet execs a command inside the container; exit code 0 = success, any other exit code = failure. The most flexible mechanism but the most expensive — every probe execution forks a process inside the container, which is significant overhead at high probe frequency.

  • grpc — the kubelet uses the gRPC Health Checking Protocol (grpc-health-probe specification) to call the standardized grpc.health.v1.Health/Check RPC on the configured port. The response’s status field must equal SERVING for success. Stable since K8s 1.27 under the GRPCContainerProbe feature gate, which graduated to GA there and no longer needs enabling (Probes docs, KEP-2727); it was alpha in 1.23 (disabled by default) and beta in 1.24 (on by default, 1.24 gRPC-probes-beta blog). Before native gRPC probes, the workaround was to embed grpc-health-probe as a binary in the image and use an exec probe — that workaround is now obsolete.

Probe parameters

Every probe takes the same six parameters (Probe v1 core API reference):

ParameterDefaultMeaning
initialDelaySeconds0Delay between container start and first probe. Useful when the app needs N seconds before it’s worth probing at all (alternatives: use a startupProbe).
periodSeconds10How often the probe runs. Lower = faster failure detection, higher kubelet load.
timeoutSeconds1How long the kubelet waits for a probe response before declaring it failed. The default of 1 second is short; HTTP probes that hit a fastapi/spring server under load may legitimately take longer.
successThreshold1 (liveness/startup: must be 1)Consecutive successes needed to flip “failing” to “succeeding.” Only meaningful for readinessProbe (where flapping is undesirable) — for liveness/startup, K8s enforces 1.
failureThreshold3Consecutive failures needed to take action (restart for liveness/startup; mark unready for readiness).
terminationGracePeriodSeconds(inherits Pod)Probe-level override of the Pod’s grace period; lets you give a container less time to terminate on a liveness/startup-driven restart than the Pod gives its containers on kubectl delete. Applies only to liveness and startup probes (a failing readiness probe never terminates the container, so the field is meaningless there). Stable since K8s 1.28 — alpha in 1.21, beta in 1.22, GA in 1.28 after missing the 1.27 cut (KEP-2238).

startupProbe suppresses liveness and readiness

This is the most important interaction. While startupProbe is running and has not yet succeeded:

  • The livenessProbe is not executed. The container cannot be liveness-killed during startup.
  • The readinessProbe is not executed. The Pod is Ready=False; no traffic is routed to it.

Once the startupProbe succeeds (or if it’s not defined at all), the kubelet stops running it for the lifetime of the container and switches to running liveness and readiness. The K8s docs phrase this as: “liveness and readiness probes will not start until the startup probe succeeds.”

The configuration recipe: set startupProbe with a very generous failureThreshold * periodSeconds budget (e.g., failureThreshold: 60, periodSeconds: 10 = 10 minutes), and the livenessProbe with tight settings (e.g., periodSeconds: 5, failureThreshold: 3 = 15 seconds to declare dead). This gives a 10-minute boot budget without weakening the steady-state liveness check.

Readiness probe and Service endpoints

A container’s readiness is one of the inputs to the Pod’s Ready condition (the other inputs: containers’ readiness statuses combined with any user-defined readinessGates). When Ready=False, the EndpointSlice controller removes the Pod’s IP from every Service that targets it (see EndpointSlice and Service (Kubernetes)). When Ready=True, the IP is re-added. The Pod itself is not restarted; the kubelet leaves it running indefinitely. This is the right hammer for “I’m temporarily overloaded” or “I’m draining my connection pool” situations.

readinessGates — external readiness control

Beyond the per-container readiness probes, the Pod’s Ready condition can be gated on additional, externally-managed conditions via spec.readinessGates[]. Each entry names a condition type; an external controller writes that condition to the Pod’s status.conditions[], and the Pod’s Ready only flips true when all gates and all container readiness probes are true. This is how the AWS Load Balancer Controller, for example, makes a Pod “Ready” only after the LB target group has successfully health-checked it — closing the race where a Pod becomes ready in K8s, gets traffic from the LB before the LB has registered it, and 503s. The mechanism is documented at Pod readiness.

Restart policy interaction

When a liveness probe fails over failureThreshold times, the kubelet kills the container (sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL) and considers a restart. Whether the container is actually restarted depends on the Pod’s restartPolicy:

  • Always (default for Deployment/StatefulSet/DaemonSet Pods) — yes, restart.
  • OnFailure (sometimes used for Jobs) — yes, since the container exited non-zero.
  • Never — no; the Pod transitions to Failed.

A container that keeps crashing under repeated liveness-driven restarts enters CrashLoopBackOff, with the kubelet applying exponential backoff: the default sequence starts at 10 s and doubles each time (10 s, 20 s, 40 s, 80 s, 160 s) until it caps at 300 s (5 minutes). As of K8s 1.33 the alpha ReduceDefaultCrashLoopBackOffDecay feature gate lowers the default cap to 60 s, and a kubelet-config maxContainerRestartPeriod (1 s–300 s) lets operators tune the ceiling (restart-backoff KEP-5593). See Pod Lifecycle for the full backoff treatment.

Configuration / API Surface

A production-grade Pod manifest combining all three probe types, annotated:

apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  # Pod-level grace period; probe-level can override.
  terminationGracePeriodSeconds: 30
 
  containers:
    - name: app
      image: myorg/spring-app:v3.2
      ports:
        - name: http
          containerPort: 8080
        - name: grpc
          containerPort: 9090
 
      # ---------- (1) Startup probe: slow JVM warm-up ----------
      # Spring Boot's full context refresh can take 1-3 minutes under
      # autowiring + JPA + Flyway migrations. Budget 5 minutes (30 * 10s)
      # before declaring boot failed.
      startupProbe:
        httpGet:
          path: /actuator/health/readiness
          port: http        # named port, not number
        # Generous failure budget; only this probe matters during startup.
        failureThreshold: 30
        periodSeconds: 10
        timeoutSeconds: 3
        # initialDelaySeconds usually 0 with a startupProbe; the failure
        # threshold provides the budget instead.
 
      # ---------- (2) Liveness probe: post-startup health ----------
      # Tight: catch a *deadlocked* JVM (long GC pause, deadlock between
      # threads). MUST NOT depend on downstream services.
      livenessProbe:
        httpGet:
          path: /actuator/health/liveness     # local-only, no downstream calls
          port: http
        # Run every 10s after the startup probe succeeds.
        periodSeconds: 10
        timeoutSeconds: 3
        # 3 consecutive failures → restart. Total time-to-restart ≈ 30s.
        failureThreshold: 3
        # If liveness fails, restart this container quickly (don't wait the
        # full 30s pod grace period).
        terminationGracePeriodSeconds: 10
 
      # ---------- (3) Readiness probe: traffic gating ----------
      # MAY depend on downstream services (e.g., is the DB pool healthy?)
      # because a readiness flip-flop just removes from endpoints, not restart.
      readinessProbe:
        httpGet:
          path: /actuator/health/readiness
          port: http
        periodSeconds: 5
        timeoutSeconds: 2
        # Faster removal from endpoints (just 2 failures), but require 2
        # successes to add back (avoid flapping).
        failureThreshold: 2
        successThreshold: 2
 
  # ---------- (4) ReadinessGate: external readiness signal ----------
  # AWS Load Balancer Controller writes the condition with this type
  # once the target is healthy in the LB target group.
  readinessGates:
    - conditionType: "target-health.elbv2.k8s.aws/k8s-default-svc-ENV"

Alternative probe mechanisms for the same logical health check, demonstrating the API:

# TCP probe — just "is the port open?"
livenessProbe:
  tcpSocket:
    port: 6379                # Redis default
  periodSeconds: 5
 
# Exec probe — arbitrary command, exit 0 = healthy
livenessProbe:
  exec:
    command:
      - sh
      - -c
      - pg_isready -h localhost -U postgres
  periodSeconds: 10
 
# gRPC probe — standard grpc.health.v1.Health/Check
livenessProbe:
  grpc:
    port: 9090
    service: ""               # empty = overall service health
  periodSeconds: 10

The grpc mechanism does not allow custom request payloads — it always calls grpc.health.v1.Health/Check with the (optional) service name. The container must implement that RPC; the grpc-health-probe README lists library support across languages.

Failure Modes

  1. Aggressive liveness probe causing crashloops on a healthy app. The most common K8s anti-pattern. Liveness probe with failureThreshold: 1 and timeoutSeconds: 1 catches a single slow GC pause and restarts the container, which is already healthy. After restart, the JVM cold-starts (slow), the next probe times out, restart, repeat. The cluster sees CrashLoopBackOff on Pods that the app team swears are fine. Mitigation: use only readiness for downstream-dependent checks; liveness should fire only on truly unrecoverable conditions (deadlock, persistent OOM). See Liveness Probe Misuse.

  2. Liveness probe that calls the database / a backing service. When the DB has a 5-second hiccup, every Pod’s liveness probe fails, the kubelet restarts every Pod simultaneously, every Pod cold-starts and hammers the just-recovered DB, the DB falls over again, repeat. This is the textbook cascading restart failure. Mitigation: liveness MUST be a local-only check; readiness MAY include downstream dependencies, because readiness failure just removes from endpoints (no restart).

  3. No startupProbe on a slow-booting app. The Pod’s livenessProbe with initialDelaySeconds: 120 is an anti-pattern — 120 seconds during startup and 120 seconds wait between failures during steady state. The right answer is a permissive startupProbe plus a tight livenessProbe; the startup probe suppresses liveness until boot is done.

  4. successThreshold > 1 on liveness or startup probe. K8s rejects this at admission. Only readinessProbe accepts successThreshold > 1.

  5. Probe failure during graceful shutdown. When the kubelet starts terminating a Pod it sends SIGTERM (after any preStop hook); a draining application will start failing its readinessProbe, which correctly pulls it from endpoints. The subtle part is that the kubelet does not spuriously restart a container just because a probe fails while the container is already being torn down — a container in the terminating path is on its way out regardless of the liveness verdict. What the probe-level terminationGracePeriodSeconds does address is the orthogonal case where a steady-state liveness or startup failure triggers a restart: that field lets the kill use a shorter grace budget than the Pod’s terminationGracePeriodSeconds (which exists for ordinary kubectl delete draining), so a wedged container is rebooted promptly instead of waiting out a long Pod-level drain window.

Uncertain

Verify: the exact kubelet behavior toward liveness/readiness probe execution during the termination grace window across versions (do probes keep being scheduled until the container exits, or are they cancelled when deletionTimestamp is set?). Reason: the official Probes and Pod-Lifecycle docs describe endpoint/readiness updates on deletion but do not spell out per-probe scheduling during termination. To resolve: read the kubelet prober/worker source (pkg/kubelet/prober) or a SIG-Node design doc that states whether probe workers are stopped on container termination. #uncertain

  1. timeoutSeconds: 1 (default) is too short under load. A heavy Java app under 100 % CPU might take 1.5s to respond to an HTTP probe, not because it’s broken but because the GC is busy. The 1-second default triggers a false failure. Mitigation: explicitly set timeoutSeconds: 3 or 5 on every probe; never rely on the default.

  2. Exec probe causing fork bombs. Every probe execution forks a process inside the container. At periodSeconds: 1 on a Pod with thousands of replicas, you’ve created a steady fork load. Memory limits compound this — every fork briefly doubles the container’s RSS until exec replaces it. Mitigation: avoid exec probes if possible (prefer httpGet/tcpSocket), or use longer periods.

  3. HTTP probe ignored by an HTTPS-only server. httpGet sends plain HTTP. If the server only listens on TLS, the probe always fails. There’s no httpsGet field; the workaround is scheme: HTTPS on the probe (which makes the kubelet skip cert verification) or terminate TLS in front of the app.

  4. gRPC probe failing because the server doesn’t implement the health protocol. A common shipping bug — the team adds grpc: to the probe spec without implementing grpc.health.v1.Health/Check on the server. The kubelet gets UNIMPLEMENTED, scores it as failure, restarts the Pod. Mitigation: add the standard health implementation to your gRPC server (most languages have a one-liner via grpc/health package).

  5. readinessGates set but no controller writing the condition. The Pod’s Ready is gated on a condition that nothing ever writes. The Pod stays unready forever; no traffic. Mitigation: check that the controller named in the conditionType actually exists in the cluster. Surfaces in kubectl describe pod as the missing condition.

Alternatives and When to Choose Them

  • No probes (defaults). A container with no probes is Ready as soon as it starts and never gets restarted by health-check logic — only by exit-non-zero plus restartPolicy. Acceptable for batch Jobs and short-lived workloads where probes are overkill. Reject for long-running services.

  • livenessProbe alone. Only restart-driven health. Reject — without a readinessProbe, a Pod that’s alive-but-not-ready receives traffic during its slow phases (cold start, dependency loss). Always pair with readiness.

  • readinessProbe alone. Traffic gating only; container never restarted by probes. Acceptable when the app is self-healing in place — if it gets stuck, it’ll get traffic-removed but human operators handle restart. Often a safer default than livenessProbe for less-mature services (lower blast radius for misconfiguration).

  • Sidecar-based health check (legacy). A health-check sidecar that exposes HTTP and itself probes the main container via local IPC. Some teams used this before probes had the expressiveness they do now. Reject in 2026 — probes are sufficient.

  • External health checker (LB-level, mesh-level). The cloud load balancer (ALB, GCLB) or service mesh (Istio, Linkerd) does its own health check. K8s probes and external health checks are independent; they can disagree. The readinessGates mechanism is how you tie them together properly. Choose external in addition to (not instead of) probes.

  • PodHealthcheck CRDs offered by various operators (e.g., Datadog Cluster Agent) — out-of-band metric-based health. Useful for “this Pod is alive but its custom metric requests_per_second has been 0 for 5 minutes; declare it unready.” Choose for sophisticated metric-driven readiness; not a substitute for kubelet-level probes.

Production Notes

  • The Lyft engineering blog has multiple posts about probe disasters that brought down their service mesh — aggressive liveness probes on Envoy sidecars cascading into mesh-wide restarts during minor upstream blips. Their resolution: relax liveness to only catch unrecoverable conditions, use readiness for everything else.

  • Spring Boot’s actuator endpoints (/actuator/health/liveness and /actuator/health/readiness) are the canonical Java pattern. The two endpoints are designed to be probed independently — liveness checks intrinsic JVM health, readiness checks downstream-dependency reachability. Most production Spring Boot apps map probes to these endpoints directly, as in the YAML above.

  • gRPC health protocol adoption. Before K8s 1.27, the standard pattern was grpc-health-probe binary in every image plus an exec probe. The 1.27 GA of native grpc: probes simplified that to a YAML field — but only on clusters that are 1.27+. As of early 2026, this is universally available.

  • k8s.af failures (Kubernetes Failure Stories) include several probe-related outages: aggressive liveness probes on Cassandra Pods during a JVM full-GC cascade, readiness flapping on a service whose ConfigMap reload was slow, startup-probe-less Pods that crashlooped during a cold-start storm. The recurring lesson: probe parameters are a load-test concern, not a config-file concern — tune them under realistic conditions.

  • The Shopify Helm chart conventions publish opinionated probe defaults for their internal services: livenessProbe.failureThreshold: 10 (very lenient — they trust readiness more than liveness for traffic gating), startupProbe.failureThreshold: 60, periodSeconds: 10 (10-minute boot budget). These numbers are larger than the K8s defaults; they reflect the production reality that defaults are too aggressive.

  • CKAD/CKA exam content (Certified Kubernetes Application Developer) routinely tests probe configuration. The exam scenarios are stripped-down versions of these failure modes — given a crashlooping Pod, identify whether the cause is liveness misconfiguration vs. application bug.

  • Version-gating both newer features. Two probe features carry version floors worth memorizing because clusters in the field span several minors. Native grpc: probes are GA from 1.27 (GRPCContainerProbe, KEP-2727); probe-level terminationGracePeriodSeconds is GA from 1.28 (ProbeTerminationGracePeriod, KEP-2238). On the long-stable cluster releases prevalent in early 2026 (1.30–1.36) both are universally available, but a manifest using either against an older control plane will be silently dropped (unknown field) or rejected.

See Also

Graph View

Backlinks