vault

Sidecar Containers

18 min read

Sidecar Containers

Sidecar containers are long-running, co-located containers that provide supporting functions to a Pod’s main containers — mesh proxies, log shippers, secret-rotation agents, metrics scrapers, profiler agents. They share the Pod’s network namespace, volumes, and (optionally) PID namespace with the main containers, allowing them to communicate with the application over localhost and observe it without external network hops. The Kubernetes documentation defines them: “Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code” (kubernetes.io — Sidecar Containers). The deployment-level concept of the sidecar pattern predates K8s and is covered separately in Sidecar Pattern (architectural framing). This note covers the K8s-native sidecar resource introduced by KEP-753: an init container with restartPolicy: Always, which graduated to alpha in K8s 1.28, became enabled-by-default (beta) in 1.29, and reached stable (GA) in K8s 1.33 (April 2025, “Octarine” release — confirmed by the docs’ FEATURE STATE: Kubernetes v1.33 [stable] banner and “the feature is active by default since Kubernetes v1.29” — kubernetes.io — Sidecar Containers, v1.33 release notes). The native sidecar resolves two structural problems with the pre-1.28 approach of “just add another container to spec.containers[]”: (1) such a sidecar would prevent a Job from completing because the sidecar never exits, and (2) there was no ordering guarantee — neither for startup (so the mesh proxy might come up after the app makes its first call) nor for shutdown (so the log shipper might exit before the app finishes flushing its final log line). Native sidecars solve both via the init-container ordering machinery combined with restartPolicy: Always to keep them running.

Mental Model

flowchart TB
    subgraph "Pod with native sidecar (K8s 1.33+)"
        direction LR
        I0[initContainer&#91;0&#93;<br/>setup<br/>run-to-completion]
        SC[initContainer&#91;1&#93;<br/>restartPolicy: Always<br/>SIDECAR — log shipper]
        I2[initContainer&#91;2&#93;<br/>more setup<br/>run-to-completion]
        APP[container&#91;0&#93;<br/>main app]
        I0 -->|exit 0| SC
        SC -.->|startupProbe<br/>succeeds, kubelet<br/>marks 'started'| I2
        I2 -->|exit 0| APP
    end
    KUBELET[kubelet]
    POD_TERM[Pod terminating]
    KUBELET -- "start order:<br/>I0 → SC → I2 → APP" --> APP
    POD_TERM -- "shutdown order:<br/>APP → I2 (already exited)<br/>→ SC (reverse of start)" --> SC

What this diagram shows. A native sidecar (the box labelled restartPolicy: Always) is just another entry in spec.initContainers[], distinguished by its restartPolicy field. Because it’s structurally an init container, it inherits the sequential startup order of init containers: the kubelet starts initContainers[0], waits for it to exit zero, starts the sidecar (initContainers[1]), waits for the sidecar’s startupProbe to succeed (or for the process to be running with no probe), then proceeds to initContainers[2], and finally starts the main containers. But unlike a run-to-completion init container, the sidecar stays running for the entire Pod lifetime; if it exits, the kubelet restarts it. On Pod termination, the order reverses: the main containers receive SIGTERM first, and the sidecar is kept alive until the main containers have fully stopped (Kubernetes docs — termination with sidecars). This is the entire point: the supporting infrastructure outlives the application it supports, ensuring the app never finds itself talking to a dead proxy or losing logs during the final flush. The insight to extract is that the native sidecar is defined by two simultaneous properties — it’s positioned in initContainers[] to inherit ordering, and restartPolicy: Always overrides the run-to-completion behavior — and those two properties together solve the historical problems with sidecars-as-regular-containers.

Mechanical Walk-through

Startup ordering

The kubelet processes a Pod’s containers in this order:

  1. For each entry in spec.initContainers[], in declaration order:
    • Start the container.
    • If the container’s restartPolicy is unset (or OnFailure/Never) — a classic run-to-completion init container — wait for it to exit zero, then move to the next.
    • If the container’s restartPolicy: Always — a native sidecar — wait until the container is “started.” The kubelet’s notion of “started” is either (a) the container’s process is running and no startupProbe is defined, or (b) the startupProbe succeeds for the first time. Then move to the next init container without waiting for the sidecar to exit (it is, after all, intended to keep running).
  2. Once all initContainers[] are in their started state, start all spec.containers[] in parallel.

The K8s docs state: “If an init container is created with its restartPolicy set to Always, it will start and remain running during the entire life of the Pod” and “sidecar containers … benefit from the same ordering and sequential guarantees as regular init containers, allowing you to mix sidecar containers with regular init containers for complex Pod initialization flows” (kubernetes.io — Sidecar Containers). The interleaving of run-to-completion init containers and native sidecars in one initContainers[] list is what enables, for example, “configure the network namespace (init), then start the mesh proxy (sidecar), then check the mesh proxy is reachable (init), then start the app (main).”

Shutdown ordering

On Pod termination, the kubelet:

  1. Sends SIGTERM to main containers only.
  2. Waits up to terminationGracePeriodSeconds for main containers to exit.
  3. Once all main containers have exited, sends SIGTERM to native sidecars in reverse order of their declaration in initContainers[].
  4. Waits a short period (whatever remains of the grace period; if exceeded, SIGKILLs the sidecars).

The doc passage that matters: “Upon Pod termination, the kubelet postpones terminating sidecar containers until the main application container has fully stopped. The sidecar containers are then shut down in the opposite order of their appearance in the Pod specification” (kubernetes.io — Sidecar Containers). Note the explicit warning in the same source: “When other containers take all allotted graceful termination time, the sidecar containers will receive the SIGTERM signal, followed by the SIGKILL signal, before they have time to terminate gracefully. So exit codes different from 0, for sidecar containers are normal on Pod termination and should be generally ignored by the external tooling.” Operationally: a sidecar with a slow shutdown is at the mercy of the main container’s promptness in handling SIGTERM.

Job-completion problem

Before native sidecars, a Job using a sidecar (e.g., a batch job whose log shipper is a sidecar container) faced an unsolvable problem: the Job is “complete” when all the Pod’s containers have terminated successfully, but the sidecar never terminates. Workarounds were ugly — shareProcessNamespace: true plus a preStop hook that sent SIGTERM to the sidecar PID, or wrapping the sidecar in a shell script that watched for a “done” sentinel file.

Native sidecars solve this structurally because they’re init containers, and init containers don’t count toward Job completion. The K8s docs state: “If you define a Job that uses sidecar using Kubernetes-style init containers, the sidecar container in each Pod does not prevent the Job from completing after the main container has finished” (kubernetes.io — Sidecar Containers). Once the Job’s main container exits successfully, the kubelet terminates the sidecar in reverse order and the Pod transitions to Succeeded.

Probes on sidecars

Native sidecars support the full probe trio — startupProbe, livenessProbe, readinessProbe — with the same semantics as on regular containers (see Pod Probes). One nuance: the sidecar’s readinessProbe contributes to the Pod’s overall Ready condition. If your mesh proxy sidecar is not ready, the Pod is not ready, and Service endpoints exclude it. This is usually what you want — a Pod whose proxy isn’t up has no business receiving traffic. Conversely, a misconfigured liveness probe on a sidecar that crashloops will manifest as a Pod that thrashes; see Liveness Probe Misuse for the general problem.

Resource accounting

Because a native sidecar runs concurrently with the main containers (it never exits), it is counted alongside them rather than via the classic init-container max-of formula. The Kubernetes documentation states the Pod’s effective request/limit for a resource is the pod overhead plus the higher of two quantities: “the sum of all non-init containers (app and sidecar containers) request/limit for a resource” or “the effective init request/limit for a resource” (kubernetes.io — Sidecar Containers). KEP-753 spells out the full expression — Max( Max(each InitContainerUse), Sum(Sidecar Containers) + Sum(each ContainerUse) ) + pod overhead — where the second term captures the steady-state in which every sidecar and every app container is running at once (KEP-753). The practical upshot: a sidecar’s request is added to, not maxed against, the app’s request. Adding a 200 MiB Envoy sidecar to every Pod genuinely costs 200 MiB per Pod, cluster-wide — the basis for the Ambient Mesh motivation to remove per-Pod sidecars in service meshes. (Contrast with Init Containers, whose run-to-completion requests are folded in via the max term because they never overlap the main containers.)

OOM score and eviction order

Under Linux memory pressure the kernel’s OOM killer chooses a victim partly by each process’s oom_score_adj — a higher value makes a process a more attractive kill target. Classical run-to-completion init containers are given a relatively aggressive (kill-friendly) adjustment because they are expected to be short-lived. A native sidecar, by contrast, must survive for the whole Pod lifetime, so KEP-753 deliberately treats it like a regular container for OOM purposes. The KEP states the design intent precisely: “In order to minimize OOM kills of sidecar containers, the OOM adjustment for these containers will match or fall below the OOM score adjustment of regular containers in the Pod” (KEP-753, addressing kubernetes/kubernetes#111356). The rationale is straightforward: losing the mesh proxy or secret agent under memory pressure usually breaks the application it serves, so the kernel should not preferentially reap the sidecar before the workload it supports. Note the wording — “match or fall below” — means a sidecar is never made more OOM-killable than the main containers, but the kubelet may still derive a slightly lower (more protected) score for it; the exact computed integer is an implementation detail of the kubelet’s QoS-based scoring and is not part of the user-facing contract.

Configuration / API Surface

A canonical Pod with a native sidecar (log shipper) plus a classical run-to-completion init container (network setup):

apiVersion: v1
kind: Pod
metadata:
  name: webapp-with-sidecar
spec:
  # Main containers exit on SIGTERM within 30s; sidecars get the remainder
  # of the grace period after that, then SIGKILL.
  terminationGracePeriodSeconds: 60
 
  initContainers:
    # ---------- (1) Classical run-to-completion init container ----------
    # Sets up the shared volume. Exits 0 and is done.
    - name: setup-logs-volume
      image: busybox:1.36
      command: ["sh", "-c", "touch /var/log/app.log && chmod 666 /var/log/app.log"]
      volumeMounts:
        - { name: logs, mountPath: /var/log }
      # No restartPolicy field → defaults to run-to-completion init semantics.
 
    # ---------- (2) Native sidecar: log shipper ----------
    # The presence of `restartPolicy: Always` is what makes this a sidecar.
    # It starts AFTER setup-logs-volume completes, runs for the life of
    # the Pod, and shuts down AFTER the main container in step (3) exits.
    - name: log-shipper
      image: fluent/fluent-bit:3.0
      restartPolicy: Always               # <-- THIS LINE makes it a sidecar
      args:
        - "/fluent-bit/bin/fluent-bit"
        - "-i"
        - "tail"
        - "-p"
        - "path=/var/log/app.log"
        - "-o"
        - "stdout"
      volumeMounts:
        - { name: logs, mountPath: /var/log }
      # Probes work like on regular containers; readinessProbe contributes
      # to the Pod's Ready condition.
      readinessProbe:
        exec:
          command: ["sh", "-c", "test -p /var/log/app.log || test -f /var/log/app.log"]
        periodSeconds: 5
      resources:
        # Sidecar requests are SUMMED with main containers' requests
        # (unlike classical init containers, which use max-of formula).
        requests: { cpu: "50m",  memory: "64Mi" }
        limits:   { cpu: "200m", memory: "128Mi" }
 
  containers:
    # ---------- (3) Main app container ----------
    - name: app
      image: myorg/webapp:v2.1.0
      command: ["sh", "-c", "while true; do echo $(date) hello >> /var/log/app.log; sleep 5; done"]
      volumeMounts:
        - { name: logs, mountPath: /var/log }
      resources:
        requests: { cpu: "100m", memory: "128Mi" }
        limits:   { cpu: "500m", memory: "256Mi" }
 
  volumes:
    - name: logs
      emptyDir: {}

What happens on startup: kubelet pulls all images, then runs setup-logs-volume (exits 0), then starts log-shipper and waits for its readiness probe (or just “process running” if no startup probe), then starts app. The log shipper is up before the app starts logging.

What happens on kubectl delete pod: kubelet sends SIGTERM to app; app exits (or is SIGKILLed at 30s if it ignores SIGTERM); kubelet then sends SIGTERM to log-shipper, giving it the remaining grace period (30s) to flush its buffer and exit. The log shipper genuinely outlives the app for those final seconds — the structural guarantee that motivated the feature.

What happens if the Pod is a Job: the Job’s main container exits successfully; kubelet terminates the sidecar in reverse order; the Pod transitions to Succeeded; the Job is Complete. No special handling needed in user code.

Failure Modes

  1. Forgetting restartPolicy: Always on the sidecar. Without that field, the entry is a classical run-to-completion init container; the kubelet waits for it to exit zero, which a log shipper never does. The Pod hangs in PodInitializing forever. Symptom: a Pod that never reaches Running, with the would-be sidecar showing as Init:Running indefinitely.

  2. Running on a cluster < 1.33 without the feature gate. Native sidecars are GA in 1.33. On 1.29–1.32 they exist as a beta feature (SidecarContainers feature gate enabled by default); on 1.28 they’re alpha (gate off by default); on ≤1.27, the restartPolicy field on an init container is silently ignored or rejected. Symptom on old clusters: surprising behavior — looks like a regular init container despite the restartPolicy field. Verify minor with kubectl version.

  3. Sidecar that crashloops blocks main containers from starting. Because the sidecar must reach “started” before the next init container or main container runs, a crashlooping sidecar with a failing startupProbe blocks Pod readiness indefinitely. Symptom: kubectl describe pod shows the sidecar in Init:CrashLoopBackOff and the main container never started.

  4. Sidecar’s slow shutdown is squeezed out of grace period. If the main container takes all 30 of 30 seconds to exit, the sidecar gets ~0 seconds plus a SIGKILL. The log shipper’s final flush gets cut off. Mitigation: set terminationGracePeriodSeconds generously (60+ seconds) so the sidecar has headroom, and design the main container’s preStop/SIGTERM handler to exit promptly.

  5. Sidecar competing with main container for resources at startup. While the sidecar is starting (between init container 1 and the main containers), the main container hasn’t started yet — fine. But if the sidecar consumes its memory limit during its own bootstrap and gets OOM-killed, the kubelet restarts it (per restartPolicy: Always), potentially starving the node. Mitigation: set resources.requests and resources.limits realistically on sidecars; their startup is not free.

  6. Mistake: putting a sidecar in spec.containers[] on a cluster that supports native sidecars. Doesn’t break anything immediately, but you lose the ordering guarantee (sidecar may start after the app) and the Job-completion behavior (sidecar blocks Job completion). Symptom: tail-end logs lost, or Jobs stuck in Running forever. Lift the sidecar into spec.initContainers[] with restartPolicy: Always.

  7. Probe configuration on a sidecar that depends on the main app. A sidecar with readinessProbe that calls into the main app’s port at localhost:8080 will fail until the main app starts — but the main app doesn’t start until the sidecar is ready. Deadlock. The fix: sidecar’s readiness should reflect the sidecar’s readiness, not the app’s (e.g., “is my Envoy admin port up?” not “is the upstream app responding?”).

  8. Sidecar sprawl. Adding a mesh proxy + log shipper + metrics collector + secret rotator + tracer to every Pod easily doubles or triples baseline memory per Pod. Cross-link Sidecar Sprawl Anti-Pattern — the architectural pressure that drove Ambient Mesh and per-node-agent designs.

Alternatives and When to Choose Them

  • Classical “just another container” sidecar (in spec.containers[]). Pre-1.28 default. Still works for Deployments and StatefulSets (where the Pod is long-lived anyway, and ordering matters less). Choose only if you must support clusters <1.29 or are willing to live with the unordered startup. For 1.33+, prefer native.

  • Init container (Init Containers). Choose for setup work that must complete before the main container starts and that you don’t need running afterwards. Init = one-shot; sidecar = long-running.

  • DaemonSet-based per-node agent (DaemonSet). Instead of one log shipper per Pod, run one per node. Pros: drastically lower resource overhead at scale; one process to monitor. Cons: doesn’t have access to per-Pod context as cleanly (Pod identity must be inferred from log path / metadata), no per-Pod isolation. Choose for cluster-wide concerns (node-level log forwarding, node-level monitoring). Cilium CNI and GKE Dataplane V2 use this for networking; Fluent Bit and Vector commonly use it for logs.

  • Ambient Mesh / sidecarless service mesh. Istio Ambient and similar designs lift the proxy from per-Pod sidecar to per-node ztunnel + per-namespace waypoint proxies. Trades sidecar’s per-Pod isolation for lower overhead. Choose for clusters with thousands of Pods where the sidecar memory tax is the dominant cost.

  • Ephemeral container (Ephemeral Containers). Not really an alternative to sidecars — ephemeral containers are for debugging, not long-running auxiliary work. But operators sometimes confuse “I want to inject a process into a Pod” cases. Ephemeral = debug, transient, no probes; sidecar = production, persistent, full probes.

  • In-process library. The OpenTelemetry SDK, the Vault Go SDK, etc. — embed the auxiliary functionality into the main container’s process directly. Pros: no extra container, no IPC overhead. Cons: every language/runtime needs its own SDK; updates require redeploying the app. Choose for tightly-integrated cases (in-process tracing); use sidecars for orthogonal concerns (proxying, log forwarding).

Production Notes

  • Istio sidecar injection is the archetypal native-sidecar use case. The Istio control plane’s mutating admission webhook injects an Envoy proxy and an init container (network setup via iptables) into every Pod in mesh-labeled namespaces. The Envoy is the sidecar; in K8s 1.33+, Istio configurations targeting “native sidecar mode” place the Envoy in initContainers[] with restartPolicy: Always, gaining ordered startup (Envoy ready before app starts), ordered shutdown (Envoy stays up until app finishes draining), and Job-completion compatibility. See Istio and Envoy Proxy.

  • HashiCorp Vault Agent Injector — same mechanism, different sidecar. Injects a Vault Agent container that authenticates to Vault, fetches secrets, writes them to a shared tmpfs, and rotates them. Native-sidecar mode prevents the agent from blocking Job completion and ensures the agent is up before the app reads secrets.

  • Sidecar sprawl in real production. Large fleets routinely run 3–5 sidecars on every Pod (mesh proxy, log shipper, metrics exporter, secret injector, debug agent), and the overhead compounds: at 10,000 Pods × 5 sidecars × 100 MiB, that is 5 TiB of cluster memory consumed by infrastructure rather than application code. The architectural responses include consolidating sidecars (one “platform sidecar” doing logs + metrics + tracing), moving node-scoped concerns to DaemonSets, or adopting Ambient Mesh. See Sidecar Sprawl Anti-Pattern for the full treatment with cited write-ups.

Uncertain

Verify: the specific per-company sidecar-overhead anecdote (an earlier draft attributed it to named Shopify/Lyft/Airbnb engineering blogs). Reason: those exact posts were not re-fetched during this revision, so the attribution is dropped rather than asserted. The arithmetic (replica × sidecar × footprint) is sound regardless. To resolve: locate and cite the specific engineering post if the named-company framing is wanted; otherwise the generic framing above stands. uncertain

  • v1.33 “Octarine” release (Kubernetes blog, April 2025) made sidecars GA. KEP-753 was first opened in 2019, so the feature spent roughly six years from initial proposal to GA — an unusually long alpha/beta tenure that reflected the need to validate the behavior against the major real-world sidecar consumers (service meshes such as Istio and Linkerd, and observability vendors) before locking the API.

  • The pre-native-sidecar workarounds that production teams used 2019–2023 are still worth knowing about: (a) shareProcessNamespace: true so a preStop hook on the main container could kill -TERM 1 the sidecar; (b) wrapping the sidecar in tini or a custom supervisor that watches for a sentinel file; (c) using a dedicated terminator sidecar whose only job was to detect main-container exit and kill the actual sidecars. These hacks are now obsolete in 1.33+ but persist in many production Helm charts.

See Also

Graph View

Backlinks