Logging on Kubernetes

Kubernetes has no built-in cluster-wide logging backend. This is the single most important — and most surprising — fact about logging on K8s. What the platform does provide is narrow: the container runtime captures each container’s stdout and stderr, the kubelet writes those streams to files on the node and rotates them, and kubectl logs reads the current file back. That is the whole of the native capability. The Kubernetes docs are explicit: “Kubernetes does not provide a native storage solution for log data” and “In a cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called cluster-level logging” (kubernetes.io — Logging Architecture). Real logging — searchable, durable, surviving Pod deletion — is something you build on top using one of three architectures, of which the node-level agent DaemonSet has decisively won. This note covers the native primitives, the three architectures, the agents (Fluent Bit, Vector, Fluentd, Promtail/Alloy), the backends (Loki, Elasticsearch/OpenSearch), and why the DaemonSet pattern beat the alternatives. The K8s-API-native event stream is a different signal — see Kubernetes Events — and trace/structured telemetry lives in OpenTelemetry on Kubernetes.

Mental Model

There are two layers. The node level is what K8s gives you: runtime captures stdout/stderr → kubelet writes /var/log/pods/... and symlinks /var/log/containers/*.log → kubectl logs reads the latest file. The cluster level is what you build: an agent reads those node files (or the app ships logs itself), enriches with Pod metadata, and forwards to a durable, queryable backend.

flowchart TB
    subgraph "Node (one of many)"
      APP[app container<br/>writes stdout/stderr]
      RT[container runtime<br/>CRI logging format]
      FILES["/var/log/pods/.../*.log<br/>+ /var/log/containers/*.log<br/>kubelet rotates"]
      AGENT[node log agent<br/>DaemonSet Pod<br/>Fluent Bit / Vector]
      APP --> RT --> FILES
      FILES -->|tail + parse| AGENT
      AGENT -->|enrich w/ Pod metadata| AGENT
    end
    KCTL[kubectl logs] -->|reads latest file only| FILES
    AGENT -->|forward| BACKEND[(Loki / Elasticsearch /<br/>OpenSearch / cloud logging)]
    BACKEND --> QUERY[Grafana / Kibana<br/>search + dashboards]

The two layers of K8s logging. The insight: everything left of BACKEND is ephemeral. Node files rotate and are capped; kubectl logs only ever sees the latest rotated file; when a Pod is deleted its logs are deleted with it. Durability begins only once the agent has forwarded a line to a real backend. The node agent is the bridge between the ephemeral node layer and the durable cluster layer.

Mechanical Walk-through

What the node gives you natively

The container runtime “handles and redirects any output generated to a containerized application’s stdout and stderr streams… the integration with the kubelet is standardized as the CRI logging format” (kubernetes.io — Logging Architecture). Each line is written to a file on the node, conventionally under /var/log/pods/<namespace>_<pod>_<uid>/<container>/, with the readable symlink tree at /var/log/containers/*.log. kubectl logs is a thin reader: it asks the kubelet, which returns the contents of that file.

The contract is deliberately minimal, and three consequences follow:

• The kubelet rotates logs. Configurable via containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5). Critically, “only the contents of the latest log file are available through kubectl logs. For example, if a Pod writes 40 MiB of logs and the kubelet rotates logs after 10 MiB, running kubectl logs returns at most 10MiB of data” (kubernetes.io — Logging Architecture). kubectl logs is not a complete history.
• Restart keeps exactly one prior container. “By default, if a container restarts, the kubelet keeps one terminated container with its logs.” That prior boot is reachable with kubectl logs --previous (-p) — the canonical move for diagnosing a CrashLoopBackOff. Two restarts back, the logs are gone.
• Pod eviction deletes the logs. “If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs.” No agent, no durability — full stop.

kubectl logs essentials:

kubectl logs my-pod                       # latest container's stdout/stderr
kubectl logs my-pod -c sidecar            # a specific container in a multi-container Pod
kubectl logs my-pod --previous            # the PRIOR boot — for crashloop diagnosis
kubectl logs -l app=api --all-containers  # by label across Pods (no cross-Pod merge ordering)
kubectl logs my-pod -f --tail=100         # follow, last 100 lines

System component logs

Kubernetes’ own components log too. The kubelet and container runtime are non-containerized — on systemd hosts their logs go to the journal (journalctl -u kubelet). The scheduler, controller-manager, API server (typically Static Pods), etcd, and kube-proxy (typically a DaemonSet) are containerized and their logs land in /var/log/pods like any other Pod (kubernetes.io — System Logs). Control-plane components use the klog library; verbosity is -v=<level>.

Architecture 1 — node-level agent (the winner)

A logging agent runs as a DaemonSet — exactly one Pod per node. Each agent Pod hostPath-mounts /var/log/containers and /var/log/pods, tails every container log file on its node, parses each line (JSON or regex), enriches it with Kubernetes metadata (namespace, Pod name, labels, node) by querying the API server, and forwards to a backend.

Why this won, decisively:

• Zero application cooperation. The app writes to stdout — the 12-factor convention (12-Factor App Methodology) — and is otherwise unaware logging exists.
• One agent per node, not per Pod. N nodes’ worth of agents, regardless of how many thousands of Pods run. The resource cost is bounded by cluster size, not workload count.
• Captures everything on the node, including Pods that crash before they could ship anything themselves.
• Enrichment is centralized in one well-maintained agent rather than reimplemented in every app.

Agents: Fluent Bit (the modern default — C, tiny footprint, fast), Vector (Rust, high throughput, rich transform language), Fluentd (the older Ruby-based CNCF graduate — heavier, plugin-rich), Promtail / Grafana Alloy (the Loki-native shippers; Promtail is being superseded by Alloy).

Architecture 2 — sidecar logging container

When the application writes logs to a file inside the container rather than stdout, the node agent never sees them. The fix is a sidecar container in the same Pod that either (a) re-streams that file to its own stdout (so the node agent then catches it) or (b) runs a full logging agent itself. The K8s docs recommend the sidecar when “node-level logging agents cannot capture logs” — e.g., a legacy app with hard-coded file logging, or a Pod producing multiple distinct log streams that need separate handling (kubernetes.io — Logging Architecture).

Cost: a sidecar per Pod — multiplied resource overhead, and one more container to manage. Use it as a targeted workaround, not a default.

Architecture 3 — direct shipping from the application

The application uses a logging library that writes straight to the backend (Elasticsearch, Loki, a cloud API) — no node files, no agent. The K8s docs list this as an option but it is discouraged:

• It couples the app to the logging backend (a backend swap is an app redeploy).
• It loses the node agent’s free metadata enrichment.
• Network failures to the backend become application failures or memory pressure.
• It violates the 12-factor “treat logs as event streams” principle — the app should emit to stdout and let the platform route.

Backends

• Grafana Loki — index-only-the-labels, store the rest cheaply (often object storage). Cost-efficient; queried with LogQL; pairs naturally with Grafana and a Prometheus-shaped mental model. The increasingly common default.
• Elasticsearch / OpenSearch — full-text index of every line; powerful search, heavier to run and store. The classic “ELK/EFK” stack (Elasticsearch + Fluentd/Fluent Bit + Kibana). OpenSearch is the Apache-2.0 fork.
• Cloud logging — CloudWatch Logs (EKS), Cloud Logging (GKE), Azure Monitor (AKS). Zero ops, billed per GB ingested/retained, less portable.

Configuration / API Surface

A node-level agent as a DaemonSet (Fluent Bit), abbreviated with commentary:

apiVersion: apps/v1
kind: DaemonSet                       # one agent Pod per node — the winning pattern
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector: { matchLabels: { app: fluent-bit } }
  template:
    metadata: { labels: { app: fluent-bit } }
    spec:
      serviceAccountName: fluent-bit  # needs RBAC to read Pod metadata for enrichment
      tolerations:
      - operator: Exists              # run on EVERY node, including tainted control-plane
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.1
        volumeMounts:
        - { name: varlog,  mountPath: /var/log,                 readOnly: true }
        - { name: varlibdockercontainers,
            mountPath: /var/lib/docker/containers, readOnly: true }
      volumes:
      - name: varlog
        hostPath: { path: /var/log }  # the node's log directory — the data source
      - name: varlibdockercontainers
        hostPath: { path: /var/lib/docker/containers }

The agent’s own pipeline config (Fluent Bit syntax):

[INPUT]
    Name              tail               # tail the per-container log files
    Path              /var/log/containers/*.log
    Tag               kube.*
    multiline.parser  cri                # reassemble CRI-format multi-line entries
 
[FILTER]
    Name                kubernetes       # the K8s metadata enrichment filter
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Merge_Log           On               # parse JSON app logs into structured fields
 
[OUTPUT]
    Name   loki                          # forward to the durable backend
    Match  kube.*
    Host   loki-gateway.logging.svc
    Labels job=fluentbit, $kubernetes['namespace_name']

The kubernetes filter is the analogue of OTel’s k8sattributes processor: it turns a bare log line into a line tagged with namespace, Pod, container, node, and labels — the difference between logs you can query by workload and logs you can’t.

Failure Modes

1. kubectl logs shows nothing useful for a crashed Pod. The current container is a fresh restart with empty logs. Use kubectl logs --previous. Beyond one restart back, only a node agent that already forwarded the lines has them.

2. Logs lost on rotation. A chatty Pod writes faster than 10Mi between agent reads; the kubelet rotates and kubectl logs returns only the latest fragment. The node agent should still have caught the rotated content if it kept up — but if the agent fell behind, rotated files can be deleted before it reads them. Mitigation: size agent resources for peak log throughput; raise containerLogMaxSize.

3. Agent can’t keep up — backpressure. A log spike outpaces the agent’s forward rate. Lines buffer in the agent (memory growth, possible OOMKill) or are dropped. Mitigation: enable disk-backed buffering, tune flush intervals, rate-limit the noisiest Pods.

4. Missing Pod metadata. The agent’s ServiceAccount lacks RBAC to list Pods, so the enrichment filter returns nothing; logs arrive unlabeled and unqueryable-by-workload. Diagnose: check the agent log for forbidden API errors.

5. Multi-line stack traces split into separate lines. Each line of a Java/Python traceback becomes its own backend entry. Mitigation: configure the agent’s multiline parser (the cri parser, plus a language-specific exception parser).

6. hostPath mount missing or wrong. If the DaemonSet doesn’t mount the correct node log directory (paths differ across runtimes and distros), the agent tails nothing. Symptom: agent healthy, backend empty.

7. Logging backend outage cascades. If the backend is down and the app ships directly (Architecture 3), the app stalls or OOMs. With a node agent, the agent buffers and the app is unaffected — a structural argument for the agent pattern.

8. Log volume cost explosion. DEBUG-level logging cluster-wide can dwarf the application’s actual data in storage cost. Mitigation: structured logging with levels, sampling of high-volume noisy logs, drop-filters in the agent.

Alternatives and When to Choose Them

• Node agent (DaemonSet) vs sidecar. Node agent is the default — bounded cost, no app changes. Use a sidecar only when the app insists on logging to a file inside the container, or needs per-stream handling the node agent can’t express.
• Node agent vs direct shipping. Direct shipping is discouraged: it couples the app to the backend and loses enrichment. The exception is a serverless/FaaS context with no node you control.
• Fluent Bit vs Fluentd. Fluent Bit (C, ~MBs of RAM) is the modern node agent; Fluentd (Ruby, heavier) survives as an aggregation tier or where its huge plugin ecosystem is needed. The common shape is Fluent Bit on nodes → Fluentd aggregator → backend.
• Vector — when you need high throughput and a rich in-agent transformation language (VRL); a strong Fluent Bit alternative.
• Promtail / Grafana Alloy — when the backend is Loki. Alloy is Grafana’s unified collector superseding Promtail; it also speaks OTLP, blurring the line with OpenTelemetry on Kubernetes.
• Loki vs Elasticsearch/OpenSearch. Loki: cheap (label-only index), LogQL, Grafana-native — choose when cost matters and label-scoped queries suffice. Elasticsearch/OpenSearch: full-text search, richer querying, heavier — choose when arbitrary text search across all logs is essential.
• OTel filelog receiver vs a dedicated log agent. The OTel Collector can tail container logs too. Choose OTel for logs when you are already running OTel for traces/metrics and want one collector; choose Fluent Bit/Vector when logging is the priority and you want the more mature log-specific tooling. See OpenTelemetry on Kubernetes.

Production Notes

• Log to stdout/stderr, always. This is 12-factor factor XI: treat logs as event streams the app emits, never as files the app manages. Everything in the node-agent architecture depends on it.
• Structured (JSON) logging is the modern standard. A JSON log line the agent parses into fields is queryable by field (level, trace_id, user_id); a free-text line is only grep-able. Structured logging plus a trace_id field is what makes logs correlatable with traces from OpenTelemetry on Kubernetes.
• The DaemonSet node agent won because the economics are unbeatable. Cost scales with node count; apps need zero awareness; crashed Pods are still captured; enrichment is centralized. Every managed K8s logging integration (CloudWatch Container Insights, GKE Cloud Logging, AKS Container Insights) is this exact pattern under the hood.
• kubectl logs is for now, not history. It reads the latest node file. For “what happened during last night’s incident” you need the durable backend. Treat kubectl logs as a live tail, not an archive.
• Mind the cost. Log ingestion/retention is frequently a top-three observability line item. Set log levels deliberately, sample noisy paths, and drop-filter known-junk lines at the agent before they hit paid storage.
• Logs, metrics, traces, Events are four distinct signals. Logs = what happened inside the container. Metrics (Prometheus on Kubernetes) = numeric time series. Traces (OpenTelemetry on Kubernetes) = request paths across services. Events (Kubernetes Events) = what K8s did to objects. Correlate them via shared IDs (trace_id, Pod name); don’t expect one to do another’s job.

See Also

• DaemonSet — the K8s object backing the winning node-agent pattern
• OpenTelemetry on Kubernetes — traces and structured telemetry; OTel’s filelog receiver overlaps with log agents
• Prometheus on Kubernetes — the metrics signal, sibling pillar of observability
• Kubernetes Events — the K8s-API-native event stream, a different signal from logs
• Pod Lifecycle and Containers Status — kubectl logs --previous is core to crashloop debugging
• 12-Factor App Methodology — factor XI (“logs as event streams”) is the foundation of the node-agent model
• Static Pods — control-plane components whose logs land in /var/log/pods
• kubelet — the component that writes and rotates node log files
• Container Runtime Interface — defines the CRI logging format the runtime emits
• kubectl — kubectl logs is the native reader
• Sidecar Containers — the deployment shape of Architecture 2
• Kubernetes MOC — §14 Observability, umbrella index