vault

Kubernetes Events

12 min read

Kubernetes Events

A Kubernetes Event is a first-class API object that records something that happened to some other object — a Pod was Scheduled, an image is Pulling, a container Started, a liveness probe failed (Unhealthy), the scheduler FailedScheduling, a container is in BackOff. Events are the cluster’s running commentary: when you run kubectl describe pod and read the timeline at the bottom, you are reading Events. They are emitted by controllers and the kubelet through an EventRecorder, stored in etcd via the API server, and — critically — deliberately short-lived: the API server expires every Event after a default one hour (kube-apiserver --event-ttl, default 1h0m0s). This single fact governs how Events should and should not be used: they are an excellent real-time debugging signal and a terrible durable audit log. The modern events.k8s.io/v1 API group reached GA in Kubernetes v1.19, modernizing the legacy core/v1 Event with better aggregation semantics. This note explains the Event object, its lifecycle, deduplication, and why durable retention requires shipping Events off-cluster. For the underlying audit trail Events are not, see Kubernetes Audit Logging.

Mental Model

An Event is a tiny K8s object with a “regarding” reference to the object it describes, a reason (machine-readable, short), a note/message (human-readable), a type (Normal or Warning), and timing. It is created by a producer (the kubelet, a controller, the scheduler), persisted by the API server with a one-hour TTL, and read by humans via kubectl or by tooling via watch.

flowchart LR
    subgraph "Producers"
      KUBELET[kubelet]
      SCHED[kube-scheduler]
      CTRL[controllers<br/>kube-controller-manager,<br/>operators]
    end
    REC[EventRecorder<br/>client-go<br/>dedup + aggregate]
    API[(kube-apiserver)]
    ETCD[(etcd<br/>TTL = 1h)]
    subgraph "Consumers"
      KCTL[kubectl describe /<br/>get events / events]
      EXP[event-exporter]
    end
    KUBELET --> REC
    SCHED --> REC
    CTRL --> REC
    REC -->|create / patch| API
    API --> ETCD
    API -->|read / watch| KCTL
    API -->|watch| EXP
    EXP -->|durable| SINK[(Elasticsearch /<br/>cloud logging /<br/>Slack)]

The Event lifecycle. The insight: the TTL on etcd is the load-bearing fact. Events vanish after an hour by design — to keep etcd from filling with stale chatter. Anything that needs Events to survive longer (audit, post-incident forensics, dashboards) must watch them out to a durable sink before they expire. kubectl is a consumer, not a store.

Mechanical Walk-through

What an Event looks like

Reading them is the daily operation:

kubectl get events                         # all events in the namespace
kubectl get events --sort-by=.lastTimestamp # chronological (default order is unstable)
kubectl get events -A                       # all namespaces
kubectl events --for=pod/my-pod             # filter to one object (kubectl events, 1.23+)
kubectl describe pod my-pod                 # Events section at the bottom

Typical output:

LAST SEEN   TYPE      REASON              OBJECT          MESSAGE
2m          Normal    Scheduled           pod/my-pod      Successfully assigned default/my-pod to node-3
2m          Normal    Pulling             pod/my-pod      Pulling image "myorg/app:v2"
118s        Normal    Pulled              pod/my-pod      Successfully pulled image in 1.4s
117s        Normal    Created             pod/my-pod      Created container app
117s        Normal    Started             pod/my-pod      Started container app
30s         Warning   Unhealthy           pod/my-pod      Liveness probe failed: HTTP 500
12s         Warning   BackOff             pod/my-pod      Back-off restarting failed container

Common reasons

The reason field is a short CamelCase token. The ones every operator should recognize:

ReasonEmitted byMeaning
ScheduledschedulerPod bound to a node.
FailedSchedulingschedulerNo node satisfies the Pod’s constraints — Warning.
Pulling / PulledkubeletImage download started / finished.
FailedkubeletImage pull failed (paired with ErrImagePull/ImagePullBackOff container state).
Created / StartedkubeletContainer object created / process started.
KillingkubeletContainer being terminated (rollout, probe failure, deletion).
BackOffkubeletCrash-loop back-off restarting a failed container — Warning.
UnhealthykubeletA liveness/readiness/startup probe failed — Warning.
PreemptingschedulerThis Pod is being preempted for a higher-priority Pod.
EvictedkubeletNode-pressure eviction.
FailedMountkubeletVolume could not be attached/mounted — Warning.
SuccessfulCreate / SuccessfulDeletecontrollersA ReplicaSet/Job created or deleted a Pod.

Warning is the type to alert on; Normal is routine narration.

events.k8s.io/v1 vs the legacy core/v1 Event

There are two Event APIs for historical reasons:

  • core/v1 Event — the original, in the core API group. Aggregates repeats by incrementing a single count field and updating lastTimestamp. Reporter info lives in source.component/source.host; the human text is message; the subject object is involvedObject (core/v1 Event reference).
  • events.k8s.io/v1 Event — the modernized API, GA in v1.19 (Kubernetes 1.19 release). It splits reason (why) from a new explicit action (what was done), uses required microsecond-precision eventTime, models repeats with a structured series sub-object (series.count, series.lastObservedTime), names the reporter with reportingController + reportingInstance, and renames the subject reference to regarding (with an optional secondary related) (events/v1 Event reference).

The two APIs are explicitly bridged for compatibility. The events/v1 type carries deprecatedCount, deprecatedFirstTimestamp, deprecatedLastTimestamp, and deprecatedSource fields whose Go doc comments state verbatim that each is “the deprecated field assuring backward compatibility with core.v1 Event type” (k8s.io/api/events/v1). The mapping is mechanical: the old count becomes deprecatedCount (new events must instead use series.count); source.component/source.host become deprecatedSource.component/deprecatedSource.host (new events must use reportingController/reportingInstance); and the old firstTimestamp/lastTimestamp become the deprecated timestamps. The API server rejects a new events/v1 Event that populates the deprecated fields instead of the modern ones, which is how it forces producers onto the new shape while still letting old core/v1 readers see a sensible object.

Uncertain

Verify: that the two Event APIs are served from one shared etcd backend (an Event created via core/v1 is visible via events.k8s.io/v1 and vice-versa). Reason: the API references and the Go type docs confirm the field-level backward-compat bridge but do not explicitly state a single shared storage backend; this is asserted from the bridging design, not from a primary source that names it. To resolve: check the apiserver registry/storage wiring (pkg/registry/core/event and pkg/registry/events) in kubernetes/kubernetes, or a KEP describing the events.k8s.io storage.

The practical takeaway: new controllers should use events.k8s.io/v1 via client-go’s events recorder; kubectl get events reads whichever is convenient and the data is the same.

Deduplication and aggregation

A crash-looping Pod could emit a BackOff Event every few seconds — thousands per hour. To stop Events from drowning etcd, the EventRecorder in client-go deduplicates and aggregates: an identical Event (same regarding object, reason, message, source) is not stored as a new object. Instead the existing Event’s count (core/v1) or series.count (events.k8s.io/v1) is incremented and its last-seen time updated. This is why kubectl get events shows one BackOff line with LAST SEEN 12s and a high count, not 300 separate lines. Aggregation happens client-side in the producer, before the API call — it is a property of the recorder library, not the API server.

How controllers emit Events — the EventRecorder

A controller does not POST raw Event objects. It uses an EventRecorder from client-go (k8s.io/client-go/tools/record). The pattern:

recorder.Eventf(pod, corev1.EventTypeWarning, "FailedScheduling",
    "0/12 nodes are available: insufficient memory")

The recorder fills in the timestamps, the involvedObject/regarding reference, the source/reporting fields, runs dedup/aggregation, and asynchronously creates or patches the Event via the API server. Every operator built with controller-runtime gets an EventRecorder from the manager — emitting Events for significant reconcile outcomes is standard operator hygiene, because it surfaces the operator’s reasoning in kubectl describe.

The one-hour TTL

The API server expires Events. kube-apiserver --event-ttl defaults to 1h0m0s (kube-apiserver reference). When an Event is created it gets a one-hour TTL in etcd; if the same Event recurs (count incremented), the last-seen timestamp is updated and the TTL is refreshed from that moment (Platform9 KB). After an hour of silence the Event is garbage-collected by etcd’s TTL mechanism. On managed Kubernetes the kube-apiserver flags are operated by the provider and --event-ttl is generally not user-tunable — the one-hour window is effectively fixed, and AWS’s own guidance is to forward control-plane events to a durable store (e.g. via EventBridge / control-plane logging) rather than rely on the in-cluster retention (Managing Kubernetes control plane events in Amazon EKS). This is the same conclusion the durability section reaches: keep Events transient in etcd and stream them out for retention.

Configuration / API Surface

An events.k8s.io/v1 Event object, annotated:

apiVersion: events.k8s.io/v1
kind: Event
metadata:
  name: my-pod.17a3f2...           # generated; <object>.<hash>
  namespace: default
eventTime: "2026-05-16T10:42:13.123456Z"   # microsecond precision
reportingController: kubernetes.io/kubelet # who emitted it
reportingInstance: node-3                  # which instance
action: Pulling                            # machine-readable: what was done
reason: Pulling                            # machine-readable: why
note: 'Pulling image "myorg/app:v2"'       # human-readable (<=1kB)
type: Normal                               # Normal | Warning
regarding:                                 # the object this is ABOUT
  apiVersion: v1
  kind: Pod
  name: my-pod
  namespace: default
  uid: 8f3a...
series:                                    # set only if this recurred
  count: 4
  lastObservedTime: "2026-05-16T10:45:02.0Z"

Reading Warnings cluster-wide — the canonical triage query:

# Every Warning event across the cluster, newest last.
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp
 
# Watch events for a specific object as they arrive.
kubectl events --for=deployment/my-app --watch

Failure Modes

  1. “The Events are gone.” An incident is investigated two hours after it started; the FailedScheduling/OOMKilled-driven BackOff Events have expired. Events are not a forensic record. Mitigation: ship Events to a durable sink (see Alternatives) before you need them.

  2. Events used as an audit log. Events answer “what is the kubelet/scheduler doing,” not “who called the API and were they authorized.” They are not authenticated-actor records and they expire. The real audit trail is Kubernetes Audit Logging. Conflating the two is a security-review red flag.

  3. Event storm fills etcd. A pathological controller emitting non-aggregatable Events (each with a unique message) defeats dedup and writes thousands of distinct Event objects. etcd write load spikes; the one-hour TTL eventually clears it but the cluster degrades meanwhile. Mitigation: ensure emitted Event messages are stable strings, not message-with-timestamp; aggregation depends on identical text.

  4. kubectl get events default ordering is not chronological. Events list in name order, not time order. Reading them without --sort-by=.lastTimestamp (or .metadata.creationTimestamp) gives a misleading timeline. This trips up nearly everyone once.

  5. Missing Events because they expired mid-debug. A flaky issue reproduces, you kubectl describe 70 minutes later, the Events are gone and the Pod looks fine. Capture state immediately, or rely on a streamed sink.

  6. Operator emits no Events. A custom operator that never calls its EventRecorder is opaque — kubectl describe <customresource> shows nothing about why reconciliation is stuck. Not a failure of K8s, but a failure of operator hygiene.

Alternatives and When to Choose Them

  • kubernetes-event-exporter — watches the Event stream and forwards to Elasticsearch, cloud logging, Slack, webhooks. The standard way to make Events durable and alertable. Choose this when you need post-incident Event history or Slack notifications on Warnings.
  • kube-state-metrics — turns object state into Prometheus metrics, and many of the signals that cause Warning events also surface there as metrics (kube_pod_container_status_restarts_total for crash loops, kube_pod_status_phase{phase="Pending"} for unschedulable Pods). Note the important caveat: KSM watches object kinds, not the Event stream — there is no Event resource in its default metric set (its documented metric families are workload/cluster/storage/etc., with no “events” category). So KSM is the right way to alert on the underlying conditions via metrics, but it is not a generic “count Events by reason” exporter — for that, use an event-exporter that emits Prometheus metrics. See kube-state-metrics.
  • kspan / kube-events — turn Events into OpenTelemetry traces / structured streams. Niche; useful for correlating Events into causal timelines.
  • Audit logs — for who did what to the API, not what the system did to objects. Different question, different mechanism — Kubernetes Audit Logging.
  • Application logs — for what happens inside a container. Events describe the container’s lifecycle from K8s’s view, not its internal behavior — Logging on Kubernetes.
  • Raising --event-ttl — possible on self-managed clusters, but it grows etcd. Streaming Events out is almost always the better answer than retaining them in etcd longer.

Production Notes

  • Events are a debugging signal, not a system of record. Internalize the one-hour TTL. The first instinct in an incident should be to capture state (kubectl get events -A -o yaml > events.yaml) before the window closes.
  • kubectl describe is the fastest Event reader. It shows only the Events regarding that object, already correlated — far more useful than scanning kubectl get events. The first-line diagnostic for a misbehaving Pod is kubectl describe pod → read the Events section → read containerStatuses (Pod Lifecycle and Containers Status).
  • Alert on Warning events, not all events. type=Warning is the actionable subset. A common pattern: kubernetes-event-exporter filters type=Warning to Slack (and can emit its own Prometheus counters), while PrometheusRule alerts on the underlying conditions use kube-state-metrics object-state metrics (restart counters, Pending-phase gauges) rather than Event counts.
  • Operators should emit Events generously. A well-behaved operator narrates its reconcile decisions through its EventRecorder — it makes kubectl describe <cr> self-explanatory and is the cheapest observability an operator can offer.
  • Don’t build dashboards on raw Events. They expire and aggregate; counts are approximate. Build dashboards on object-state metrics (kube-state-metrics) and export Events to a searchable sink for history — use each for what it’s good at.
  • The kubectl events command (1.23+) is a better Event reader than kubectl get events — --for=<kind>/<name> filtering and --watch make it the modern default for live triage.

See Also

Graph View

Backlinks