vault

Kubernetes Audit Logging

11 min read

Kubernetes Audit Logging

Kubernetes auditing is the chronological, security-relevant record of who did what, when, and with what result in a cluster (Kubernetes — Auditing). Because every cluster mutation passes through kube-apiserver — the sole writer of cluster state, see API Server Request Flow — the apiserver is the natural and authoritative place to record this trail. An audit handler inside the apiserver generates an audit event for every request and processes it according to an audit policy file: a list of rules matching (users, groups, verbs, resources, namespaces) tuples to an audit level (None, Metadata, Request, RequestResponse) that controls how much of the request is captured. Events are written to a log backend (a file) and/or a webhook backend (a remote service, typically a SIEM). The central design tension is forensic completeness vs. volume: capturing full request and response bodies for everything is unaffordable, so audit policy is the craft of logging enough to investigate an incident without drowning in noise.

Mental Model

Audit logging is not application logging and not the Events API. Application logs are what containers write to stdout; Events are short-lived, best-effort status notifications. The audit log is the immutable evidentiary record of API access — the thing a security team reads after an incident to answer “which credential deleted the production database, from which IP, at what time, and did it succeed?”

Two knobs control every audit event. The level (a vertical axis) controls depth — from “don’t log this at all” to “log the full request and response body.” The stage (a horizontal axis) controls when in the request’s life an event is emitted. A single request can produce up to four events, one per stage. The audit policy is a list of rules; the first matching rule wins and sets the level — so rule ordering is load-bearing, exactly like a firewall ACL.

flowchart TD
    REQ["API request hits kube-apiserver"] --> H["audit handler"]
    H --> P{"audit policy:<br/>first matching rule"}
    P -->|"level: None"| DROP["not logged"]
    P -->|"level: Metadata"| M["who / verb / resource / when<br/>(no bodies)"]
    P -->|"level: Request"| RQ["metadata + request body"]
    P -->|"level: RequestResponse"| RR["metadata + request body<br/>+ response body"]
    M --> STG
    RQ --> STG
    RR --> STG
    STG["emit at stages:<br/>RequestReceived → ResponseStarted<br/>→ ResponseComplete → Panic"] --> B1["log backend<br/>(JSONlines file)"]
    STG --> B2["webhook backend<br/>(remote API → SIEM)"]

The audit pipeline. Insight to extract: level chooses depth (the four boxes after the policy gate), stage chooses when (the four-stage box), and the first matching policy rule decides the level. A request can emit several events; the same event can fan out to both backends.

Mechanical Walk-through

The audit policy file

Auditing is off by default — without --audit-policy-file, the apiserver logs nothing. The policy is a Policy object (apiVersion: audit.k8s.io/v1) with a required rules list. Each rule matches some subset of requests and assigns a level. The first rule that matches a request wins; a final catch-all rule sets the default. Rules can match on users, userGroups, verbs, resources (group + resource + optional resourceNames), namespaces, nonResourceURLs, and per-rule omitStages.

Audit levels — the depth axis

Four levels, from least to most data (Kubernetes — Auditing):

  • None — do not log requests matching this rule. Used to suppress high-volume noise (leader-election ConfigMap updates, health probes, the kubelet’s relentless node-status patches).
  • Metadata — log request metadata only: the requesting user, the source IP, the verb, the resource, the timestamp, the response status code — but not the request or response body. This is the sane default for a catch-all rule: it answers “who did what” cheaply.
  • Request — log metadata plus the request body. You see what was submitted — the full Pod spec that was created, the patch that was applied. Does not log non-resource requests.
  • RequestResponse — log metadata, the request body, and the response body. The most complete and the most expensive: you see both what was asked and what the apiserver returned (including, for a GET/LIST, the full object contents). Does not log non-resource requests.

A subtle point: the level applies per matching rule, and the level is also clamped by the request — a RequestResponse rule on a watch produces no body, since a watch has no single response object.

Audit stages — the timing axis

Each request passes through up to four stages, and an event can be emitted at each (audit.k8s.io reference):

  • RequestReceived — emitted the moment the audit handler receives the request, before any processing. Useful to detect requests that the apiserver received but never finished. Frequently suppressed via omitStages because it roughly doubles event volume.
  • ResponseStarted — emitted once response headers are sent but before the body. Relevant only for long-running requests like watch, where the body streams indefinitely.
  • ResponseComplete — emitted when the response body has finished and no more bytes will be sent. This is the primary event for most requests — it carries the outcome.
  • Panic — emitted when the apiserver panics handling the request.

omitStages (global or per-rule) drops uninteresting stages; omitting RequestReceived is standard practice.

The audit event

Each event is a structured JSON object: an auditID (correlates the multiple stage-events of one request), stage, level, verb, user (username, groups, UID), sourceIPs, objectRef (resource, namespace, name), requestReceivedTimestamp, responseStatus, and — at Request/RequestResponse levels — requestObject and responseObject. For a PATCH, the requestObject is the JSON-patch array of operations, not a full object.

Backends

Two backends, usable simultaneously:

  • Log backend — writes events to a file in JSONlines format, configured by --audit-log-path, with rotation via --audit-log-maxage (days retained), --audit-log-maxbackup (number of rotated files), and --audit-log-maxsize (megabytes before rotation). The file is local to the control-plane node, so on multi-master clusters each apiserver writes its own file.
  • Webhook backend — POSTs audit events to a remote HTTP API, configured by --audit-webhook-config-file (a kubeconfig-shaped file pointing at the receiver). This is how audit events reach a central collector. There is also a dynamic audit-sink mechanism in some versions, but the static webhook is the stable path.

Shipping to a SIEM

The audit log only has forensic value if it is aggregated, retained, and queryable somewhere durable and outside the cluster — an attacker who roots a control-plane node can tamper with a local file. The standard pattern: the webhook backend (or a log shipper tailing the JSONlines file) forwards events to a SIEM (Splunk, Elastic, a cloud logging service) where they are indexed, retained per compliance policy, and alerted on. Managed Kubernetes services do this for you: EKS streams audit logs to CloudWatch, GKE to Cloud Logging, AKS to Azure Monitor diagnostic settings — see the same point in API Server Request Flow’s production notes.

Configuration / API Surface

A representative audit policy plus the apiserver flags that enable it:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"                              # globally drop the RequestReceived stage (halves volume)
rules:
  # 1. Suppress noisy, low-value, high-frequency traffic FIRST (first match wins).
  - level: None
    users: ["system:kube-scheduler", "system:kube-controller-manager"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version"]
 
  # 2. Secrets/configmaps: log WHO touched them, but NEVER the body.
  #    RequestResponse here would dump secret values into the audit log.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
 
  # 3. Writes to RBAC objects: full request body — privilege-escalation forensics.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
 
  # 4. Writes to workloads: request body is enough to reconstruct what was deployed.
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "apps"
        resources: ["deployments", "daemonsets", "statefulsets"]
      - group: ""
        resources: ["pods"]
 
  # 5. Catch-all: metadata for everything else (who/what/when, cheap).
  - level: Metadata
# kube-apiserver flags (typically a static-pod manifest edit on the control plane)
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log   # log backend
--audit-log-maxage=30                                  # retain rotated logs 30 days
--audit-log-maxbackup=10                               # keep 10 rotated files
--audit-log-maxsize=100                                # rotate at 100 MB
--audit-webhook-config-file=/etc/kubernetes/audit-webhook.yaml   # webhook backend (optional)

Line-by-line: rule order is everything — the None suppression rules (1) come first, or the catch-all would log the scheduler’s chatter. Rule 2 is the canonical secrets decision: Metadata, never Request/RequestResponse, because the body is the secret. Rule 3 uses RequestResponse on RBAC writes — privilege escalation is the highest-value forensic target, worth the full body. Rule 4 uses Request (not RequestResponse) on workloads — the submitted spec is enough; the response would just duplicate it. Rule 5 is the cheap Metadata floor. The global omitStages: [RequestReceived] roughly halves event count.

Failure Modes

  • Audit log explosion. A Request or especially RequestResponse rule on a high-traffic resource generates gigabytes per hour. Events are the classic trap — extremely high write volume and low forensic value. RequestResponse on secrets or on LIST verbs is the most expensive possible choice. The dispatch’s framing is exact: the volume problem is the audit-policy design problem.
  • Secret leakage into the log. A Request/RequestResponse rule matching secrets writes plaintext secret values into the audit log — which then sits on disk and flows to a SIEM, multiplying the blast radius of a leak. Secrets must be Metadata-level. This is the single most important audit-policy safety rule.
  • Apiserver memory pressure. Audit logging increases apiserver memory consumption; buffering events for the webhook backend, and holding request/response bodies for high-level rules, both cost RAM. An aggressive policy on a busy cluster can push the apiserver toward OOM.
  • Local file is not tamper-proof. The log backend writes a file on the control-plane node. An attacker with node access can edit or delete it. Forensic integrity requires shipping events off-node to an append-only store as they are produced.
  • Webhook backend backpressure. If the audit webhook receiver is slow or down, events buffer in the apiserver (batched mode) — and the buffer is bounded, so a prolonged receiver outage means lost audit events (a silent gap in the forensic record).
  • Multi-master fragmentation. Each apiserver writes its own log file; an investigation must aggregate across all control-plane nodes. Centralized shipping solves this.
  • No retroactivity. Auditing only records from the moment a policy is enabled. There is no way to reconstruct what happened before --audit-policy-file was set — enable it on day one.

Alternatives and When to Choose Them

  • Kubernetes Events — the API’s own event stream (kubectl get events). Events are best-effort, short-TTL operational notifications, not a security record; they are for “why did my Pod fail to schedule,” not “who deleted it.” Complementary, not an alternative.
  • Falco with the Kubernetes Audit Logs plugin — Falco can consume the audit event stream as a plugin source and apply detection rules to it (alerting on, say, a privileged Pod creation or an anonymous request). This turns the passive audit trail into active detection. The natural pairing: the apiserver produces audit events, Falco evaluates them in real time, a SIEM retains them.
  • Cloud provider audit pipelines — EKS/GKE/AKS expose the audit stream into their native logging service with retention, query, and alerting built in. On managed clusters this is usually the path of least resistance and you do not control the apiserver flags directly.
  • eBPF-based runtime auditing (Falco syscalls, Tetragon) — audits kernel-level behavior rather than API access. Different layer entirely: API audit logging sees kubectl delete pod; syscall auditing sees the execve inside the container. Both are needed for full coverage.

Production Notes

  • The audit policy is iteratively tuned. Start with a mostly-Metadata policy plus targeted None suppressions, measure the log volume, then selectively raise specific high-value resources (RBAC objects, ServiceAccounts, admission webhook configs) to Request/RequestResponse.
  • Compliance frameworks (SOC 2, PCI-DSS, FedRAMP, HIPAA) effectively mandate API audit logging with defined retention — the --audit-log-maxage and SIEM-retention numbers are usually dictated by the relevant standard, not chosen freely.
  • High-value monitoring targets to alert on in the SIEM: requests from system:anonymous, any use of the system:masters group, create/patch of RBAC and ServiceAccount objects, exec/attach into pods, and changes to ValidatingWebhookConfiguration/MutatingWebhookConfiguration (an attacker disabling policy enforcement).
  • On self-managed clusters the policy file and apiserver flags are edited in the apiserver static Pod manifest (/etc/kubernetes/manifests/kube-apiserver.yaml); kubelet restarts the apiserver Pod on manifest change.
  • Pair audit logging with Kubernetes RBAC: RBAC limits what each identity can do; audit logging records what each identity actually did. One is prevention, the other is detection and forensics — you want both.

See Also

Graph View

Backlinks