vault

API Server Request Flow

16 min read

API Server Request Flow

Every interaction with a Kubernetes cluster — a kubectl apply, a controller-loop watch, a kubelet status patch, a service-account token request — is an HTTP request against kube-apiserver, the sole component permitted to read or write the cluster’s source of truth in etcd. Each request passes through a fixed pipeline: TLS termination → authentication → authorization → mutating admission → schema validation → validating admission → storage → watch fan-out (Kubernetes — Controlling Access). Stages reject early on failure, mutate the request mid-flight, and finally publish a change event to every interested watcher. Understanding this pipeline is the prerequisite for understanding everything else in Kubernetes: RBAC, webhooks, operators, even the existence of Watch and Informers.

Mental Model

The apiserver is not a database front-end; it is a policy-checking, schema-enforcing, audit-logging proxy in front of etcd, with a publish/subscribe sidechannel on every successful write. The eight stages form a fixed linear pipeline for mutating verbs (POST/PUT/PATCH/DELETE); read verbs (GET/LIST/WATCH) skip admission entirely (admission cannot reject reads — Kubernetes — Admission Controllers). Every stage is short-circuiting: any rejection terminates the request immediately with an HTTP error, and only the storage stage produces persistent side effects.

sequenceDiagram
    autonumber
    participant C as Client<br/>(kubectl / controller / kubelet)
    participant TLS as TLS Layer
    participant AuthN as Authentication
    participant AuthZ as Authorization
    participant MA as Mutating Admission
    participant SV as Schema Validation
    participant VA as Validating Admission
    participant ETCD as etcd Storage
    participant WC as Watch Cache
    participant W as Subscribed Watchers

    C->>TLS: HTTPS POST /apis/apps/v1/...
    TLS->>AuthN: decrypted request + client cert chain
    AuthN->>AuthZ: User, Groups, UID, Extra
    AuthZ->>MA: allow (verb x resource x apiGroup x ns)
    MA->>SV: object (possibly mutated)
    SV->>VA: schema-valid object
    VA->>ETCD: validated object + dryRun flag
    ETCD-->>WC: revision N committed
    WC-->>W: ADDED/MODIFIED/DELETED event<br/>resourceVersion=N
    ETCD-->>C: HTTP 201/200 + object

The eight-stage pipeline. The insight to extract: each stage has a single, narrow responsibility and a clear contract with the next stage. The arrows are not symmetric — read verbs skip stages 4–6 entirely, and only stage 7 mutates etcd. The watch fan-out (stage 8) is fundamentally a side-effect of storage commit, not a separate request handler.

Mechanical Walk-through

Stage 1: TLS handshake (transport)

All apiserver traffic is TLS-terminated. In-cluster traffic typically negotiates HTTP/2 + protobuf for efficiency — protobuf is the default wire format between core control-plane components and reduces both CPU and bytes-on-the-wire compared to JSON (Kubernetes — API Concepts). External kubectl traffic typically uses HTTP/1.1 + JSON because human-facing tooling expects readability. The TLS layer also extracts the client certificate, if presented, for use by stage 2.

The apiserver’s serving certificate, the client-CA bundle for cert-based authentication, the request-header CA (for the aggregation proxy), and the etcd client cert are configured at startup via --tls-cert-file, --client-ca-file, --requestheader-client-ca-file, and --etcd-cafile respectively.

Stage 2: Authentication — who are you?

The apiserver runs a chain of authenticator modules (Kubernetes — Authenticating). Each module inspects the request and either:

  • returns an authenticated identity (short-circuits the chain), or
  • declines (passes to the next module).

Supported authenticators include:

  • X.509 client certificates — the certificate’s CN becomes the username, O fields become groups. Enabled by --client-ca-file. The primary mechanism for control-plane components and the original kubeadm admin kubeconfig.
  • Bearer tokens — sent as Authorization: Bearer <token>. Includes static-token files (deprecated for production), bootstrap tokens (system:bootstrappers: for kubeadm node joins), and projected service-account tokens (audience-scoped, time-bound JWTs signed by the apiserver).
  • Service-account tokens — the legacy and projected variants. Projected tokens (the post-1.21 default; BoundServiceAccountTokenVolume stable in 1.22) include audience, expiry, and a binding to the Pod’s UID, replacing the long-lived static-Secret pattern.
  • OpenID Connect (OIDC) — JWT verification against a configured issuer; --oidc-issuer-url, --oidc-client-id, --oidc-username-claim, --oidc-groups-claim. The standard way to integrate corporate identity (Okta, Auth0, AAD, Google).
  • Webhook token authenticator — POST the bearer token to a remote HTTPS endpoint, which returns a TokenReview with the identity. Used by managed services (EKS aws-iam-authenticator is the canonical example) to plug cloud IAM into K8s authn.
  • Authenticating proxy — trust a front proxy (verified via --requestheader-client-ca-file) to set X-Remote-User, X-Remote-Group, X-Remote-Extra-* headers. The mechanism by which the API Aggregation Layer forwards identity to extension API servers.
  • Anonymous — requests that no authenticator claims become user system:anonymous, group system:unauthenticated. Controlled by --anonymous-auth (default true outside AlwaysAllow mode).

On success, the request carries a user identity object:

User:    "alice@example.com"
UID:     "73fbcf2c-..."
Groups:  ["system:authenticated", "developers", "team-payments"]
Extra:   {"scopes.authentication.k8s.io": ["openid","email"]}

These fields are opaque to the authenticator; they acquire meaning only in stage 3.

Kubernetes 1.30 Beta introduced Structured Authentication Configuration (Kubernetes 1.30 blog) — a YAML config file pointed to by --authentication-config that lets administrators configure multiple JWT issuers, apply CEL expressions to extract or transform claims, and reload without restarting the apiserver. This is the modern replacement for the per-flag OIDC configuration.

Stage 3: Authorization — what may you do?

The apiserver runs a chain of authorizers (Kubernetes — Authorization Overview). Each authorizer is invoked with the request’s attribute tuple:

  • user, groups, extra (from stage 2)
  • HTTP verb mapped to API verb: POST → create, GET → get/list, PUT → update, PATCH → patch, DELETE → delete/deletecollection, plus watch (a GET with ?watch=true)
  • resource (e.g. pods), subresource (e.g. status, scale, exec, log)
  • apiGroup ("" for core, apps, batch, networking.k8s.io, …)
  • namespace (empty for cluster-scoped resources)
  • name (when targeting a specific object)
  • resourceRequest flag (true for API resources, false for non-resource URLs like /healthz)

Each authorizer returns allow, deny, or no opinion. The decision logic is:

  • Any authorizer denies → reject (HTTP 403).
  • Any authorizer allows → permit, skip remaining authorizers.
  • All authorizers return no opinion → reject (default deny).

Configured authorizers (in order, set by --authorization-mode):

  • Node — special-purpose authorizer; restricts each kubelet to read/write only the resources tied to its own Node (its Pods’ status, its Pod’s mounted Secrets/ConfigMaps, etc.). Enabled in conjunction with the NodeRestriction admission plugin.
  • RBAC — Role/ClusterRole + RoleBinding/ClusterRoleBinding lookup over the verb × resource × apiGroup × namespace tuple. The canonical authorizer for human and ServiceAccount identities. See Kubernetes RBAC.
  • ABAC — flat-file policy of attribute predicates; pre-RBAC era; rarely used.
  • Webhook — POST the SubjectAccessReview to a remote service. The mechanism behind cloud-IAM-mapped authorization on managed services and the OPA-driven authorization patterns.
  • AlwaysAllow / AlwaysDeny — for testing and bootstrap; never for production.

Stage 4: Mutating admission — modify the object

For create / update / delete (admission does not see reads), the apiserver invokes mutating admission plugins in a fixed compiled-in order (not the order they appear in --enable-admission-plugins), each receiving the current object and either modifying it or rejecting the request (Kubernetes — Admission Controllers). Built-in mutators include:

  • ServiceAccount — injects the namespace’s default ServiceAccount and a projected token volume into Pods that don’t specify one.
  • DefaultTolerationSeconds — adds the node.kubernetes.io/not-ready and unreachable tolerations to Pods that lack them (the “300 seconds before eviction” defaults).
  • DefaultStorageClass — assigns the default StorageClass to PVCs that don’t specify one.
  • DefaultIngressClass — same for Ingress.
  • AlwaysPullImages — forces imagePullPolicy: Always (typically off; security-hardening clusters enable it).
  • Priority — resolves a Pod’s PriorityClass name to a numeric priority.
  • MutatingAdmissionPolicy — declarative CEL-based mutation, the in-process counterpart to mutating webhooks. It went GA / stable in v1.36 (alpha v1.32, beta v1.34), enabled by default (Kubernetes — Mutating Admission Policy; v1.36 release blog). It applies ApplyConfiguration (server-side-apply-style) or JSON Patch mutations expressed in CEL, avoiding the latency and availability cost of a webhook callout — the mutating analogue of ValidatingAdmissionPolicy.
  • MutatingAdmissionWebhook — invokes registered external webhooks. The mechanism behind sidecar injectors (Istio, Linkerd), secrets injectors (Vault), and admission-time defaulting in operators.

Mutators may reject as well as modify; if any returns an error the request fails. The order in which built-in plugins are listed in --enable-admission-plugins does not matter — the apiserver runs its compiled-in plugins in a fixed predetermined order, and the flag only toggles which are active (Kubernetes — Admission Control). Within the mutating phase, built-in mutating plugins are re-run if a mutating webhook changes the object, so that a built-in plugin can react to a sub-structure a webhook just added; this is the reinvocationPolicy: IfNeeded convergence behavior, deliberately conservative because no single ordering of mutators works for every case.

Stage 5: Schema validation — does the object even type-check?

The post-mutation object is validated against the OpenAPI schema registered for that resource type. Schema validation rejects requests with unknown fields (under fieldValidation=strict, the modern default for kubectl apply since 1.25), wrong types, or constraint violations (enum, pattern, min/max). For CRDs, the schema is the one declared in the CRD spec; for built-in resources, it is compiled into the apiserver binary.

This stage is the cheap, structural check before the more expensive validating admission stage.

Stage 6: Validating admission — final policy gate

Validating admission plugins are invoked after all mutations have settled and the schema is known good. The validating phase itself has a documented internal order: ValidatingAdmissionPolicy evaluations run before ValidatingAdmissionWebhook callouts (Kubernetes — Admission Control). Built-in validators include:

  • PodSecurity — enforces Pod Security Standards (Privileged/Baseline/Restricted) per the namespace’s pod-security.kubernetes.io/enforce label. Replaces the deprecated PodSecurityPolicy (removed 1.25).
  • ResourceQuota — enforces per-namespace caps on object counts, total CPU/memory requests/limits, etc.
  • LimitRanger — enforces min/max Pod/Container request and limit bounds per namespace.
  • NamespaceLifecycle — rejects creates in terminating or non-existent namespaces, protects default and kube-*.
  • ValidatingAdmissionWebhook — invokes registered external webhooks. The mechanism behind OPA Gatekeeper, Kyverno (validation mode), and any custom policy engine.
  • ValidatingAdmissionPolicy — declarative CEL-based validation, GA in 1.30. The in-process alternative to webhooks, avoiding the latency and reliability cost of a webhook callout.

Stage 7: Storage — commit to etcd

The validated object is serialized (protobuf for core resources, JSON for CRDs by default) and written to /registry/<group>/<resource>/<namespace>/<name> in etcd. If encryption at rest is configured (Kubernetes — Encrypting Data at Rest), the value is encrypted with the configured provider (aescbc / aesgcm / kms) before storage; the apiserver retains the keys, etcd never sees plaintext.

etcd’s commit advances the cluster’s resourceVersion (which is the etcd revision); this becomes the new object’s metadata.resourceVersion. The apiserver replies to the client with the canonical post-storage object.

Stage 8: Watch fan-out — notify subscribers

Every successful mutating write is published to the apiserver’s watch cache (a per-resource in-memory ring buffer; see List-Watch Semantics). Every currently-active watch whose filter matches the new/modified/deleted object receives an ADDED, MODIFIED, or DELETED event with the new resourceVersion. This is what wakes up the Deployment controller two milliseconds after kubectl apply -f deployment.yaml returns. See Watch and Informers for how clients consume this stream.

Throughout, the audit subsystem (Kubernetes — Audit Logging) records events at four configurable stages (RequestReceived, ResponseStarted, ResponseComplete, Panic) at four levels (None, Metadata, Request, RequestResponse) per the cluster’s audit-policy file.

Configuration and Inspection

Watch the pipeline yourself

# -v=6 shows the HTTP round trips kubectl makes
# -v=8 shows full request/response bodies
kubectl -v=8 apply -f deployment.yaml

-v=6 reveals timing: TLS, authn, authz, admission webhooks each show up as latency. -v=8 dumps the full request body — invaluable when debugging admission webhook mutations (you can see exactly what your sidecar injector did).

Inspect what you may do

# "Can I create deployments in production?"
kubectl auth can-i create deployments --namespace production
# yes / no
 
# Impersonate another user (requires the impersonate verb on users/groups)
kubectl auth can-i create deployments \
  --namespace production \
  --as alice --as-group developers

auth can-i issues a SelfSubjectAccessReview API request, which the apiserver evaluates by running the request attributes through the authorization chain without actually performing the action. This is the diagnostic for “why is this denied?”

Inspect the admission chain

# List all enabled admission plugins (recent versions expose this)
kubectl get --raw='/livez?verbose' | grep admission
 
# List dynamic webhooks
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations
kubectl get validatingadmissionpolicies

Sample MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector.example.com
webhooks:
- name: sidecar-injector.example.com
  rules:
  - apiGroups:   [""]
    apiVersions: ["v1"]
    operations:  ["CREATE"]
    resources:   ["pods"]
    scope:       "Namespaced"
  clientConfig:
    service:
      name: sidecar-injector
      namespace: kube-system
      path: "/mutate"
    caBundle: <PEM-encoded CA>
  admissionReviewVersions: ["v1"]
  sideEffects: None              # MUST be None or NoneOnDryRun in v1
  timeoutSeconds: 5              # default 10s, max 30s
  failurePolicy: Fail            # or Ignore
  reinvocationPolicy: IfNeeded   # re-run if a later mutator changed object
  namespaceSelector:
    matchExpressions:
    - key: sidecar-injection
      operator: In
      values: ["enabled"]

Line-by-line: rules is the (verb × resource × apiGroup) trigger; clientConfig.service points at an in-cluster Service that serves the webhook over HTTPS (the apiserver does mTLS using the caBundle); timeoutSeconds caps each call; failurePolicy decides whether a timeout or unreachable webhook fails the request (Fail) or is silently skipped (Ignore). The namespaceSelector scopes the webhook to namespaces labeled sidecar-injection=enabled — a critical hygiene knob.

Failure Modes

  • Admission webhook latency blow-up. Each webhook has a timeoutSeconds (default 10, max 30); the apiserver enforces a roughly 30-second budget on the whole admission phase (Kubernetes — Dynamic Admission Control). A slow webhook with failurePolicy: Fail turns the apiserver’s kubectl apply latency into the webhook’s response time, and a webhook that becomes unreachable makes the cluster unable to create the affected resources. The classic outage pattern: an admission webhook lives in the same cluster as the workloads it gates, the webhook’s Pods die, and the cluster can no longer schedule new Pods because the webhook is unreachable and failurePolicy: Fail is in effect. Mitigation: namespaceSelector excluding kube-system, failurePolicy: Ignore for non-critical webhooks, and running webhooks redundantly outside the workloads they gate.

  • Authentication misconfiguration cascade. If the OIDC issuer becomes unreachable, all OIDC-authenticated kubectl commands fail; in-cluster ServiceAccount tokens are unaffected (signed locally by the apiserver). The split is important: ServiceAccount-based control loops keep working through an identity-provider outage.

  • system:masters group footgun. The default kubeadm admin cert binds to group system:masters, which the apiserver treats as superuser bypassing RBAC entirely (Authorization Overview). Loss of that kubeconfig file is loss of cluster control; sharing it widely defeats RBAC.

  • etcd quorum loss. Once authn/authz pass and admission rules out the request, the storage write blocks on etcd consensus. If etcd has lost quorum, the apiserver returns 5xx and the request must be retried after recovery. This is why etcd HA (3 or 5 nodes) is non-optional.

  • Encryption-at-rest key loss. If the encryption provider’s keys are lost, objects encrypted under them become permanently unreadable. The apiserver’s reading path decrypts on the way out of etcd; missing keys produce errors that look like “etcd corruption” but are really crypto-failure.

  • Dry-run drift. kubectl --dry-run=server runs through stages 1–6 (admission included) but stops before storage. It is the correct way to preview what an apply would do; client-side dry-run skips admission and is misleading for webhook-heavy clusters.

  • Audit log explosion. A Request or RequestResponse-level audit policy on a high-traffic resource (Events especially) can generate gigabytes of audit log per hour. Audit policies must scope levels by resource carefully.

Alternatives and Comparisons

  • ValidatingAdmissionPolicy (in-process CEL) vs ValidatingAdmissionWebhook (out-of-process HTTP) — ValidatingAdmissionPolicy reached GA in 1.30 and is the recommended path for new policy that fits into CEL expressions. Webhooks remain necessary for policies that need to call out (e.g., signature verification against an external Rekor log) or that have state.

  • Built-in admission plugins vs webhook-based replacements — the in-process plugins are cheaper and cannot fail open in a webhook-unreachable scenario. Prefer built-ins where they suffice (PodSecurity, ResourceQuota, LimitRanger) before reaching for Gatekeeper or Kyverno.

  • OPA Gatekeeper vs Kyverno — both ride on ValidatingAdmissionWebhook + (optional) MutatingAdmissionWebhook. Gatekeeper uses Rego; Kyverno uses pure YAML. Kyverno also supports image verification (cosign integration) and generation policies. ValidatingAdmissionPolicy’s CEL is increasingly cannibalizing the simple cases of both.

  • MutatingAdmissionPolicy — the mutating counterpart to ValidatingAdmissionPolicy (KEP-3962). It progressed alpha in v1.32, beta in v1.34, and reached GA / stable in v1.36 (April 2026), enabled by default (Mutating Admission Policy; v1.36 release). It is now the recommended in-process path for CEL-expressible mutations, displacing many MutatingAdmissionWebhook deployments (sidecar injection and field defaulting in particular), just as ValidatingAdmissionPolicy displaced validating webhooks.

Production Notes

  • Latency budget: production-grade clusters expect p99 apiserver write latency in the tens of milliseconds. Each admission webhook contributes its own latency; teams running >5 webhooks routinely see p99 of hundreds of milliseconds. Operators consuming the watch stream amplify the apparent slowness — if their reconcile is gated on a fresh resourceVersion they wait for the watch fan-out too.

  • EKS, GKE, AKS all expose an audit log stream (CloudWatch, Cloud Logging, Diagnostic Settings) and let admins configure RBAC and webhooks but do not allow control over the apiserver’s --enable-admission-plugins flag. If you need a non-default built-in plugin, that’s an indicator you should run a webhook policy engine instead.

  • The in-process trend (as of v1.36, April 2026) has largely landed: ValidatingAdmissionPolicy (GA v1.30) and now MutatingAdmissionPolicy (GA v1.36) are both stable and enabled by default, alongside Structured Authentication Configuration (beta v1.30) and Structured Authorization Configuration (the analogous structured config for the authorizer chain). The strategic direction is “describe policy declaratively in YAML/CEL, evaluate in-process, reload on file change” — cutting webhook latency and the operational pain of running webhook deployments. With both policy types GA, new clusters can express most defaulting and validation logic without standing up a single admission-webhook server.

On the admission timeout and ordering (resolved)

Two previously-flagged claims, now pinned:

  • Per-webhook timeout, not an aggregate cap. The dynamic-admission docs document only a per-webhook timeout: the default is 10 seconds and each webhook’s timeoutSeconds is set individually (the GA admissionregistration.k8s.io/v1 API allows 1–30 seconds), with timeout behavior governed by the webhook’s failurePolicy (Kubernetes — Dynamic Admission Control). There is no documented single aggregate budget across all webhooks in a request — the often-quoted “30 seconds total” is the per-webhook maximum, not a phase-wide cap. In practice the limiting factor is the sum of the webhook timeouts plus the client’s HTTP request timeout.
  • Built-in plugin order is fixed; flag order is ignored. The order in which plugins are listed in --enable-admission-plugins does not matter — the apiserver runs its compiled-in built-in plugins in a single predetermined order, in two phases (all mutating, then all validating), and the flag merely toggles which built-ins are active (Kubernetes — Admission Control). Within the validating phase, ValidatingAdmissionPolicy runs before ValidatingAdmissionWebhook. The only ordering knob an operator controls is dynamic webhook ordering, which is itself not strictly guaranteed — hence the reinvocation mechanism that re-runs built-in mutators after a webhook mutation.

See Also

Graph View

Backlinks