vault

kube-apiserver

14 min read

kube-apiserver

The kube-apiserver is the REST front door of a Kubernetes cluster and the only control-plane component permitted to read or write etcd directly. Every actor in the system — kubectl, controllers, the kube-scheduler, the kubelet on each node, admission webhooks, monitoring stacks — interacts with cluster state exclusively through this HTTP/2 endpoint, which authenticates the caller, authorizes the verb against the requested resource, runs mutating and validating admission, validates the object against its OpenAPI schema, persists the result to etcd, and fans the change out to every long-lived watch subscriber (Kubernetes — Components, API Concepts). Architecturally this is a deliberate single-point-of-mediation: every other component is stateless and idempotent because the apiserver is the canonical authority. Operationally it is the most important component to scale and protect — a degraded apiserver does not crash workloads, but it freezes the entire reconciliation universe of the cluster.

Mental Model

The kube-apiserver is best understood as a schema-aware REST gateway in front of etcd, augmented with a long-lived event multiplexer (the watch fan-out). Each HTTP request enters a fixed pipeline whose stages are independent and short-circuit on the first refusal. Reads bypass admission entirely (admission controllers cannot block get, list, or watch per the admission-controllers reference); writes traverse the full pipeline. The apiserver itself is stateless except for an in-memory watch cache that absorbs read load and lets thousands of clients share one etcd subscription per resource kind.

flowchart LR
    CLIENT[kubectl / controller / kubelet] -->|HTTP/2 + protobuf or JSON| AUTHN[1. Authentication<br/>cert / token / OIDC / webhook]
    AUTHN --> AUTHZ[2. Authorization<br/>RBAC / Node / Webhook]
    AUTHZ --> MUTATE[3. Mutating Admission<br/>MutatingAdmissionWebhook,<br/>ServiceAccount, DefaultStorageClass]
    MUTATE --> SCHEMA[4. Schema Validation<br/>OpenAPI / CRD schema]
    SCHEMA --> VALIDATE[5. Validating Admission<br/>ValidatingAdmissionWebhook,<br/>PodSecurity, ResourceQuota,<br/>ValidatingAdmissionPolicy CEL]
    VALIDATE --> STORE[(6. etcd write<br/>encrypted-at-rest if configured)]
    STORE --> CACHE[7. Watch cache update]
    CACHE -.->|long-lived HTTP/2 stream| W1[Controller A watch]
    CACHE -.-> W2[kubelet watch on node X]
    CACHE -.-> W3[Scheduler watch on Pods]
    CACHE -.-> W4[CRD operator watch]

What this diagram shows. The horizontal axis is the write path: authentication → authorization → mutating admission → schema validation → validating admission → etcd write. The vertical fan-out at the right is the watch path: any successful write updates the in-memory watch cache, which immediately notifies every subscribed client whose selector matches. The insight to extract is that the apiserver is two systems glued together — a sequential, transactional write pipeline that must complete in milliseconds to keep kubectl apply responsive, and a streaming pub/sub layer that must scale to thousands of long-lived HTTP/2 connections per resource kind. Both share the same process but stress completely different operational properties (write pipeline = latency-bound, watch fan-out = memory- and goroutine-bound).

Mechanical Walk-through

A request begins as an HTTP/2 frame against the secure port (default 6443). The apiserver supports three wire encodings — JSON, YAML, and Kubernetes’ protobuf flavor — selected via the Accept and Content-Type headers; protobuf (application/vnd.kubernetes.protobuf) is preferred for internal component-to-component traffic because it is 3–10× smaller and faster to decode than JSON at scale (API Concepts). HTTP/2 multiplexing is essential because every controller holds dozens of watch streams open simultaneously; without it, each watch would consume a full TCP connection.

The first pipeline stage is authentication. Plugins are tried in order until one returns a non-anonymous identity, mapping the request to a (user, groups, extra) tuple. Supported mechanisms include X.509 client certificates verified against the cluster CA, service-account JWT tokens (issued via the projected TokenRequest API with audience binding), static bearer tokens (discouraged), OIDC ID tokens (--oidc-issuer-url, --oidc-client-id), and external webhook authenticators (for cloud-IAM integrations like AWS IRSA). If --anonymous-auth is true — still the default as of 1.36 (kube-apiserver flag reference) — unauthenticated requests are mapped to user system:anonymous in group system:unauthenticated — this is the single most common path to misconfiguration, because RBAC bindings to system:unauthenticated accidentally expose APIs to the world. Note that the blanket anonymous mapping has been tightened over time: the AnonymousAuthConfigurableEndpoints feature (beta in 1.32) lets operators restrict anonymous access to a specific allowlist of endpoints (typically just /healthz, /livez, /readyz) via the structured authentication config, closing the accidental-exposure hole without fully disabling anonymous health probing.

The second stage is authorization. The flag --authorization-mode is an ordered list (typical: Node,RBAC or Node,RBAC,Webhook); the first authorizer to return allow or deny wins, otherwise the next is consulted. The Node authorizer is a hard-coded specialization that grants each kubelet permission only to read its own Pods and Secrets; without it, a compromised kubelet could read every Secret in the cluster. RBAC is the general-purpose role-binding system covered in Kubernetes RBAC. Webhook delegates to an external HTTP service — the most flexible but slowest option, since it adds a network round-trip to every write.

The third stage is mutating admission. Built-in mutators run first in a fixed order — ServiceAccount injects the default SA token volume, DefaultStorageClass fills in storageClassName for PVCs that omit it, DefaultTolerationSeconds adds 300-second tolerations for node.kubernetes.io/not-ready. After built-ins, the MutatingAdmissionWebhook controller calls every external mutating webhook registered via a MutatingWebhookConfiguration. This is how sidecar injection works: an Istio or Linkerd webhook receives the Pod spec and returns a JSON Patch adding the proxy container. The order across webhooks is alphabetical by webhook name and not deterministic across re-registrations, which is a frequent source of subtle bugs when multiple mutators touch the same field.

The fourth stage is schema validation: the apiserver decodes the request body against the generated OpenAPI schema for the resource (for core types, compiled in; for CRDs, derived from the CRD’s openAPIV3Schema). Validation failures here return 400 Bad Request with a structured field-level error. This stage is also where server-side apply does its three-way-merge work, computing the new object from the user-supplied YAML, the previous version, and the field-ownership manager metadata (API Concepts).

The fifth stage is validating admission, which is symmetrical to mutating admission but cannot modify the object. Built-in validators include PodSecurity (enforcing one of Privileged / Baseline / Restricted at namespace scope, see Pod Security Admission), ResourceQuota, LimitRanger, and ValidatingAdmissionPolicy — the latter being a 1.30-stable in-process CEL engine that replaces many webhook use cases without the network round-trip. External ValidatingAdmissionWebhook calls run after the built-ins; OPA Gatekeeper and Kyverno hook in here.

If all stages pass, the apiserver writes to etcd via a gRPC client. Writes are wrapped in optimistic-concurrency Txn operations keyed on the current resourceVersion; a concurrent writer that increments the resourceVersion first causes a 409 Conflict returned to the loser. Encryption at rest, if configured via --encryption-provider-config, encrypts the value before the etcd write using AES-GCM, AES-CBC, secretbox, or KMS-backed envelope encryption (Encrypt Data at Rest); without it, Secrets sit in etcd as base64-decoded plaintext.

After the etcd write succeeds, the watch cache receives the update. Each resource kind has its own in-memory ring buffer (size capped per kind, default ~1000 events) plus an index over the current snapshot. Every open watch stream is represented by a goroutine that reads from the ring and filters by the client’s label/field selector before pushing the event over the open HTTP/2 stream. The watch cache is the only stateful subsystem in the apiserver — restart of the process forces every client to re-list before resuming watches, which is the apiserver-restart “thundering herd” that capacity-plans must account for.

Configuration / API Surface

A representative production HA setup runs three apiserver instances behind a TCP load balancer. The startup flags typify the surface:

kube-apiserver \
  --advertise-address=10.0.1.42 \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --etcd-servers=https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379 \
  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
  --authorization-mode=Node,RBAC \
  --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota \
  --service-account-issuer=https://kubernetes.default.svc \
  --service-account-key-file=/etc/kubernetes/pki/sa.pub \
  --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
  --api-audiences=https://kubernetes.default.svc \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver.key \
  --audit-log-path=/var/log/audit.log \
  --audit-log-maxsize=100 \
  --audit-log-maxage=30 \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --max-requests-inflight=400 \
  --max-mutating-requests-inflight=200

Line-by-line: --advertise-address is what other components see in the Endpoints for the kubernetes Service; --bind-address=0.0.0.0 listens on all interfaces (production should bind to a specific IP). --etcd-servers is the comma-separated list of etcd peers — note that the apiserver does not load-balance reads, it connects to all of them and the embedded etcd client handles failover. The three etcd-cert flags establish the mutual TLS between apiserver and etcd; this is the single most security-sensitive trust boundary in Kubernetes because anyone with apiserver→etcd credentials can read every Secret. --encryption-provider-config enables envelope encryption of Secrets and any other listed resources before they hit etcd. The --authorization-mode=Node,RBAC orders Node first (cheap, hard-coded) before RBAC (more expensive). Note there is no --authentication-mode flag — authentication is not an ordered “mode” list the way authorization is; instead each authenticator is enabled by its own dedicated flag (--client-ca-file, --oidc-issuer-url, --service-account-key-file, --authentication-token-webhook-config-file, etc.) or, since the structured authentication config graduated to GA in 1.34, via a single --authentication-config file (kube-apiserver flag reference). --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota augments the defaults — NodeRestriction is the partner to the Node authorizer that limits what a kubelet can mutate (a kubelet can’t update Pods on other nodes). The three --service-account-* flags configure the apiserver as the issuer and signer of projected service-account JWT tokens used by Pods to authenticate back to the apiserver. --audit-policy-file defines which requests get logged at which level (None / Metadata / Request / RequestResponse); a production audit policy logs Request for writes against Secrets and ConfigMaps and Metadata for everything else. --max-requests-inflight=400 and --max-mutating-requests-inflight=200 are the priority-and-fairness fallback caps — once a cluster exceeds them, requests get 429 Too Many Requests until traffic subsides.

Failure Modes

429 Too Many Requests storm. A controller bug that spams writes — or a CRD operator that re-reconciles every object every few seconds — saturates --max-mutating-requests-inflight. The apiserver throttles all clients (priority-and-fairness mitigates by giving leader-election traffic a dedicated lane), but in extreme cases the kubelets cannot renew their Node leases and the cluster marks healthy nodes NotReady. Diagnosis: kubectl get --raw /metrics | grep apiserver_request_total plus the audit log to identify the noisy client.

Watch cache lag and 410 Gone. If a client falls behind and its requested resourceVersion is older than the oldest entry in the watch cache ring, the apiserver returns HTTP 410 Gone forcing a full re-list. A cluster under heavy churn (mass Pod creation) can blow past the ring buffer and force the entire fleet of controllers to re-list simultaneously, hammering the apiserver further. Mitigations are larger watch caches (--watch-cache-sizes) and bookmarks (the apiserver emits BOOKMARK events periodically so clients can advance their resourceVersion without an actual change — see API Concepts).

etcd disconnect. When the apiserver loses connection to etcd quorum, all writes fail with 500 Internal Server Error; reads continue from the watch cache for a brief window then also fail once the cache becomes stale. The cluster is effectively read-only until quorum returns. Existing workloads keep running because kubelet does not need the apiserver to keep containers alive — but no scaling, deployment, or recovery happens.

Webhook timeout cascade. A misbehaving admission webhook with failurePolicy: Fail (the default) and a slow backend can stall every write to a resource kind. The 10-second default webhook timeout times --max-mutating-requests-inflight worth of stuck requests blocks the apiserver from accepting new writes. Production webhooks must set aggressive timeouts (1–2 seconds) and use failurePolicy: Ignore for non-security mutators.

Slow leader-election lease renewal. The apiserver itself does not use leader election (it is multi-active behind a load balancer), but kube-controller-manager and kube-scheduler both renew leases through the apiserver every few seconds. An overloaded apiserver causes leader-election thrash, with controllers and schedulers continuously failing-over between replicas and burning CPU on warm-start (re-listing all resources after winning the lease).

Alternatives and When to Choose Them

There is no production alternative to the kube-apiserver within Kubernetes. Every API group, every CRD, every operator depends on the single API surface; replacing it would mean re-implementing Kubernetes. The interesting variants are:

  • Aggregated API servers (via the apiregistration.k8s.io/v1 API): mount a second HTTP server (metrics-server, custom-metrics adapters, the service-catalog API) under a sub-path of the main apiserver. The main apiserver proxies requests for the registered API group to the aggregated server. This is how kubectl top works and how virtual-cluster systems like vCluster expose tenant-specific APIs.
  • Embedded apiserver in non-Kubernetes orchestrators: HashiCorp Nomad has its own API; OpenShift adds Red-Hat-specific routes to its forked apiserver; lightweight distributions like k3s embed the apiserver in a single binary with etcd replaced by SQLite or Postgres for the smallest deployments.
  • Mock / fake apiservers: envtest (controller-runtime) starts an apiserver+etcd pair in-process for unit tests; kwok simulates an entire cluster’s apiserver+kubelet behavior for scheduler benchmarks without running real Pods.

Production Notes

The apiserver’s scaling ceiling is what gates 5,000-node clusters per the large-cluster best practices. The dominant pressures are watch fan-out (memory grows linearly with watchers × objects per resource), etcd write throughput (a single etcd cluster sustains ~10–30 k writes/sec depending on hardware), and audit-log I/O. Production playbook lessons:

  • Run 3 apiserver replicas behind a TCP load balancer, one per failure zone. Load-balance with TCP (not HTTP) because HTTP/2 long-lived watches require sticky-by-connection routing — though watches re-establish gracefully, hot-spotting one replica is fine until it isn’t.
  • Separate Events into their own etcd cluster. Events are high-volume, low-value churn; running them in the main etcd ages out real state. The flag --etcd-servers-overrides=/events#https://events-etcd-0:2379,... redirects only the events resource to a dedicated etcd.
  • Tune --default-watch-cache-size up for high-churn resource kinds (e.g., Pods at 10000). The trade-off is process memory. The static default of 100 is largely superseded: since 1.19 the apiserver sizes per-resource watch caches dynamically from a memory budget rather than from a fixed per-kind constant (kubernetes/kubernetes #90058), so the headline “100” is a floor for unlisted kinds rather than the operative number for hot resources.
  • Profile with apiserver_request_duration_seconds by verb and resource — LIST of high-cardinality resources (Pods, Events) is the typical culprit when the apiserver’s p99 latency climbs.
  • Cluster-API and managed services (EKS, GKE, AKS) all run hidden apiservers — but the operational behavior is identical and you still need to size your client load to avoid 429s on the managed control plane. AWS’s EKS publishes apiserver request quotas; GKE’s are softer but symmetric.
  • OpenAI’s “Scaling Kubernetes to 7,500 nodes” write-up surfaced kube-apiserver as the dominant scaling pressure, but their conclusion ran the opposite way from “shard into smaller clusters”: they kept one large cluster and attacked the apiserver load directly. At ~1,000 nodes the apiservers were reading >500 MB/s from etcd, traced to Datadog and Fluentd DaemonSets polling the apiserver from every node; the fix was to make those agents poll less aggressively and to interpose caching/aggregation services rather than have per-node DaemonSets hit the apiserver directly. At 7,500 nodes they ran 5 apiserver replicas (each using up to ~70 GB heap) against a 5-member etcd cluster, and credited EndpointSlices (GA-tracked from 1.17) with cutting network-related apiserver load by roughly 1000×. The broad lesson is the inverse of cluster proliferation: minimise what talks to the apiserver (especially DaemonSets) before you minimise cluster size.

Uncertain uncertain

Verify: the exact per-resource watch-cache ring-buffer event counts (e.g. “~1000 events” cited in the walk-through above). Reason: since 1.19 these are computed by an internal memory-budget heuristic, not a published per-kind constant, and the Kubernetes docs explicitly do not document --watch-cache-sizes / --default-watch-cache-size defaults (kubernetes/kubernetes #57105); the values also shift by minor version. To resolve: read pkg/storage/cacher and setDefaultWatchCacheSizes in the kube-apiserver source for the specific minor version in use. The flag names and the static fallback of 100 for unlisted kinds are confirmed; the dynamic per-kind ring sizes are not.

See Also

Graph View

Backlinks