vault

API Aggregation Layer

11 min read

API Aggregation Layer

The Kubernetes API Aggregation Layer (“kube-aggregator”) lets cluster administrators mount entirely separate API servers next to the core kube-apiserver so that, from a client’s perspective, paths like /apis/metrics.k8s.io/v1beta1/nodes look indistinguishable from native paths like /apis/apps/v1/deployments (Kubernetes — API Aggregation Layer). The registration mechanism is the cluster-scoped APIService resource; the runtime mechanism is a reverse proxy inside the apiserver that forwards matching requests to a configured Service endpoint and forwards the authenticated identity along with them. This is the machinery behind kubectl top (metrics-server registers metrics.k8s.io), behind HPA scaling on Prometheus metrics (prometheus-adapter registers custom.metrics.k8s.io and external.metrics.k8s.io), and behind any cluster operator that wants its own REST semantics, own storage, and own validation rather than the CRD model’s etcd-backed object handling.

Mental Model

A core distinction in K8s extensibility: CRDs let you add data; aggregation lets you add a server. CRDs reuse the apiserver’s stack — etcd storage, OpenAPI schema validation, watch cache, RBAC — and just append new “kinds” to the namespace. The aggregation layer surrenders all of that to a process you run, and the apiserver becomes a proxy gating the front door (TLS, authn, authz) but not the back end. The trade is: more flexibility, more operational responsibility.

flowchart LR
    Client[kubectl / controller / kubelet]
    Aggregator[kube-aggregator<br/>inside kube-apiserver]
    CoreHandler[Core API handlers<br/>/api, /apis/apps, /apis/batch ...]
    ETCD[(etcd)]
    APIService[APIService<br/>v1beta1.metrics.k8s.io]
    Ext[Extension API Server<br/>metrics-server Pod]
    NodeKubelet[kubelet<br/>summary API on each node]

    Client -->|HTTPS| Aggregator
    Aggregator -->|/apis/apps/...| CoreHandler
    CoreHandler --> ETCD
    Aggregator -->|/apis/metrics.k8s.io/...<br/>after authn+authz| Ext
    APIService -.registers proxy target.-> Aggregator
    Ext -->|in-memory scrape| NodeKubelet

The aggregator splits incoming requests by URL prefix. Core paths land on the apiserver’s own handlers and end up in etcd. Aggregated paths (registered by an APIService) are reverse-proxied to a registered Service. The key insight: the extension server can store its data anywhere — in memory (metrics-server), in Prometheus (custom-metrics-apiserver via prometheus-adapter), in an external database, or even compute it on demand. Etcd is not involved.

Mechanical Walk-through

The APIService resource

An APIService is a cluster-scoped resource registered with the apiregistration.k8s.io API group:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  version: v1beta1
  groupPriorityMinimum: 100
  versionPriority: 100
  service:
    namespace: kube-system
    name: metrics-server
    port: 443
  caBundle: <PEM-encoded CA>
  insecureSkipTLSVerify: false

This object tells the aggregator: “All requests under /apis/metrics.k8s.io/v1beta1/* should be reverse-proxied to the Service kube-system/metrics-server over HTTPS, using caBundle to verify the certificate.” The aggregator continuously watches APIService objects and rebuilds its routing table on change.

The request path

When a client makes GET /apis/metrics.k8s.io/v1beta1/nodes:

  1. The aggregator receives the request inside the apiserver, performs the standard authentication chain (stage 2 of the API Server Request Flow).
  2. It performs authorization against the configured authorizers — the extension server does not re-run RBAC; the aggregator is the policy gate.
  3. It opens an mTLS connection to the registered Service, forwarding the request body, headers, and — critically — the authenticated user identity via the request-header authentication headers: X-Remote-User, X-Remote-Group, X-Remote-Extra-*.
  4. The extension server verifies the request-header client certificate against the apiserver’s request-header CA (the same --requestheader-client-ca-file bundle used everywhere) and trusts the identity headers.
  5. The extension server processes the request using its own storage, validation, and conversion logic.
  6. The response flows back through the proxy unchanged.

The aggregator handles TLS, authn, authz; the extension server handles storage, validation, semantics.

Health and discovery

The aggregator hits the extension’s /apis and /healthz endpoints periodically. If an APIService fails its health probe, its status.conditions[Available] flips to False, and discovery for the entire API group fails until it recovers. This is the proximate cause of the canonical incident: kubectl get deployments taking 30 seconds because metrics-server is unhealthy, because kubectl does a discovery sweep on first use and the aggregator waits for the unhealthy APIService to time out.

The discovery requirement is strict — extension servers must return discovery results within five seconds (Kubernetes — API Aggregation Layer). This places a hard latency budget on the extension server’s startup and its /apis endpoint.

The canonical extension servers

  • metrics-server (kubernetes-sigs/metrics-server) registers metrics.k8s.io/v1beta1. It scrapes the kubelet /metrics/resource (formerly summary) endpoint on every node every 15 seconds, keeps the results in-memory (no etcd), and serves NodeMetrics and PodMetrics resources via the aggregated API. kubectl top and the Horizontal Pod Autoscaler (HPA) read from it.

  • custom-metrics-apiserver (kubernetes-sigs/custom-metrics-apiserver) is a Go framework for building APIServices that register custom.metrics.k8s.io and external.metrics.k8s.io. The most common production deployment is prometheus-adapter (prometheus-community/prometheus-adapter), which translates HPA queries into PromQL against a Prometheus server and returns the results as Metrics objects. This is how HPA scales on “request rate” or “queue depth” rather than CPU.

  • service-catalog (now archived) registered servicecatalog.k8s.io to provide a higher-level API over the OSB Open Service Broker. A historical example of an aggregated API for non-metric data.

Configuration

Setup checklist for a new extension API server

# 1. The aggregation layer must be enabled in kube-apiserver (default on for ~all distros)
#    Flags involved:
#      --requestheader-client-ca-file=<CA bundle for trusted proxies>
#      --requestheader-allowed-names=<comma-separated allowed CNs>
#      --requestheader-extra-headers-prefix=X-Remote-Extra-
#      --requestheader-group-headers=X-Remote-Group
#      --requestheader-username-headers=X-Remote-User
#      --proxy-client-cert-file=<apiserver client cert>
#      --proxy-client-key-file=<apiserver client key>
 
# 2. Deploy the extension server with its own serving cert
kubectl apply -f extension-apiserver-deployment.yaml
kubectl apply -f extension-apiserver-service.yaml
 
# 3. Bind the SA that the extension runs as to the "extension-apiserver-authentication-reader"
#    Role in kube-system — this lets it read the request-header CA bundle from a ConfigMap
kubectl create rolebinding extension-apiserver-auth-reader \
  --role=extension-apiserver-authentication-reader \
  --serviceaccount=my-ns:my-extension-sa \
  --namespace=kube-system
 
# 4. Register the APIService
kubectl apply -f apiservice.yaml
 
# 5. Verify
kubectl get apiservice v1.myextension.example.com
# NAME                          SERVICE                AVAILABLE   AGE
# v1.myextension.example.com    my-ns/my-extension     True        1m

The extension-apiserver-authentication-reader Role binding is the most overlooked step. Without it the extension server cannot verify the request-header client cert chain, and the aggregator will receive 401s back from the extension on every forwarded request.

Inspecting the registered APIServices

# All registered APIServices, including the ones the core apiserver registers for itself
kubectl get apiservice
 
# Drill into an unhealthy one
kubectl describe apiservice v1beta1.metrics.k8s.io
# Look at: Conditions: Available: False — Reason: FailedDiscoveryCheck

Failure Modes

  • Cascading discovery slowness. If any registered APIService is unavailable, the aggregator probes it on every discovery sweep. kubectl and controllers that perform discovery (e.g., controller-runtime on startup) wait for the probe. Symptom: every kubectl <anything> becomes slow. Mitigation: delete the offending APIService when its server is permanently gone (kubectl delete apiservice v1beta1.broken.example.com), or set short timeouts on the apiserver’s webhook calls and aggressively retry.

  • Identity propagation misconfiguration. If the extension server’s request-header CA does not match the apiserver’s --proxy-client-cert-file issuer, the extension server cannot verify the proxied identity, and every request looks anonymous. Symptom: 401s in the extension server’s logs even though kubectl reaches it.

  • Extension storage divergence from etcd. Because aggregated APIs control their own storage, an extension server crash means all its data is lost unless it has its own persistence. metrics-server explicitly accepts this — its in-memory snapshot rebuilds in seconds — but extension authors who store domain state must run their own HA stack. CRDs do not have this problem; etcd is HA by design.

  • Per-resource-version skew between extension and core. The extension server’s resourceVersion namespace is its own. A controller that watches both core resources and aggregated resources must track two independent resourceVersion streams. This is a frequent source of bugs in custom controllers (see List-Watch Semantics).

  • mTLS rotation hell. When the apiserver’s proxy-client cert rotates (e.g., during managed-cluster control-plane upgrade), the extension’s request-header CA bundle must be re-fetched. metrics-server and prometheus-adapter handle this; ad-hoc extensions often don’t, leading to mysterious 401-after-control-plane-upgrade failures.

  • Five-second discovery budget. Extension servers that need >5s to compute discovery (e.g., aggregating from external systems on each call) miss the aggregator’s probe and flap Available=False/True.

Alternatives and When to Choose Them

The canonical comparison is aggregation vs CRDs. The upstream summary (Kubernetes — Custom Resources) puts the trade-off in one sentence — “CRDs are simple and can be created without any programming. API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.” Expanded across the dimensions that actually drive the choice:

ConcernCRDAggregated API Server
Storageetcd, managed by kube-apiserverAnything — in-memory, external DB, computed on demand
Schema validationOpenAPI v3 schema in CRD spec; CEL validation rules via x-kubernetes-validationsWhatever the extension implements
Versioning / conversionDeclared in CRD; conversion webhook for non-trivial transformsFully under your control
RBAC / admissionReuses the apiserver’s stackAggregator authz at the front; extension can layer its own
Watch & resourceVersionReuses kube-apiserver’s watch cacheMust be implemented by the extension
Operational costNear zeroRun the extension server as an HA deployment
PerformanceNative apiserver performanceNetwork hop + extension server CPU
Effort to buildDeclarative YAMLReal Go server code (or kubebuilder + custom-metrics-apiserver framework)

Choose CRDs when the resource fits the “spec/status object stored in etcd, reconciled by a controller” model. This is 95% of operator use cases.

Choose aggregated API when any of:

  • Data should not live in etcd (high cardinality metrics, computed on demand).
  • You need custom REST semantics (sub-resources with non-CRUD behavior, weird streaming endpoints).
  • You need full control over conversion (a non-CRD conversion webhook isn’t expressive enough).
  • The resource volume is too high for etcd (Pods on a 5,000-node cluster already strain etcd; per-Pod metrics across a 5,000-node cluster would crush it).

For most operator authors today the answer is CRD. The aggregation layer is reserved for platform-level extensions — observability metrics, autoscaling sources, and Kubernetes-style APIs over non-Kubernetes systems (e.g., cluster-API providers, kcp).

Production Notes

  • EKS, GKE, AKS all run metrics-server (or the cloud-vendor equivalent) as an aggregated API by default. Customers cannot remove the aggregation layer — it is part of the apiserver — but they can register their own APIServices.

  • Managed control plane caveat: the apiserver-to-extension network path must traverse the same network as the cluster’s Pod network. On EKS, this means the metrics-server Pod must be reachable from the AWS-managed apiserver, which is enforced by the security-group default. If a custom aggregated API is registered and reachability breaks, the entire cluster’s API surface stutters until the APIService is removed or fixed.

  • The trajectory of API extensibility: SIG-API-Machinery’s long-running effort is to push more capabilities into CRDs (CEL validation rules, declarative defaults, ratcheting, server-side apply) so that fewer use cases require the aggregation layer. The aggregation layer is increasingly the escape hatch for non-etcd-shaped data (computed metrics, on-demand resources, custom storage), not for adding “more types” — and the upstream docs reinforce this framing by characterising aggregation specifically as the path for control over storage and version conversion rather than as a general-purpose flexibility lever.

  • kcp (kcp.io) is a notable consumer: it uses the aggregation pattern (and beyond) to build a Kubernetes-API-shaped multi-tenant control plane out of multiple workspaces. Aggregation is foundational to non-cluster K8s-style deployments.

The five-second discovery latency budget is officially specified — “Discovery requests are required to round-trip from the kube-apiserver in five seconds or less” (Kubernetes — API Aggregation Layer). The canonical framing of the CRDs-vs-aggregation trade-off comes from the Custom Resources concept page (Kubernetes — Custom Resources): “CRDs are simple and can be created without any programming. API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.” That sentence is the official distinction — aggregation is for storage control and conversion control, not merely “more flexibility.”

Resolved (2026-05-30)

The aggregator fails fast with HTTP 503 when an APIService is Available=False — no retry. Verified at staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go at release-1.36 — the proxy handler short-circuits with proxyError(w, req, "service unavailable", http.StatusServiceUnavailable) when handlingInfo.serviceAvailable is false; there is no in-handler retry loop. There is also no per-call HTTP timeout constant in handler_proxy.go — the per-request timeout the client experiences is the apiserver’s overall request timeout (default 60s for non-watch) applied through the underlying transport, not an aggregator-specific budget. The five-second discovery budget (separately documented) is for the kube-aggregator’s APIServiceRegistrationController resolving the aggregated discovery doc, not the data-path proxy.

See Also

Graph View

Backlinks