Aggregated API Server

An aggregated API server (also “extension API server”) is a way to extend the Kubernetes API by running a whole separate API server alongside the core kube-apiserver and registering it so that, to clients, its endpoints look like native parts of the Kubernetes API (kubernetes.io — API Aggregation Layer). This is the second of the two K8s API-extension mechanisms, and it sits in deliberate contrast to the first: a Custom Resource Definition (CRD) extends the API within the core server (your new type is stored in etcd and served by kube-apiserver itself), whereas an aggregated API server extends the API by bolting on another server you build, deploy, secure, and scale yourself. The registration object is the cluster-scoped APIService; the runtime mechanism is the kube-aggregator — a reverse proxy inside kube-apiserver — which forwards requests for a matching API group/version to your extension server’s HTTPS endpoint. The canonical production example is Metrics Server, which serves the metrics.k8s.io API group this way. This note is about the aggregated server itself; the registration/proxying machinery — the APIService, the kube-aggregator request flow, identity propagation — is covered in depth in API Aggregation Layer, which this note assumes as background.

Mental Model

The choice is add data vs add a server. A CRD reuses the entire kube-apiserver stack — etcd storage, OpenAPI/CEL schema validation, the watch cache, RBAC, admission, server-side apply — and merely appends a new “kind.” An aggregated API server reuses almost none of that: you get the front-door services (TLS termination, authentication, authorization delegation) from the aggregator for free, but storage, watch, validation, conversion, and the entire data path are yours to implement. You are writing, in effect, a second kube-apiserver — usually using the upstream apiserver library (k8s.io/apiserver) and a scaffolding tool (sample-apiserver, apiserver-builder) so you do not start from zero, but it is still a real, stateful, HA-requiring server process.

flowchart TD
    subgraph CORE["kube-apiserver process"]
        AGG["kube-aggregator<br/>(reverse proxy)"]
        COREH["core API handlers<br/>/api, /apis/apps, /apis/batch"]
    end
    ETCD[("etcd")]
    APISVC["APIService object<br/>v1beta1.metrics.k8s.io<br/>→ Service kube-system/metrics-server"]
    EXT["Extension API Server<br/>(a Pod you run)<br/>OWNS: storage, watch,<br/>validation, conversion"]
    STORE[("its own storage:<br/>in-memory / external DB /<br/>computed on demand")]

    APISVC -. "registers route" .-> AGG
    AGG -- "/apis/apps/... → in-process" --> COREH
    COREH --> ETCD
    AGG -- "/apis/metrics.k8s.io/... → proxy<br/>(after authn + authz)" --> EXT
    EXT --> STORE

The diagram contrasts the two data paths. The insight to extract: the aggregator splits requests by URL prefix. Core groups stay in-process and land in etcd. An aggregated group is reverse-proxied out to a Pod you operate, and that Pod’s storage is whatever you built — etcd is not involved. A CRD would have ridden the left-hand path; an aggregated API server is the entire right-hand subgraph, and all of it is your operational responsibility.

Mechanical Walk-through

Registration: the APIService

You register an extension server by creating an APIService in the apiregistration.k8s.io group, naming the API group/version it serves and the in-cluster Service that fronts your extension Pods:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io          # <version>.<group>
spec:
  group: metrics.k8s.io
  version: v1beta1
  groupPriorityMinimum: 100             # ordering vs other groups in discovery
  versionPriority: 100                  # ordering of versions within the group
  service:                              # the Service to reverse-proxy matching requests to
    namespace: kube-system
    name: metrics-server
    port: 443
  caBundle: <PEM CA>                    # CA the aggregator uses to verify the extension's TLS cert
  insecureSkipTLSVerify: false

From this point, any request under /apis/metrics.k8s.io/v1beta1/* is, after the aggregator runs authentication and authorization, reverse-proxied to the metrics-server Service over mTLS. The full request-path mechanics — the request-header identity propagation (X-Remote-User/X-Remote-Group), the five-second discovery budget, the Available health condition — are detailed in API Aggregation Layer.

What the extension server must implement

Because the aggregator only proxies, the extension server has to be an API server. Concretely it must serve, for the API group it owns:

• Discovery — the /apis and /apis/<group>/<version> endpoints that tell clients which resources exist (must return within the aggregator’s 5-second budget).
• CRUD — GET/LIST/POST/PUT/PATCH/DELETE for each resource, with correct status codes and the standard list/get response envelopes.
• Storage — somewhere to keep the data. This is the whole point of the mechanism. Options: in-memory (metrics-server), an external database, computed-on-demand (no storage at all — synthesize the response per request), or even etcd (your own etcd, separate from the cluster’s).
• Watch — if clients/controllers need to watch your resources, you must implement the watch protocol and a resourceVersion scheme. The aggregator does not give you the core server’s watch cache.
• OpenAPI — publishing schema so kubectl explain and clients work.

In practice nobody writes this from scratch. The upstream k8s.io/apiserver library supplies the generic server machinery; k8s.io/sample-apiserver is a worked example; apiserver-builder scaffolds a project. But even with scaffolding, the storage and semantics layer is genuine, custom server code — far more than a CRD’s declarative YAML.

The canonical example: metrics-server

Metrics Server is the textbook aggregated API server. It registers metrics.k8s.io/v1beta1 and serves two resources — NodeMetrics and PodMetrics. Its storage is in-memory: it scrapes every node’s kubelet /metrics/resource endpoint every 15 seconds and keeps the latest snapshot in RAM. There is no etcd, no persistence — and that is exactly right for its use case. CPU/memory readings are high-cardinality, high-churn, and worthless once stale; storing them in etcd would be both expensive and pointless. kubectl top and the Horizontal Pod Autoscaler read metrics.k8s.io and get fresh data from RAM. metrics-server is the proof that “I need API semantics but etcd-backed object storage is the wrong model” is a real and recurring need — which is precisely when the aggregation layer beats a CRD.

Configuration / API Surface

Verifying a registered extension server’s health:

kubectl get apiservice                                    # all APIServices, core + extension
# NAME                     SERVICE                  AVAILABLE   AGE
# v1beta1.metrics.k8s.io   kube-system/metrics-server   True    40d
 
kubectl describe apiservice v1beta1.metrics.k8s.io        # drill into an unhealthy one
# Status:
#   Conditions:
#     Type:    Available
#     Status:  False                                       # extension server unreachable/unhealthy
#     Reason:  FailedDiscoveryCheck

A consumer reading the aggregated API is indistinguishable from one reading a core API:

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes      # served by metrics-server, NOT etcd
kubectl top nodes                                         # the friendly front end for the same call

The key operational fact: an unhealthy APIService is not a contained failure. The aggregator probes every registered APIService during the discovery sweep that kubectl and controller-runtime perform on startup, so a broken extension server can make every kubectl command slow cluster-wide — see Failure Modes and API Aggregation Layer.

Failure Modes

• A broken extension server degrades the whole cluster’s API. Because discovery aggregates across all APIServices, an unavailable extension server makes kubectl <anything> wait for the failed discovery probe. A CRD can never cause this — it lives inside kube-apiserver. Mitigation: delete the APIService of a permanently-dead extension server (kubectl delete apiservice <name>).
• Data loss on crash. The extension server owns its storage. If that storage is in-memory and the Pod restarts, the data is gone (metrics-server accepts this — it rebuilds in ~15s). An extension server holding domain state without its own HA persistence loses it on every restart. A CRD never has this problem: etcd is HA by design.
• mTLS / request-header CA rotation. When the cluster’s --proxy-client cert or request-header CA rotates (e.g. during a managed control-plane upgrade), an extension server that does not re-fetch the CA bundle starts rejecting the proxied identity — every forwarded request looks anonymous. Symptom: 401s after a control-plane upgrade.
• You now operate a server. Scaling, HA, upgrades, security patching, leader election (if stateful), capacity planning — all of it is yours. This is not a failure mode so much as the standing cost; underestimating it is the failure.
• resourceVersion skew. The extension server’s resourceVersion namespace is independent of the core server’s. A controller watching both must track two streams; conflating them is a classic custom-controller bug.

Alternatives and When to Choose Them

The decision is almost always aggregated API server vs CRD (kubernetes.io — Custom Resources):

Concern	CRD	Aggregated API Server
Storage	etcd, managed by kube-apiserver	Anything — in-memory, external DB, computed
Validation	OpenAPI v3 schema + CEL rules	Whatever you code
Watch / resourceVersion	Core watch cache, free	You implement it
RBAC / admission	Reuses the core stack	Aggregator authz at the door; rest is yours
Conversion between versions	Declarative + conversion webhook	Fully under your control
Operational cost	~zero (just a YAML)	A real HA server to run
Effort to build	Declarative YAML	Real Go server code

Choose CRDs — and this is the answer ~95 % of the time. No separate server, no separate storage, the kube-apiserver handles watch/RBAC/admission/server-side apply for you. The recurring SIG-API-Machinery trend is to push more capability into CRDs (CEL validation rules, defaulting, ratcheting, server-side apply) so that fewer use cases ever need the aggregation layer.

Choose an aggregated API server only when CRDs genuinely cannot serve the requirement:

• The data should not live in etcd. High-cardinality, high-churn, or ephemeral data (metrics) — etcd is the wrong store, or the volume would crush it. Per-Pod metrics across a 5,000-node cluster is the classic example.
• Custom storage. The data already lives in an external system (Prometheus, a SQL database) and you want a Kubernetes-API-shaped front end over it without copying it into etcd.
• Custom business logic on read. The response is computed per request, not stored — synthesized, aggregated, or transformed from another source.
• REST semantics a CRD cannot express. Non-CRUD sub-resources, special streaming endpoints, protobuf serialization, or conversion logic richer than a conversion webhook allows.
• Request volumes too high for the CRD path.

The honest summary: an aggregated API server is a real server you must build, deploy, secure, scale, and keep alive — and a bug in it can stutter the whole cluster’s API surface. Reach for it only when the CRD model is genuinely inadequate. For adding types, CRDs win; the aggregation layer is the escape hatch for non-etcd-shaped data.

Production Notes

• EKS, GKE, AKS all run metrics-server (or a vendor equivalent) as an aggregated API by default. Customers cannot remove the aggregation layer — it is part of kube-apiserver — but they can register their own APIServices. On managed clusters, the apiserver-to-extension network path matters: the managed (off-customer-VPC) apiserver must be able to reach the extension Pod, or the APIService flaps Available=False.
• prometheus-adapter is the other widely deployed aggregated API: it registers custom.metrics.k8s.io/external.metrics.k8s.io and translates HPA queries into PromQL against Prometheus — storage is “Prometheus,” not etcd. A clean illustration of “custom storage” as the deciding factor.
• kcp (kcp.io) builds an entire multi-tenant Kubernetes-style control plane out of the aggregation pattern (and beyond) — aggregation is foundational to non-cluster K8s-style deployments.
• service-catalog (now archived) registered servicecatalog.k8s.io as an aggregated API over the Open Service Broker — a historical example of an aggregated API for non-metric domain data, and a cautionary one: it was eventually deprecated partly because the operational weight of an extra API server outweighed the benefit over CRDs.
• The strategic direction is fewer aggregated API servers, not more. As CEL validation, declarative defaults, and server-side apply mature on CRDs, the set of use cases that require an aggregated server keeps shrinking. Treat aggregation as a specialist tool for platform-level extensions (observability, autoscaling sources, K8s-shaped APIs over non-K8s systems), not a default.

See Also

• API Aggregation Layer — the kube-aggregator, the APIService, the proxy/identity machinery (sibling note; assumed background)
• Custom Resource Definition — the other, default extension mechanism
• Metrics Server — the canonical aggregated API server
• Horizontal Pod Autoscaler — the canonical consumer of an aggregated metrics API
• Operator Pattern — the CRD-based extension pattern, the usual alternative
• kube-apiserver — the host process the aggregator runs inside
• API Server Request Flow — the pipeline the aggregator sits within
• CEL in Kubernetes — the validation engine that keeps shrinking the need for aggregation
• Kubernetes API Groups and Versions — how the aggregated group slots into the API surface
• Kubernetes MOC — umbrella index (§13 Extensibility)