vault

Kubernetes

21 min read

Kubernetes

Kubernetes (commonly abbreviated K8s) is the open-source container-orchestration platform that has become the de facto industry standard for running containerized workloads in production. The Kubernetes documentation defines it as “a portable, extensible, open source platform for managing containerized workloads and services that facilitate both declarative configuration and automation” (kubernetes.io — Overview). Google open-sourced the project in 2014 (first GitHub commit on 6 June 2014, attributed to Joe Beda), reached version 1.0 on 21 July 2015, simultaneously announced the donation of the project to the newly-formed Cloud Native Computing Foundation (CNCF), formally joined CNCF on 10 March 2016 as its seed project, and graduated as CNCF’s first graduated project on 6 March 2018 (CNCF — Kubernetes is ten years old). The name comes from Ancient Greek κυβερνήτης (kybernḗtēs, “helmsman” or “pilot”) — etymologically the same root as “cybernetics” and English “governor” — which is why the project’s logo is a seven-spoked ship’s wheel (Wikipedia — Kubernetes). The abbreviation K8s is a “numeronym” — the digit 8 counts the eight letters elided between the leading K and the trailing s (K-ubernete-s) — a convention borrowed from the older i18n (internationalization) and l10n (localization) abbreviations. The platform’s design draws explicitly on Google’s decade-plus internal experience with the Borg (Verma et al. 2015) and Omega cluster managers, as documented in the 2016 retrospective by Burns, Grant, Oppenheimer, Brewer, and Wilkes (ACM Queue 14(1)). See Borg Omega and Kubernetes Lineage for the architectural-genealogy detail and Container Orchestration Architecture for the broader pattern of which Kubernetes is the dominant instance.

This note is the umbrella for the Kubernetes vault: it answers what is K8s, where did it come from, how is it shaped, and how does the rest of the vault decompose it. The deeper architectural treatment — control plane vs nodes, etcd, scheduler, kubelet, the API surface, workload resources, networking, storage, autoscaling, security, extensibility, observability — lives in the leaf notes linked throughout and is indexed by Kubernetes MOC.

Mental Model

flowchart TB
    subgraph "Layer 4: Ecosystem (CNCF projects, vendors)"
        ECO[Helm · Kustomize · ArgoCD · Istio · Prometheus<br/>Cilium · Karpenter · KEDA · Velero · Operators]
    end
    subgraph "Layer 3: Workload Resources (built-in + CRDs)"
        WL[Pod · Deployment · StatefulSet · DaemonSet · Job · CronJob<br/>Service · Ingress · ConfigMap · Secret · PVC · NetworkPolicy · CRDs]
    end
    subgraph "Layer 2: Control Plane + Nodes"
        CP[kube-apiserver · etcd · kube-scheduler<br/>kube-controller-manager · cloud-controller-manager]
        NODE[kubelet · kube-proxy · container runtime<br/>CRI plugin · CNI plugin · CSI plugin]
    end
    subgraph "Layer 1: Foundational ideas"
        IDEA[declarative desired state · level-triggered reconciliation<br/>label-based grouping · API-first extensibility · Pods as atomic unit]
    end
    IDEA --> CP
    CP <--> NODE
    CP --> WL
    NODE --> WL
    WL --> ECO

What this diagram shows. Kubernetes is best understood as four layers stacked on a small set of foundational ideas. Layer 1 is the conceptual core — declarative desired state reconciled by level-triggered controllers (see Kubernetes Control Loop Pattern and Desired State vs Observed State), labels as the universal connector, API-first extensibility, and Pods as the atomic scheduling unit. Layer 2 is the physical anatomy — a control plane that mediates all reads and writes through the API server, persists state in etcd, and orchestrates per-node agents (kubelet, kube-proxy) that delegate runtime / networking / storage to pluggable CRI / CNI / CSI drivers. Layer 3 is the catalog of built-in workload, service, configuration, and policy resources, plus the Custom Resource Definitions that user-written controllers add. Layer 4 is the CNCF and vendor ecosystem — package managers, GitOps engines, service meshes, observability stacks, schedulers, operators — which is what most production teams actually interact with day-to-day. The insight to extract is that Kubernetes itself is small at the bottom and wide at the top: every later layer is built from a few primitives in Layer 1, and the platform’s apparent complexity is dominated by the ecosystem (Layer 4) and the CRD-extended workload catalog (Layer 3) rather than by the core (Layers 1–2).

Origin and History

Kubernetes was conceived in late 2013 / early 2014 by Google engineers Joe Beda, Brendan Burns, and Craig McLuckie, with early contributions from Ville Aikas, Tim Hockin, Dawn Chen, Brian Grant, and Daniel Smith (Wikipedia — Kubernetes; v1-32.docs.kubernetes.io — 10 Years of Kubernetes). The team was responding to two converging pressures. The first was the rise of Docker (open-sourced March 2013), which made Linux containers operationally accessible to teams outside hyperscalers and immediately generated demand for an open-source orchestrator. The second was the dawning realization inside Google that the company’s decade of cluster-management know-how — embodied in Borg (running internally since approximately 2003–2004) and its research successor Omega (begun around 2013) — was a competitive advantage that would erode if the rest of the industry built a non-Borg-shaped orchestration substrate. Mid-2014 Google therefore announced Kubernetes, internally codenamed “Project Seven” — a Star Trek reference to Seven of Nine, a former Borg drone severed from the collective (GeekWire — Kooky Kubernetes name). The seven-spoked ship’s wheel logo (designed by Tim Hockin) preserves the codename inside the public branding: seven spokes, seven sides of the heptagonal background, and the helmsman’s wheel that echoes the Greek meaning of kybernḗtēs.

The earliest public commit landed on 6 June 2014. Version 1.0 shipped on 21 July 2015, and on the same day Google and the Linux Foundation jointly announced the formation of the Cloud Native Computing Foundation (CNCF) with Kubernetes as its seed project (Aqua — Kubernetes History). The formal donation to CNCF completed on 10 March 2016, and on 6 March 2018 Kubernetes became the foundation’s first graduated project (CNCF blog). The donation pattern — Google preserving design influence through the Steering Committee and the Kubernetes Special Interest Group (SIG) structure while ceding ownership and trademark to a neutral foundation — became the template that subsequent CNCF projects (Prometheus, Envoy, containerd, etcd) followed.

By 2024–2026 Kubernetes runs on every major cloud provider as a managed service (Amazon EKS, Google GKE, Azure AKS, Oracle OKE, DigitalOcean Kubernetes, and equivalents), drives the production infrastructure of essentially every hyperscale internet company (Spotify, Shopify, Airbnb, Reddit, GitHub, Pinterest, Uber, the New York Times, and so on), and is the substrate on which the entire CNCF ecosystem (Helm, Prometheus, Istio, Linkerd, ArgoCD, Flux, Knative, Crossplane, KEDA, Karpenter — cf. Kubernetes MOC sections §15–§17) is built. The Kubernetes documentation itself describes the platform in deliberately understated terms: “Kubernetes is not a mere orchestration system. In fact, it eliminates the need for orchestration … Kubernetes comprises a set of independent, composable control processes that continuously drive the current state towards the provided desired state” (kubernetes.io — Overview) — a sentence that compresses the entire conceptual model into a single claim and which the Kubernetes Control Loop Pattern note unpacks.

The official Kubernetes documentation states the etymology plainly: “The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the ‘K’ and the ‘s’” (kubernetes.io — Overview). The Greek word κυβερνήτης is the source of both the modern term “cybernetics” (Norbert Wiener’s 1948 coinage for the science of feedback and control) and the English word “governor” via Latin gubernator. The semantic resonance is deliberate: a Kubernetes cluster is governed by control loops that exhibit goal-seeking feedback, and the platform’s intellectual ancestry runs through cybernetics, control theory, and Borg’s reconciliation-driven design.

The K8s numeronym (K + 8 + s, eight elided letters) follows the pattern established by i18n (i + 18 letters + n) and l10n (l + 10 letters + n), themselves coined in the early 1990s in the internationalization community. Other vault notes you may see using the same pattern: there are none in this vault yet, but k8s shows up in shell scripts, file names, namespace prefixes (kube-system, kube-public), and many cloud-vendor tools. The abbreviation is treated as canonical and is used in talks, blog posts, and even the URLs of CNCF properties (k8s.io redirects to kubernetes.io).

The seven-spoked ship’s wheel logo combines three references in one mark: (1) the literal helmsman’s wheel from the Greek etymology; (2) the heptagonal frame and seven spokes echo the project’s “Project Seven” internal codename; (3) the Seven of Nine / Borg subtext was a private joke among ex-Borg-team engineers (GeekWire). The logo’s blue color was chosen for visibility on CNCF marketing materials and has remained essentially unchanged since 2014.

Surface Area: What Kubernetes Provides

The kubernetes.io overview enumerates what the platform delivers out of the box (kubernetes.io — Overview):

The same overview is explicit about what Kubernetes is not: “Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. … Kubernetes does not deploy source code and does not build your application. … Does not provide application-level services, such as middleware, data-processing frameworks, databases, caches, nor cluster storage systems as built-in services. … Does not provide nor mandate a configuration language/system. … Does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems.” The platform is therefore a kernel on top of which higher-level platforms (OpenShift, Rancher, Tanzu, Knative, Backstage-driven IDPs) and tools (Helm, ArgoCD, Crossplane) are built — see Kubernetes MOC §15–§17 for the ecosystem layer.

The Anatomy at a Glance

This section is the briefest possible survey of K8s’ physical structure; each component has its own atomic note linked from Kubernetes MOC §2.

A cluster consists of a control plane (one or more highly-available machines, often three or five) and a fleet of worker nodes (VMs or bare metal). The control plane runs:

  • kube-apiserver — the REST front door; “the API server is the front end for the Kubernetes control plane” (kubernetes.io — Architecture). Every read and write goes through it; it is the sole writer to etcd.
  • etcd — the consistent, highly-available key-value store backing all cluster state; backed by Raft consensus.
  • kube-scheduler — assigns Pods to Nodes via a filter-and-score pipeline.
  • kube-controller-manager — bundles built-in controllers (Deployment, ReplicaSet, Job, Node, Endpoint, ServiceAccount, …) into a single process; each controller is an instance of the Kubernetes Control Loop Pattern.
  • cloud-controller-manager — the cloud-provider-specific controllers (LoadBalancer provisioning, node lifecycle integration), separated so the core remains cloud-agnostic.

Each worker node runs:

  • kubelet — the per-node agent that watches the API server for Pod assignments and instructs the container runtime to start, stop, and probe containers.
  • kube-proxy — programs node-local network rules (iptables / IPVS / nftables / eBPF) so Service virtual IPs route to backend Pods.
  • A container runtime (containerd or CRI-O in the post-dockershim world — dockershim was removed in v1.24, see Platform Evolution below; communicating via the Container Runtime Interface gRPC contract).

The cluster’s resource model is the universal {apiVersion, kind, metadata, spec, status} shape covered in Kubernetes Object Model. Every workload, configuration, networking, storage, scheduling, and security primitive is an instance of this shape. The full taxonomy is documented in Kubernetes MOC §4–§14.

Scalability Envelope

Kubernetes is not infinitely scalable, and the documented limits are a useful sanity check on cluster-design proposals. As of v1.36 the project states that a single cluster is designed to accommodate configurations meeting all of the following criteria simultaneously (kubernetes.io — Considerations for large clusters):

  • No more than 5,000 nodes.
  • No more than 110 Pods per node.
  • No more than 150,000 total Pods.
  • No more than 300,000 total containers.

These are supported envelope numbers, not hard kernel limits — the binding constraints in practice are etcd’s write throughput and database size (etcd defaults to a multi-gigabyte storage quota and degrades sharply if saturated), the API server’s watch fan-out cost, and the scheduler’s per-Pod decision latency. Teams that need to exceed the envelope generally run multiple clusters rather than one giant cluster (see Multi-Cluster Kubernetes); a fleet of 50 clusters of 1,000 nodes each is operationally healthier than one 50,000-node cluster, which no upstream-supported configuration reaches.

Platform Evolution: Three Milestone Removals

Three large breaking-or-structural changes shaped modern Kubernetes and are worth knowing because stale tutorials still assume the old behavior:

  • Dockershim removal (v1.24, May 2022). The kubelet’s built-in shim for talking to the Docker Engine was deprecated in v1.20 and removed in v1.24 (kubernetes.io — Dockershim historical context). Since then the kubelet speaks only the Container Runtime Interface (CRI) gRPC contract, and production clusters run containerd or CRI-O directly. Docker-built images are unaffected — they are standard OCI images — but Docker-the-daemon is no longer a Kubernetes runtime.
  • master → control-plane rename (v1.20–v1.26). Covered in depth in Kubernetes Cluster Architecture: the node-role.kubernetes.io/master label/taint was superseded by control-plane, with the legacy label removed and the legacy taint dropped over v1.24–v1.26.
  • In-tree cloud provider removal (v1.31, August 2024). Cloud-specific code (AWS, Azure, GCP, OpenStack, vSphere) that once lived inside core Kubernetes binaries was extracted into external cloud-controller-manager components and fully removed from the core tree in v1.31 (CNCF — Removing in-tree cloud providers). OpenStack went first (v1.26), AWS in v1.27, the remainder in v1.31. The payoff is leaner, vendor-neutral core binaries and cloud integrations that release on their own cadence rather than gating on the Kubernetes release.

Version Cadence and Support Window

Kubernetes ships approximately three minor releases per year (down from four; the cadence changed with the 2021 release cycle), each receiving twelve months of patch support for releases 1.19 and newer — earlier releases (1.18 and before) had only nine months (kubernetes.io — Release Cycle, as of v1.36, April 2026). The cycle is approximately fourteen weeks: roughly eleven weeks of feature development followed by a code-freeze-and-stabilization window of about two-to-three weeks before the General Availability (GA) release. The pattern is:

  • Enhancement Freeze at roughly week 4 — every feature targeted for the release must have an accepted Kubernetes Enhancement Proposal (KEP) in github.com/kubernetes/enhancements.
  • Code Freeze around week 12 — only critical bug fixes (kind/bug, kind/failing-test) accepted; everything else uses the cherry-pick process from master onto the release branch.
  • GA around week 14.

Each minor version moves through three maturity gates: Alpha (off by default, an explicit feature gate required, may break or be removed between minor releases), Beta, and Stable / GA (covered by the deprecation policy, an API guaranteed for at least one year or one minor release — whichever is longer — after deprecation). The behavior of Beta deserves precision because it is widely misstated: a feature gate in Beta and an existing Beta API version are enabled by default, but since Kubernetes 1.24 the policy under KEP-3136 is that brand-new Beta REST APIs are off by default and must be opted into via --runtime-config — existing Beta APIs that were already on stay on for backward compatibility. So “Beta is on by default” is true for feature gates and pre-1.24 APIs but false for new API groups introduced 1.24 onward.

The support window covers the three most recent minor releases (the “N-2” window: the current release plus the two prior, in effect for the patch-support period since v1.19). For example, when 1.36 is current, 1.35 and 1.34 also remain in support. Clusters that fall behind must upgrade through each intermediate minor in sequence — skipping minor versions is not supported (kubeadm upgrade enforces a one-minor-version step). The cadence has slipped occasionally in practice (1.20 was delayed by several weeks in late 2020), so specific future release dates should be checked against the Kubernetes release schedule rather than assumed from the canonical roughly-April/August/December rhythm.

Mechanical Walk-through: What “Running on Kubernetes” Actually Means

Consider the canonical example: a user runs kubectl apply -f deployment.yaml to deploy a stateless web application with three replicas. The chain of events involves nearly every component listed above and exhibits the platform’s eventually-consistent, level-triggered, declarative operating model end to end:

  1. kubectl reads the YAML, applies the declarative-apply three-way merge (or Server-Side Apply if the --server-side flag is set), and POSTs the resulting object to the API server.
  2. The API server authenticates the user, authorizes the request against Kubernetes RBAC, passes the request through admission controllers (mutating then validating), validates the schema, and persists the resulting Deployment object to etcd at the key /registry/deployments/<namespace>/<name>.
  3. The Deployment controller, watching all Deployments via the API server’s watch stream, observes the new object. Its reconcile function creates a corresponding ReplicaSet.
  4. The ReplicaSet controller observes the new ReplicaSet and creates three Pod objects with spec.nodeName unset.
  5. The scheduler observes the unscheduled Pods, runs its filter / score pipeline against the cluster’s nodes, picks a winner for each, and writes a Binding updating spec.nodeName.
  6. The kubelet on each assigned node, watching Pods filtered to its own node, observes the assignment, instructs the container runtime via CRI to pull the image and start the containers, performs initial probes, and PATCHes status.phase to Running via the API server.
  7. The cluster has now converged to the desired state. If a node fails, its kubelet stops renewing its etcd-backed Lease in kube-node-lease, the Node controller marks it NotReady, after a grace period evicts its Pods (subject to Pod Disruption Budget), and the ReplicaSet controller observes that fewer than three Pods exist and creates replacements — at which point the scheduler picks new nodes and the kubelets there start new containers. No global coordinator orchestrates the recovery (kubernetes.io — Controllers). Every step is a small, independent reconciliation loop.

That entire chain — six independent components, each running its own control loop against shared state in etcd — is the operating model of Kubernetes. Once internalized, every feature in the platform feels inevitable: a Deployment is a controller managing ReplicaSets; an HPA is a controller managing the Deployment’s replica count; a Job is a controller managing Pods until completion; an Ingress controller is a controller programming an external load balancer; an Operator is a controller managing application-specific resources. Everything is a controller. The full elaboration is in Kubernetes Control Loop Pattern.

Configuration Surface: A Minimal Manifest

The shape of every Kubernetes resource is uniform; here is the minimal Deployment manifest the kubernetes.io overview uses (kubernetes.io — Kubernetes Objects), annotated:

apiVersion: apps/v1              # API group "apps", version "v1" — the GA Deployment surface
kind: Deployment                 # the resource type (controller's responsibility)
metadata:
  name: nginx-deployment         # uniquely identifies the object within its namespace
spec:                            # desired state — what the user wants
  selector:
    matchLabels:
      app: nginx                 # the ReplicaSet/Pod label selector
  replicas: 2                    # desired number of running Pod replicas
  template:                      # the Pod template stamped out to create replicas
    metadata:
      labels:
        app: nginx               # label matches the selector — this is mandatory
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2      # container image (immutable reference recommended)
        ports:
        - containerPort: 80      # port the container listens on
# status: {}                     # status is owned by the controller, not the user

Every field above maps cleanly onto a concept the rest of the vault elaborates: apiVersion and kind to Kubernetes API Groups and Versions; metadata (labels, annotations, namespace, owner references) to Kubernetes Labels and Selectors / Kubernetes Annotations / Namespace / Owner References and Garbage Collection; spec to Desired State vs Observed State; template and replicas to ReplicaSet / Deployment; containers[].image to Container Runtime Interface / containerd. The uniformity is the platform’s most consequential design choice: every higher-level abstraction is just another resource with the same five-field shape, and every tool that handles one resource type can — by construction — handle all of them.

Failure Modes and Common Misunderstandings

Three misunderstandings recur in production:

  1. “Kubernetes guarantees that my workload will be running.” It does not. Kubernetes promises convergence given a stable desired state — that the controllers will keep trying to drive observed state toward desired state. It promises nothing about when convergence will happen, whether transitions will be atomic, or that all replicas will be simultaneously healthy. See Desired State vs Observed State for the conceptual fix and Kubernetes Control Loop Pattern for why this is the right guarantee given the operating environment.

  2. “kubectl apply is idempotent so I can run it as often as I want without consequence.” Mostly true, but apply runs a three-way merge that may delete fields owned by other writers (controllers, operators, manual edits). The introduction of Server-Side Apply (stable since K8s 1.22, August 2021) replaces the client-side merge with server-tracked field ownership and explicit conflict detection (kubernetes.io — Server-Side Apply). The deeper discussion is in Declarative vs Imperative Configuration.

  3. “K8s is just a container scheduler.” The scheduler is one of dozens of controllers, and is the smallest part of what makes the platform distinctive. The substance is the API surface, the resource model, the controller pattern, and the extensibility hooks — none of which are about scheduling per se. The kubernetes.io overview is blunt: “Kubernetes is not a mere orchestration system. In fact, it eliminates the need for orchestration” (kubernetes.io — Overview). The full elaboration of why this distinction matters is in Container Orchestration Architecture.

Alternatives and When to Choose Them

The architectural-pattern alternatives to Kubernetes are catalogued in Container Orchestration Architecture; in brief:

  • HashiCorp Nomad — simpler single-binary scheduler, works for containers / VMs / bare binaries, much smaller ecosystem. Choose when operational simplicity outweighs ecosystem leverage.
  • Apache Mesos + Marathon / DC/OS — older orchestrator with a two-level scheduler design that influenced Omega and Kubernetes. Largely supplanted by Kubernetes by 2024–2026; persists in some legacy deployments and the Twitter / X infrastructure.
  • Amazon ECS — AWS-proprietary orchestrator with deep AWS integration. Choose for AWS-only workloads where you want to avoid Kubernetes’ learning curve.
  • Docker Swarm — Docker’s own orchestrator; largely abandoned by 2020s.
  • Netflix Titus — internal container scheduler built on Mesos; Netflix migrated workloads to Kubernetes over 2020–2024 but retained Titus for legacy.
  • Serverless (Serverless and Function as a Service Architecture) — different problem shape: stateless, bursty, event-driven workloads where the platform handles all scaling. Use Kubernetes for steady, stateful, long-running workloads; serverless for spiky stateless ones. The two compose via Knative inside a Kubernetes cluster.

The dominance of Kubernetes is not absolute, but its ecosystem effects (the CNCF landscape, the depth of available operators, the talent pool of K8s-fluent engineers) make it the path of least resistance for any new orchestration deployment.

Production Notes

  • Spotify migrated from Helios to Kubernetes beginning around 2017–2018 (Spotify Engineering — How Spotify Accidentally Deleted All its Kube Clusters is a famous post-mortem from this period). Their migration involved retrofitting a Helios-shaped internal platform onto Kubernetes’ very different model.
  • Shopify has documented its Kubernetes operations through Black Friday / Cyber Monday peaks at shopify.engineering, including specific posts on Pod Disruption Budgets, graceful rollouts, and cost management.
  • Pinterest, Reddit, GitHub, Airbnb, the New York Times, Uber, and Adobe all run production Kubernetes; engineering blogs at each company catalogue the recurring operational themes — multi-cluster patterns, autoscaling configuration, GitOps adoption.
  • Kubernetes Failure Stories at k8s.af — curated by Henning Jacobs — is the canonical compendium of production K8s incidents and the single best reading for understanding the platform’s failure modes empirically.
  • CNCF case studies at cncf.io/case-studies document hundreds of named-company adoptions across industries.

See Also

Graph View

Backlinks