vault

Self-Managed Kubernetes

12 min read

Self-Managed Kubernetes

Self-managed Kubernetes means you operate the control plane yourself — you run, patch, back up, and scale kube-apiserver, etcd, kube-scheduler, and kube-controller-manager on machines you provision, instead of paying a cloud provider to hide them (Kubernetes — Installing with deployment tools). This is the opposite of the Managed Kubernetes Services model: there is no $0.10/hour cluster fee and no SLA, because there is no provider on the hook — the cluster’s availability is entirely your team’s problem. The canonical tooling spans a spectrum of automation: kubeadm (the official, low-level cluster-bootstrap tool that wires up one node at a time), kops (declarative full-lifecycle management, historically AWS-focused), kubespray (Ansible-based, multi-provider, strong on bare metal), and at the pedagogical extreme “Kubernetes The Hard Way” (Kelsey Hightower’s deliberately un-automated walkthrough where every certificate and every component flag is wired by hand). Self-managed Kubernetes makes sense for regulatory isolation, bare-metal fleets, air-gapped environments, very large scale, or when you need control over component flags that managed services lock down — but it transfers the entire day-2 burden (etcd backups, control-plane HA, upgrades, certificate rotation, security patching) onto you.

Mental Model

flowchart TB
    subgraph SPECTRUM["The self-managed automation spectrum"]
        HW["Kubernetes The Hard Way<br/>(zero automation — pedagogy)"]
        KA["kubeadm<br/>(bootstraps ONE node per invocation)"]
        KS["kubespray<br/>(Ansible orchestrates kubeadm across the fleet)"]
        KO["kops<br/>(declarative — also provisions the VMs)"]
        CAPI["Cluster API<br/>(K8s resources manage K8s clusters)"]
    end
    HW -->|"more automation"| KA
    KA -->|"+ OS prep, runtime, CNI, multi-node"| KS
    KA -->|"+ infra provisioning, declarative state"| KO
    KO -->|"+ reconciliation as controllers"| CAPI

    subgraph OWN["What 'self-managed' transfers to YOU"]
        E1["etcd backup &amp; restore"]
        E2["control-plane HA &amp; quorum"]
        E3["version upgrades, version skew"]
        E4["certificate rotation"]
        E5["security patching, CVE response"]
    end

What this shows. The four tools are not competitors so much as points on an automation gradient. Kubernetes The Hard Way automates nothing — its value is pedagogical. kubeadm automates the per-node bootstrap but nothing around it. kubespray and kops wrap kubeadm-class bootstrapping with fleet-wide orchestration; kops additionally provisions the infrastructure. Cluster API is the modern endpoint: cluster lifecycle expressed as Kubernetes resources reconciled by controllers. The lower box is the real point of the note — whichever tool you pick, the day-2 operational burden is now yours.

Mechanical Walk-through

kubeadm — the official low-level bootstrapper

kubeadm is the Kubernetes-project-blessed cluster-bootstrap tool, and the foundation almost every other self-managed installer builds on. It does not provision machines, install a container runtime, or pick a CNI plugin — it assumes a Linux host with a CRI runtime already present, and from there bootstraps the Kubernetes layer (Kubernetes — kubeadm).

The two commands are kubeadm init and kubeadm join:

  • kubeadm init on the first control-plane node runs pre-flight checks, generates a cluster Certificate Authority and all component TLS certificates, writes static-Pod manifests for the API server, etcd, scheduler, and controller-manager into /etc/kubernetes/manifests/, and starts the local kubelet which picks those manifests up and runs the control plane. The mechanics are covered in detail in Static Pods — the control plane bootstraps itself because static Pods don’t need the API server to exist first.
  • kubeadm join adds further nodes. Additional control-plane nodes get their own static-Pod control plane and join the etcd quorum; worker nodes simply register their kubelet with an existing API server. The join command carries a bootstrap token plus a CA discovery hash so the new node can authenticate and verify it’s joining the right cluster.

For high availability, kubeadm supports the same two etcd topologies described in Kubernetes Cluster Architecture — stacked (etcd colocated on control-plane nodes) and external (etcd on dedicated machines) — but kubeadm does not set up the load balancer in front of the API servers; you provision that (HAProxy + keepalived, kube-vip, or a cloud LB) yourself (Kubernetes — HA with kubeadm). kubeadm is “one node per invocation” — it has no concept of a fleet. That gap is what kubespray and kops fill.

kops — Kubernetes Operations

kops (“Kubernetes Operations”) treats a cluster and its infrastructure as a single declarative spec. Unlike kubeadm, kops provisions the VMs themselves — it makes cloud API calls directly (historically AWS via the AWS API; later GCP, and other providers in varying maturity). You describe the desired cluster with kops create cluster, kops stores the spec in a state store (an S3 bucket on AWS), and kops update cluster reconciles reality toward the spec — creating EC2 instances, Auto Scaling Groups, load balancers, and routing, then bootstrapping Kubernetes on them (kops).

kops’s strength is being declarative and infrastructure-aware — kops rolling-update cluster performs a node-by-node rolling upgrade of the whole fleet. Its historical weakness is cloud coupling: kops is most mature on AWS, and on other providers or bare metal it is less battle-tested than kubespray (Altoros — comparison).

kubespray — Ansible-based, provider-agnostic

kubespray is a collection of Ansible playbooks that stand up a production-grade cluster on bare metal or most clouds. Crucially, since version 2.3 kubespray uses kubeadm internally for the actual Kubernetes bootstrap — kubespray’s job is everything around kubeadm: OS preparation (kernel modules, sysctls, swap), container-runtime installation, CNI plugin deployment, and post-install configuration (kubespray — comparisons).

kubespray’s strengths are provider-agnosticism and bare-metal friendliness — it does not assume a cloud API, only SSH access to a set of machines — and its fit with shops that already operate Ansible. The cost is the operational weight of Ansible itself: large playbook runs, slow incremental changes, and the need for Ansible fluency on the team.

Kubernetes The Hard Way — pedagogy, not production

“Kubernetes The Hard Way” is Kelsey Hightower’s deliberately un-automated tutorial: you generate the CA and every certificate by hand, distribute them, write every component’s systemd unit and flag set yourself, configure the kubelet TLS bootstrap manually, and debug each piece in isolation (Kubernetes The Hard Way). It is not a way to run production clusters — it has no automation, no upgrade story, no HA convenience. Its sole purpose is understanding: after doing it once, “what is inside this cluster” stops being a mystery, and kubeadm’s static-Pod magic becomes legible rather than opaque.

When self-managed makes sense — and the real cost

Self-managed Kubernetes is the right call for a minority of clusters:

  • Regulatory / sovereignty isolation — industries or jurisdictions that forbid running on a hyperscaler’s shared control plane.
  • Bare metal — owning the hardware (cost at very large scale, specialized NICs/GPUs, latency-sensitive on-prem workloads).
  • Air-gapped environments — no internet egress, so a cloud-managed control plane is impossible by construction.
  • Full control over component flags — managed services lock down API-server feature gates, admission plugins, and scheduler config; self-managed exposes all of it.

The cost is total ownership of day-2 operations. You — not a provider — own: etcd snapshot + restore (lose etcd, lose the cluster); control-plane HA and Raft quorum sizing; version upgrades respecting the version-skew policy; certificate rotation (kubeadm-generated certs expire after one year by default — a notorious “cluster died on its first birthday” failure); and security patching of the OS, runtime, and Kubernetes binaries against CVEs. The Kubernetes MOC decision framework is blunt about this: choose managed Kubernetes unless you have a specific reason not to.

Configuration / API Surface

# --- kubeadm: bootstrap a single control-plane node ---
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.example.com:6443" \  # the LB VIP/DNS — set this
  --upload-certs \                                       # share CA certs for HA joins
  --pod-network-cidr 10.244.0.0/16 \                     # must match the CNI you install
  --kubernetes-version v1.32.3
# kubeadm prints two join commands: one for control-plane nodes, one for workers.
 
# Add a SECOND control-plane node (joins the etcd quorum):
sudo kubeadm join k8s-api.example.com:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>
 
# Add a WORKER node (kubelet registers; no static-Pod control plane):
sudo kubeadm join k8s-api.example.com:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:<hash>
 
# kubeadm does NOT install a CNI — you must, or every node stays NotReady:
kubectl apply -f https://raw.githubusercontent.com/.../calico.yaml
# --- kops: a declarative cluster spec (state stored in an S3 bucket) ---
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: prod.k8s.example.com
spec:
  kubernetesVersion: 1.32.3
  cloudProvider: aws
  networkCIDR: 10.0.0.0/16
  etcdClusters:
    - name: main
      members:                          # 3 etcd members across 3 AZs
        - { name: a, instanceGroup: master-us-east-1a }
        - { name: b, instanceGroup: master-us-east-1b }
        - { name: c, instanceGroup: master-us-east-1c }
  topology:
    masters: private
    nodes: private
# `kops update cluster --yes` reconciles AWS infrastructure toward this spec;
# `kops rolling-update cluster --yes` does a node-by-node fleet upgrade.

Line-by-line. In the kubeadm block, --control-plane-endpoint is the single most important flag for HA — it must point at the load balancer’s stable address, not a node IP, because every node and kubectl will target it; getting this wrong bakes a single-node dependency into the cluster permanently. --upload-certs stages the CA so additional control-plane nodes can join without manual cert copying. The final kubectl apply line is the reminder that kubeadm gives you a cluster whose nodes are all NotReady until you install a CNI. In the kops spec, the explicit three-member etcdClusters across three AZs is kops doing what kubeadm leaves to you — declaring the HA etcd topology as data.

Failure Modes

Certificate expiry on the cluster’s first birthday. kubeadm-issued certificates default to a one-year lifetime. A cluster bootstrapped and then left alone fails ~365 days later: the API server rejects component certs, kubectl stops working, the cluster is effectively down. The fix is kubeadm certs renew all (and ideally automating it, or upgrading regularly since kubeadm upgrade renews certs as a side effect). This is one of the most common self-managed outages and has no analog on managed services.

Forgetting to install a CNI. A fresh kubeadm init cluster has every node NotReady and CoreDNS Pods stuck Pending because there is no Pod networking. Operators new to kubeadm often think the install failed; it didn’t — kubeadm deliberately delegates CNI choice to you.

etcd quorum loss with no backup. Self-managed clusters that never ran etcdctl snapshot save and then lose an etcd majority have lost the entire cluster state irrecoverably. The discipline (etcd Backup and Restore) is yours alone — no provider snapshots etcd for you.

API-server load balancer is a single point of failure. kubeadm does not provision the LB; teams that put a single non-HA HAProxy in front of three API servers have replaced control-plane HA with LB fragility. The LB needs its own HA story (keepalived VIP, kube-vip, cloud LB).

Version-skew violations during upgrades. Upgrading kubelets ahead of the API server, or skipping a minor version, violates the version-skew policy and produces subtle, hard-to-diagnose breakage. Managed services enforce the ordering; self-managed leaves it to your runbook.

Ansible drift with kubespray. A kubespray cluster modified out-of-band (manual kubectl edits, hand-tuned manifests) diverges from the playbook state; the next playbook run can overwrite or conflict with the manual changes. The discipline is “all changes go through the playbook” — which not every team maintains.

Alternatives and When to Choose Them

Managed Kubernetes Services (EKS, GKE, AKS). The default recommendation. The per-cluster fee is dwarfed by the cost of operating etcd, control-plane HA, upgrades, and certificate rotation. Choose self-managed only against the specific reasons listed above.

Cluster API. The modern, Kubernetes-native take on self-managed lifecycle: cluster topology expressed as Kubernetes resources (Cluster, MachineDeployment, KubeadmControlPlane) reconciled by controllers in a management cluster. Cluster API still uses kubeadm under the hood for bootstrap but adds declarative, controller-driven reconciliation — “Kubernetes managing Kubernetes.” Choose it when operating many self-managed clusters; it scales the operational model that kops and kubespray apply to one cluster at a time.

Lightweight Kubernetes Distributions (k3s, k0s, MicroK8s). A different kind of self-managed: single-binary distributions that collapse the whole control plane into one process and bundle the CNI and datastore. Choose these for edge, IoT, single-node, or CI — where the full kubeadm stack is too heavy.

kops vs kubespray. On AWS, kops is the more mature, infrastructure-aware choice. For bare metal, non-AWS clouds, or Ansible-fluent teams, kubespray is the more flexible and provider-agnostic option.

Production Notes

The Kubernetes project’s own documentation positions kubeadm as the reference bootstrap path — every conformance test, every “production environment” tutorial, and most on-prem installs build on it. Cluster API, kops, kubespray, microk8s, k3s (in some modes), and even cloud-managed services lean on kubeadm-class bootstrap mechanics; understanding kubeadm is understanding the substrate.

The cloud-industry consensus, reflected in the Kubernetes MOC decision framework and echoed across vendor write-ups, is that most organizations should not self-manage. The savings from skipping the per-cluster control-plane fee — on the order of $70/month per cluster — are trivially outweighed by the engineering time a self-managed control plane consumes in etcd operations, upgrade choreography, and incident response. Self-managed remains a real and necessary option for the regulated, the air-gapped, the bare-metal, and the hyperscale — but it is a deliberate choice with a known, large, ongoing cost, not a default.

“Kubernetes The Hard Way” deserves a final word: most production engineers do not need to do it, and need to do it at most once. Its return is not a running cluster but a demystified mental model — after it, the static-Pod bootstrap, the certificate web, and the kubelet TLS handshake stop being magic.

See Also

Graph View

Backlinks