vault

Blue-Green Deployment on Kubernetes

10 min read

Blue-Green Deployment on Kubernetes

Blue-green deployment is a release pattern in which two complete, full-capacity environments exist side by side — “blue” (the version currently serving production) and “green” (the new version) — and the release is an atomic switch of all traffic from blue to green at once. There is no gradual ramp and no mixed-version interval: until the cutover, 100% of traffic hits blue and 0% hits green; after it, the reverse. On Kubernetes the pattern has a famously direct expression — run two Deployments with distinct labels, and a single Service whose selector is flipped from blue’s Pod labels to green’s, an essentially instantaneous edit (kubernetes.io — Service). The payoffs are an instant rollback (flip the selector back) and the ability to fully test green with real infrastructure before any production traffic touches it; the costs are roughly 2× resource consumption during the overlap and the perennial difficulty of stateful workloads and shared databases, which do not “switch” atomically. This note covers blue-green as a Kubernetes pattern, both hand-rolled and via Argo Rollouts. The siblings are Rolling Update Strategy (gradual, mixed-version, no extra capacity) and Canary Deployment on Kubernetes (gradual, fractional, metric-gated).

Mental Model

flowchart TB
    subgraph before["Before cutover"]
        SVC1["Service: app<br/>selector: version=blue"]
        BLUE1["Deployment app-blue<br/>version=blue — v1.4 — LIVE"]
        GREEN1["Deployment app-green<br/>version=green — v1.5 — running, 0 traffic"]
        SVC1 -- "100% traffic" --> BLUE1
        SVC1 -. "0% traffic" .-> GREEN1
    end
    subgraph after["After cutover — one selector edit"]
        SVC2["Service: app<br/>selector: version=green"]
        BLUE2["Deployment app-blue<br/>v1.4 — kept warm for rollback"]
        GREEN2["Deployment app-green<br/>v1.5 — LIVE"]
        SVC2 -. "0% traffic" .-> BLUE2
        SVC2 -- "100% traffic" --> GREEN2
    end
    before == "kubectl patch svc:<br/>selector.version blue→green" ==> after

What this diagram shows. Two Deployments coexist, each at full replica count; a single Service selects exactly one of them. The “deployment” is not a rollout at all — it is a one-field edit to the Service selector that re-points every future connection. Blue stays running after the switch so that rollback is the same one-field edit in reverse. The insight to extract: blue-green converts the release risk from “a gradual transition that can stall halfway” into “a binary state with an atomic toggle.” You trade continuous capacity efficiency (you pay for two fleets during overlap) for a release with exactly two states and an O(1) rollback.

Mechanical Walk-through

The selector-swap implementation

The canonical hand-rolled blue-green on Kubernetes uses two Deployments + one Service:

  1. Blue Deployment runs v1.4, every Pod labelled app=shop, version=blue. The Service shop has selector: {app: shop, version: blue}. All traffic reaches blue.
  2. Apply the green Deployment running v1.5, Pods labelled app=shop, version=green, at full replica count. Green Pods start, pass readiness — but the Service does not select them, so they receive zero production traffic.
  3. Test green out-of-band. Reach green directly — kubectl port-forward to a green Pod, or a second “preview” Service shop-preview with selector.version=green exposed only internally. Run smoke tests, integration checks, manual QA against the real green Pods.
  4. Cut over. kubectl patch service shop -p '{"spec":{"selector":{"version":"green"}}}'. The moment the API server persists this, the EndpointSlice controller recomputes endpoints to the green Pods, kube-proxy reprograms every node, and new connections go to green. The switch is “atomic” at the API-object level; the data-plane reprogramming across nodes still takes a short, real interval (sub-second to low seconds).
  5. Hold blue warm. Keep the blue Deployment running for a soak period. If green misbehaves, kubectl patch the selector back to blue — instant rollback, because blue never stopped running.
  6. Decommission. Once green is trusted, scale blue to zero or delete it. The roles swap for the next release: green becomes the new “blue.”

Cutover is not perfectly atomic in the data plane

A selector edit is one atomic write in etcd, but the effect propagates through two more controllers (Service (Kubernetes)): the EndpointSlice controller must rewrite the slice, and every node’s kube-proxy must reprogram its rules. For a few hundred milliseconds, different nodes may route to different versions. In-flight requests on existing connections continue to blue — a TCP connection already established to a blue Pod is not migrated; it finishes on blue and only new connections go to green. This matters for long-lived connections (gRPC streams, WebSockets): a blue-green cutover does not forcibly cut them. Keep blue alive long enough for them to drain.

The Ingress / Gateway route-swap variant

Instead of one Service flipped between label sets, an alternative keeps two stable Services (shop-blue, shop-green, each permanently selecting its own colour) and flips an Ingress / Gateway API route to point at one or the other. The cutover is then an edit to the routing object’s backend/backendRefs. This is cleaner when an L7 router already fronts the app, and it generalises smoothly into Canary Deployment on Kubernetes — a route can carry weights, so “blue-green” and “canary” become the same routing object with different weight values (100/0 vs 90/10).

Automating it with Argo Rollouts

Argo Rollouts has a first-class blueGreen strategy that automates the whole dance (argo-rollouts.readthedocs.io — blue-green). Instead of two Deployments and a manual kubectl patch, you write one Rollout referencing two Services:

  • activeService (required) — the Service carrying production traffic. The Rollout controller manages its selector, pointing it at the stable ReplicaSet and switching it to the new ReplicaSet on promotion.
  • previewService (optional) — a Service the controller points at the new ReplicaSet before promotion, so green can be tested. After promotion the preview Service also tracks the new (now active) ReplicaSet.
  • autoPromotionEnabled — defaults to true; when false, the Rollout pauses indefinitely after green is healthy and waits for a human kubectl argo rollouts promote. autoPromotionSeconds auto-promotes after a delay.
  • prePromotionAnalysis — runs an AnalysisRun against green before the traffic switch; a failure blocks promotion.
  • postPromotionAnalysis — runs after the switch; a failure triggers automatic rollback to the previous ReplicaSet.
  • scaleDownDelaySeconds — defaults to 30; how long the old (blue) ReplicaSet is kept running after the cutover, both to let kube-proxy rules propagate and to keep rollback instant.

Configuration / API Surface

Hand-rolled, the swap is one command:

# blue is live; green Deployment already applied at full capacity and Ready.
# Cut over — atomic API write:
kubectl patch service shop \
  -p '{"spec":{"selector":{"app":"shop","version":"green"}}}'
 
# Rollback if green misbehaves — same edit, reversed:
kubectl patch service shop \
  -p '{"spec":{"selector":{"app":"shop","version":"blue"}}}'

The Argo Rollouts blueGreen form, with commentary:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop
spec:
  replicas: 6
  selector:
    matchLabels: { app: shop }
  template:                              # identical shape to a Deployment Pod template
    metadata:
      labels: { app: shop }
    spec:
      containers:
        - name: shop
          image: registry.example.com/shop:v1.5
  strategy:
    blueGreen:
      activeService: shop-active         # Service the controller flips on promotion
      previewService: shop-preview       # Service pointed at green for pre-cutover testing
      autoPromotionEnabled: false        # require an explicit `promote`; do not flip automatically
      scaleDownDelaySeconds: 300         # keep blue running 5 min post-cutover for fast rollback
      prePromotionAnalysis:              # gate the cutover on metrics from the preview Service
        templates:
          - templateName: smoke-success-rate
      postPromotionAnalysis:             # auto-rollback if green degrades after the switch
        templates:
          - templateName: prod-success-rate

Line-by-line:

  • activeService / previewService must be pre-existing Services with a selector the controller is allowed to manage; the controller injects the rollouts-pod-template-hash label to point them at the right ReplicaSet.
  • autoPromotionEnabled: false is the human-gate: green runs and is testable via shop-preview, but production keeps hitting blue until someone runs kubectl argo rollouts promote shop.
  • scaleDownDelaySeconds: 300 is the rollback window — blue stays warm for 5 minutes after the switch; an abort within that window is instantaneous because blue’s Pods never went away.

Failure Modes

  1. 2× capacity not available. During overlap both blue and green run at full replica count. On a cluster sized for one fleet, green’s Pods stay Pending with FailedScheduling and the cutover never becomes possible. Plan capacity for the overlap, or use Cluster Autoscaler / Karpenter to absorb the temporary doubling.
  2. Shared database does not switch. Blue and green almost always share one database. If green needs a schema change, blue must tolerate that schema during the overlap and rollback window — otherwise rollback is impossible (the DB is already migrated). The discipline is expand-and-contract migrations: make the schema backward-compatible so both versions work against it. Blue-green does not solve database migration; it constrains it.
  3. Long-lived connections survive the cutover on blue. gRPC streams and WebSockets on existing connections stay on blue after the selector flip. If you scale blue down too soon (scaleDownDelaySeconds too short, or a hasty manual delete), those connections are killed. Size the soak window to the connection lifetime.
  4. Stateful in-Pod data is stranded. If blue Pods accumulated in-memory state (session caches, queues), the cutover abandons it on blue. Blue-green assumes stateless or externally-stored state. Stateful workloads need StatefulSet-aware strategies, not a selector flip.
  5. “Atomic” oversold. The selector write is atomic; the data-plane convergence is not. For a brief window nodes disagree. Usually harmless for HTTP; for strict consistency requirements, do not assume an instantaneous global switch.
  6. Rollback window closed by garbage collection. If the old ReplicaSet/Deployment is deleted (or scaleDownDelaySeconds elapses) before green is trusted, rollback is no longer O(1) — it becomes a fresh deploy of the old version. Keep blue until green has soaked.

Alternatives and When to Choose Them

  • Rolling Update Strategy — choose when 2× capacity is unaffordable and a mixed-version interval is acceptable. RollingUpdate is continuous and cheap on resources; blue-green is discrete and resource-hungry. The default for most services.
  • Canary Deployment on Kubernetes — choose when you want to limit blast radius by exposing the new version to a small traffic fraction and watching metrics before widening. Blue-green is all-or-nothing; canary is graduated. They are points on the same routing spectrum — a weighted route at 100/0 vs 0/100 is blue-green; intermediate weights are canary.
  • Argo Rollouts blueGreen — choose when you want the pattern automated with preview Service, analysis gates, and a managed scale-down delay, rather than scripting kubectl patch. Strictly better than hand-rolling for any team past the toy stage.
  • Recreate Strategy — superficially similar (full swap) but the opposite trade: Recreate has downtime and no second environment; blue-green has no downtime and two environments. Recreate is for “versions must not overlap”; blue-green is for “I want an instant, tested, reversible cutover.”

Production Notes

  • Netflix popularised blue-green (their “red/black”) via Spinnaker; the historical lesson that carried into Kubernetes practice is that the database, not the application, is the hard part — blue-green only works smoothly when schema changes are expand-and-contract so both colours run against one schema.
  • The 2× cost is real but bounded. The overlap is minutes to hours, not permanent — teams treat it as an acceptable transient, especially with Karpenter/Cluster Autoscaler absorbing the spike and releasing it after scale-down.
  • Preview environments as a feature, not just a step. Many teams keep previewService wired to internal tooling permanently, so “the green environment” doubles as a staging surface that always runs the next release candidate against production-grade infrastructure.
  • Why teams graduate to canary. Blue-green still exposes 100% of users to a bad release the instant the switch flips — the blast radius is the whole user base for however long it takes to notice and roll back. Teams that want a bounded blast radius move to Canary Deployment on Kubernetes with metric-gated analysis, which catches a bad release while only a few percent of users are affected.

See Also

Graph View

Backlinks