vault

Canary Deployment on Kubernetes

10 min read

Canary Deployment on Kubernetes

A canary deployment releases a new version to a small, controlled fraction of traffic or users first, observes its behaviour under real production load, and only then progressively widens the fraction until the new version carries everything (the name borrows the “canary in a coal mine” — a small early-warning sample). Its defining virtue over Blue-Green Deployment on Kubernetes is a bounded blast radius: a bad release degrades 5% of users, not 100%, and is caught while the damage is small. On Kubernetes there is a crude path and a real path. The crude path runs a small-replica canary Deployment behind the same Service as the stable Deployment, so traffic splits roughly in proportion to replica counts (kubernetes.io — canary). The real path uses weighted L7 routing — an Istio VirtualService, a Gateway API HTTPRoute, or a canary-aware Ingress Controller — to send a precise percentage to the canary independent of how many replicas it runs. The most mature form is automated, metric-gated canary via Argo Rollouts or Flagger, which define rollout steps interleaved with metric analysis and roll back automatically when a metric breaches its threshold. This note covers the spectrum; the siblings are Blue-Green Deployment on Kubernetes (atomic, all-or-nothing) and Rolling Update Strategy (gradual but not traffic-controlled and not metric-gated).

Mental Model

flowchart TB
    CLIENT[Incoming traffic]
    ROUTER["Weighted router<br/>Istio VirtualService /<br/>Gateway API HTTPRoute /<br/>Ingress canary annotations"]
    STABLE["Stable ReplicaSet<br/>v1.4 — weight 95%"]
    CANARY["Canary ReplicaSet<br/>v1.5 — weight 5%"]
    ANALYSIS{"Metric analysis<br/>success rate / latency / errors<br/>(Prometheus, Datadog, …)"}
    WIDEN["Promote: shift weight<br/>5% → 25% → 50% → 100%"]
    ABORT["Abort: weight → 0%<br/>scale canary down<br/>(automatic rollback)"]

    CLIENT --> ROUTER
    ROUTER -- "95%" --> STABLE
    ROUTER -- "5%" --> CANARY
    CANARY --> ANALYSIS
    ANALYSIS -- "metrics healthy" --> WIDEN
    ANALYSIS -- "metrics breached" --> ABORT
    WIDEN -. "next step" .-> ROUTER

What this diagram shows. A weighted router splits traffic between a stable and a canary ReplicaSet. The canary’s real-world metrics feed an analysis step; healthy metrics advance the rollout to a larger weight, breached metrics abort it and roll back — all without exposing more than the current fraction of users to risk. The insight to extract: canary is blue-green’s routing machinery (two versions, a router) plus a dimmer instead of a switch plus a feedback loop. The dimmer bounds blast radius; the feedback loop (metric analysis) is what makes “progressive delivery” progressive — the decision to widen is data-driven, not a timer.

Mechanical Walk-through

The crude path — replica-ratio splitting

The simplest canary needs no mesh and no extra tooling (kubernetes.io — canary). Run two Deployments sharing one set of selector labels:

  • app-stable — v1.4, replicas: 9, Pods labelled app=shop, track=stable.
  • app-canary — v1.5, replicas: 1, Pods labelled app=shop, track=canary.

A Service with selector: {app: shop} (note: not including track) selects all ten Pods. Since kube-proxy load-balances roughly evenly across endpoints, ~10% of traffic lands on the single canary Pod. To widen, scale the canary up and the stable down.

This is genuinely useful and zero-dependency, but its limits are real: the split granularity is 1/total replicas (you cannot do 2% with 10 Pods), the split is coupled to capacity (more canary traffic forces more canary Pods even if the canary doesn’t need them), and there is no metric gate — a human watches dashboards and scales by hand.

The real path — weighted L7 routing

Decoupling traffic weight from replica count requires a Layer-7 router that can split by percentage:

  • Istio VirtualService. Two Kubernetes Services (shop-stable, shop-canary); the VirtualService’s http.route lists both as destinations with explicit weight values that must sum to 100. Shift the canary’s weight from 0 → 5 → 25 → 100 by editing the VirtualService (istio.io — VirtualService). Istio also supports header-based canary (route requests with x-canary: true to the canary regardless of weight) for targeting internal users first.
  • Gateway API HTTPRoute. A single HTTPRoute rule lists two backendRefs, each with a weight. Weights are relative, not required to sum to 100 — weight: 95 and weight: 5 splits ~95/5; weight: 9 and weight: 1 splits the same (Gateway API — HTTPRoute). This is the vendor-neutral successor to Ingress and is increasingly the recommended substrate for canary (oneuptime, 2026).
  • Ingress Controller canary annotations. The NGINX ingress controller accepts nginx.ingress.kubernetes.io/canary: "true" plus canary-weight, canary-by-header, or canary-by-cookie annotations on a second Ingress object pointing at the canary Service. A lighter-weight option when a full mesh is unwanted.

The mature path — automated, metric-gated canary

Manually editing weights and watching dashboards does not scale. Argo Rollouts and Flagger automate the loop.

Argo Rollouts replaces the Deployment with a Rollout CRD whose canary strategy defines steps — an ordered list interleaving traffic shifts and pauses (argo-rollouts.readthedocs.io — canary):

  • setWeight: N — shift N% of traffic to the canary.
  • pause: {} — pause indefinitely until a human promote; pause: {duration: 10m} — pause for a fixed time then auto-advance.
  • analysis — run an AnalysisTemplate (a Prometheus/Datadog/etc. query) as a step or in the background; if it fails, the Rollout aborts and rolls back automatically.
  • setCanaryScale — control canary replica count independently of weight (only meaningful with trafficRouting configured).

A typical steps block: setWeight 10 → pause 5m → analysis (success rate ≥ 99%) → setWeight 30 → analysis → setWeight 60 → analysis → (implicit setWeight 100). At any analysis failure the controller drives the weight back to 0 and scales the canary down. Without trafficRouting, the Rollout makes a best-effort replica-ratio split (the crude path, automated); with trafficRouting (Istio, Gateway API, NGINX, ALB, SMI) it sets precise weights.

Flagger does the same job with a different architecture: it keeps the standard Deployment and drives a copy of it externally via its own Canary CRD — see Argo Rollouts for the full Flagger contrast.

Configuration / API Surface

A Gateway API HTTPRoute doing a 95/5 weighted split:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs:
    - name: prod-gateway
  rules:
    - backendRefs:
        - name: shop-stable        # stable Service
          port: 80
          weight: 95               # relative weight — ~95% of traffic
        - name: shop-canary        # canary Service
          port: 80
          weight: 5                # relative weight — ~5% of traffic

An Argo Rollouts canary strategy with metric-gated steps:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: shop-canary   # Service the controller points at the canary ReplicaSet
      stableService: shop-stable   # Service pointed at the stable ReplicaSet
      trafficRouting:
        gatewayAPI:                # decouple weight from replica count via Gateway API
          httpRoute: shop
          namespace: prod
      steps:
        - setWeight: 10            # 10% to canary
        - pause: { duration: 5m }  # soak 5 min
        - analysis:                # query Prometheus; abort + rollback if it fails
            templates:
              - templateName: success-rate
        - setWeight: 30
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 60
        - pause: { duration: 5m }
        # implicit final step: setWeight 100, canary becomes the new stable
  selector:
    matchLabels: { app: shop }
  template:
    metadata: { labels: { app: shop } }
    spec:
      containers:
        - name: shop
          image: registry.example.com/shop:v1.5

Line-by-line:

  • canaryService / stableService are two Services whose selectors the controller manages (it injects the pod-template-hash) so the router can address each ReplicaSet distinctly.
  • trafficRouting.gatewayAPI is what makes the setWeight values precise — without a trafficRouting block, setWeight: 10 becomes a best-effort replica ratio (~1 of 10 Pods), not a true 10% traffic weight.
  • analysis steps reference an AnalysisTemplate (see Argo Rollouts); a failed analysis at any step aborts the rollout and shifts weight back to 0 — automatic rollback.

Failure Modes

  1. Crude path: granularity and capacity coupling. Replica-ratio splitting cannot express weights finer than 1/replicas, and forcing 50% canary traffic forces 50% canary replicas whether or not the canary needs that capacity. For precise or low single-digit percentages you need L7 weighted routing.
  2. Metrics too noisy or window too short. A canary step that analyses for 2 minutes on a low-traffic service gathers too few requests for a statistically meaningful success-rate signal — the rollout either promotes a bad release (false negative) or aborts a good one (false positive). Size the soak window to traffic volume; consider request-count-based rather than time-based steps.
  3. Canary metrics polluted by warm-up. Fresh canary Pods have cold caches and empty connection pools; their first-minute latency/error metrics look bad even for a healthy release. Without a warm-up exclusion the analysis aborts a fine canary. Add an initial pause before the first analysis, or scope queries to exclude startup.
  4. Session affinity defeats the split. If the router pins a client to one backend (sessionAffinity: ClientIP, sticky cookies), a “5% canary” can mean 5% of clients see only the canary for the whole session — fine for canarying, but surprising if you expected 5% of every client’s requests. Know which one your router does.
  5. Shared database / stateful coupling. As with Blue-Green Deployment on Kubernetes, canary and stable share a database; the canary’s schema must be backward-compatible with stable for the whole rollout. Canary does not solve migrations.
  6. Wrong metric gated. Gating on HTTP 5xx rate alone misses a release that returns HTTP 200 with wrong data, or one that degrades a downstream dependency. Choose analysis metrics that actually capture “is this release good” — success rate, latency percentiles, and business KPIs, not just error codes.

Alternatives and When to Choose Them

  • Blue-Green Deployment on Kubernetes — choose when you want a single atomic cutover with pre-cutover testing and instant rollback, and don’t need a graduated blast radius. Canary trades blue-green’s simplicity for a bounded-risk ramp.
  • Rolling Update Strategy — choose for ordinary low-risk releases where you don’t need traffic-percentage control or metric gates. RollingUpdate replaces Pods gradually but exposes the new version to a random replica-proportional traffic share with no analysis loop — it is a canary only in the crudest sense.
  • Argo Rollouts / Flagger — choose when you want the canary automated and metric-gated with automatic rollback. Hand-editing Istio/Gateway weights is fine for an occasional release; it does not scale to many services or fast cadence.
  • A/B testing / feature flags — choose when the goal is experimentation on user segments (does variant B convert better) rather than safe rollout. Canary asks “is the new version healthy”; A/B asks “is the new version better.” They are often layered: ship via canary, expose features via flags.

Production Notes

  • Industry origin. Canary as a named practice was popularised by Netflix, Google, and Facebook; the Kubernetes ecosystem standardised it via Argo Rollouts and Flagger. The CNCF’s own guidance treats progressive delivery — canary with automated analysis — as the mature state, with plain RollingUpdate as the baseline (CNCF, 2024).
  • Start with header-based canary for internal users. Many teams route x-internal: true (or an employee identity header) to the canary at 100% before opening any percentage to the public — staff dogfood the release with zero public blast radius, then the weighted ramp begins.
  • Tooling pairs with the GitOps stack. Teams on ArgoCD reach for Argo Rollouts; teams on Flux reach for Flagger — the pairing follows the existing CD tool, not a technical superiority of one canary engine (CNCF, 2024).
  • The analysis is the hard part, not the routing. Splitting traffic is mechanically easy once a mesh or Gateway API is in place; the engineering effort is in choosing metrics, queries, thresholds, and windows that reliably distinguish a good release from a bad one without false aborts. Budget for tuning AnalysisTemplates, not just for installing the controller.

See Also

Graph View

Backlinks