vault

Argo Rollouts

11 min read

Argo Rollouts

Argo Rollouts is a Kubernetes controller and a set of CRDs — part of the Argo project family alongside ArgoCD — that provides progressive delivery: advanced, metric-gated deployment strategies that the built-in Deployment cannot do. Its central design move is to replace the Deployment with a Rollout custom resource whose spec is almost identical (same Pod template, same replicas, same selector) but whose spec.strategy offers first-class canary and blueGreen strategies driven by an ordered list of steps (argo-rollouts.readthedocs.io). Steps interleave traffic shifts (setWeight), pauses, and analysis — and analysis is the feature that makes Argo Rollouts more than a fancier rollout: an AnalysisTemplate runs a real metric query (against Prometheus, Datadog, CloudWatch, New Relic, Kayenta, a Kubernetes Job, and more) and, if the metric breaches its threshold, automatically aborts the rollout and rolls back — the thing native Deployments conspicuously do not do on ProgressDeadlineExceeded. Rollouts integrates with traffic-management layers (Istio, Gateway API, NGINX, AWS ALB, SMI) for precise weight control, supports standalone Experiments, and ships a dashboard and a kubectl plugin. Its closest sibling is Flagger (the Flux/CNCF equivalent), which solves the same problem with a different architecture — covered below. This note is the controller; the strategy patterns live in Canary Deployment on Kubernetes and Blue-Green Deployment on Kubernetes.

Mental Model

flowchart TB
    GIT[Git: Rollout manifest<br/>image bumped]
    ARGOCD[ArgoCD / Flux<br/>applies the manifest]
    ROLLOUT["Rollout CR<br/>(replaces Deployment)<br/>strategy.canary.steps[]"]
    CTRL[Argo Rollouts controller]
    RS_STABLE[Stable ReplicaSet]
    RS_CANARY[Canary ReplicaSet]
    STEP["Execute next step:<br/>setWeight / pause / analysis"]
    ANALYSIS["AnalysisRun<br/>query Prometheus / Datadog /<br/>CloudWatch / Kayenta"]
    PROMOTE[Step passed →<br/>advance to next step]
    ABORT[Metric breached →<br/>abort + auto-rollback<br/>weight → 0, scale canary down]

    GIT --> ARGOCD --> ROLLOUT --> CTRL
    CTRL -- "manages" --> RS_STABLE
    CTRL -- "manages" --> RS_CANARY
    CTRL --> STEP
    STEP -- "analysis step" --> ANALYSIS
    ANALYSIS -- "pass" --> PROMOTE
    ANALYSIS -- "fail" --> ABORT
    PROMOTE -. "loop" .-> CTRL

What this diagram shows. A GitOps tool applies a Rollout (not a Deployment); the Argo Rollouts controller owns the stable and canary ReplicaSets and walks the steps list, executing weight shifts and pauses and spawning AnalysisRuns. A passing analysis advances the rollout; a failing one aborts and rolls back automatically. The insight to extract: Argo Rollouts is the reconciliation loop of a Deployment, plus a step program and a metric feedback gate. The Deployment controller’s loop is “make actual match desired”; the Rollout controller’s loop is “make actual match desired, and only advance if the metrics say it’s safe.” That feedback gate is the entire value proposition.

Mechanical Walk-through

The Rollout CRD replaces the Deployment

A Rollout (argoproj.io/v1alpha1) has the same spec.replicas, spec.selector, and spec.template as a Deployment — migration is mostly changing kind: Deployment to kind: Rollout and adding a strategy. The Rollout controller manages ReplicaSets underneath exactly as the Deployment controller does (injecting a rollouts-pod-template-hash label), so the ReplicaSet/Pod layer is unchanged. What differs is the strategy.

A Rollout can also wrap an existing Deployment via workloadRef instead of replacing it — the Rollout references the Deployment and scales it to zero, taking over Pod management. This eases migration but the Rollout is still the resource you operate on.

The canary strategy and steps

The canary strategy (argo-rollouts.readthedocs.io — canary) is an ordered steps list. Each step is one of:

  • setWeight: N — shift N% of traffic to the canary ReplicaSet. With a trafficRouting integration this is a precise weight; without one it is a best-effort replica ratio.
  • pause: {} — pause indefinitely until a human runs kubectl argo rollouts promote. pause: {duration: 10m} — pause for a fixed time, then auto-advance.
  • analysis — spawn one or more AnalysisRuns from named AnalysisTemplates; the rollout advances only if they succeed, and aborts + rolls back if they fail.
  • setCanaryScale — set the canary ReplicaSet’s replica count independently of traffic weight (replicas, weight, or matchTrafficWeight); only meaningful when trafficRouting is configured.
  • experiment — run a transient side-by-side Experiment (see below).

The canary strategy’s own maxSurge / maxUnavailable default to 25% — the same knobs as a Rolling Update Strategy, governing how Pods of the canary ReplicaSet are created.

The blueGreen strategy

The blueGreen strategy (argo-rollouts.readthedocs.io — blue-green) manages two Services:

  • activeService (required) — carries production traffic; its selector is switched to the new ReplicaSet on promotion.
  • previewService (optional) — points at the new ReplicaSet pre-promotion for testing.
  • autoPromotionEnabled — defaults to true; false pauses until a manual promote. autoPromotionSeconds auto-promotes after a delay.
  • scaleDownDelaySeconds — defaults to 30; how long the old ReplicaSet stays running after cutover (IP-table propagation + fast rollback).
  • prePromotionAnalysis / postPromotionAnalysis — analysis gates before and after the switch; a failed post-promotion analysis auto-rolls-back.

See Blue-Green Deployment on Kubernetes for the pattern in depth.

AnalysisTemplate and AnalysisRun

Analysis is the differentiator (argo-rollouts.readthedocs.io — analysis). An AnalysisTemplate is a reusable definition of one or more metrics, each specifying a query against a provider and a success/failure condition. An AnalysisRun is one execution of a template, spawned by the Rollout (as a step, or in the background across the whole canary, or pre/post promotion in blue-green). Supported providers include Prometheus, Datadog, New Relic, Wavefront, Graphite, InfluxDB, CloudWatch, a Kubernetes Job, a generic Web (HTTP) probe, and Kayenta (Netflix’s automated canary analysis service). Each metric has successCondition / failureCondition CEL-like expressions, a count, an interval, and failureLimit / inconclusiveLimit tolerances. If a metric fails enough times, the AnalysisRun fails, and the Rollout aborts and reverts to the stable version — automatic rollback.

Traffic management integrations

To turn setWeight into a real traffic percentage (decoupled from replica count), Rollouts integrates with a router (argo-rollouts.readthedocs.io — traffic management): Istio (VirtualService weights), Gateway API (HTTPRoute backendRefs weights), NGINX ingress (canary annotations), AWS ALB (target-group weights), Apache APISIX, Traefik, and SMI (the Service Mesh Interface TrafficSplit). Without a traffic-management plugin the canary degrades gracefully to replica-ratio splitting.

Experiments

An Experiment is a standalone CRD that runs one or more ReplicaSets for a bounded duration with optional analysis, without progressing a release — used for A/B-style comparisons or load tests of a candidate version. Canary steps can embed experiment steps to spin up an ephemeral baseline-vs-canary comparison mid-rollout.

Dashboard and kubectl plugin

The kubectl argo rollouts plugin adds get rollout, promote, abort, retry, pause, set image, and a live --watch view. A bundled dashboard (kubectl argo rollouts dashboard) renders rollout progress, step position, and analysis results in a browser.

Configuration / API Surface

A canary Rollout with traffic routing and a metric gate:

apiVersion: argoproj.io/v1alpha1
kind: Rollout                          # NOT Deployment — the core substitution
metadata:
  name: shop
spec:
  replicas: 10
  revisionHistoryLimit: 5
  selector:
    matchLabels: { app: shop }
  template:                            # identical to a Deployment Pod template
    metadata: { labels: { app: shop } }
    spec:
      containers:
        - name: shop
          image: registry.example.com/shop:v2.0
  strategy:
    canary:
      canaryService: shop-canary       # Service pointed at the canary ReplicaSet
      stableService: shop-stable       # Service pointed at the stable ReplicaSet
      trafficRouting:
        istio:                         # makes setWeight a PRECISE traffic percentage
          virtualService:
            name: shop-vsvc
            routes: [primary]
      steps:
        - setWeight: 5                 # 5% of traffic to the canary
        - pause: { duration: 10m }     # soak 10 min
        - analysis:                    # query Prometheus; abort + rollback on failure
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: {}                    # pause INDEFINITELY — human runs `promote`
        # implicit final step: setWeight 100
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      count: 5                         # sample 5 times
      successCondition: result[0] >= 0.99   # ≥99% success keeps the rollout going
      failureLimit: 1                  # one breach fails the AnalysisRun → rollback
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{job="shop",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{job="shop"}[2m]))

Line-by-line:

  • kind: Rollout — the one mandatory change from a Deployment manifest.
  • trafficRouting.istio — without this, setWeight: 5 is a best-effort replica ratio; with it, the controller edits the Istio VirtualService so exactly 5% of requests hit the canary.
  • pause: {} vs pause: {duration: 10m} — the empty form blocks for a human promote; the durational form auto-advances. Mixing them gives “auto-ramp to 50%, then require human sign-off for 100%.”
  • AnalysisTemplate.successCondition — a CEL expression over the provider’s result; failureLimit: 1 means a single breach fails the run and triggers automatic rollback.

Failure Modes

  1. Migration friction — everything references Deployment. HPAs, PodDisruptionBudgets, and tooling that target Deployment by kind will not find the Rollout. The HPA must use scaleTargetRef of kind: Rollout; PDBs match by label and still work. Audit every controller pointed at the old Deployment.
  2. trafficRouting omitted — setWeight is a lie. Without a traffic-management plugin, setWeight: 5 cannot produce 5% traffic — it falls back to replica ratio, so 5% needs 1 of 20 Pods and finer values are impossible. Teams expecting precise canary percentages must configure Istio/Gateway API/etc.
  3. Analysis provider unreachable. If Prometheus is down or the query is malformed, the AnalysisRun returns Error/Inconclusive. Depending on inconclusiveLimit the rollout may pause indefinitely waiting for a verdict it will never get. Monitor the monitoring; set sane inconclusiveLimit.
  4. Bad successCondition aborts good releases. A threshold tuned to a high-traffic peak fails during a low-traffic trough (too few samples, noisy ratio). False aborts erode trust in progressive delivery. Tune thresholds and windows to real traffic, and prefer count-based steps on low-volume services.
  5. Abandoned indefinite pause. A pause: {} step with no one to promote leaves the rollout parked forever, the canary stuck at a partial weight. Wire alerting on rollouts in Paused state past an SLA.
  6. scaleDownDelaySeconds too short (blueGreen). Scaling the old ReplicaSet down before kube-proxy converges or before in-flight long connections drain causes resets. Size it to connection lifetime; default 30 is often too low for gRPC/WebSocket workloads.

Alternatives and When to Choose Them

  • Flagger — the closest sibling: a CNCF project in the Flux family that delivers the same canary / blue-green / A/B progressive delivery. The architectural contrast is fundamental (CNCF, 2024, Buoyant): Argo Rollouts replaces the Deployment with a Rollout CRD and the controller drives the rollout directly via explicit steps; Flagger keeps the standard Deployment unchanged and a separate Canary CRD drives a copy of it externally, requiring zero manifest changes to the workload. Choose Flagger when you want progressive delivery without re-architecting manifests and you are already on Flux; choose Argo Rollouts when you want explicit step-based control with manual approval gates and you are already on ArgoCD. The pairing usually follows your existing GitOps tool, not a capability gap.
  • Native Deployment RollingUpdate — choose for low-risk releases that need neither traffic-percentage control nor metric gates. It is simpler and has no extra controller; it just lacks analysis and automatic rollback.
  • Service-mesh weights by hand — choose for an occasional release where editing an Istio VirtualService or Gateway API HTTPRoute manually is acceptable. It does not scale to many services or fast cadence; that is exactly the gap Argo Rollouts / Flagger fill.
  • Knative — choose for request-driven autoscale-to-zero with revision-based traffic splitting. Knative does percentage splitting between revisions but is a different model (serverless), not a general progressive-delivery controller.

Production Notes

  • GitOps pairing. ArgoCD + Argo Rollouts is the canonical Argo-stack combination: ArgoCD syncs the Rollout manifest from Git, Argo Rollouts executes the progressive delivery. ArgoCD even renders Rollout health natively. Flux + Flagger is the symmetric pairing on the other side (CNCF, 2024).
  • The migration is the cost. The single most-cited adoption friction is converting Deployment to Rollout across an estate — every HPA, PDB, and CI script that names the kind must be updated. Teams often start with workloadRef (wrap, don’t replace) to stage the migration, or pick Flagger specifically to avoid it.
  • Analysis tuning dominates the effort. As with Canary Deployment on Kubernetes generally, installing the controller is quick; the real work is authoring AnalysisTemplates with metrics, thresholds, and windows that reliably catch bad releases without false aborts. Most teams iterate on these for weeks.
  • Kayenta for sophisticated analysis. Argo Rollouts can delegate analysis to Kayenta, the automated canary analysis engine extracted from Netflix’s Spinnaker, when statistical comparison of canary-vs-baseline (rather than a single threshold) is wanted — relevant for high-stakes, high-traffic services.

See Also

Graph View

Backlinks