Argo Workflows

Argo Workflows is an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes, implemented as a Kubernetes Custom Resource Definition (CRD) (Argo Workflows docs). Its defining principle is that each step of a workflow is a container (Argo docs): you describe a computation as either a linear sequence of steps or a directed acyclic graph (DAG) capturing dependencies between tasks, and Argo runs every node as its own Kubernetes pod, scheduling them in parallel wherever the dependency graph allows. Because a Workflow is just a CRD object, the whole thing is kubectl apply-able, RBAC-governed, and observable like any other Kubernetes resource. Argo Workflows is part of the broader Argo project, which graduated within the Cloud Native Computing Foundation (CNCF) on 6 December 2022 (CNCF, 2022) — a status it still holds as of 2026-07-25. While frequently used for CI/CD, its container-per-step DAG model makes it equally at home for batch data processing, machine-learning pipelines, and infrastructure automation.

Mental Model — A DAG Where Every Node Is a Pod

The right way to think about Argo Workflows is: a Makefile or Airflow DAG whose every task is a container, executed on Kubernetes. You do not write a monolithic pipeline script; you declare templates (reusable units of work, like functions) and template invocators (control-flow that calls those functions in a steps sequence or a dag graph), name an entrypoint template as main, and let the workflow controller drive the graph to completion by creating and watching pods.

flowchart TD
    subgraph WF["Workflow (CRD) — the 'what' + live state"]
        ENTRY["entrypoint: main"]
        subgraph TPLS["templates ('functions')"]
            direction LR
            T1["container / script<br/>(do work)"]
            T2["dag / steps<br/>(invoke others)"]
            T3["resource / suspend<br/>(k8s op / pause)"]
        end
    end
    ENTRY --> T2
    T2 -->|"each task/step"| POD["Kubernetes Pod<br/>main + init + wait containers"]
    CTRL["workflow-controller<br/>watches Workflow + Pods"] -.->|"reconciles → creates"| POD
    POD -.->|"status"| CTRL
    CTRL -.->|"updates status subresource"| WF

What it shows and the insight to take: the Workflow object is dual-purpose — it both “defines the workflow to be executed” and “stores the state of the workflow” as it runs (Argo, Workflow Concepts). Templates split into two families: work executors (container, script, resource, suspend, http, plugin, containerSet) that do something, and template invocators (steps, dag) that orchestrate other templates. The workflow-controller is the reconciler: it watches Workflow objects and their pods via Kubernetes Informers, creates a pod per graph node, and writes progress back into the Workflow’s status. The insight: there is no separate scheduler or state database — the CRD is the state, and the controller is the engine.

The CRD Family — Workflow, WorkflowTemplate, CronWorkflow

Argo Workflows ships several CRDs that together give you run-once, reusable-library, and scheduled execution:

erDiagram
    WORKFLOW ||--|{ TEMPLATE : "contains"
    TEMPLATE ||--o| CONTAINER : "work executor"
    TEMPLATE ||--o| DAG : "invocator"
    DAG ||--o{ TASK : "nodes (dependencies)"
    WORKFLOW ||--o| WORKFLOWTEMPLATE : "workflowTemplateRef"
    WORKFLOWTEMPLATE ||--|{ TEMPLATE : "stores reusable"
    CLUSTERWORKFLOWTEMPLATE ||--|{ TEMPLATE : "cluster-scoped reusable"
    CRONWORKFLOW ||--|| WORKFLOW : "spawns on schedule"

Workflow — a single runnable job instance submitted to the cluster; it holds both the definition and the live state (Argo docs).
WorkflowTemplate — “definitions of Workflows that live in your cluster,” letting you “create a library of frequently-used templates and reuse them” (Argo, Workflow Templates). Namespace-scoped.
ClusterWorkflowTemplate — the same idea but cluster-scoped, so templates are shareable across all namespaces.
CronWorkflow — wraps a Workflow spec with a cron schedule, spawning a new Workflow on each tick (Argo’s built-in “run this pipeline twice a day”).

There are two distinct ways to reference reusable definitions, and they are easy to confuse (Argo, Workflow Templates):

templateRef — used inside a steps or dag node to call one specific template out of a WorkflowTemplate. Fine-grained: “run the print-message template from workflow-template-1.”
workflowTemplateRef — used at the top of a Workflow spec (available since v2.9) to instantiate an entire WorkflowTemplate as a runnable Workflow, merging arguments and optionally overriding the entrypoint.

Mechanical Walk-through — How a Step Runs as a Pod

The most important internal mechanism is what a workflow pod actually contains, because Argo — like any Kubernetes-native engine — has to solve “get inputs in, run the user’s container, get outputs out” using containers alone.

sequenceDiagram
    participant U as User / Event
    participant API as Kube API server
    participant C as workflow-controller
    participant P as Workflow Pod

    U->>API: apply Workflow (or CronWorkflow tick)
    API->>C: informer event (new Workflow)
    C->>C: evaluate DAG — which nodes are ready?
    C->>API: create Pod per ready node
    Note over P: Pod = init + main + wait containers
    P->>P: init container: pull input artifacts/params
    P->>P: main container: user image, argoexec runs the command
    P->>P: wait container: capture outputs/artifacts, save results
    P->>C: pod phase → Succeeded/Failed
    C->>C: mark node done, unlock dependents
    C->>API: update Workflow status; repeat until graph complete

What it shows and the insight to take: each workflow step or DAG task “generates a Pod containing three containers” (Argo, Architecture):

init container — fetches input artifacts and parameters and stages them for the main container.
main container — runs the user’s image; the argoexec utility is mounted in and executes the configured command as a subprocess so Argo can observe it.
wait container — handles cleanup: saving output parameters and artifacts after the main container exits.

The controller “only ever processes a single Workflow at a time” through worker goroutines fed by an informer-backed queue (Argo, Architecture) — the standard controller pattern. It repeatedly evaluates the DAG: nodes whose dependencies are all Succeeded become “ready,” get a pod, and on completion unlock their dependents, until the whole graph terminates. The insight: parallelism is emergent from the DAG — Argo launches every currently-unblocked node’s pod at once, so a wide graph runs wide automatically.

Executor evolution

Older Argo versions offered several workflow executors (docker, kubelet, k8sapi, pns); as of recent releases the emissary executor is the default and the legacy ones were removed. There is also an optional “init-less” layout using a supervisor container in place of the init/wait pair, with artifact handling as regular sidecars (Argo, Architecture). The three-container (init/main/wait) description above is the classic model.

Configuration — A DAG Workflow

apiVersion: argoproj.io/v1alpha1     # note: still v1alpha1 despite graduation/maturity
kind: Workflow
metadata:
  generateName: ci-dag-              # a generated, unique run name
spec:
  entrypoint: main                   # which template is the 'main' function
  templates:
    - name: main
      dag:                           # a template invocator: the DAG
        tasks:
          - name: build
            template: run-container
            arguments:
              parameters: [{name: cmd, value: "make build"}]
          - name: test
            template: run-container
            dependencies: [build]    # DAG edge: test waits for build
            arguments:
              parameters: [{name: cmd, value: "make test"}]
          - name: lint
            template: run-container   # no dependency → runs in PARALLEL with build
            arguments:
              parameters: [{name: cmd, value: "make lint"}]
    - name: run-container            # a work-executor template ('function')
      inputs:
        parameters: [{name: cmd}]
      container:                     # container template type
        image: golang:1.24
        command: ["sh", "-c"]
        args: ["{{inputs.parameters.cmd}}"]   # {{...}} is Argo's substitution syntax

Line-by-line: entrypoint: main names the starting template. The main template is a dag invocator whose tasks list forms the graph — test declares dependencies: [build], so it waits, while lint has no dependency and therefore runs in parallel with build. Each task calls the reusable run-container container template, passing a cmd parameter. Argo’s {{inputs.parameters.cmd}} moustache syntax (distinct from Tekton’s $(params.cmd)) is substituted at run time. Swap dag: for steps: and you get sequential outer-list / parallel inner-list ordering instead of an explicit graph (Argo, Workflow Concepts).

Passing outputs between steps

A container/script template’s output is exported to Argo variables such as {{tasks.<NAME>.outputs.result}} (Argo, Workflow Concepts) — a downstream task consumes an upstream task’s result the same way it declares a dependency, and Argo threads the value through automatically. Larger data moves as artifacts (files staged by the init/wait containers to/from S3/GCS/etc.), not as parameters.

The Argo Project Family

Argo Workflows is one of four tools under the Argo project umbrella (argoproj.github.io):

flowchart LR
    subgraph ARGO["Argo Project (CNCF graduated, 2022-12-06)"]
        WF["Argo Workflows<br/>DAG/step job engine"]
        CD["Argo CD<br/>GitOps continuous delivery"]
        RO["Argo Rollouts<br/>canary / blue-green"]
        EV["Argo Events<br/>event-driven triggers"]
    end
    EV -->|"event → submit"| WF
    WF -->|"build artifact"| CD
    CD -->|"deploy via"| RO

What it shows and the insight to take: the four projects compose but are independent. ArgoCD is declarative GitOps continuous delivery (reconcile cluster to Git); Argo Rollouts provides advanced deployment strategies “such as Canary and Blue-Green made easy”; Argo Events is “event based dependency management for Kubernetes” that can submit Workflows on external triggers (argoproj.github.io). The four graduated together as the single “Argo” CNCF project on 6 December 2022 (CNCF, 2022; CNCF Argo project page). The insight: “Argo Workflows” (the job engine, this note) is not the same as “Argo CD” (the GitOps deployer) — they are siblings often confused because they share the name and the DAG heritage, but they solve different problems.

Failure Modes and Gotchas

Argo Workflows ≠ Argo CD. The most common conceptual error. Workflows runs jobs/pipelines to completion; Argo CD continuously reconciles deployed state from Git. A “workflow” is a finite computation; an “application” in Argo CD is a long-lived desired state. Use Workflows for CI/build/batch, Argo CD for CD/deploy. See ArgoCD.
Pod-per-step cost. Like Tekton, every node is a pod with init/main/wait containers, so a fine-grained graph pays scheduling and image-pull latency per node. Wide DAGs also hit parallelism limits (a controller and per-workflow cap) — unbounded fan-out can overwhelm the cluster or the API server.
Artifacts vs parameters confusion. parameters are small strings substituted into templates; artifacts are files staged through object storage. Trying to pass a large file as a result/parameter fails — Argo caps output parameter/result size. Bulk data must be an artifact with a configured repository (S3/GCS/MinIO).
generateName and orphaned Workflows. Submitting with generateName creates a new object every run; without a Workflow TTL / garbage-collection policy, completed Workflow objects (and their pods) accumulate in etcd and can bloat the cluster.
Controller is a singleton bottleneck. Because the controller processes one Workflow at a time per worker and holds the whole graph in memory, very large workflows (tens of thousands of nodes) can pressure controller memory and reconcile latency.

Alternatives and When to Choose Them

Dimension	Argo Workflows	Tekton	Apache Airflow	GitHub Actions
Substrate	Kubernetes pods (CRD)	Kubernetes pods (CRD)	Workers (K8s optional)	Vendor runners
Core abstraction	DAG/steps of containers	Task/Pipeline building blocks	Python-defined DAG of operators	YAML workflows/jobs
Sweet spot	Batch, ML, data pipelines, CI on K8s	Reusable CI/CD Task catalog for platform teams	Scheduled data-engineering DAGs	Turnkey repo-integrated CI
Definition language	YAML CRD (`{{ }}` templating)	YAML CRD (`$( )` templating)	Python	YAML
Reuse model	WorkflowTemplate / ClusterWorkflowTemplate	Catalog Tasks (Artifact Hub)	Python modules / plugins	Reusable workflows / Marketplace
CNCF status (2026-07-25)	Graduated	Incubating	Apache Software Foundation	vendor product

When to choose Argo Workflows: you have DAG-shaped batch work — CI build/test fan-out, ML training/inference pipelines, ETL/data processing — that you want to run natively on Kubernetes with per-step containers and automatic parallelism. Its container-per-step model and artifact passing make it the de-facto Kubernetes batch/ML orchestrator (it underpins Kubeflow Pipelines). When not to: for turnkey repo-integrated CI where you just want a green PR check, GitHub Actions/GitLab CI CD are simpler; for a reusable-Task-catalog CI/CD framework that a platform team curates, Tekton is the closer philosophical match; for Python-authored scheduled data DAGs with a rich operator ecosystem off Kubernetes, Airflow fits better. See CI CD Platform Models Compared and Pipeline DAGs Stages and Gates.

Production Notes

Argo Workflows is heavily used for machine-learning and data pipelines — it is the execution engine beneath Kubeflow Pipelines — and for CI at organizations that already standardize on Kubernetes; maintainers and adopters listed by the project include Akuity, BlackRock, Intuit, Octopus Deploy, and Red Hat (argoproj.github.io). The Argo project’s CNCF graduation (Dec 2022) was underpinned by a graduation-level security audit and broad production adoption (Red Hat, 2022). The canonical delivery architecture pairs Argo Workflows / Tekton for CI (build, test, produce an immutable artifact) with ArgoCD for CD (GitOps reconcile that artifact into the cluster) — Workflows produces, Argo CD deploys, and Argo Rollouts governs the rollout strategy. The security controls around what those pipelines build (scanning, signing, SBOMs) are owned by DevSecOps and Supply Chain Security MOC; the deployment strategy Rollouts implements is owned by Site Reliability Engineering MOC — cross-linked, not re-taught here.

vault

Explorer

Argo Workflows

Argo Workflows

Mental Model — A DAG Where Every Node Is a Pod

The CRD Family — Workflow, WorkflowTemplate, CronWorkflow

Mechanical Walk-through — How a Step Runs as a Pod

Configuration — A DAG Workflow

Passing outputs between steps

The Argo Project Family

Failure Modes and Gotchas

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents

Backlinks