vault

ArgoCD

12 min read

ArgoCD

Argo CD is a declarative, pull-based GitOps continuous-delivery tool for Kubernetes: it runs inside the cluster, treats a Git repository as the single source of truth for application manifests, and continuously reconciles the live cluster state toward what Git declares. It is one of the four sub-projects of Argo (alongside Argo Workflows, Argo Rollouts, and Argo Events); the umbrella Argo project was accepted to the Cloud Native Computing Foundation (CNCF) as Incubating in March 2020 and graduated on 6 December 2022 (CNCF announcement). Argo CD’s defining abstractions are the Application Custom Resource (a {Git repo + path + target cluster/namespace + sync policy} bundle) and ApplicationSet (a templated generator that produces many Applications). Its most-cited differentiator versus the sibling tool Flux is a polished web UI that renders the desired-vs-live diff and the resource health tree visually. This note covers Argo CD’s component architecture, the reconcile loop, the sync-wave / sync-phase model, the app-of-apps pattern, and multi-cluster operation. The general GitOps philosophy lives in GitOps; the head-to-head comparison lives in ArgoCD vs Flux.

Mental Model

Argo CD is an instance of the Kubernetes Control Loop Pattern where the desired state lives in Git rather than in an in-cluster spec. The application-controller watches Application CRDs, asks the repo-server to render the manifests that the referenced Git revision + path imply, diffs that rendered set against what is actually running in the target cluster, and reports two orthogonal verdicts: a sync status (Synced / OutOfSync — does the cluster match Git?) and a health status (Healthy / Progressing / Degraded / Missing / Suspended — is the running thing actually working?). It then either waits (manual sync policy) or acts (automated sync policy) to converge.

flowchart TB
    Dev[Developer] -->|git push manifests| Git[(Git Repository<br/>Helm / Kustomize / plain YAML)]
    subgraph ArgoCD[Argo CD - runs in-cluster]
        API[argocd-server<br/>gRPC/REST API + Web UI + RBAC/SSO]
        Repo[repo-server<br/>clones Git, renders manifests<br/>helm template / kustomize build]
        Ctrl[application-controller<br/>diff desired vs live, sync, hooks]
        Redis[(Redis<br/>manifest + diff cache)]
        API --- Redis
        Ctrl --- Redis
        Ctrl -->|render request| Repo
        Repo -->|fetch revision| Git
    end
    Ctrl -->|GET live state| TargetA[(Target Cluster A)]
    Ctrl -->|GET live state| TargetB[(Target Cluster B)]
    Ctrl -->|apply on sync| TargetA
    Ctrl -->|apply on sync| TargetB
    User[Operator / Platform team] -->|kubectl apply Application CRD<br/>or click Sync in UI| API

What this diagram shows. Git is the source of truth; nobody runs kubectl apply against the cluster directly in steady state. The repo-server is the pure manifest-rendering engine (it knows Helm, Kustomize, Jsonnet, plain YAML, and config-management plugins). The application-controller is the reconciler — it never trusts a cached diff blindly; on every wake-up it re-reads live cluster state. The argocd-server is the human/automation front door (UI, CLI, CI webhooks) and the home of RBAC, SSO, and AppProject tenancy. Redis is a non-durable cache — losing it costs a re-render, not data. The insight to extract: Argo CD can manage many target clusters from one control plane, because the controller holds credentials (as Kubernetes Secrets) for each registered cluster and the diff/apply is just an API call to whichever cluster the Application names.

Mechanical Walk-through

A new Application CRD is created (by kubectl apply, the CLI, the UI, or — recursively — by an app-of-apps parent). The application-controller picks it up via its informer watch. It extracts spec.source (a Git repoURL, a targetRevision such as HEAD or v2.3.1, and a path inside the repo) and asks the repo-server to render that path. The repo-server maintains local clones, detects the tooling (a Chart.yaml → Helm, a kustomization.yaml → Kustomize, otherwise plain YAML), runs the renderer, and returns a flat list of fully-rendered Kubernetes manifests — the desired state.

The controller then fetches the live state: every resource in the target cluster/namespace that Argo CD tracks (tracked via the app.kubernetes.io/instance label or an annotation-based tracking method). It computes a structured diff. If desired ≠ live, the Application is OutOfSync. Independently, the controller runs resource health assessment: for built-in kinds it has health Lua scripts (a Deployment is Healthy only when its updated replicas are available; a Pod is Healthy when Running/Succeeded; a Service of type LoadBalancer is Progressing until it gets an external IP), and custom resources can ship their own health checks.

What happens next depends on spec.syncPolicy:

  • Manual — Argo CD reports OutOfSync and waits. A human clicks Sync in the UI or runs argocd app sync.
  • Automated — the controller syncs on its own. automated.prune: true deletes resources removed from Git; automated.selfHeal: true reverts manual kubectl edit drift (it re-applies Git’s version when someone tampers with the live object).

A sync operation applies the rendered manifests to the cluster, ordered by sync phases and sync waves (below). The default reconcile interval is 3 minutes (timeout.reconciliation), but Git webhooks make syncs near-immediate.

Sync Phases and Sync Waves

Argo CD orders a sync along two axes (Sync Phases and Waves docs):

Sync phases are coarse stages driven by resource hooks (an annotation argocd.argoproj.io/hook). The order is always PreSync → Sync → PostSync, with SyncFail on failure. PreSync is where database migrations and backup jobs go (run-to-completion Jobs that must finish before app manifests apply); Sync is the main phase carrying the actual workload manifests; PostSync is for smoke tests and notifications that run only after Sync resources report Healthy.

Sync waves are fine-grained ordering within a phase, driven by the integer annotation argocd.argoproj.io/sync-wave (default 0, may be negative). Argo CD applies the lowest wave first, waits for those resources to be Healthy, then proceeds. A -1 wave runs before a 0 wave. The inter-wave delay defaults to 2 seconds (ARGOCD_SYNC_WAVE_DELAY, per the Sync Phases and Waves docs). Typical use: wave -1 for CRDs and namespaces, wave 0 for the database StatefulSet, wave 1 for the application Deployment.

A critical scope rule: sync waves operate only within a single Application. The Argo CD docs describe the algorithm as ordering “all resources according to their wave (lowest to highest)” within one Application’s rendered manifest set, and a long-standing community guide states the limit explicitly — “Sync waves only work for individual resources within an Application. They do not work across different Argo CD applications” (Octopus / Codefresh sync-waves guide). Two consequences follow. First, an ApplicationSet produces a fleet of independent Application resources that the application-controller reconciles concurrently — there is no implicit wave ordering across them. Second, the conventional way to order whole applications is the app-of-apps pattern below: when the children are themselves Application CRDs sitting in a parent Application’s manifest set, putting argocd.argoproj.io/sync-wave on each child is a wave-ordering of resources within the parent, which happens to mean “create child Application X before child Application Y.” Wave ordering of the children’s own contents is a separate per-child concern.

For ApplicationSet specifically, the controller-level mechanism for staged cross-Application rollout is Progressive Syncs — a beta feature since Argo CD v3.3.0, enabled by --enable-progressive-syncs (or the ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS env var, or applicationsetcontroller.enable.progressive.syncs: "true" in argocd-cmd-params-cm). Progressive Syncs let an ApplicationSet declare a strategy.type: RollingSync with ordered steps (each step matches a subset of generated Applications by labels and a maxUpdate percentage), and the controller waits for each step’s Applications to report Healthy before advancing — the canonical rollout being “dev first, then QA, then prod in 10% batches” (ApplicationSet Progressive Syncs docs). Progressive Syncs and sync waves are complementary, not duplicative: Progressive Syncs sequences the Applications, sync waves sequence the resources inside each Application.

App-of-Apps Pattern

The app-of-apps pattern bootstraps a cluster from one root Application. The root’s Git path contains not workloads but a set of child Application CRDs. Argo CD syncs the root, which creates the children, each of which Argo CD then reconciles in turn (cluster bootstrapping docs). This gives a single declarative entry point for an entire cluster’s footprint and lets you sync-wave-order whole applications.

ApplicationSet

ApplicationSet is a separate controller and CRD that templates Applications from a generator. Generators include: List (a static list of parameter sets), Cluster (one Application per registered cluster — deploy the same app fleet-wide), Git directory/file (one Application per directory or per config file in a repo — the monorepo pattern), Pull Request (one ephemeral Application per open PR — preview environments), Matrix and Merge (compose generators). It replaces hand-maintained app-of-apps sprawl with a single templated object. Optional Progressive Syncs (beta since Argo CD v3.3.0, opt-in via the controller flag) layers a staged-rollout strategy on top, so the generated Applications are not all flipped in lockstep — see the Sync Phases and Sync Waves section above for the relationship between Progressive Syncs (ordering Applications) and sync waves (ordering resources inside an Application).

Configuration / API Surface

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd                       # Applications live in the Argo CD namespace
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade-delete managed resources on Application delete
spec:
  project: payments-team                  # AppProject — the multi-tenancy boundary (RBAC, allowed repos/clusters)
  source:
    repoURL: https://github.com/acme/manifests.git
    targetRevision: main                  # branch, tag, or commit SHA — the pinned desired revision
    path: apps/payments-api/overlays/prod # path inside the repo to render
    # helm: { valueFiles: [values-prod.yaml] }   # tooling-specific knobs go here
  destination:
    server: https://kubernetes.default.svc # the target cluster API (in-cluster shorthand);
                                           # a remote cluster URL here = multi-cluster delivery
    namespace: payments
  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true     # revert out-of-band kubectl drift back to Git's version
    syncOptions:
      - CreateNamespace=true        # create the destination namespace if absent
      - ApplyOutOfSyncOnly=true     # apply only the resources that actually drifted
    retry:
      limit: 5
      backoff: { duration: 5s, factor: 2, maxDuration: 3m }

Line-by-line: the finalizer makes deleting the Application cascade-delete everything it manages — without it, deletion orphans the workloads. spec.project ties the app to an AppProject, the tenancy object that whitelists which Git repos, destination clusters/namespaces, and resource kinds this app may touch — the backbone of Argo CD multi-tenancy. targetRevision is the pin: tracking main means “always latest commit”; pinning a tag or SHA means “never move without a Git change.” destination.server pointing at a remote cluster URL is all it takes to deliver to another cluster. automated.selfHeal is the GitOps enforcement teeth — manual kubectl edit is reverted within a reconcile cycle.

A hook resource — e.g. a PreSync migration Job — is just an ordinary manifest with:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded   # clean up the Job after it passes
    argocd.argoproj.io/sync-wave: "-1"

Failure Modes

  1. OutOfSync that never converges — usually a rendering nondeterminism: a Helm chart that injects timestamps, random suffixes, or Job names that change every render. The diff is permanently non-empty. Diagnose with argocd app diff; fix by making rendering deterministic or excluding the noisy field via ignoreDifferences.
  2. selfHeal fighting another controller — if a mutating admission webhook or another operator legitimately modifies a field Argo CD manages, selfHeal reverts it, the other controller re-applies, and the two oscillate. Fix with ignoreDifferences on the contested field.
  3. Sync-wave stall — wave N+1 never starts because a wave-N resource never reaches Healthy (e.g. a Deployment with an unschedulable Pod). The whole sync hangs. Read the health of the blocking resource in the UI’s resource tree.
  4. repo-server OOM / timeout — rendering a huge monorepo or a Helm chart with thousands of objects exhausts the repo-server. Symptom: ComparisonError / render timeouts. Scale repo-server replicas, raise memory limits, or split the repo.
  5. Stale cluster cache — the application-controller maintains a per-cluster informer cache; under heavy churn it can lag, showing brief false OutOfSync. Usually self-corrects.
  6. Drift outside Argo CD’s tracking — resources created without the tracking label aren’t seen; prune won’t remove them and the diff won’t show them. The “Argo CD doesn’t know about it” blind spot.

Alternatives and When to Choose Them

  • Flux — the other CNCF-graduated GitOps tool. A set of composable controllers with no built-in UI, native image-update automation, lighter footprint. Choose Flux for minimal Kubernetes-native building blocks; choose Argo CD for the UI and the managed-Application model. The full treatment is ArgoCD vs Flux.
  • Plain CI pushing kubectl apply — push-based delivery from a CI runner. Simpler to start, but the CI system needs cluster credentials, there is no continuous drift correction, and there is no in-cluster source-of-truth reconciler. GitOps tools exist precisely to fix this — see GitOps.
  • Helm / Kustomize alone — these render manifests but do not reconcile or deliver; Argo CD wraps them. Not competitors — Argo CD’s repo-server uses them.
  • Argo Rollouts — a sibling Argo project for progressive delivery (canary, blue-green with analysis). Complementary: Argo CD delivers the manifests, Argo Rollouts governs how a new version takes traffic.

Production Notes

  • Argo CD’s web UI is its most-cited operational advantage: the resource tree, the live-vs-desired diff renderer, and the per-resource health/events view make “why is this app degraded” answerable without kubectl. Flux deliberately has none of this (see ArgoCD vs Flux).
  • The app-of-apps and ApplicationSet patterns are how large platform teams manage hundreds of apps across dozens of clusters from one Argo CD; the Cluster generator deploying a baseline app set to every registered cluster is a common platform-engineering primitive.
  • A common multi-tenant topology: one Argo CD per environment (or one central Argo CD), AppProject per team, SSO via OIDC/Dex, RBAC mapping SSO groups to project-scoped permissions. AppProjects are the real isolation boundary — Applications themselves are not namespaced tenancy units.
  • The 2022 simultaneous CNCF graduation of Argo CD and Flux (The New Stack) marked GitOps moving from emerging practice to default production delivery model.
  • Argo CD is itself often bootstrapped via GitOps — an “Argo CD managing Argo CD” app-of-apps so the tool’s own config is version-controlled and self-healed.

See Also

Graph View

Backlinks