vault

GitOps

13 min read

GitOps

GitOps is an operational philosophy — not a single tool — in which a Git repository is the single source of truth for the declarative description of an entire system (its infrastructure and its applications), and an automated agent running inside the target environment continuously reconciles the live state of that environment to match what Git says it should be (opengitops.dev). The term was coined by Weaveworks in 2017 (weave.works — GitOps) and later formalized by the OpenGitOps working group at the CNCF into four principles. GitOps takes the declarative reconciliation idea that already runs inside a Kubernetes cluster — every controller watches desired state and drives observed state toward it (see Kubernetes Control Loop Pattern) — and extends that same loop one level up: instead of a Deployment controller reconciling Pods toward a Deployment spec, a GitOps controller reconciles the whole cluster toward a Git repository. The cluster becomes, in effect, the observed state of a control loop whose desired state lives in version control. The two dominant implementations are ArgoCD and Flux. This note covers the four OpenGitOps principles, the pull-vs-push distinction that is GitOps’s defining architectural commitment, the operational benefits that follow, and the one genuinely hard problem — secrets — that GitOps does not solve on its own.

Mental Model

flowchart LR
    DEV["Developer / Operator"]
    PR["Pull Request<br/>(review · approval · CI checks)"]
    GIT[("Git repository<br/>declarative desired state:<br/>manifests / Helm / Kustomize")]
    subgraph CLUSTER["Target cluster"]
        AGENT["GitOps agent<br/>(ArgoCD / Flux controller)"]
        LIVE["Live cluster state<br/>(observed)"]
        AGENT -- "watch / diff / apply" --> LIVE
    end
    DEV --> PR
    PR -- "merge" --> GIT
    GIT -. "agent PULLS desired state<br/>on a poll / webhook" .-> AGENT
    AGENT -- "compare Git vs live" --> AGENT
    LIVE -. "drift?" .-> AGENT
    AGENT -. "drift → re-apply Git's version<br/>(self-healing)" .-> LIVE

What this diagram shows. Every change to the system flows through one funnel: a Git commit, normally via a reviewed-and-approved Pull Request. No human runs kubectl apply against production. Inside the cluster, a GitOps agent runs a perpetual loop — pull the desired state from Git, diff it against the live cluster, and if they differ, apply Git’s version. The two dotted feedback edges are the heart of the model: the agent pulls (Git does not push to the cluster — the agent reaches out to Git), and the agent continuously reconciles (a manual kubectl edit that drifts the cluster away from Git is detected and either flagged or automatically reverted — self-healing). The insight to extract: GitOps is the Kubernetes Control Loop Pattern applied with Git as the desired-state store and the entire cluster as the managed resource. Once you see that, every property of GitOps — auditability, reproducibility, drift correction — is just a property of a control loop whose spec happens to be version-controlled.

Mechanical Walk-through

The four OpenGitOps principles

The CNCF OpenGitOps project (PRINCIPLES.md v1.0.0, open-gitops/documents) defines GitOps as a system satisfying four properties:

  1. Declarative. “A system managed by GitOps must have its desired state expressed declaratively.” Not a script of steps — a description of the end state. This is non-negotiable and is exactly why Kubernetes, whose entire API is declarative (Declarative vs Imperative Configuration), is GitOps’s natural home.

  2. Versioned and Immutable. “Desired state is stored in a way that enforces immutability, versioning, and retains a complete version history.” Git satisfies this directly: every state is a commit, the history is append-only, and any past state is recoverable. (The store need not literally be Git — an OCI registry of immutable artifacts also qualifies — but Git is the overwhelmingly common choice.)

  3. Pulled Automatically. “Software agents automatically pull the desired state declarations from the source.” The agent pulls; nothing external pushes into the cluster. This is the principle with the most architectural consequence — see below.

  4. Continuously Reconciled. “Software agents continuously observe actual system state and attempt to apply the desired state.” The loop never stops. Drift — divergence between Git and the cluster — is continuously detected and corrected, not caught only at the next deploy.

Pull vs push — the defining distinction

The third principle, “pulled automatically,” is what separates GitOps from a CI-driven deployment pipeline, and the distinction is worth dwelling on:

Push-based (CI-driven) delivery. A CI pipeline, on merge, runs kubectl apply (or helm upgrade) against the cluster from the pipeline runner. This is the traditional model. Its costs:

  • Cluster credentials live in CI. The pipeline must hold kubeconfig/tokens with write access to production. CI systems are a large attack surface; credentials there are credentials everywhere the CI can reach.
  • No drift correction. The pipeline acts only when it runs. Between pipeline runs, anything — a manual kubectl edit, a half-finished hotfix, a flapping operator — can drift the cluster away from the intended state, and the pipeline neither knows nor cares until the next merge.
  • The cluster is opaque to the repo. The pipeline pushes and forgets; there is no continuous statement of “the cluster currently matches commit abc123.”

Pull-based (GitOps) delivery. An agent inside the cluster pulls from Git and reconciles. Its advantages:

  • No cluster credentials outside the cluster. The agent already runs in the cluster with a ServiceAccount; it needs only read access to Git. CI never needs cluster write access. The blast radius of a compromised CI shrinks dramatically.
  • Self-healing. Because the loop runs continuously, drift is detected and (if configured) auto-reverted. A manual production edit is undone within the reconcile interval.
  • The repo is the always-current statement of intent. The cluster’s state is, by construction, “whatever Git says,” and the agent reports continuously whether reality matches.

Pull-based is the architecturally distinctive GitOps property; push-based pipelines that merely store manifests in Git are sometimes called “GitOps” loosely, but they fail principles 3 and 4.

The benefits, derived

Every operational benefit of GitOps is a consequence of the four principles, not an independent feature:

  • Audit trail. Every change is a Git commit with an author, a timestamp, a diff, and (via PR) a reviewer and an approval. “Who changed production and when” is git log.
  • Review and approval as a deployment gate. Because the only path to production is a merged PR, code review and branch-protection rules become the deployment-approval mechanism for free.
  • Disaster recovery. The cluster’s entire declared state is in Git. Rebuilding a destroyed cluster is “point a fresh agent at the repo” — the agent reconciles the empty cluster up to the declared state. RPO is “the last commit.”
  • Drift detection and correction. The agent continuously compares Git to live state; drift is surfaced (ArgoCD’s UI literally shows OutOfSync) and optionally auto-corrected.
  • Reproducibility and rollback. Rolling back is git revert; the agent then reconciles the cluster to the reverted state. Promotion across environments is a commit/merge between repo paths or branches.

Repository topology

Two common shapes. A monorepo holds all environments in one repository (often base/ + overlays/dev|staging|prod/ — the Kustomize layout, or per-environment Helm value files). A multi-repo split keeps application source code in one repo and deployment manifests in a separate “config repo” that the GitOps agent watches — so an app build produces an artifact and a commit to the config repo, decoupling “code changed” from “deployment changed.” The config-repo split is the more common production shape because it keeps the GitOps agent’s input clean of application source churn.

Configuration / API Surface

GitOps is a philosophy; its API surface is whatever the agent exposes. A representative ArgoCD Application — the object that declares “reconcile this Git path into this cluster” — with commentary:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: argocd                       # ArgoCD's own namespace
spec:
  project: default
  source:
    repoURL: https://github.com/acme/deploy-config.git   # the GIT source of truth
    targetRevision: main                  # the branch/tag/commit to reconcile to
    path: overlays/prod                   # the directory within the repo (a Kustomize overlay here)
  destination:
    server: https://kubernetes.default.svc # the cluster to reconcile INTO
    namespace: web-prod
  syncPolicy:
    automated:
      prune: true                         # delete live objects no longer in Git
      selfHeal: true                      # revert manual drift back to Git's state — principle 4
    syncOptions:
      - CreateNamespace=true

The conceptual lines:

  • source is the desired state — a Git repo, a revision, a path. Principles 1 (declarative) and 2 (versioned) are satisfied by what this points at.
  • syncPolicy.automated is the reconciliation loop. selfHeal: true is principle 4 made operational: the agent doesn’t just deploy on commit, it continuously re-asserts Git’s state over any drift. prune: true lets the agent delete objects removed from Git — making Git authoritative for existence, not just configuration.
  • The agent pulls repoURL (poll or webhook); nothing in this object grants any external system write access to the cluster — principle 3.

Flux expresses the same idea with a different shape — a GitRepository source object plus a Kustomization or HelmRelease that references it — but the semantics are identical. The comparison is the subject of ArgoCD vs Flux.

The Secrets Problem

GitOps has one genuinely hard, unavoidable problem: you cannot commit plaintext secrets to Git. Git history is immutable and (often) widely readable; a leaked credential in a commit is leaked permanently, even after a later “removal” commit. Yet the four principles demand that all desired state — including secret material — be declared in the versioned store. The reconciliation of this tension is a small ecosystem of patterns:

  • Sealed Secrets (Bitnami) — an in-cluster controller holds a private key; you encrypt a Secret into a SealedSecret custom resource with the matching public key and commit that. Only the controller can decrypt it, so the ciphertext in Git is safe. Encryption is cluster-specific.
  • SOPS (Mozilla) — encrypts the values of a YAML/JSON file (leaving keys readable) using a KMS key (AWS KMS, GCP KMS, age, PGP). The encrypted file is committed; Flux and ArgoCD can decrypt it at reconcile time. Common with Flux’s decryption field.
  • External Secrets Operator — does not store the secret in Git at all. You commit an ExternalSecret reference object; the operator fetches the actual value from an external secret manager (Vault, AWS Secrets Manager, GCP Secret Manager) and materializes a Kubernetes Secret. Git holds only the pointer, never the secret.

The conceptual split: Sealed Secrets and SOPS keep an encrypted form in Git; External Secrets Operator keeps the secret out of Git entirely and stores only a reference. Both approaches preserve the GitOps property “everything needed to rebuild the cluster is declared” while not exposing plaintext.

Failure Modes

  1. “GitOps” that is actually push-based. Storing manifests in Git but still kubectl apply-ing them from CI satisfies principle 1 (declarative) and 2 (versioned) but not 3 (pulled) or 4 (continuously reconciled). It gets the audit trail but neither the credential-isolation nor the self-healing benefit. This is the most common dilution of the term.

  2. Drift fights between the agent and an autoscaler. An HPA writes spec.replicas; the GitOps agent re-applies the Git manifest’s replicas: 3 and resets it; the HPA re-scales; loop. Mitigation: omit replicas from Git, or use the agent’s “ignore differences” feature, or rely on Server-Side Apply field ownership. Detailed in Declarative vs Imperative Configuration.

  3. prune: true deleting more than intended. If the agent’s tracked-resource set is mis-scoped, removing or moving a manifest in Git can prune a live object the operator needed. Mitigation: scope Applications narrowly; test prune behavior in staging; use resource-level “prune protection” annotations for stateful objects.

  4. Plaintext secret committed before the encryption pattern was set up. The single most damaging GitOps mistake — and irreversible, because Git history is immutable. Mitigation: enforce a pre-commit secret scanner and never let a repo go GitOps-live without one of the secrets patterns in place from commit one.

  5. Reconcile loop too slow for incident response. During an outage, “commit to Git and wait for the agent to pull” can be slower than the situation tolerates. Mitigation: every GitOps shop needs a documented break-glass procedure (temporary direct access, agent webhook to force immediate sync) — and a discipline to commit the emergency change back to Git afterward so the agent doesn’t revert it.

  6. Bootstrap chicken-and-egg. The GitOps agent itself must be installed before it can manage anything — including, ideally, itself. Mitigation: a bootstrap step (flux bootstrap, ArgoCD’s app-of-apps) installs the agent and then hands its own configuration to the agent to self-manage thereafter.

Alternatives and When to Choose Them

  • CI-driven push delivery (kubectl apply/helm upgrade from a pipeline). Simpler to start, no agent to run. Acceptable for small teams, non-production, or environments where the credential-isolation and self-healing benefits don’t justify the operational overhead. The trade-off is exactly the pull-vs-push table above.
  • Imperative operations (kubectl by hand). Fine for development and genuine emergencies; never a delivery system for production state — see Decision Framework 6 in Kubernetes MOC.
  • Cloud-provider deployment services (AWS CodeDeploy, Cloud Deploy) — push-based, provider-coupled; reasonable inside a single cloud, weaker on drift correction and multi-cluster.
  • Higher-level GitOps platforms (Argo CD + Argo Rollouts for progressive delivery, Flux + Flagger) layer canary/blue-green analysis on top of the base GitOps loop — choose these when you need metric-gated progressive delivery, not just reconcile-to-Git.

Production Notes

  • Weaveworks coined “GitOps” in 2017 describing how they ran their own SaaS; the term spread fast enough that the CNCF chartered the OpenGitOps working group (2021) to keep the definition rigorous — the four principles exist precisely to stop “GitOps” from degrading into “we keep some YAML in Git.”
  • ArgoCD and Flux are both CNCF graduated projects — the highest maturity tier — and are the two dominant implementations; the choice between them is the subject of ArgoCD vs Flux.
  • The app-of-apps / GitOps-bootstrap pattern — the agent’s own configuration is itself a Git-managed Application — is how mature shops avoid a special-cased, un-versioned agent install. The agent manages the agent.
  • Disaster-recovery drills are GitOps’s most-cited operational payoff: teams running GitOps can credibly test “delete the cluster, recreate it, point the agent at the repo” and watch the environment rebuild itself — a drill that is far harder when cluster state is the product of an imperative pipeline’s history.
  • GitOps is a standard platform-engineering interview topic; the strong answer leads with the pull-vs-push architectural distinction and the four principles, not with a tool name.

See Also

Graph View

Backlinks