June 14, 2025 · 7 min read

Hyperautomation via GitOps: The Code-Driven Infrastructure

A practitioner's guide to GitOps with Argo CD: sync and self-heal semantics, ApplicationSets, secrets, prune safety, and when GitOps is the wrong tool.

By Husain Abbas, Enterprise Architect & former CTO

The Problem

Most infrastructure still changes in one of two ways. In the first, engineers click through cloud consoles (“ClickOps”). In the second, an imperative CI/CD pipeline runs kubectl apply or shell scripts against production. Both fail the same way: nothing keeps checking that the live environment still matches what was approved.

Picture a common outage. During an incident, an engineer hand-edits a Deployment or loosens a firewall rule to restore service. The live environment no longer matches the repository. The next pipeline run then does one of two things. It silently overwrites the fix and causes a second outage, or it fails on a conflict nobody can explain at 2 a.m. Push-based pipelines also create a security problem: the CI runner needs cluster-admin-grade credentials for every target cluster. That makes the CI system one of the most valuable things an attacker could compromise.

The root cause is that deployment is treated as a one-off event, not as a continuously enforced state.

The Architectural Solution

GitOps makes a version-controlled repository the only way to change a system. A reconciler running inside the target environment enforces it. The CNCF’s OpenGitOps project defines four principles:

Declarative: The desired state is expressed declaratively (Kubernetes manifests, Helm charts, Kustomize overlays).
Versioned and immutable: Desired state is stored with full history. Every change is a reviewable commit, and every rollback is a git revert.
Pulled automatically: Software agents pull the desired state from the source. The CI system never pushes into the cluster.
Continuously reconciled: Agents keep comparing actual state with desired state and correct any drift.

Argo CD or Flux?

Both are CNCF-graduated projects, and both implement the same principles well. The choice usually comes down to how your organisation operates:

Concern	Argo CD	Flux
Model	`Application` / `ApplicationSet` CRDs; often one “hub” instance managing many clusters	Composable controllers (`GitRepository`, `Kustomization`, `HelmRelease`), usually installed in each cluster
UI	Built-in web UI with diff and sync views	No built-in UI; CLI and third-party dashboards
Multi-tenancy	AppProjects plus Argo CD RBAC	Kubernetes RBAC and service-account impersonation
Secrets	Recommends in-cluster operators	Native SOPS decryption in the Kustomization controller

Reconciliation Timing Matters

“Continuous” does not mean “instant”. By default, Argo CD checks Git every 120 seconds, plus up to 60 seconds of jitter (timeout.reconciliation). Configure Git webhooks if you need commits to deploy within seconds. Drift in the live cluster is handled separately. Argo CD watches cluster resources, and with selfHeal enabled it re-syncs whenever the live state diverges. The default self-heal retry delay is 5 seconds.

Kubernetes Is the Natural Boundary

GitOps controllers reconcile Kubernetes objects. To apply the same model to cloud resources such as VPCs, databases, or IAM, those resources need a Kubernetes-native representation, for example through Crossplane or a cloud provider’s operator. Otherwise, keep Terraform in a plan/apply pipeline with scheduled drift detection, and run GitOps for everything above the cluster API. Many mature platforms deliberately use this hybrid.

Architecture Diagram

GitOps reconciliation loop: a Git repository is pulled by Argo CD or Flux running in the cluster, applied through the Kubernetes API, and continuously compared against live state so that drift is reverted

Core Implementation

A Production-Grade Application

The Application below syncs one service into a cluster. It fixes the common mistakes in tutorial manifests: a missing project, a production deployment that tracks HEAD, and replica-count fights with the Horizontal Pod Autoscaler (HPA).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-service-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # delete child resources with the app
spec:
  project: payments                            # AppProject scopes repos, clusters, kinds
  source:
    repoURL: https://github.com/example-org/platform-config.git
    path: clusters/prod/payment-service
    targetRevision: release-2025.06.1          # pin prod to a tag, not HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: payments-prod
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas                       # owned by the HPA, not Git
  syncPolicy:
    automated:
      prune: true      # delete resources that were removed from Git
      selfHeal: true   # revert manual changes made in the live cluster
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

A few semantics are easy to get wrong:

automated turns on auto-sync when Git changes.
prune controls deletion only. It is off by default.
selfHeal controls the response to drift in the live cluster. It is also off by default.
Auto-sync will not retry a failed sync against the same commit. Push a fix or revert to move forward.

Scaling to a Fleet with ApplicationSet

Copying that manifest for 50 clusters creates a maintenance problem. An ApplicationSet with a cluster generator creates one Application per registered cluster that matches a label selector:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-service
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:
        selector:
          matchLabels:
            tier: production
  template:
    metadata:
      name: 'payment-service-{{.name}}'
    spec:
      project: payments
      source:
        repoURL: https://github.com/example-org/platform-config.git
        path: 'clusters/{{index .metadata.labels "region"}}/payment-service'
        targetRevision: release-2025.06.1
      destination:
        server: '{{.server}}'
        namespace: payments-prod
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

With missingkey=error, a cluster without a region label fails to render. Without it, Argo CD would quietly generate a broken path.

Edge Cases & Limitations

Secrets do not belong in Git, even encrypted ones, unless you have a plan. Argo CD’s secret management guidance strongly recommends managing secrets in the destination cluster, for example with External Secrets Operator or Sealed Secrets. It advises against injecting secrets during manifest generation, partly because Argo CD caches generated manifests in plaintext in Redis. Flux users often choose SOPS, but key rotation and access to the decryption key become part of your threat model.
Self-heal changes incident response. With selfHeal: true, an emergency kubectl scale or kubectl set image is reverted within seconds. That is the point of the feature, but responders need a sanctioned break-glass path: commit to Git, or temporarily disable auto-sync (argocd app set <app> --sync-policy none, or automated.enabled: false in recent Argo CD versions) and record that in the incident timeline. Deleting a crash-looping pod is unaffected. The Deployment’s ReplicaSet recreates it whether or not GitOps is in use.
Prune is a loaded gun. If you move a directory, rename a path, or make a bad Kustomize change, resources disappear from the rendered output and get deleted. Protect stateful or expensive objects such as PVCs, namespaces, and CRDs with the argocd.argoproj.io/sync-options: Prune=false annotation. Keep allowEmpty at its default (false) so that an empty render never wipes an application.
Fields owned by other controllers cause permanent OutOfSync. HPA replica counts, injected sidecars, and defaults from mutating webhooks all differ from Git. Remove those fields from the manifest or declare them in ignoreDifferences. Otherwise self-heal will fight the other controller.
Git becomes production-critical. If the Git host is unavailable, nothing can be deployed, including rollbacks. Mirror critical repositories, protect branches, require signed commits on environment branches, and treat write access to the config repository as production access.
When not to use it. Avoid this pattern for a single small cluster where a reviewed pipeline is enough, for workloads that are not on Kubernetes and have no operator, and for imperative one-off operations such as database schema migrations, which need ordering and backout logic beyond “apply desired state”. In those cases, use sync hooks or keep the operation in a pipeline.

Key Strategic Takeaways

Make Git the only write path. Remove human write access to production clusters except through an audited break-glass role. Drift you allow becomes drift you have to debug.
Pull, don’t push. In-cluster controllers mean CI never holds cluster credentials, which removes a high-value target from the attack surface.
Pin and promote. Production should track tags or commits that are promoted by pull request, never a moving HEAD.
Respect the boundary. Use GitOps for anything with a Kubernetes API. Cover cloud resources with Crossplane or a Terraform pipeline that has drift detection.
Guard the sharp edges. Use Prune=false on stateful resources, ignoreDifferences for controller-owned fields, and a documented break-glass procedure. Pair this with shift-left policy checks so that what reaches Git is already compliant.