Before this migration, here was the state of my Kubernetes platform:
- 7 Helm releases running on AWS EKS — kube-prometheus-stack, Tempo, Loki, Promtail, Chaos Mesh, plus the microservices overlay
- Each installed manually with
helm installorkubectl apply -kfrom my laptop - No drift detection: if a resource was patched directly in the cluster, I'd find out when something broke
- No audit trail: the deployment history was a mix of terminal history and my own memory
- Deployment ordering enforced by me running commands in the right sequence
This is how most Kubernetes platforms look after the "it works" phase. The cluster is functional but fragile. The state of your infrastructure lives in your head, not in git.
This post documents how I changed that — using ArgoCD's app-of-apps pattern to migrate the entire platform to GitOps in a single session.
What ArgoCD App-of-Apps Actually Is
The app-of-apps pattern is simpler than it sounds: you create one "root" ArgoCD Application that watches a directory in your git repository. Every YAML file in that directory is itself an ArgoCD Application manifest. When you push a new Application file to that directory, ArgoCD automatically creates the corresponding release. When you delete the file, ArgoCD prunes the release from the cluster.
One directory. One git push. ArgoCD handles the rest.
The root application for this platform is 34 lines:
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: platform-root namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/Rummy43/ai-microservices-platform.git targetRevision: HEAD path: k8s/gitops/apps destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true That's the entire bootstrap. One kubectl apply -f k8s/gitops/app-of-apps.yaml. Everything after that is a git push.
prune: true means ArgoCD deletes resources from the cluster when their definition is removed from git. selfHeal: true means ArgoCD automatically reverts manual cluster changes back to the git state. These two flags together are what makes this GitOps rather than just "ArgoCD-managed deploys."
The Seven Child Applications
Inside k8s/gitops/apps/, there are seven Application manifests:
| Application | Namespace | Deployment Type |
|---|---|---|
kube-prometheus-stack | monitoring | Helm multi-source |
tempo | monitoring | Helm multi-source |
loki | monitoring | Helm multi-source |
observability-config | monitoring | Kustomize |
promtail | monitoring | Helm multi-source |
chaos-mesh | chaos-testing | Helm multi-source |
microservices | microservices | Kustomize prod overlay |
ArgoCD auto-detects the deployment type. If the target path contains a kustomization.yaml, it uses Kustomize. If it contains Chart.yaml, it uses Helm. No explicit type annotation required.
The microservices application points directly at the production overlay that was already managing the platform services before this migration:
spec: source: path: k8s/overlays/prod syncPolicy: automated: prune: true selfHeal: true syncOptions: - ApplyOutOfSyncOnly=true ApplyOutOfSyncOnly=true is worth calling out: it tells ArgoCD to only apply resources that have actually drifted from git, rather than re-applying every resource on every sync cycle. For a Kustomize overlay with 20+ manifests, this matters for cluster stability — you don't want rolling restarts on every sync just because ArgoCD did a reconciliation pass.
Sync Waves: Encoding Deployment Order in YAML
The observability stack must be ready before microservices start. Prometheus needs to be scraping before services emit metrics. Chaos Mesh needs to be installed before you run experiments against it. This ordering exists whether you encode it or not — the question is whether it lives in a Confluence page, in your head, or in the deployment system itself.
ArgoCD sync waves encode it in the deployment system:
# Wave 0 — observability stack (must be ready first) metadata: annotations: argocd.argoproj.io/sync-wave: "0" # Applied to: kube-prometheus-stack, tempo, loki, observability-config # Wave 1 — log shipping and chaos tooling (needs observability running) metadata: annotations: argocd.argoproj.io/sync-wave: "1" # Applied to: promtail, chaos-mesh # Wave 2 — platform microservices (needs everything else ready) metadata: annotations: argocd.argoproj.io/sync-wave: "2" # Applied to: microservices (user-service, notification-service, api-gateway) ArgoCD will not begin wave 1 until every application in wave 0 reports Healthy. It will not begin wave 2 until wave 1 is complete. If kube-prometheus-stack fails its health check, the microservices never start — which is correct behavior. You want to know about the observability failure before your services are running without metrics.
The practical value goes beyond ordering. It eliminates an entire category of incidents that begin with "which order did you install things?" You don't answer that question from memory anymore. You read the wave number.
Multi-Source Helm: Separating Chart Version from Values Config
Standard Helm with ArgoCD gives you one source: a chart from a Helm repository with values inlined or from a file in the same application repo. Multi-source, available from ArgoCD 2.6+, lets you reference a Helm chart from one repository and your values files from another.
For kube-prometheus-stack:
spec: sources: - repoURL: https://prometheus-community.github.io/helm-charts chart: kube-prometheus-stack targetRevision: "91.8.0" helm: releaseName: kube-prometheus-stack valueFiles: - $values/k8s/helm/kube-prometheus-stack/values.yaml - $values/k8s/helm/kube-prometheus-stack/values-eks.yaml - repoURL: https://github.com/Rummy43/ai-microservices-platform.git targetRevision: HEAD ref: values The second source defines the $values reference — a named pointer to your git repository. The first source uses $values/path/to/file.yaml to load values from that reference.
The same pattern applies to Tempo, Loki, Chaos Mesh, and Promtail — each has a base values.yaml for local kind cluster configuration and a values-eks.yaml for EKS-specific overrides (S3 backends, IRSA service accounts, gp3 storage classes).
What this gives you:
Chart version pinned in git. targetRevision: "91.8.0" lives in a commit in your platform repo. Upgrading the chart is a PR with a diff, not a manual helm repo update && helm upgrade. Rollback is git revert.
Environment-specific values separated cleanly. The base chart never sees your EKS configuration. Your EKS values never pollute your local cluster setup. The two layers compose at deploy time — ArgoCD merges them.
Drift detection on Helm values. If someone runs helm upgrade with modified values directly in the cluster, ArgoCD will detect the drift and self-heal back to the git state within minutes. Without GitOps, that modification is permanently invisible.
The 262KB Problem
Chaos Mesh ships with large Custom Resource Definitions — the schemas that define PodChaos, NetworkChaos, StressChaos, IOChaos, and the rest of the fault injection primitives. When ArgoCD applies a large CRD using the standard kubectl apply path, it stores the entire last-applied configuration as an annotation on the resource. For Chaos Mesh's CRDs, that annotation exceeds the 262KB limit that Kubernetes imposes on annotation values.
The error on the first sync attempt:
metadata.annotations: Too long: may not be more than 262144 bytes The fix is a single sync option in the ArgoCD Application manifest:
syncOptions: - ServerSideApply=true Server-side apply moves the apply logic from the client (ArgoCD/kubectl) to the Kubernetes API server. The last-applied-configuration annotation is no longer stored on the resource, so the annotation size limit doesn't apply. The API server tracks field ownership in a different, more efficient way.
This same fix was needed for kube-prometheus-stack — the Prometheus Operator CRDs have the same characteristic. The pattern to remember: if an ArgoCD application fails to sync with an annotation size error, ServerSideApply=true is the resolution. It's safe to enable for all Helm applications.
What "Healthy/Synced" Actually Looks Like
After the bootstrap and the first automated sync cycle, the ArgoCD UI showed:
platform-root Synced Healthy ├── kube-prometheus-stack Synced Healthy (wave 0) ├── tempo Synced Healthy (wave 0) ├── loki Synced Healthy (wave 0) ├── observability-config Synced Healthy (wave 0) ├── promtail Synced Healthy (wave 1) ├── chaos-mesh Synced Healthy (wave 1) └── microservices Synced Healthy (wave 2) Seven applications. All Healthy. All Synced. Wave ordering verified in the event timeline — wave 0 completed before wave 1 started; wave 1 completed before wave 2 started.
The first real test came immediately after. I updated an image tag in a deployment manifest, committed, and pushed. ArgoCD detected the diff within 30 seconds. It applied only the changed resource (ApplyOutOfSyncOnly=true in action), triggered a rolling update, and reported Healthy within two minutes of the push.
No kubectl apply. No helm upgrade. No SSH into a bastion host. The git push was the deployment.
Two Gaps the Game Day Found
GitOps solved the deployment pipeline. It didn't fix the gaps in the monitoring stack that the Phase 11 chaos engineering game day exposed.
Gap 1: Kafka consumer lag alerts need a lag exporter to have a data source. The PipelineConsumptionStalled alert rule is in git, properly formatted, correctly deployed via ArgoCD. But the metric it queries — kafka_consumergroup_lag — does not exist in this cluster because no Kafka lag exporter is running. The alert rule evaluates to NaN and never fires. An alert with no metric source is a hypothesis, not a safeguard.
Gap 2: SLO burn-rate alerts need real traffic to generate error budget burn. AvailabilityFastBurn fired correctly in prior testing. During the EKS game day, with only Prometheus scraping actuator endpoints as "traffic," there was no request error budget to consume. The burn rate calculation worked correctly — it correctly identified no errors — which masked the actual latency degradation.
Both fixes will themselves be delivered via GitOps. The kafka-lag-exporter will be a new Application manifest in k8s/gitops/apps/. The traffic generator will be a CronJob added to the microservices overlay. When those files are pushed, ArgoCD deploys them automatically.
The irony of GitOps: it's very good at deploying your solutions. You still have to find the gaps first.
What Changed
Before:
- Deployment state lived in terminal history and my memory
- Helm releases were upgraded in place with no record of what values changed
- Ordering was enforced by convention and documentation
- Drift was invisible until something broke
- "What's running in production?" was a question that required interrogating the cluster
After:
- Every resource in the cluster has a corresponding file in git
- Every change is a commit — with a diff, a reason, and a timestamp
- Deployment order is encoded in sync wave annotations, not in a runbook
- ArgoCD detects drift within minutes and self-heals automatically
- "What's running in production?" is
git log
The bootstrap was one command. Everything since has been git push origin main.
If you're running multiple Helm releases on Kubernetes and managing their state mentally, app-of-apps is the migration path. The pattern is straightforward once you've seen it once — the main decision is how to structure your sync waves to reflect the actual dependency order of your stack.
The platform source is at github.com/Rummy43/ai-microservices-platform. Phase 12 covers the kafka-lag-exporter and traffic generator — closing the game-day blind spots via, naturally, another git push.
