Introduction Before you move furniture into a new house, you install the locks, the cameras, and the gates. Kubernetes is no different, yet engineers constantly build clusters inside-out, treating security as a cosmetic bolt-on to be handled "later." Not this time. The order here is entirely deliberate. You don't move into a finished house and then start wondering about the security system—you wire the alarms while the building is still empty. With my 29-node RKE2 infrastructure fully framed across three distributed Proxmox datacenters, the platform is technically alive. But it is also completely vacant. Before a single tenant workload steps foot inside—long before MLflow, Kubeflow, Tekton, or Harbor’s first private project—the security stack goes live. When a homelab scales past a handful of nodes and begins hosting your actual identity providers, encrypted secrets engines, private registries, and core data pipelines, the blast radius of a compromise becomes real. Security transforms from a theoretical checklist into an absolute prerequisite. This marks the definitive boundary where this project stops behaving like a handcrafted science project and starts operating like a zero-trust production platform. It also marks the moment I transition completely to a 100% GitOps workflow via ArgoCD. The days of ad-hoc terminal manipulation and local kubectl apply commands are over. From here on out, every single policy configuration is a version-controlled commit. This article documents the exact mechanics of how I wired four distinct, cloud-native security engines into a unified defensive front covering the full workload lifecycle: The Four Guards Here's what I wired together: Guard Engine Job 🚪 The Bouncer Kyverno Blocks bad manifests at admission 🔒 The Locks KubeArmor Watches syscalls at the kernel — and can block them 📹 The CCTV Falco Detects and alerts on abnormal behavior 🔍 The Inspector Trivy Continuously scans for CVEs and misconfigurations Each works independently. If one fails, the others don't notice. That's the point. What Makes This Different This episode covers the full workload lifecycle: Before workloads enter the API (admission control) While workloads run at the kernel tier (runtime audit, enforcement on demand) When workloads behave anomalously (behavioural detection) Long after workloads were deployed (continuous scanning) This isn’t a pristine reference diagram copied from a vendor's whitepaper. These are real, unvarnished operational lessons, architectural deadlocks, and configuration missteps encountered while making these engines work together cleanly. My hope is that sharing these traps saves you time on your own infrastructure journey. Where I Am in the Build This security layer is part of a larger infrastructure series documenting the evolution of the cluster: Episode 1 — The Foundation Infrastructure provisioning with Terraform across multiple Proxmox datacenters Episode 2 — The Framing Building the HA RKE2 control plane using Ansible, kube-vip, and Cilium Episode 3 — The Security System Hardening the platform using Kyverno, KubeArmor, Falco, and Trivy Operator The infrastructure exists. Now the platform gets secured before anything important is allowed to run inside it. The repository layout for this stack lives inside kubernetes-addons/security-stack/ with an intentionally clean directory structure: security-stack/ ├── README.md ← Walkthrough + disaster recovery runbook │ ├── kyverno/ ← The Bouncer: admission control │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── kyverno-kubearmor-falco-trivy-values.yaml │ └── policies/ ← shipped as its OWN ArgoCD app │ ├── kustomization.yaml │ ├── charts/kyverno-policies-3.8.0/ ← vendored upstream PSS baseline │ ├── security-policies.yaml ← my custom hardening rules │ ├── compliance-policies.yaml │ ├── kubeflow-user-namespace-governance.yaml │ ├── gpu-node-scheduling.yaml │ ├── job-ttl-after-finished.yaml │ ├── cleanup-finished-jobs.yaml │ ├── cleanup-stale-machineconfig-jobs.yaml │ └── rook-csi-apparmor-unconfined.yaml ← the Ceph CSI scar, as code │ ├── kubearmor/ ← The Locks: runtime LSM (audit today) │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── kubearmor-falco-trivy-values.yaml │ ├── example-policies.yaml │ ├── webhook-skip-terminating-hook.yaml ← the wedged-pod workaround │ └── charts/kubearmor-operator-v1.6.18/ ← vendored operator chart │ ├── falco/ ← The CCTV: runtime detection │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── falco-trivy-values.yaml │ ├── custom-rules.yaml ← scoped exceptions live here │ └── servicemonitor.yaml │ └── trivy/ ← The Inspector: continuous scanning ├── kustomization.yaml ├── namespace.yaml ├── trivy-values.yaml └── servicemonitor.yaml Two things in that tree are worth pointing at before I go further, because both are scars rather than design: kubearmor/webhook-skip-terminating-hook.yaml and kyverno/policies/rook-csi-apparmor-unconfined.yaml only exist because KubeArmor's AppArmor annotations wedged terminating pods and confined my Ceph CSI plugins. I get to those later. Each stack also carries its own README.md. Five ArgoCD child Applications, one single master ApplicationSet, and two distinct sync waves. The structural walls of my cluster haven't shifted an inch since Episode 2; what is about to change is what I allow to get inside, and what gets flagged the second it steps out of line. The Threat Surface Let’s be completely transparent: a modern homelab is no longer a casual testing environment where safety defaults can be skipped. This 29-node environment hosts HashiCorp Vault instances, enterprise Keycloak identity providers, MLflow registries, Kubeflow modeling pipelines, and private Harbor registries. Every single one of these software components holds explicit corporate identities, cryptographic keys, machine learning models, or proprietary container images. The blast radius of a single container escape or internal compromise is incredibly high. More importantly, this lab is my production testing sandbox; if I cut corners on security configurations here, I build bad architectural habits that will inevitably leak into the enterprise architectures I design for clients. The cluster threat vectors break down into four critical layers: Misconfigured Manifests (The Illegal Entry): Workloads containing unsafe specifications that should never be written to etcd. Think of a deployment requesting runAsUser: 0, a pod requesting host namespaces hostNetwork: true, images pointing to mutable tags like :latest, or a generic network policy leaving wide-open public egress. These should be caught and stopped dead at the API gateway during kubectl apply. Malicious Post-Admission Execution (The Broken Lock): Pods that legitimately pass validation checks but once scheduled onto a node, attempt to violate boundaries—such as writing directly to /etc/shadow, attempting to run an unexpected exec /bin/sh inside a production database pod, or mapping block devices. The local Linux kernel needs to refuse these operations natively. Subtle Runtime Anomalies (The Stealth Intruder): Exploits that bypass standard rule boundaries entirely. Fileless malware executions, unauthorized sudo executions inside an active application container, or an application process spawning unexpected binary payloads. These system calls must be observed, parsed, and logged instantly. The Zero-Day Decay (The Expired Clearance): Images that are fully verified, pristine, and compliant when pushed to the cluster on Friday, but become highly vulnerable on Monday morning because a new critical CVE was disclosed over the weekend. The cluster must continuously audit its own passive inventory. No single security engine can mitigate this entire lifecycle. That is the core tenet of the defense-in-depth model: I install four distinct guards that share zero dependency lines, hook into entirely different layers of the OS and Kubernetes lifecycle, and run side-by-side without interference. The Four Guards Architecture Here is the exact layout of how my security tooling intercepts threat vectors across the entire deployment and runtime lifecycle: The dotted line is deliberate. Everything drawn with a solid arrow is running in my cluster today; the alert fan-out is one chart flag away and is the one piece I have not turned on yet. I would rather draw the gap than pretend the box is filled. By segregating these duties, I protect against systemic failure. If the admission webhook is temporarily bypassed or down during a system upgrade, the kernel layer is still watching — and, once policies land, still enforcing. If the kernel layer has no rule for a specific zero-day vector, the behavioral CCTV logging system still records and exposes the activity. The Bouncer at the Door — Kyverno : Kyverno runs as my primary admission controller. Every single API interaction targeting my cluster passes directly through its webhook filters before the specification can be saved to etcd. If a defined policy evaluates to a failure, the request is outright denied. If a policy specifies an auto-remediation rule, the specification is seamlessly mutated in flight. My architecture runs Kyverno Chart 3.8.0 isolated completely within the kyverno namespace, consisting of five core deployments: kyverno-admission-controller — The primary engine processing real-time validation and mutation webhooks. kyverno-background-controller — Manages asynchronous generate policy reconciliation loops. kyverno-cleanup-controller — Garbage cycles dead PolicyReport and EphemeralReport metrics. kyverno-reports-controller — Aggregates policy data cluster-wide. kyverno-policy-reporter — The reporting visualization plane (disabled in my default values to preserve RAM). While clean on paper, syncing this setup via GitOps under ArgoCD presented a massive hurdle. Out of the box, the 3.8.0 chart immediately breaks synchronization pipelines due to mutating resource values. To establish an entirely hands-off automated sync loop, I had to introduce four specific structural fixes directly into the ArgoCD configuration: # Inside my master ApplicationSet / Application definitions: spec: ignoreDifferences: # 1. Neutralize the Kustomize CRD naming hash mutation - group: apiextensions.k8s.io kind: CustomResourceDefinition jsonPointers: - /metadata/annotations/kustomize.config.k8s.io~1needs-hash # 2. Prevent permanent diffs on ClusterPolicies defaulting validation modes - group: kyverno.io kind: ClusterPolicy jsonPointers: - /spec/validationFailureAction # 3. Strip dynamic API mutations from webhook manifests - group: apiextensions.k8s.io kind: CustomResourceDefinition jsonPointers: - /spec/conversion/webhook/clientConfig/caBundle Additionally, I forced forceFailurePolicyIgnore: true across all helm configurations. This ensures that during a major cluster upgrade or node restart where the Kyverno pods are initializing, the Kubernetes control plane switches to an Ignore mode rather than locking up the API server entirely. One major day-two operational trap to memorize (feedback_kyverno_chart_upgrade_job_leftovers): during every chart upgrade, a temporary hook job named Job/kyverno-migrate-resources is run alongside associated service accounts and cluster roles. ArgoCD marks these resources as unmanaged orphans (sync=None requiresPruning=True). You can manually execute a safe kubectl delete once you confirm the job completed its task; however, never trigger a blind global argocd sync --prune, as it will inadvertently purge active sibling metadata from unassociated applications. Here is the Bouncer's actual roster, and what happens to a walk-in who ignores the dress code: kubectl get clusterpolicy kubectl run nginx --image=nginx the output of both commands — 36 ClusterPolicies all READY=True, then the plain kubectl run nginx bounced before it ever reaches a node. Worth reading that second command's output carefully, because it shows two guards, not one: Error from server (Forbidden): pods "nginx" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false ... runAsNonRoot != true ... seccompProfile ... That rejection says PodSecurity, not Kyverno — it is Kubernetes' native Pod Security Admission, the free floor built into the API server, catching the most naive possible mistake before Kyverno is even consulted. The 36 ClusterPolicies sitting above it are what PSA can't do: mutate a spec in flight, generate a NetworkPolicy into every new namespace, verify a cosign signature, pin GPU pods to GPU nodes. I keep both deliberately, and the layering is the point — the cheapest guard catches the cheapest mistake. A plain kubectl run is exactly the kind of thing a tired operator does at 2am, and it never gets far enough to need the expensive machinery. The Locks on Every Door — KubeArmor : Kyverno decides who gets in. KubeArmor governs what they can do once they're inside. This is the distinction most homelabs miss: a pod can pass every admission check — correct securityContext, no host namespaces, a pinned image digest — and still, the moment it's running, try to read /etc/shadow, spawn /bin/sh inside a production database, or map a block device. Admission control is blind to all of it, because admission already happened. KubeArmor closes that gap at the only place that can't be argued with: the Linux kernel itself. It enforces per-pod policies through the kernel's Linux Security Modules (AppArmor + BPF-LSM), so a blocked action isn't reported — it's refused, natively, before the syscall completes. That is the capability. What I actually have turned on is a step short of it, and I'll show you exactly where I stopped. My architecture runs the KubeArmor operator v1.6.18 in the kubearmor namespace, which stands up three pieces: kubearmor (DaemonSet) — the per-node enforcement engine that talks to the kernel LSM. kubearmor-relay — aggregates the realtime block/audit event stream from every node into one place. kubearmor-controller — the admission + annotation controller that wires each pod to its AppArmor profile. Policies are per-workload and live right next to the thing they protect — KubeArmor watches every namespace. Locking a pod down is deliberately boring: apiVersion: security.kubearmor.com/v1 kind: KubeArmorPolicy metadata: name: no-shell-in-my-app namespace: my-ns spec: selector: matchLabels: { app: my-app } process: matchPaths: - path: /bin/sh action: Block - path: /bin/bash action: Block With that applied, an attacker who gets code execution inside my-app reaches for a shell and the kernel hands them nothing. Except I haven't applied it. Here is the honest state of my cluster, and it is the most useful thing in this section: kubectl get kubearmorpolicy -A No resources found Zero policies. Not one. And the operator's default posture backs that up: kubectl -n kubearmor get kubearmorconfig -o yaml | grep Posture defaultCapabilitiesPosture: audit defaultFilePosture: audit defaultNetworkPosture: audit So what is KubeArmor doing on my 29 nodes right now? Watching, in enormous detail. The relay aggregates a live stream of every process exec, file access and network connection, resolved to pod and namespace: kubectl -n kubearmor logs -l kubearmor-app=kubearmor-relay the kubearmor-relay stream — a live feed of resolved connections (remotehost=pod/harbor/harbor-nginx-..., svc/rook-ceph/rook-ceph-rgw-...), alongside kubectl get kubearmorpolicy -A returning No resources found. Visibility without enforcement, which is exactly what audit posture looks like. That gap is deliberate, and it is the correct order of operations. You do not deploy an LSM in enforce mode on a cluster you have not first watched. Audit posture tells you what normal looks like — which binaries actually exec, which paths actually get written, which pods actually talk to which — and that is what a least-privilege policy is written from. Ship action: Block on day one and you will spend your week explaining to yourself why CNPG can't start. I know this because KubeArmor has already blocked things on this cluster — just not attackers. The baseline AppArmor profile it generates for every pod carries its own deny-list (deny mount, deny /sys/... writes), and that alone was enough to break two critical subsystems. Which brings me to the traps. Here's the honest part. KubeArmor is the guard that will bite you before it ever bites an attacker, because it doesn't only lock down your workloads — by default it generates a restrictive AppArmor profile for every pod, including the critical infrastructure DaemonSets that keep the cluster alive. Three traps cost me real time: Trap 1 — KubeArmor vs. Cilium's eBPF (feedback_cilium_kubearmor_apparmor_block). After any change that rolls a BPF-using DaemonSet, some nodes wedged with: mkdir /sys/fs/bpf/tc: permission denied KubeArmor's default-deny profile was refusing Cilium's own eBPF mount. And it survives reboots — the restrictive profile is reapplied on every pod start, so a reboot "fix" lasts exactly until the next cilium-agent restart. The fix is to opt the DaemonSet out with a pod-template annotation: spec: template: metadata: annotations: container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined For Cilium this is now baked into the Ansible RKE2 bootstrap manifest (rke2-cilium-config.yaml) so a fresh install never rediscovers it. Any other BPF-touching DaemonSet (kube-vip, NFD) can need the same treatment. Trap 2 — the terminating-pod deadlock (feedback_kubearmor_apparmor_annotation_stuck_pod_k8s134). On newer Kubernetes, the controller's AppArmor-annotation mutation can leave pods stuck Terminating indefinitely — the annotation it wants to apply races the pod's own teardown. The release valve is blunt but reliable: scale kubearmor-controller to 0, let the stuck pods drain, then scale it back. A PostSync hook that skips terminating pods keeps it from recurring. Trap 3 — don't confine the storage plumbing (feedback_ceph_csi_rook_apparmor_rbd_module). KubeArmor's profile will happily confine the Rook-Ceph CSI plugin pods, and when it does, RBD volume mounts start failing cluster-wide — a storage outage wearing a security costume. The CSI plugins get the unconfined treatment for the same reason Cilium does: infrastructure that manipulates the kernel directly cannot run under a default-deny LSM profile. The lesson across all three: KubeArmor is a scalpel, not a blanket. Lock down tenant workloads aggressively; leave the infrastructure that talks to the kernel alone. The CCTV on the Ceiling — Falco : Locks stop the break-ins you anticipated. But security's oldest truth is that the attack you didn't write a rule for is the one that gets you. That's what the cameras are for. Falco doesn't block anything — it watches everything, and the second something moves that shouldn't, it shouts. Where KubeArmor enforces a known-bad list at the kernel, Falco taps kernel tracepoints via eBPF and pattern-matches the entire syscall firehose against a ruleset. A fileless payload executing from memory, an unexpected outbound connection from a pod that should only ever talk to Postgres, a sudo inside a container that has no business having one — none of those may trip a specific KubeArmor lock, but all of them light up the CCTV. I run the Falco chart 8.0.2 as a DaemonSet in the falco namespace on the modern_ebpf driver — one sensor per node, no kernel module to compile. Today, detections go exactly one place: stdout, which means the camera feed is kubectl logs: kubectl -n falco logs falco-2289j --tail=5 --max-log-requests=5 | grep -i warning And that is the honest weak point in this stack. A camera nobody watches is a camera that catches nothing after the fact. stdout is fine while I'm actively tailing it and useless at 3am — the detection fires, scrolls past, and ages out of the log buffer. Falco is the one guard here whose value collapses without somewhere to send the alert. Routing it is a chart flag, not a rebuild — Falco ships Falcosidekick as a subchart, so there is no second Helm install: # falco/falco-trivy-values.yaml falcosidekick: enabled: false # ← the honest current state Flip that to true, add a backend (Slack, Discord, Loki, PagerDuty), and the fan-out is done. The only reason it isn't already on is that the webhook is a credential, and I am not putting one in Git — so it waits for the same treatment everything else in this homelab gets: Terraform writes it to Vault, VSO syncs it in as a Secret. That is the next commit on this stack, and I would rather publish the gap than describe a pipeline I have not wired. a live Falco detection in kubectl -n falco logs falco-2289j --tail=5 --max-log-requests=5 | grep -i warning -f — rule name, pod, namespace, the offending command — the CCTV catching someone mid-motion. Two traps here, both about the sensor itself: The kernel floor. The modern_ebpf driver needs CAP_PERFMON + CAP_BPF and a kernel ≥ 5.8. On a compliant node it just works; drop in an older distro and Falco CrashLoopBackOffs on startup with a driver-load failure. The fallback is the legacy ebpf driver (or the kmod driver, which then wants kernel headers on the node) — set via values. The homelab's Ubuntu 24.04 image is well clear of the floor, but it's the first thing to check when a new node's Falco pod won't come up. Tuning the sensitivity, not silencing the camera. A fresh Falco install is noisy — legitimate operations trip generic rules. The wrong reflex is to delete the rule. The right one is to add a scoped exception in falco/custom-rules.yaml, commit, and push; ArgoCD reconciles within 30 seconds and Falco hot-reloads its config with no pod restart. You narrow the camera's blind spot deliberately, in version control — you never unplug it. The Inspector on Patrol — Trivy : The first three guards all police the present: what's entering, what's running, what's misbehaving right now. Trivy polices the thing none of them can see — time. An image can be pristine, signed, and fully compliant when it's pushed on Friday, and be a critical CVE waiting to happen by Monday morning, because a vulnerability was disclosed over the weekend and nothing about the image changed. The lock is fine. The camera sees nothing. The image just quietly went bad. So the Trivy operator (0.32.1), in the trivy-system namespace, walks the beat: it continuously re-scans the cluster's entire passive inventory and writes every finding as a Kubernetes custom resource — no database, no PVC, no UI. Six report types cover the surface: Report CR What it audits VulnerabilityReport Image CVEs, per workload, auto-rescanned ConfigAuditReport Workload misconfiguration (Pod security) RbacAssessmentReport Over-permissioned RBAC InfraAssessmentReport Node / control-plane CIS checks SbomReport A full SBOM for every pulled image ClusterComplianceReport NSA / CIS Benchmark posture, cluster-wide Because it's all just CRs, your SIEM is kubectl: # Every workload with a CRITICAL CVE, right now kubectl get vulnerabilityreport -A -o json | \ jq '.items[] | select(.report.summary.criticalCount > 0) | {ns:.metadata.namespace, name:.metadata.name, crit:.report.summary.criticalCount}' # Cluster-wide compliance posture — note the REAL report names kubectl get clustercompliancereport # k8s-cis-1.23 # k8s-nsa-1.0 # k8s-pss-baseline-0.1 # k8s-pss-restricted-0.1 kubectl describe clustercompliancereport k8s-nsa-1.0 the jq output above with its CRITICAL counts, and kubectl describe clustercompliancereport k8s-nsa-1.0 — the Inspector's findings board. And since the whole point of this stack is that I report what it actually finds rather than what I wish it found: the worst offender on my cluster is my own registry. Harbor's components come back at 26–33 CRITICAL apiece — harbor-portal at 33, harbor-core, registry and jobservice at 26 each — followed by my CNPG Postgres pods at 14, Elasticsearch at 8, and vault-transit at 6. That is an uncomfortable result to publish, and it is exactly why the Inspector earns its place. Harbor is the thing that scans everything else on the way in, and it is carrying the heaviest CVE load in the cluster. Nothing in the first three guards would ever have told me that — the manifests are valid, the syscalls are ordinary, the behaviour is normal. Only the thing that polices time finds it. Worth being precise about what those numbers mean: a CRITICAL count is not a breach, and most of these are unreachable code paths in base images. The signal is the trend and the ranking, not the absolute number — and the ranking is telling me which upstream images to pin, rebuild, or replace first. Pair the SbomReport output with a Kyverno verifyImages policy and the Bouncer and the Inspector start working together: Trivy tells you what's actually in every image, Kyverno refuses anything unsigned at the door. That's a real supply-chain story, not a checkbox. The one trap worth naming: private images silently fail to scan. Trivy's k8schain reads imagePullSecrets from the workload's ServiceAccount — so anything in an auth-required registry (some of Harbor's projects) won't scan until the namespace SA carries the pull secret: apiVersion: v1 kind: ServiceAccount metadata: name: default namespace: my-ns imagePullSecrets: - name: harbor-tekton-builder-creds Miss it and you get the most dangerous kind of failure — a silent one. The reports look clean not because the images are clean, but because Trivy never got to look. The Honest Head-to-Head — Why These Four, and Not the Alternatives First, the framing most security comparisons get backwards: these four tools are not competing with each other. "Should I use Falco or Kyverno?" is the wrong question — it's like asking whether a building needs a door lock or a smoke alarm. Security isn't a tool, it's a pipeline, and each guard owns a different stage of it: Build time → Trivy scans the image for CVEs before it ships. Admission time → Kyverno refuses the manifest (and cosign-verifies the signature) before it ever reaches etcd. Runtime, detect → Falco watches syscalls and alerts on the abnormal. Runtime, enforce → KubeArmor refuses the malicious syscall at the kernel. So the real question is never "which one" — it's "for this layer, which tool?" Here's that answer, honestly, layer by layer: Layer My pick The alternatives Why I picked mine Admission / policy-as-code Kyverno OPA/Gatekeeper, Kubewarden, native PSA YAML-native policies (no Rego to learn) + mutate + generate Runtime enforcement (LSM) KubeArmor raw AppArmor/SELinux, Tetragon K8s-native CRDs over the kernel LSM; per-workload least-privilege Runtime detection Falco Tetragon, Tracee, Sysdig Secure CNCF-graduated; biggest rule ecosystem; plugin model Vuln / misconfig scanning Trivy Grype, Clair, Snyk, Docker Scout All-in-one (CVE + IaC + secrets + SBOM); free; already in Harbor Where the alternatives genuinely win — because every one of my picks has a real rival: OPA/Gatekeeper vs Kyverno. Gatekeeper's Rego is a genuine policy language — more expressive than Kyverno's YAML for complex cross-resource logic, and the incumbent in heavily-regulated enterprises. I chose Kyverno because I'd rather write policies in the same YAML as everything else, and because Kyverno's generate (auto-stamp a NetworkPolicy / ResourceQuota / alias Service into every new namespace) and verifyImages (cosign — the same signatures my Tekton Chains pipeline produces) are first-class. That's load-bearing in my Kubeflow multi-tenancy. If I needed Rego's raw expressiveness, Gatekeeper would win. Kubewarden (WASM policies in any language) is the elegant dark horse; a smaller community kept it off my cluster. Native Pod Security Admission is free and built in — three levels, zero install. If all you need is "no privileged pods," use PSA and skip a policy engine entirely. I run both: PSA sets the floor, Kyverno does everything PSA can't (mutate, generate, image verification, custom logic). Tetragon vs Falco (and KubeArmor). This is the sharpest call in the stack, and the most honest thing I can say is: I run Cilium, so Tetragon is already right there — eBPF-based, and it can both detect and enforce in one tool, potentially collapsing two of my guards into one. I stayed with Falco + KubeArmor because Falco's rule ecosystem and CNCF maturity are unmatched for detection, and KubeArmor's CRD model is friendlier than hand-driving Tetragon TracingPolicies for enforcement. A leaner cluster could genuinely do this with Tetragon alone — it's on my honest "re-evaluate" list. Grype / Clair / Snyk / Docker Scout vs Trivy. All fine scanners. Snyk has the best developer DX (and a bill); Grype is fast and SBOM-native; Clair is the registry classic. Trivy won because it's all-in-one — CVEs and IaC misconfiguration and leaked secrets and SBOM in one binary — it's free, and my Harbor already scans every push with Trivy natively. One scanner, build-time and registry-time, no new tool to learn. The honest cost of running all four, stated plainly: it's real overhead — two DaemonSets (KubeArmor + Falco), Kyverno's five controllers, and the Trivy operator — and KubeArmor is the finickiest thing in my whole cluster. Its AppArmor annotations have wedged terminating pods, blocked Cilium's own BPF mounts, and confined CSI plugins — each of those is a scar in my notes. Falco will drown you in alerts until it's tuned. Kyverno's immutable-ConfigMap policy has bitten me more than once. If you want one layer to start, start with Kyverno — admission is the highest-leverage gate — and add the others as your threat model demands. Defense-in-depth is a spectrum, not all-or-nothing. But the four together are what turns "I hope nothing's wrong" into "I can prove nothing's wrong." Bringing the Four Guards Together Step back and look at what's now standing watch over an empty house: Layer Guard Mode When it acts Admission Kyverno gate / mutate before the workload enters etcd Runtime audit KubeArmor LSM, audit posture while it runs, at the kernel Runtime detect Falco tracepoint match when it behaves anomalously Continuous Trivy scheduled scan long after it was deployed Four engines, zero shared dependencies — no common operator, no shared CRDs, no single point of failure. If Kyverno's webhook is down mid-upgrade, the kernel layer is still watching. If a zero-day slips past every lock, the cameras still record it. If everything passes today, the Inspector still finds the CVE that lands tomorrow. Defense-in-depth isn't four tools; it's four tools that don't know about each other, so no single failure blinds the whole system. And notice what isn't here: a dashboard to log into, an SSO redirect to configure, a UI to gate. The entire security posture is kubectl get against custom resources. Security you query, not security you click. For a GitOps platform, that's the point — every policy, every rule, every exception is a commit, reviewable and revertible, with no console drift. What's Next? (Don't Miss Out!) The locks are set. The cameras are live. The inspector is on patrol. The house is finally safe to move into — and that's exactly what happens next. With the security system armed first, the tenants can finally arrive: the private Harbor registry, Tekton pipelines, MLflow, Kubeflow, the data platform — every one of them landing in a cluster that rejects a bad manifest at the door, records every syscall at the kernel, alerts on anything anomalous, and never stops scanning its own inventory. That was the whole point of doing this in the empty house. And the two honest gaps stay on the board where I can see them: Falco's alerts still only reach stdout, and KubeArmor is still in audit posture with zero policies. Neither is a surprise and neither is an accident — one needs a webhook in Vault, the other needs a few weeks of audit data before enforcement is anything other than an outage generator. Writing them down is how they stay next actions instead of quietly becoming the permanent state. That's the next episode: moving the workloads in — safely. Follow along so you don't miss how the platform goes from secured and vacant to secured and busy. 🔐 the ArgoCD security-stack ApplicationSet — all five child apps across two sync waves, Synced + Healthy. The whole security system, GitOps-reconciled, in one view.