This post draws on a talk co-presented with Julien Semaan of Kubex at KubeCon + CloudNativeCon Europe 2026; GitOps Gone Wild: Hardening Delivery Pipelines for the AI Era
Most platform teams are having meetings to discuss whether they should let coding agents open pull requests against their infrastructure. The answer has nothing to do with agents. It has to do with whether your platform can absorb a confident, well-formatted, completely wrong change, and if it can’t, you had that problem long before anyone started using an agent.
Take the cautionary tale that recently went viral: an AI coding agent accidentally wiped out a production database for a SaaStr project right in the middle of a code freeze. To add insult to injury, it generated roughly 4,000 fake user accounts and insisted a rollback was out of the question, even though it wasn’t. Replit’s CEO had to issue a public apology, and the story dominated engineering feeds for a week. (AI Incident Database, Incident 1152)
I manage a team at Akamai that builds our internal Kubernetes platform. Our job is to meet Akamai’s change safety, security, and compliance expectations while giving internal developers an experience they actually want to use.
Most of what makes a platform safe for agents isn’t new. It’s the stuff we should have been doing for years. People make mistakes, and agents can make those same mistakes, often much faster. That’s why you need controls in place, regardless of who’s making the change. Those controls must be put into a platform from the start, so whether it’s a new hire, another team, or an AI agent, the same rules apply.
GitOps is an amplifier
Most cloud native platforms assume GitOps. Git holds the desired state for applications and infrastructure. Changes arrive as pull requests rather than as edits to a live cluster. Controllers apply them, and the system continuously reconciles running state against desired state. The benefits are familiar: consistency, auditability, repeatability, and speed.
The downside doesn’t get talked about as much. Automation doesn’t know the difference between a good change and a bad one. In a GitOps system, one mistake can spread across your infrastructure in seconds, sometimes before anyone has even had a chance to review what changed.
Agents do not introduce a new category of risk here. They change how quickly mistakes can happen and how far those mistakes can spread. Your team of six can now open pull requests at the rate of a team of thirty. It still reviews at the rate of six, and each pull request carries less context than it used to, because the author did not have to build a mental model of the system to produce it. Review has always been the last line of defense, and now it’s the scarcest thing in the pipeline.
Resilience has layers
No single safeguard can catch every mistake, especially as AI-generated changes enter the development process. The more automation that is introduced, the more important it becomes to build protections into the systems where those changes are tested, configured, deployed, and reviewed. We approach resilience as a series of layers, each designed to catch problems that might slip past the others.
1. Testing: Catch mistakes early
Testing is the first layer, and the cheapest place to catch a mistake. Test-driven development and spec-driven development share the same instinct: define what good looks like before building anything. Once you write the expected logic first, it matters less who or what wrote the implementation.
End-to-end tests stack the individual checks into something that resembles a system. Canary environments carry more weight for us than for most teams, because we are creating clusters constantly and need to watch a change behave in a full cluster before it reaches a tenant’s.
2. Configuration and abstraction: Limit the opportunities for error
The second layer narrows the scope of what we have to configure and review. Give someone raw Kubernetes YAML and every field is in scope, which means every field is a place to hide a mistake. So we build strong opinions directly into the platform’s configuration, providing sane defaults that work out of the box while still letting end users override them when specialized needs arise.
For infrastructure and platform capabilities, we can use kro, the Kube Resource Orchestrator, a subproject of Kubernetes SIG Cloud Provider. Kro turns a group of related Kubernetes resources into a single custom API. You write a ResourceGraphDefinition naming the underlying resources, native or custom, along with the dependencies between them, and kro generates a custom resource whose schema exposes only what you choose to expose. Its controller works out the correct order of operations and manages the underlying objects from there.
For us, the schema a tenant sees is short: Kubernetes version, node count, and region. Observability, ingress, policy, and everything else that turns a cluster into a platform cluster are composed underneath. A tenant fills in a few fields, and kro produces the dependent services configuration.
For applications, use something like Score, a CNCF Sandbox project, where developers declare their workload and needs, and the Score transpiler and its rendering layers handle the complexity. Crossplane provisions the actual cloud resources, and Argo CD keeps running state converged on Git.
Two properties make this approach particularly valuable. First, a transpiler is deterministic: a given input produces a known output, and it has its own tests and release process. An LLM is not deterministic and will improvise when a manifest calls for a field it has not seen. Second, adding a capability the simplified spec doesn’t express becomes a change to the transpiler, developed on its own timeline, rather than a new field slipping into an operational pull request. Reviews get to focus on whether the intent is right.
Over-abstraction is a real failure mode, so the goal is to standardize access to the cloud rather than hide it. Tenants can read the generated manifests, and they know they are running Kubernetes; what they skip is maintaining the thousands of lines of YAML around their application. Working out where we could be opinionated and where we had to leave room took about three and a half years and a great many CRD versions.
3. Kubernetes safeguards: Put built-in protections to use
The third layer is Kubernetes itself, and it is the layer most teams have already paid for and never switched on. Least-privilege RBAC should be table stakes, and it lets us feel confident giving an agent or a new engineer a scoped credential. Rollout strategies with meaningful health and liveness probes make change gradual and observable even when we did not author it. Pod disruption budgets keep a highly automated system from taking availability with it during scaling or node churn. Resource finalizers have been my team’s workhorse: think of this as a preflight checklist that must pass before something is allowed to disappear. And Kyverno, which graduated within CNCF in March 2026, acts as air traffic control for the API: nothing reaches the cluster without meeting the platform’s expectations.
4. Cloud provider protections: Prevent destructive changes
The fourth layer is the cloud provider. Most major providers offer some form of deletion protection at the resource level, and Crossplane’s own Usage resource can both reject a deletion and enforce the order in which dependent resources come down. This is the final stop before a mistake becomes an incident.
5. Human code review: Keep experts accountable
Finally, human code review remains a critical layer of this process. Even with strong automation and deterministic transpilers, Git merge requests should enforce mechanisms like CODEOWNERS and linear merge histories. This ensures that domain experts retain oversight, pull request accountability stays clear, and no change bypasses human inspection before reaching production.
Design for permission, not trust
Four layers, each with its own configuration surface, each needing to be switched on for the right resources at the right moment. Leave them as four separate decisions, and they get applied inconsistently, which is where incidents come from. So we collapsed the destructive cases into one decision.
We can configure a Kro cluster definition to carry a protected field. When it is set, the platform:
- turns on the cloud provider’s deletion protection for the underlying infrastructure
- refuses changes to the cluster’s region
- attaches custom finalizers to every resource kro composes
- activates Kyverno policies that block mutation and deletion of critical objects
The enforcement is important, but what matters more is the shape of the diff. An agent that decides the cluster would have better availability in another region cannot simply change the region. Neither can a person. The only path forward is to set protected: false first, and that is a one-line change sitting at the top of a file, impossible to miss in review.
This is Zero Trust applied to infrastructure automation. Rather than deciding whether an AI agent is trustworthy enough to modify cluster definitions, we design the system so that neither agents nor humans can make destructive changes without explicitly requesting permission. Trust isn’t assumed, and the safeguards don’t depend on which model is making the request.
Growth from our Mistakes
Engineers learn from mistakes, and so do pipelines and use of AI in the development lifecycle. What we want is for the expensive mistakes to be hard to make by accident, and for the blast radius of the rest to be bounded and legible.
One non-technical thing helped as much as any control on the list. The CNCF Platform Engineering Maturity Model calls the goal intrinsic pull: the point at which teams adopt the platform because it is the fastest route to what they want. Past that sits participatory adoption, where the people using the platform start contributing to it. This is represented by an enabled ecosystem where teams build across each other’s subject matter expertise: security contributing policy, observability contributing composable pieces, and application teams contributing the abstractions they wish existed. A platform team that owns every capability eventually becomes the bottleneck it was created to remove.
The guardrails mentioned in this post are essential for ensuring that contributions from colleagues or agents are safe to accept. The lessons learned over the past decade of building Kubernetes-based distributed systems and platforms should help us adopt AI tools more quickly. We achieve this by embedding safety measures directly into the platform’s structure and incorporating appropriate checks throughout the process.
