A developer deployed a customer-support agent that got stuck in a retry loop with a CRM tool. No hard circuit breaker. It spent six hours overnight repeating the same broken call while he slept, and he woke up to roughly a $4,200 OpenAI bill for doing nothing useful. That's the failure mode. Not a dramatic hack — a boring loop with no external stop. Roughly 95% of enterprise generative-AI pilots in 2025 delivered no measurable return, and almost none of those were model failures. Teams optimized the brain and skipped the nervous system. A read-only chatbot is a wiki with better search. A deployed agent does things — writes to a database, calls an API, refunds a customer. That shift from reading to doing is where pilots die. The useful mental model: treat the LLM as a fallible kernel, not a magic box. You don't trust a kernel blindly. You wrap it in checks, limits, and a way to roll back. Agents break three assumptions your ops playbook depends on Non-determinism. The same input can produce different actions on different days. An agent succeeds Monday and fails on the identical request Tuesday. Call it ghost debugging — the bug won't sit still long enough to catch. Design for a 3% to 15% tool-call failure rate as a normal state, not an exception. Cost grows with loop length, not request count. This is the mechanism most guides skip. The agent loops: think, call a tool, read the result, think again. Frameworks append every step and every tool error to the running history, then resend the whole cumulative log on the next call. Token use grows quadratically. A 20-step loop is not twice a 10-step loop — each step re-pays for everything before it. Related trap: past roughly the 40% context-fill mark, answers get worse. Load the window with tool definitions and raw JSON and you're doing your real work in the dumb zone. Every tool you connect widens the blast radius. Which brings us to the controls. Four controls the agent cannot touch Wire these before you touch agent logic, not after. The principle is determinism of enforcement: limits live outside the model, in code it cannot rewrite. An agent told to stay under budget will, often enough, talk itself into one more call. Circuit breaker. Hard cap on retries and loop iterations. Hit it and the agent stops, full stop. Deny-by-default allowlist. The agent calls only tools you explicitly permit. Everything else is blocked, not warned. Confirmation gate. Anything destructive or irreversible — delete, refund, send — waits for a human yes. Cost ceiling. Spend limit per run and per day, enforced by infrastructure rather than by asking the model nicely. Research on action-level privilege control found enforcement cut attack success from 70.3% to 7.3%, and to zero with manual policies. The control plane protects you, not the prompt. Worth being honest about the limit: external controls stop catastrophes. They don't make a weak agent good. They buy you the safety to improve it in production instead of gambling with it. Three identities, not one Most teams give an agent one set of credentials. That's how a single prompt injection becomes a full breach. Split it: The user — who asked for the action. The agent — its own machine identity, separate from any human. The tool token — a scoped, on-behalf-of credential per tool call. In a DORA- or PCI-DSS-bound system, "the AI did it" is not an answer an auditor accepts. There's a quieter risk in the tool descriptors themselves. A study of 1,899 live MCP servers found 7.2% carried general vulnerabilities and 5.5% were open to tool poisoning, where a malicious description hijacks the agent. Pin and version your descriptors. Treat them as supply-chain code, not config. And assume prompt injection — direct, where a user types it, and indirect, where the agent reads a poisoned page or document and follows hidden orders. You cannot prompt your way out of this. Checks run outside the model, in deterministic code. One scan of 5,000 AI-built apps found 60% vulnerable. Ship it in five stages Never all at once. Each stage answers one question before the next earns traffic. Offline eval suite — proves broadly correct on known cases, before any real traffic. Shadow mode — the agent decides, nothing executes. Proves it behaves sanely on live data at zero risk. Canary — a small slice of low-risk real traffic. Proves it survives messy production input. Graduated rollout — widen in steps, confirmation gates on anything destructive. Post-deploy validation — behaviour drifts as inputs and model versions change. Proves it stays correct. The rollback detail teams get wrong: an agent's behaviour is set by four things together — code, prompts, tool catalog, model version. Reverting code while keeping yesterday's prompt is not a rollback. It's a new configuration you have never run. Revert all four as one atomic bundle. Monitor the trajectory, not the endpoint Uptime dashboards tell you the agent returned a 200. They don't tell you the answer was confidently wrong. Three surfaces: Operational — is it running? p95 latency, error rates, cost per run. Cognitive — what did it decide, and why? Full reasoning trace and every tool call, in order. Contextual — what did it see? Exact inputs, retrieved documents, tool outputs. Track task-completion rate, tool-call success rate, tokens per run, and latency. Drift in any of them is the early warning. Almost-right output passes every uptime check. It's the expensive failure mode precisely because completely wrong gets caught — tests fail, the build breaks — while almost right ships and compounds for months. AI-generated pull requests average 10.8 issues against 6.4 in human code, and the tell is often suppression rather than error: one review turned up a file carrying eleven eslint-disable comments where the agent had silenced the type errors instead of fixing them. Three questions before any agent-written change merges: does it reuse what exists or reinvent it badly, does it follow your conventions or invent its own, and can a developer explain it without reading the AI's comments? What it actually costs Inference is rarely the big number. Integration and compliance run 40% to 60% of total deployment spend. Get the data layer wrong and everything else inflates. Pick hosting by workload shape rather than by which is "cheaper": serverless for spiky or unpredictable volume, containers for steady high volume, hybrid if you genuinely have both and can afford to run two systems. Cloud is rented elasticity — if your load is steady, you're paying a premium for flexibility you aren't using. Then cap the variable part: token budgets per run and per day enforced in infrastructure, semantic caching so repeated questions don't re-pay the model, and model tiering so the expensive model only runs on the steps that need it. The ten-line go/no-go Empty box means you're shipping a demo: Eval suite passes. Staged rollout ready — shadow, canary, graduated, with gates. Circuit breaker live, enforced outside the model. Cost ceiling set in infrastructure. Tool allowlist is deny-by-default. Three-identity auth. Trajectory logging on. Atomic rollback bundle — code, prompts, tools, model version. Spec-first verification with the three-question test. Named incident owner — a specific human, not a team alias. If you went quiet on three or four of those, you're in the same place as most teams shipping their first production agent. Writing the agent is the cheap part. The gap between a working demo and something you trust at 2 a.m. is everything above. Full guide — architecture patterns, identity delegation, cost modelling, and deploying onto a legacy core: teamvoy.com/blog/ai-agent-deployment-best-practices Written by Taras Voytovych, Founder & CEO at Teamvoy. More engineering writing at teamvoy.com/blog.