I set up AI agents for small businesses, and the most expensive bugs I've seen so far didn't throw a single error.
13 runs, 8 leads, 5 silent zeros
Day one of a lead-finding agent: it ran 13 rounds and came back with 8 qualified leads. Each round hit 9 companies in total, the dedupe against our old list dropped 1, and every one of the 8 came with an evidence link explaining why it was a prospect. No evidence, no lead.
Looked great. Then I read the run log.
Five rounds in a row had returned nothing. No exception, no warning, no retry storm. The sites it was searching had flagged it as a bot and quietly served empty pages, so the agent did exactly what it was told: searched, found nothing, moved on.
If I'd only looked at the final number, "8 leads on day one" would have gone into a report and nobody would have asked about the gap.
Green dashboards lie in the same way
Two more from the same month:
- A status panel for our model endpoints showed all green while the three accounts behind it had been logged out for two days. The panel checked that the process was alive, not that a request could actually be served.
- A session cookie said it was valid for more than a year. The server kicked it in under 3 days. We switched to opening the real settings page every 4 hours instead of trusting the expiry date.
Same pattern each time: the health check measured something adjacent to the work, not the work.
Old rules keep running
Long-running agents get a standing rulebook. We cap ours at 140 lines of index, and anything over that has to be deleted or merged. One retired rule wasn't fully removed, and a fresh session obeyed it and stopped dead on a sign-up page. It took a while to realize the agent wasn't broken. It was following orders nobody meant anymore.
Count cost per finished task, not per call
Price per token tells you very little. SemiAnalysis's GDPx board (206 expert-level tasks, numbers I read on Nov 9) lists DeepSeek Flash at about $0.94 per task with a 5.4% completion rate, and Opus 5.5 at about $51.41 per task with 12%. Divide cost by completion and you get roughly $17 versus $428 per finished task. Which one is "cheap" depends on whether your job can tolerate failing and retrying.
What I check now
- Every result carries its evidence. Empty results are logged as a result, not skipped.
- Several empty runs in a row get a human look, even with zero errors.
- Health checks hit the real thing: a real page, a real request.
- The rulebook has a hard size cap, and retired rules get deleted, not commented out.
- Each agent reports one number a day: how many things it actually finished, and what each one cost.
What's the quietest failure you've caught in an agent of yours?
