I asked an agent to tighten one slow query. It came back, said done, pointed at the single test I cared about, and the change merged. The next morning I diffed the branch and found the actual fix was three lines. The tree had also carried along a commented-out alternative query, a debug script that printed a stack trace, and a half-finished refactor it had abandoned on the second pass.
The agent's 'done' meant the target test passed. It did not mean the working tree matched what it claimed. That gap is the failure, and it is far more common than the dramatic bugs.
Why the scratch survives the loop
An agent works by trial. It edits, runs a check, edits again, and on each pass it leaves a little scaffolding: a backup file it 'just in case'd, a block it commented out to compare timing, a console.log it meant to remove, an experiment branch saved as a sibling file. When the target check finally goes green, the loop stops — it has no step that re-reads the tree state from the outside and asks, 'is this everything I intended to ship?'
So the last green run is where the review starts, and the scratch is already inside the diff.
| What the agent leaves | Why it rides to review |
|---|---|
| Commented-out alternative | It compared two versions and never deleted the loser |
| Debug logging | Added to diagnose, forgotten once green |
| Untracked helper file | Created during exploration, never git cleaned |
| A second approach half-applied | Pivoted mid-task, left the first attempt in place |
The reviewer's blind spot
Reviewers read the diff to judge the intended change. They trust that the diff is the change. A commented-out block or an untracked scratch file looks like the author's local noise, not a defect, and it gets waved through. The problem compounds on a large diff: the more the agent touched, the easier it is for one stray hunk to hide.
An agent that can produce a convincing target diff can also produce a tree that is not that diff. 'LGTM' on the hunk you expected says nothing about the hunks you did not.
Three cheap checks before accepting done
- Require a clean tree as part of the definition of done. Before you review, look at
git statusand the full diff against the branch base. If there is an untracked file or a commented-out alternative, the agent is not finished — it just stopped iterating. - Ask for the final diff on its own, with no unrelated files. When the agent lists what it changed, every path in that list must appear in the diff and vice versa. A path it did not mention but that is present is exactly the scratch to remove.
- Strip debug scaffolding by rule. Grep for the debug idioms your codebase uses (unconditional logging, TODO markers,
console.log, debug print, backup suffixes) and treat them as uncommitted unless they are the point of the change.
None of this needs a fancier model. It needs the agent to verify its own claim about the end state of the tree, not just the one test it was optimizing. 'Done' should mean the working tree is the intended change and nothing else.
How much of the agent's last merged diff was actually scratch you waved through, and do you have a step that checks the tree is clean before review?
