A front-end release broke my agent evals overnight. The agent had not changed

My computer-use agent runs a nightly suite against a staging copy of an internal web app. It reads the screen from screenshots, clicks, types, and files a form at the end. One morning the suite was a wall of red.

I spent an hour reading trajectories before I checked the obvious thing. The front-end team had shipped a release the evening before. A button had moved, a field had been renamed, and the flow had grown a confirmation step. My verifier was a stack of hard-coded checks on the final state, and every one of them was now looking for a page that no longer existed.

The agent was fine. The grader was wrong. And a grader that can be wrong in that direction can be wrong in the other direction too, which is the direction that actually scares me.

A computer-use agent's output is a trajectory, not an answer

With a chat model you grade one output. With a computer-use agent you grade a path: a sequence of clicks, keystrokes, and scrolls across a moving interface.

Outcome-only grading reads the final screenshot and calls it a day. The problem is that two runs can end on the same confirmation page while one of them was clean and the other did something reckless on the way there. A run can reach the right final state through a wrong path, and a final-state check will score both runs identically.

Once I understood that, my wall of red looked like the cheap failure. The expensive failure is the false pass, where the check finds the record it was told to find and never notices what else happened.

Programmatic verifiers fail in both directions

My hard-coded checks were fast, deterministic, and nearly free to run. I am not giving them up. But they have two failure directions and I had only ever planned for one.

False fails come from UI drift. Buttons move, fields get renamed, flows grow steps. Every drift costs you a morning of triage and another patch to the suite, and the suite slowly becomes a second codebase you maintain.

False passes come from rigidity. A check that asks "does the order exist in the database" passes when the order exists and is wrong. A check that asks "did we reach the confirmation page" passes when the form was submitted twice on the way there. The check is answering its own narrow question, and the question was never "did the agent do the task well".

A model judge is not automatically the fix

The tempting move is to replace the brittle checks with an LLM or VLM judge that reads the screenshots and the trajectory against a rubric. It handles UI drift without a code change, and it can reason about the whole path instead of only the destination. WebVoyager, one of the public computer-use benchmarks, uses a vision model to score runs for exactly this reason.

But model judges carry documented biases. The research on LLM judges (arXiv 2306.05685) shows they favor longer answers, lean toward their own style, and can be swayed by option order. A vague rubric makes the score wobble between runs on identical behavior. And every judged run adds latency and spend that a programmatic check does not.

So the judge did not replace my checks. It joined them.

Layer the graders, then grade the graders

The structure that survived in my suite has two layers. Hard programmatic gates own the non-negotiables: no charge without a matching order, no success claim without the record. The judge owns the fuzzy middle: was the path reasonable, did the agent do what the task meant rather than what the rubric letter said.

The part I had never done before was validating the verifier itself. The core question is agreement: when my verifier and a careful human grade the same runs, how often do they land on the same verdict? If they disagree a third of the time, the verifier is not grading. It is guessing.

Raw agreement flatters you, because two graders agree by chance more often than you would think. Cohen's kappa corrects for that chance agreement, so that is the number I track against a small hand-graded gold set.

Two more habits came out of this verifier design guide and earned their keep. Mutation testing: feed the verifier trajectories you know are bad and confirm it rejects them, because a grader that passes everything looks exactly like a healthy suite. And canaries: plant a few gameable runs where the agent could reach the success state without doing the task, and see whether the verifier notices. Deterministic checks and judges get gamed differently, which is the real argument for running both.

The full walkthrough, with the validation checklist and a worked kappa example, is in this verifier design guide.

The mistakes I would not repeat

• Grading only the final state. The trajectory is the output. A clean destination does not certify the path.
• Treating the red wall as the bad day. False fails cost mornings. False passes cost incidents. Plan for the second one.
• Swapping brittle checks for an unvalidated judge. That trades visible breakage for invisible bias.
• Never testing the tester. Mutation runs and a kappa score against hand-graded runs turned my verifier from an opinion into an instrument.
• Letting the suite rot between audits. Scores that climb while real task success stays flat are the quiet signature of a grader being gamed.

Here is my question for you: when your agent eval goes green for a month straight, how sure are you that the agent earned it and not the grader? I was sure, right up until I hand-graded twenty runs and my verifier disagreed with me on four of them.