A benchmark with flawed tests measures the tests. So I built two suites — one clean, one with 12 flaws planted in the tests — and watched a do-nothing agent start scoring points while my reference agent lost 8.3 points to the tests themselves.
This is the full experiment: calibrating the graders first, pre-registering the audit bar, catching 11 of 12 flaws — and being honest about the one that got away.
The setup
Local, deterministic, no API keys — a lab for auditing your eval before your eval audits your agent. The setup is a clean 60-task suite (arithmetic, extraction and classification; gold answers cross-checked by an independent second implementation) and an identical twin with 12 flaws planted in the tests.
What the flaws did
- Null agent (outputs nothing): 0/60 on the clean suite, 2/60 on the flawed suite
- Echo agent: 0/60 on the clean suite, 4/60 on the flawed suite
- Reference agent: 60/60 on the clean suite, 55/60 on the flawed suite — 8.3 points lost to the tests, not the tasks
Calibrate the graders first
Three graders were validated on 150 labelled items before any audit run: exact-match 80.0%, normalized 100%, rubric 99.3%. The point is stated plainly in the lab: an uncalibrated grader is just another flawed test.
The audit
The audit bar was registered before the run: at least 10 of the 12 flaws caught, at most 3 false flags. The audit caught 11 of 12 planted flaws with zero false flags. The one miss, flaw-09, had its gold and its source document disagreeing (300 vs 315) from the same table — shared-source flaws are the hard class, and that is exactly what to test for next.
If you build or trust agent benchmarks, audit the proof itself, not just the score.
