A coding agent got a ticket: a customer with exactly 100 points can't redeem the 100-point reward. It found redeemable() in src/loyalty/redeem.js, changed a > to >= and reported it fixed. Four turns, one line, no test.

The same comparison was also in src/emails/points.js. It stayed wrong.

That wasn't bad luck. On that ticket, with no tool, the second copy was fixed in 0 sessions of 5. With one note at the moment of the edit, in 15 of 15.

I'm building Ockham, a plugin for Claude Code, to find out what you can and can't get from a coding agent by telling it things. These are the measurements so far. Most of what they say applies whether you use Ockham or not.

The same ticket, without Ockham and with it

How it's measured

Every number below is one headless agent session on a fresh copy of a project of about forty source files. The agent gets a ticket and nothing else. What it leaves behind is scored by checks it never saw: whether it works, whether it holds on input the ticket left out, whether it reached every place the bug was in, whether it reused what the project already had, whether it left a test where the logic needs one.

There are 44 tasks: 39 to tune on and 5 held out. Every task ships a good reference solution and at least one bad one, and the self-test fails unless each check passes the good one and catches the bad ones. The reference arm is ponytail (MIT), the ruleset that made "make the agent write less" popular. Ockham is written from scratch.

Lesson 1: rules move a lot, but not everything

Seven rules, 314 words, tuned one change at a time. On the 22 tasks they were tuned on, five sessions a task:

No tool ponytail Ockham rules
Passed every hidden check 87/110 86/110 105/110
Held on input the ticket left out 103/110 100/110 110/110
Reached every place the bug was in 14/25 11/25 20/25
Left a test where the logic needs one 0/70 7/70 60/70
Cost per session $0.030 $0.036 $0.054

The first rule says which half wins: "the smallest change that does the whole job. If they pull apart, the whole job wins." It's there because of what a plain "do less" rule does. In the baseline, the agent with that rule reused everything, then left out part of the job and reported the omission as a saving: a CSV export with no quoting, five times out of five.

What no wording fixed was the copy of a bug in another module. One wording made it worse: the agent started saying it had looked for other copies. Four sessions of five closed with a search or a check that the project contradicts, and three of those had run nothing.

If you keep one thing from this post: a rule can't make an agent find what it doesn't know exists. That's a job for a program.

Lesson 2: tell it at the moment of the edit

So Ockham keeps an index of the project. When a session starts, the agent is told which helpers the files share and where the tests are. When an edit replaces a line that another file also has, it's told which file and which line.

On six tasks nothing had been tuned on:

No tool Rules alone Rules and index
Passed every hidden check 21/30 26/30 27/30
Reached every place the bug was in 9/15 14/15 15/15
Cost per session $0.030 $0.050 $0.047
Time per session 16 s 25 s 21 s

Across every run so far, the note about a copy went to 70 sessions, all on tasks where a bug is written twice, and all 70 fixed both places. No false warning in 546 sessions with the index.

The part I didn't expect: the index doesn't make a session cheaper. It makes it shorter. A shorter list and no list at all were tried, and they cost the same. What the text saves on every call, the agent spends on more calls.

Lesson 3: check once at the end, and say what you found

When the agent means to stop, Ockham compares the files the task changed with what they were. Seven checks: a dependency the request didn't name, a second version of something another file exports, a copy that's still wrong in another file, a check the code used to make and no longer makes, risky logic with no test, tests not run after the last change, and a summary that says the tests pass when none ran.

If one fires, the agent is told once and goes on.

  • Of 64 sessions stopped so far, 63 fixed what they were told. The other one was right: the check was wrong.
  • On five tasks with a bug written twice, a session with no tool fixed the second place once in 25. With the checks alone, 15 of 15.
  • Two of the mistakes the checks look for, this model doesn't make. In 1,673 sessions nobody added a package or dropped a check while shortening a function.

What it costs, and what's open

It costs more: about $0.05 a session against $0.03 with no tool. Most of that is the tests it asks for. Without the rule that asks for them, a session costs a quarter less and passes the same hidden checks. I kept the rule, because "checked" is the promise. OCKHAM_TESTS=off leaves it out.

Still open:

  • One task nothing helps on: a redirect after sign-in that three sessions of five leave open to another site, with the rules or without.
  • A second version of a helper written inline is rarely caught at the end: 1 of 7 planted ones.
  • Review, memory and the status band (phases 5 to 7) aren't built yet, and the Codex side is still open.

Try it

In Claude Code, with Node.js 20 or later:

claude plugin marketplace add Octonove/ockham claude plugin install ockham@ockham 

Every table above, the harness and the 44 tasks are in the repo: https://github.com/Octonove/ockham

If you run the benchmark on your own setup and get different numbers, I want to see them.