97 of 99 Sounds Great. It Would Have Been a Bad Release Number.

From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is the coverage section of the v0.1.0 field test report.

AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.

The final field test report for my agent policy engine says 97 pass, 0 fail, 0 declared, 0 not run. Ninety-seven out of ninety-seven. That is a release number I am proud of. It is also a number that only means anything if I show you the two cases that are not in it.

They are the most honest part of the release. Both are named, with reasons and re-entry conditions, in the field test report (the master table note and Field Notes 13).

What a pass rate hides

My plan originally had 99 cases. Two of them were Tier-1 integrations: FT-T1-CURSOR and FT-T1-GEMINI. Test the real Cursor agent loop against the policy engine. Test the real Gemini CLI against the policy engine.

Neither can run on my machine. Not "I ran out of time." Cannot run, for structural reasons:

  • Cursor is a GUI IDE, a VS Code fork. Its agent loop and its permission surface live inside a desktop Electron app driven by a human. There is no supported headless surface, no server mode for the permissions.json flow, and no GUI host on this box. There is no experiment to run.
  • Gemini CLI is installed, and the sanctioned local path (gemini gemma setup) genuinely works. I got a real LiteRT-LM binary and a local gemma model serving on port 9379. And then the CLI crashed at startup with IneligibleTierError, the OAuth tier gate running before any model routing. I read the bundle source: the gemma router is classifier-only. Intent classification goes to the local 1B model. The main LLM always goes to Google-hosted Code Assist, which is sunset for this CLI build. No proxy fixes a client-side auth gate.

Here is the fork in the road. If I quietly dropped those two cases and published the remaining 97 as "97/97, 100% pass," I would have been technically accurate and completely dishonest. Count them honestly as untested and the same release reads 97/99, about 98%. Either number looks green. Neither one tells you that the two missing cases are the two hardest integrations I claimed I would test. The pass rate was never the problem. The denominator was.

The vocabulary that clarified it

Somewhere around the third time I caught myself about to write "didn't get to it," I codified the rule into the field-test vocabulary. Every case gets exactly one of:

  • pass: it ran, and an independent gate re-derived the assertion from the evidence on disk. Not exit code zero. Re-derived.
  • fail: it ran and the assertion did not hold.
  • declared: it cannot run here, and I wrote down the precise reason, what is covered instead, and the condition under which it becomes runnable.
  • not run: nothing yet. Visible. Counted.

The load-bearing distinction was between unsupported here and passed here. During the investigation, that vocabulary kept me honest about what I knew, what I could prove locally, and what would need a different host or a future vendor build. For Gemini, that meant the exact crash, the classifier-only finding from the source, and the local parity stand-in that did run. For Cursor, it meant admitting there was no headless surface to exercise on this box.

That gave me a complete answer while I was still investigating. The shipped release took the stricter step: Cursor and Gemini were removed from scope entirely, not left hanging as permanent declared rows.

Why this costs me something

The hard part is publishing the limitation plainly. Anyone reading the report can see exactly which integrations were exercised live and which were not claimed at v0.1.0. A marketing page would say "works with your favorite agent CLIs." My report says: Claude Code and Codex CLI, exercised live, both passing. Cursor and Gemini CLI: not claimed as supported at v0.1.0, here is why, here is what would change it.

I removed them from the plan, the registry, and the PRD rather than leaving them as eternally skipped rows. The scoping decision and the re-entry conditions are written into the plan itself, not just my private notes. Removing a feature from scope because you cannot test it is slower than claiming it. It is also the only version of the sentence I can defend.

What worked

  • The four-state vocabulary. Two days into the sweep I could not tell you which "didn't run" was flaky and which was impossible. The vocabulary forced me to find out.
  • Clear intermediate statuses while investigating. Writing down the reason, the lower-layer stand-in that does run, and the re-entry condition made it much easier to decide the final shipped scope honestly.
  • Writing the report by hand. Every number in it was typed by a person who knows where it came from. The generator is explicitly banned from touching it.

What did not work

  • My initial instinct, twice, to leave untestable cases as "skip" rows. Skips rot. Nobody audits a skip backlog.
  • Trusting "local mode" in a vendor's marketing. Gemini's local mode is real, the install works, and the main loop still cannot use it. Partial local is a trap worth naming out loud.

More detail and evidence

I now read every "100% pass" dashboard with one question: how many cases did it take to get to that denominator? How do you handle the things you cannot test in your reports, skip them silently, or declare them loudly?