What Fact-Checking My Own Claude Code Guide Taught Me About Verifiable AI

Most talk about verifiable AI is about the model: proofs of inference, attested hardware, reproducible weights. That matters. But most of the AI output people act on every day isn't a model call they could audit. It's a claim: in a doc, a README, an answer, a config someone copied.

I write about Claude Code. In February, I published a long guide to building with it, plus a companion GitHub repo of the configs I actually use. The repo is the part a reader can copy straight into their own setup, which turns out to matter. As Veera Ravindra Divi put it here on HackerNoon, Claude Code doesn't ship production code: your config does. A wrong config does more damage than a wrong sentence.

Since then I've re-checked every claim in it twice, once in August against Claude Code v2.1.234 and again this month against v2.1.311, the second time with four Claude Code agents working in parallel. I expected stale facts. What I didn't expect was ten claims that were wrong on the day I wrote them. One of them was a correction from the August pass.

This month, the agents did most of the checking. Some of their findings looked like errors until I traced the source, and some of my old claims looked fine until I did the same. Either way, what mattered was whether a verdict could be checked by someone other than whoever produced it.

What "Verifiable" Has to Mean for AI Output

A verified claim needs four things. I learned each one by watching a check fail without it.

Property	Question it answers	Failure without it
Source	Where does this come from?	Plausible numbers nobody measured
Version	True as of when?	"Corrections" that were never true
Reproducibility	Can someone else re-run the check?	Works on my machine, fails on yours
Independence	Is the evidence separate from the claim?	A checker citing the article it's checking

None of these is about the model. You can run the best model available and still get output you can't verify. You can also run a cheaper one and get output anyone can audit, if the process makes it leave a trail.

The Ten Claims That Failed

From the August pass:

1. "Delegate Mode (Shift+Tab)." It never existed. Shift+Tab cycles permission modes. I'd described a feature that sounded right in three different files.
2. @-imports defer loading. They don't. Imported files are expanded into context at launch. Imports organise your instructions; they don't save context.
3. An eight-layer memory hierarchy. It had ten. I'd missed the managed-policy CLAUDE.md entirely, a layer that existed the whole time and can't be excluded.
4. "Keep CLAUDE.md under 500 lines." That was my own guess. Anthropic's published guidance is under 200.

From the October pass:

5. "Before v2.1.179 the default display mode was auto." This one hurts most, because I added it in August as a correction. No changelog entry supports it.
6. "Forks cannot spawn further forks." Unverified. What is documented is that forks count toward the three-level nesting cap.
7. "Auto-compaction triggers at about 95% of context." There's no fixed percentage. By default it compacts at the model's context limit, about 967K tokens on the 1M-context models, and /autocompact sets an earlier trigger.
8. "A skill costs about one line of context at startup." The real cost is the skill's description plus its "when to use" text, capped at 1,536 characters per skill. The whole listing is budgeted at 1% of the context window.
9. "Unspecified teammates inherit the lead's model." The lead's model is the last fallback, after the spawn prompt, the agent definition's model and CLAUDE_CODE_SUBAGENT_MODEL.
10. "Fable investigates before acting and verifies its own work." No official source says that. Anthropic describes Fable 5.1 as being "for demanding reasoning and long-horizon agentic work". I'd absorbed a paraphrase and repeated it as a capability.

Every one of these is fluent, specific, and plausible. That is the kind of claim language models produce easily, and agents repeat confidently. It is also the kind a reader can't check without a source.

Source: Unsourced Precision Is Unverifiable

Look at numbers 5, 7 and 8. A version number, a percentage, and a line count. Each one survived review because it was specific.

"Compaction triggers at about 95%" sounds like someone measured it. "Before v2.1.179" sounds like someone read the changelog. Nobody had. The precision did the persuading, and the precision was invented.

My rule now: every number gets a source next to it, or it gets rewritten as a range or removed. When the first correction pass suggested "fixed around v2.1.63" for a GitHub issue, there was no changelog entry behind it. That number came out too. The guide now says the behaviour is no longer reproducible and tells readers to check it themselves with /context.

Version: Sources Describe Specific Systems

Checking against official documentation sounds definitive until two official sources disagree.

In August, Claude Code's subagents page still listed a 200-subagent-per-session cap. The changelog said it was removed in v2.1.224. The docs hadn't caught up.

In October, Anthropic's API models page listed Sonnet 5.5's default effort as high. Claude Code's model configuration page says the 5.5 models default to medium. Both are right: one describes the API, the other describes Claude Code. An agent that reads only one of them will "correct" a true claim into a false one.

So provenance has to include which system and which version a source describes. For anything version-specific, I treat the changelog as the primary source. When sources disagree, the verdict says which surface each one describes rather than quietly picking one.

Reproducibility: Run It, Don't Read It

The worst error in the February guide isn't on the list above. Nothing I wrote was false; I left something out.

Agent Teams is experimental and disabled by default. It needs CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1. I never mentioned it, because the flag had sat in my own settings.json so long I'd stopped seeing it. Every Agent Teams pattern in the guide worked on my machine and did nothing on yours.

Reading docs can't catch that. The docs are right, and the claim is right. What's missing is the environment. The only check that works is re-running the instructions somewhere that isn't your machine.

The same applies to the repo's example configs. This month's pass ran them instead of reading them, and two hooks were broken:

• The PostToolUse hook that runs Prettier piped file paths through xargs, which splits on spaces. My own project lives in a directory called "Ben Ryan".
• The PreToolUse guard against destructive commands blocked git push --force-with-lease, the safe variant. It missed rm -fr / and rm -r -f /. It blocked rm -rf on any absolute path, including /tmp/build.

The README told readers to test the guard by asking Claude to run rm -rf /tmp/something. If the hook is wired up wrong, that command runs. The demo now pipes test JSON straight into the script, so anyone can reproduce the check without risk:

echo '{"tool_input":{"command":"rm -rf ~"}}' | .claude/hooks/block-destructive.sh 

The rewritten guard passes 18 test cases, including the ones that should be allowed. Testing only what you mean to block is how you end up blocking the safe cases too. Sonar's walkthrough of Claude Code hooks covers the stdin and exit-code contract in more depth.

The same lesson shaped the Claude Code toolkit I built for checking copy traders: rules in a prompt help, but missing abilities are stronger. A guard you can test is worth more than an instruction you hope the model follows.

Independence: Circular Verification Is No Verification

This one came from my content pipeline rather than the guide, and it's the clearest failure of the four.

My pipeline has an automated fact-checker that searches the web to verify claims. In September it "verified" two statistics by finding them on benryan.com.au, in the very article it was checking. The search ranked my own page, the summariser quoted it back, and two claims attributed to sources that never said them passed as confirmed.

Nothing in the check was technically broken. The search worked, and the model summarised accurately. The evidence just wasn't independent of the claim. If your verification can find your own writing, it will eventually use it to confirm itself.

A Verification Protocol You Can Copy

Here's how the October pass ran. It works for any AI-assisted docs, not just mine.

1. Split the work by domain. I gave four Claude Code subagents one area each, in parallel: memory and context, skills/plugins/hooks, parallelism, and models and pricing.
2. Rank the sources. The changelog first, then the official docs, then the installed CLI's own --help output, then GitHub issues.
3. Exclude your own material. No checker could cite my site or my GitHub.
4. Make "unknown" a real verdict. Anything unconfirmable is UNVERIFIABLE, never CORRECT. "I couldn't find it" is not "it's fine".
5. Require a trail. Every row returns location, claim, verdict, what's true now, and the source. A verdict without a source is discarded.
6. Spot-check the surprises yourself. One checker reported a Haiku 5.5 price of $0.10 per million input tokens, a 10x drop. I read Anthropic's pricing page directly before believing it. It was right.
7. Publish what changed, and why. The repo's changelog records each pass with the version it was verified against, and separates "wrong at time of writing" from "changed since".

Step 6 caught something useful, too. One agent reported that effort now defaults to medium on the 5.5 models. That looked like a conflict with the API docs until I read Claude Code's own page. If you moved to the 5.5 models and Claude feels a bit shallower, check /effort before you blame the model.

Step 7 is the one most people skip. Quietly editing a doc makes it impossible for readers to know which of your past claims to distrust. A public correction log is the cheapest audit trail there is.

What This Means for Verifiable AI

The agents did most of the work here, and they did it well. But "an AI checked it" carried no weight on its own. What made the result trustworthy was a trail any reader could follow: a source per claim, a version per source, checks that re-run, and evidence that isn't the claim itself.

That is the same standard we should want from AI systems in general. As agents write more of the docs, configs, and answers we act on, verification can't stay a property of the model. It has to be a property of the output, and the output has to be open enough that someone else can check it.

The full guide is updated for v2.1.311. The companion repo has the configs, the fixed hooks, and a changelog listing every correction from both passes. If you find something I got wrong a third time, open an issue. Apparently, I'm due.