Falsify Before You Read: A Claim Registry That Runs Before Anyone Reviews Model-Drafted Docs

The fastest way to make model-drafted documentation trustworthy is not a better prompt; it is a script that tries to disprove every factual sentence before a human opens the file. In the workflow below, drafting and verification stay as separate steps with separate artifacts, and the human review queue contains only sentences no script could settle. The claim registry and the runner are the original artifacts; the process around them is what keeps review attention pointed at intent instead of syntax.

The review-order problem

Most teams verify documentation in the wrong order. A reviewer opens a model-drafted page, spends the first ten minutes checking whether a config key actually exists, and only then starts reading for meaning. Those ten minutes are mechanical work, and mechanical work should fail in CI rather than inside a person's head. When a page reaches review already falsified, the reviewer is left with the questions that automation genuinely cannot answer.

There is a second-order effect worth stating plainly. Model-drafted pages fail in a small number of predictable classes: stale symbol names, invented parameters, plausible but wrong runnable snippets, and confident rationale that nobody can source. Each class needs a different mechanical check, so a single generic review pass hides the fact that one class is cheap to automate while another is almost impossible to verify at all.

Step 1: declare claims in a sidecar file

Do not ask the model to annotate its own output; treat the prose as data that something else checks. For each generated section, write a sidecar file naming the sentences worth falsifying and stating which check applies to each one.

# docs/claims/retries.claims.yaml section: api/retries source: docs/api/retries.md claims: - id: config-key class: config_key check: {type: toml_key, file: config/client.toml, key: retries.max_attempts} on_fail: block - id: signature class: signature_match check: {type: python_signature, module: backoff, symbol: delay, params: [attempt]} on_fail: block - id: snippet class: example_runs check: {type: run_block, fence_index: 1, timeout: 5} on_fail: demote - id: why-capped class: intent check: null owner: '<team-or-person>' 

The owner field is mandatory whenever check is null, because an unfalsifiable sentence without a named owner is an orphan claim. The runner treats a missing owner as a blocking error rather than a warning, which keeps the review queue from filling with sentences nobody has agreed to defend in public.

Step 2: register one checker per claim class

Every checker returns a pass/fail flag plus a short reason string, and nothing else. The reference implementation below is roughly eighty lines and covers three classes: config keys in TOML, imported function signatures, and runnable fenced snippets.

#!/usr/bin/env python3 '''docfalsify.py - try to disprove documentation claims before review.''' from __future__ import annotations import importlib, inspect, re, subprocess, sys, tomllib from pathlib import Path CHECKS = {} def check(name): def register(fn): CHECKS[name] = fn return fn return register @check('toml_key') def toml_key(c, root: Path): path = root / c['file'] if not path.exists(): return False, 'missing ' + c['file'] node = tomllib.loads(path.read_text(encoding='utf-8')) for part in c['key'].split('.'): if not isinstance(node, dict) or part not in node: return False, 'key ' + c['key'] + ' not found' node = node[part] return True, 'key present' @check('python_signature') def python_signature(c, root: Path): sys.path.insert(0, str(root)) mod = importlib.import_module(c['module']) obj = getattr(mod, c['symbol'], None) if obj is None: return False, c['module'] + '.' + c['symbol'] + ' missing' params = list(inspect.signature(obj).parameters) missing = [p for p in c.get('params', []) if p not in params] return not missing, 'params=' + str(params) + ' missing=' + str(missing) @check('run_block') def run_block(c, root: Path): text = (root / c['source']).read_text(encoding='utf-8') blocks = re.findall(r'``` [a-z]*\n(.*?) ```', text, flags=re.S) if c['fence_index'] >= len(blocks): return False, 'fence index out of range' proc = subprocess.run( [sys.executable, '-I', '-c', blocks[c['fence_index']]], capture_output=True, text=True, cwd=root, timeout=c.get('timeout', 10), ) return proc.returncode == 0, (proc.stderr or 'ok').strip()[:200] 

Two details matter more than they look at first glance. The -I flag isolates the snippet from user site-packages and environment variables, which removes the most common reason a documented example only works on the author's machine. The regex extracts fenced blocks in document order, so a sentence referring to the snippet below is checked against the block a reader would actually copy.

One caveat belongs here rather than at the end: this runner is a reference implementation written for the article, not a benchmarked tool, and the checkers your repository needs will differ by language and layout.

Step 3: route each verdict

The runner writes a JSON report and exits with a code that a merge gate can read directly. Route on the combination of claim class and verdict rather than on severity labels alone, because the same failure means different things in different classes.

def route_for(status, claim): if status == 'pass': return 'publish' if status == 'unfalsifiable': return 'human' return claim.get('on_fail', 'demote') 

The table is the part worth arguing about with your team, not the code. If a class appears in your docs that the table does not cover, add a row and decide explicitly whether failure blocks or demotes, because silent defaults are how claim registries quietly rot.

Step 4: keep drafting separate from verification

The drafting pass produces prose from a fixed input set: the section outline, the extracted facts, and the previous version of the page. The verification pass reads only the produced page plus the claims file, and it must never call a model at all. That separation is what makes the report reproducible, since identical inputs and identical checks produce identical verdicts regardless of which model drafted the text.

For the drafting step you can use whatever model access your team already pays for, and the verification step runs identically in both cases. That symmetry is the whole point of splitting the two passes into separate artifacts with separate owners.

Where free model access and a free server change the calculus

The expensive part of this workflow is not drafting, because prose is cheap to produce and cheap to throw away. The expensive part is having somewhere disposable to execute untrusted snippets, plus enough model calls to regenerate a page after each fix. A free tier helps most at exactly that loop: draft, falsify, fix, redraft, without a budget conversation for every iteration.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states that the project currently offers a 10-million-token free allowance and a free server option for running work like this. Treat both numbers as time-sensitive and check the project's current terms before you design a pipeline around them, since free allowances change and I cannot promise this one persists.

Treat the free server as a sandbox decision rather than a convenience decision. Snippet checks execute code a model wrote, so run them in an environment with no credentials, no network access, and no host mounts. If you cannot verify that isolation yourself, set every run_block claim to on_fail: demote and let a human run the snippet manually, which costs reviewer time but removes the worst failure mode in this design.

If you want to try the split before provisioning anything, the project's free server and free model access are the cheapest place to start; verify the environment first, then decide. One command runs the entire pass, locally or on a remote box:

python tools/docfalsify.py docs/claims/*.claims.yaml --root . --offline echo "exit code: $?" 

What a human still owns

Falsification narrows review; it does not replace review. Four sentence classes stay with a named person no matter how good the checks become, because their failure mode is disagreement rather than simple incorrectness.

1. Intent and rationale. Why a cap exists, why an option defaults to off, and which trade-off was accepted cannot be extracted from code that already implements the decision.
2. Promises with a clock. Deprecation timelines, support windows, and compatibility guarantees are commitments to future users, not descriptions of current behaviour.
3. Security and privacy statements. A claim that a path is safe needs a threat-model reference and an owner who will answer questions after publication.
4. Numbers without a benchmark in the repo. Any latency or cost figure the repository cannot reproduce should be cut, or attributed to a dated primary source.

The queue makes this ownership visible instead of implicit. A demoted or unfalsifiable claim appears in the report with the owner field already populated, so the request has a specific addressee rather than sitting in a general review backlog.

Limitations and who should skip this

The registry is overhead, and overhead needs a reason. A repository with a handful of prose pages and almost no factual claims will spend more time maintaining checkers than it saves in review, and a team without CI cannot enforce exit codes anyway. Pages that are mostly narrative, tutorials built on external services, and anything whose facts live outside the repository will produce a long unfalsifiable queue with very little automation value.

There is also a maintenance cost worth naming explicitly. Every check type is code that breaks when the repository layout changes, so budget a small owner rotation and keep checkers short enough to delete without ceremony. If a checker needs more than thirty lines, the claim it guards is usually too vague to verify in the first place.

Start with one page, the one your users complain about most, and a single claims file. If the report catches two wrong keys and one broken snippet in the first week, the loop has paid for itself; if it catches nothing at all, delete it and go back to reading carefully.