Doc Claims Should Fail CI: Turning Prose Promises Into Runnable Proofs

Most documentation rot is not a writing problem; it is an unverified-claim problem that accumulates silently until a reader trusts a sentence the code stopped honoring months ago. The workflow below converts checkable doc sentences into small proof blocks that run in CI, so a stale claim becomes a red build instead of a support ticket. The design rule is narrow on purpose: models may draft the mechanical parts of a proof, and humans own the expect value, the owner field, and every claim that cannot be executed.

The failure mode this replaces

Reviewers read prose and approve intent, but nothing re-reads the prose after the next refactor. A sentence like "retries default to three" survives three config changes because it lives in a file no test imports. Attribution and sign-off rituals help while a human is looking, yet they do not survive the six weeks after that human rotates off the service.

The claim needs a mechanical witness attached to it, not a signature above it. Once the witness exists, the interesting question stops being "who approved this paragraph" and becomes "which sentences can be proven, and who owns the proof when it breaks".

Step 1: Sort claims before you generate any of them

Do this classification first, because it decides what a model is even allowed to touch. The table is a policy artifact you can commit next to your docs and cite in review.

Two rows matter more than the rest. Performance adjectives and policy statements must never be drafted as if a test could confirm them, because a passing command would create false confidence around a claim the command never touched.

Step 2: A proof-block convention the parser can trust

Keep the claim in normal prose and put the witness in an HTML comment directly above it, so GitHub renders cleanly and the block travels with the sentence during edits.

<!-- proof id: retry-default owner: @avery command: python -c "from cfg import load; print(load()['retries'])" expect: 3 watches: src/cfg.py, src/defaults.toml verified_at: 4f2a91c --> Retries default to 3 and are adjustable through `RETRIES`. 

Your lint job should reject any block missing id, owner, command, or expect. That sounds pedantic, but the failure you are defending against is a proof that nobody can be paged for, and the owner field is the only part of the block that answers that question.

Step 3: Run the proofs with one dependency-free script

The script below is a working sketch rather than a measured result; its runtime and catch rate depend entirely on your claim set. It parses blocks, runs each command with a timeout, compares trimmed stdout to expect, and exits non-zero on the first mismatch set.

#!/usr/bin/env python3 """docproof.py - execute claims embedded in Markdown.""" from __future__ import annotations import argparse, json, re, subprocess from dataclasses import dataclass from pathlib import Path BLOCK = re.compile(r"<!--\s*proof\s*\n(.*?)-->\s*\n", re.S) REQUIRED = {"id", "owner", "command", "expect"} @dataclass class Proof: id: str owner: str command: str expect: str path: str line: int def parse(path: Path) -> list[Proof]: text = path.read_text(encoding="utf-8") proofs = [] for match in BLOCK.finditer(text): fields = {} for raw in match.group(1).splitlines(): if not raw.strip(): continue key, _, value = raw.partition(":") fields[key.strip()] = value.strip() missing = REQUIRED - fields.keys() if missing: raise SystemExit(f"{path}: proof block missing {sorted(missing)}") line = text[: match.start()].count("\n") + 1 proofs.append(Proof(fields["id"], fields["owner"], fields["command"], fields["expect"], str(path), line)) return proofs def run(proof: Proof, cwd: Path, timeout: int) -> tuple[bool, str]: try: done = subprocess.run(proof.command, shell=True, cwd=cwd, capture_output=True, text=True, timeout=timeout) except subprocess.TimeoutExpired: return False, f"timeout after {timeout}s" if done.returncode != 0: return False, f"exit {done.returncode}: {done.stderr.strip()[:200]}" if done.stdout.strip() != proof.expect: return False, f"expected {proof.expect!r}, got {done.stdout.strip()!r}" return True, "ok" def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("docs", nargs="+", type=Path) ap.add_argument("--cwd", type=Path, default=Path(".")) ap.add_argument("--timeout", type=int, default=30) ap.add_argument("--json", action="store_true") args = ap.parse_args() proofs = [p for doc in args.docs for p in parse(doc)] failures = [] for proof in proofs: ok, detail = run(proof, args.cwd, args.timeout) if not ok: failures.append({"id": proof.id, "owner": proof.owner, "where": f"{proof.path}:{proof.line}", "detail": detail}) if args.json: print(json.dumps({"total": len(proofs), "failures": failures}, indent=2)) else: for fail in failures: print(f"FAIL {fail['id']} ({fail['owner']}) at {fail['where']}: {fail['detail']}") print(f"{len(proofs) - len(failures)}/{len(proofs)} claims verified") return 1 if failures else 0 if __name__ == "__main__": raise SystemExit(main()) 

Run it locally with python docproof.py docs/*.md --timeout 15 before you wire it into anything. A command that needs network access will make the gate flaky, so prefer commands that read local config, import local modules, or hit a fixture server you control.

Step 4: Lint for tautologies and prose drift

A model drafting proof blocks will occasionally produce one that passes for the wrong reason, such as expect: "" or a command that echoes a constant. Add two checks to the same script, and both belong in review as hard failures.

The first rejects empty expectations and commands whose only effect is printing a literal. The second asserts that the expect token actually appears in the paragraph following the block, which keeps the witness and the sentence from drifting apart during a rewrite.

# reject trivial witnesses grep -n 'expect: *$' docs/*.md && exit 1 # confirm the expected value is still visible in prose python docproof.py docs/*.md --lint-prose 

That second check is the cheapest defense you have against a well-meaning edit that changes the sentence and leaves the old proof untouched. It is also the point where a human decision cannot be automated away.

Step 5: Route failures to owners on a schedule

Run the gate on every pull request, then run it again on a weekly cron so drift surfaces even in a quiet repository. Group the JSON output by owner and post one summary per owner rather than a single wall of failures.

name: docproof on: pull_request: schedule: - cron: "17 6 * * 1" jobs: proofs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: { fetch-depth: 0 } - uses: actions/setup-python@v5 with: { python-version: "3.12" } - run: python docproof.py docs/*.md --json > docproof.json - if: always() uses: actions/upload-artifact@v4 with: { name: docproof, path: docproof.json } 

Drift reporting needs history, which is why the checkout step keeps full depth. With verified_at and watches in each block, you can list commits that touched the watched files since the last green run and hand that list to the owner as context.

Where a model helps and where it must stop

Drafting proof blocks is mechanical work that suits a model well: it can read a diff, find claims whose source files changed, and propose a command plus a watcher list for each. MonkeyCode's free model access covers that drafting loop, and the free server option can host the weekly drift job so the gate does not depend on a maintainer's laptop staying awake. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The line to hold is simple and non-negotiable. A model may propose a command; a named human sets expect, accepts owner, and decides whether a claim is provable at all. If a draft arrives with an expectation the model invented, treat it as an unreviewed assertion, not as evidence.

Limitations and who should skip this

Executable proofs only cover claims that reduce to a command and a string. Anything about intent, security posture, third-party consoles, or comparative performance stays with a human, and a green build must never be read as approval of surrounding prose.

Skip this workflow if your docs are pre-launch and churning weekly, if you have no CI to run the gate, or if fewer than roughly a dozen claims are genuinely checkable, because the maintenance cost of owners and watchers will exceed the value. Start with the five claims that have already misled someone, and let the convention grow only where it earns its keep.

If you want a starting point, copy the script into scripts/, convert one numeric claim, and see whether the gate catches the next config change before a reader does.