The PR Description Is a Hypothesis: Falsifying Agent Claims with Differential Probes

An agent-generated PR usually arrives with a confident paragraph. "Refactored the cache layer. No behavior change. Tests pass." That paragraph is not evidence. It is a hypothesis written by the entity whose work you are about to approve.

The cheapest way to test that hypothesis is not to read the diff more slowly. It is to run the same probe against origin/main and against HEAD, then compare the outputs. This post builds that harness: a claim format, a runner, a verdict table, and an honest list of where the method produces nothing useful.

One signal from this week's DEV feed overlapped with this problem — the argument that agents should be given scripts instead of Markdown instructions. Same instinct here, different target: stop writing review checklists, start writing executed comparisons.

Three ways a prose review fails

Reading the diff feels rigorous because it is expensive. But the failure modes are structural.

1. Silent widening. The diff adds a branch for a new input class. Nothing on screen says the old input class now takes that branch too.
2. Vacuous tests. A new test passes on HEAD — and also passes on origin/main, because it asserts nothing that the change actually affects. The green checkmark carries zero information.
3. Unfalsifiable claims. "No behavior change" is a statement about two states of the world. You cannot verify it by inspecting one of them.

All three are cheap to detect with output comparison and expensive to detect by reading.

The probe contract: claim to verdict

Every reviewable claim gets a probe, and every probe gets a mechanical verdict. No prose in between.

• CONFIRMED — the claim held on both refs.
• REFUTED — the claim failed; the observed delta is in the report.
• INCONCLUSIVE — the probe could not run on the base ref, so the claim is untestable by differential means.

That third verdict matters more than the first two. It tells you exactly which claims you still have to read the diff for, instead of pretending you reviewed everything.

Triage table

The artifact

A claim file describes one hypothesis. It is deliberately small so it can be reviewed in ten seconds.

# probes/cache-layer.yaml claim: "The new cache returns identical values for all previously supported keys." base: origin/main head: HEAD probes: - name: same-value-roundtrip setup: "python -m pip install -q -e ." run: "python scripts/cache_replay.py --fixture tests/fixtures/keys_2025.json" expect: identical_stdout - name: p50-not-worse run: "python scripts/latency_probe.py --n 200" expect: numeric_not_worse tolerance: 0.10 - name: miss-path-exit run: "python scripts/miss_probe.py" expect: exit_code value: 0 

The runner is a reference implementation, not a product. It is unexecuted here and needs PyYAML plus a clean working tree.

#!/usr/bin/env python3 """probe.py — run each probe on base and head worktrees, compare, print a verdict.""" import subprocess, sys, tempfile from pathlib import Path import yaml def sh(cmd, cwd): p = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True) return {"exit": p.returncode, "stdout": p.stdout.strip(), "stderr": p.stderr.strip()} def add_worktree(repo, ref, dest): subprocess.run(["git", "worktree", "add", "--detach", str(dest), ref], cwd=repo, check=True, capture_output=True) return dest def run_probe(root, probe): if probe.get("setup"): sh(probe["setup"], root) return sh(probe["run"], root) def verdict(probe, base, head): expect = probe["expect"] if base["exit"] != 0 and expect != "exit_code": return "INCONCLUSIVE" # base cannot run it: claim untestable  if expect == "identical_stdout": return "CONFIRMED" if base["stdout"] == head["stdout"] else "REFUTED" if expect == "exit_code": return "CONFIRMED" if head["exit"] == probe["value"] else "REFUTED" if expect == "numeric_not_worse": b, h = float(base["stdout"]), float(head["stdout"]) return "CONFIRMED" if h <= b * (1 + probe.get("tolerance", 0)) else "REFUTED" return "INCONCLUSIVE" def main(spec_path, repo="."): spec = yaml.safe_load(Path(spec_path).read_text()) print(f"claim: {spec['claim']}") with tempfile.TemporaryDirectory() as tmp: base_root = add_worktree(repo, spec["base"], Path(tmp) / "base") head_root = add_worktree(repo, spec["head"], Path(tmp) / "head") try: for probe in spec["probes"]: b = run_probe(base_root, probe) h = run_probe(head_root, probe) v = verdict(probe, b, h) print(f"\n[{v}] {probe['name']}") if v != "CONFIRMED": print(f" base exit={b['exit']} stdout={b['stdout'][:300]!r}") print(f" head exit={h['exit']} stdout={h['stdout'][:300]!r}") finally: subprocess.run(["git", "worktree", "prune"], cwd=repo, capture_output=True) if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "probes/claim.yaml") 

Run it in CI as a job that is allowed to fail loudly:

python probe.py probes/cache-layer.yaml # any REFUTED line blocks the merge; INCONCLUSIVE lines route to manual diff review git worktree prune 

The output is a short list with a verdict per claim. Reviewers read the REFUTED lines first, then only the diff hunks belonging to INCONCLUSIVE claims. Review time drops because attention is allocated by evidence, not by habit.

Where a free-tier model actually helps

Writing probes is the tedious part. Reading a PR description and proposing three candidate probes is a task a model does well, because a wrong probe is caught mechanically — it returns INCONCLUSIVE or a probe whose base run already fails. The probe is the judge; the model is only the drafter.

MonkeyCode exposes free model access, which makes that drafting step cheap enough to run on every agent PR rather than only on the ones that look suspicious. Its operator states the free tier includes a pool of free tokens (I was given the figure of 10 million) and a free server option. Quotas and capacity change, so verify the current terms on the project's own pages before you plan a workflow around them, and never let a token budget decide whether a merge is safe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters for a mundane reason: the harness above creates two worktrees per claim and installs dependencies twice. A persistent box with a warm package cache turns a two-minute probe cycle into a ten-second one, which is the difference between running probes on every PR and running them on none. Nothing about the method depends on that box being fast or free — only on it existing.

Limitations and who should skip this

Be honest about the edges before adopting the workflow.

• Non-deterministic output kills identical_stdout. Timestamps, dict ordering, floats, and UUIDs will produce false REFUTED. Normalize first or use a tolerant comparator.
• The base ref often cannot run the probe. A new feature does not exist on origin/main. INCONCLUSIVE is the correct answer there, and it means the diff still needs a human.
• Two installs per claim is expensive. Monorepos with slow dependency graphs will spend more time in setup than in review.
• It says nothing about security, migrations, or data loss. A probe compares outputs; it cannot compare the blast radius of a schema change. Those PRs need the rollback question first, not the probe.
• Do not adopt this for docs-only or dependency-bump PRs. The harness costs more than it reveals.

Skip the whole approach if your repository cannot produce a deterministic fixture. Without one, every verdict becomes INCONCLUSIVE and you have built a slower way to read a diff.

What to take away

Treat the PR description as a set of claims, not a summary. Convert the falsifiable ones into probes, run them against both refs, and let the machine issue the verdict. Then read the diff — but only where the machine admitted it could not.

If you try this, start with a single claim: "no behavior change," one fixture, one probe. If that probe ever returns REFUTED, the harness has already paid for itself.