Turn Review Comments Into Shell Checks: A Falsification Gate for Open-Source Patches

A nine-line pull request sits in the review queue of a small Python utility that converts user input into integer cents. CI is green, two reviewers agree the change handles the edge cases, and the patch merges. A week later someone notices the new test also passed on the original commit, so nothing in the pipeline ever proved the bug existed. The reviewers were not careless; they were handed claims that had no executable form.

This walkthrough turns review claims into shell checks that confirm or reject themselves. The fixture is five files, every command below runs locally, and nothing depends on a particular upstream project. The same loop scales to a real contribution, where a wrong claim costs a revert instead of a red test.

Three lanes, one rule

The workflow separates three kinds of work, because each one fails differently.

1. Reproduction lane — prove the defect on the base commit before reading the patch.
2. Patch lane — keep the diff small enough that the reproduction still applies afterwards.
3. Verification lane — turn every review claim into a command with a predicted exit code.

The rule that holds them together: a claim without a command is a question, not a finding. Anything a reviewer cannot express as a check moves into an open-questions list in the pull request body, which makes uncertainty visible instead of laundering it into confident prose.

Step 1: Build the fixture and record the red state

The repository holds one function with one obvious defect, which is enough to exercise the entire loop.

mkdir tinycalc && cd tinycalc && git init -q cat > calc.py <<'PY' def parse_amount(text: str) -> int: # BUG: assumes the input is a bare integer string. return int(text) PY mkdir tests && cat > tests/test_calc.py <<'PY' from calc import parse_amount def test_plain_integer(): assert parse_amount('42') == 42 def test_thousands_separator(): assert parse_amount('1,200') == 1200 PY git add -A && git commit -qm 'fixture: parse_amount without separator handling' python3 -m pytest tests/test_calc.py -q; echo "exit=$?" 

The second test raises ValueError and the shell reports exit=1. Paste that tail into the pull request before the diff, because the verification lane later compares the base commit against the patched tree. Then apply a candidate fix as a second commit:

cat > calc.py <<'PY' def parse_amount(text: str) -> int: return int(text.strip().replace(',', '')) PY git commit -qam 'fix: accept thousands separators and surrounding whitespace' 

The diff stays small on purpose. A wide refactor makes every reproduction claim ambiguous, and the verification lane can no longer tell which hunk caused a check to flip.

Step 2: Write findings as claims with a check file

Review output, whether from a colleague or a model, becomes a manifest plus one script per claim. The manifest stores only what the gate needs: identity, lane, expected outcome, and the script path.

- id: F1 claim: Base commit raises ValueError for the input 1,200 lane: free run_against: base expect: nonzero_exit check: checks/F1.sh - id: F2 claim: Patched tree parses 1,200 with surrounding whitespace into 1200 lane: free run_against: head expect: zero_exit check: checks/F2.sh - id: F3 claim: Patch also accepts negative amounts, which the caller forbids lane: human expect: informational check: checks/F3.sh 

Each check is a small, boring shell script that exits non-zero when its claim is false.

mkdir checks cat > checks/F1.sh <<'SH' #!/usr/bin/env bash set -euo pipefail python3 -c 'from calc import parse_amount; parse_amount("1,200")' SH cat > checks/F2.sh <<'SH' #!/usr/bin/env bash set -euo pipefail python3 -c 'from calc import parse_amount; assert parse_amount(" 1,200 ") == 1200' SH cat > checks/F3.sh <<'SH' #!/usr/bin/env bash set -euo pipefail python3 -c 'from calc import parse_amount; print(parse_amount("-500"))' SH 

F3 is deliberately not machine-gated. Whether a negative amount is legal depends on the caller's contract, and no exit code settles that question; the check exists to show the reviewer the current behaviour, not to block the merge.

Step 3: The gate script

The gate reads the manifest, runs each check in the correct tree, and compares the actual exit code against the prediction.

#!/usr/bin/env bash # verify_findings.sh <manifest>; manifest fields are pipe-separated. set -uo pipefail MANIFEST="${1:-findings.yml}" BASE_REF="${BASE_REF:-HEAD~1}" REPO="$PWD" confirmed=0; unsupported=0; skipped=0 rows() { python3 - "$MANIFEST" <<'PY' import sys, yaml for item in yaml.safe_load(open(sys.argv[1])) or []: print('|'.join([ item['id'], item.get('lane', 'free'), item.get('run_against', 'head'), item.get('expect', 'nonzero_exit'), item['check'], ])) PY } while IFS='|' read -r id lane against expect check; do if [ "$lane" = human ] || [ "$expect" = informational ]; then printf 'SKIP %-3s %s\n' "$id" "$check"; skipped=$((skipped+1)); continue fi root="$REPO" if [ "$against" = base ]; then root="$(mktemp -d)"; git archive "$BASE_REF" | tar -x -C "$root" fi ( cd "$root" && bash "$REPO/$check" ) >/dev/null 2>&1 rc=$? verdict=UNSUPPORTED if [ "$expect" = nonzero_exit ] && [ "$rc" -ne 0 ]; then verdict=CONFIRMED; fi if [ "$expect" = zero_exit ] && [ "$rc" -eq 0 ]; then verdict=CONFIRMED; fi printf '%-11s %-3s rc=%-3s %s\n' "$verdict" "$id" "$rc" "$check" if [ "$verdict" = CONFIRMED ]; then confirmed=$((confirmed+1)); else unsupported=$((unsupported+1)); fi done < <(rows) echo echo "confirmed=$confirmed unsupported=$unsupported skipped=$skipped" [ "$unsupported" -eq 0 ] || exit 2 

Run it and the output is short enough to paste straight into a review thread:

CONFIRMED F1 rc=1 checks/F1.sh CONFIRMED F2 rc=0 checks/F2.sh SKIP F3 checks/F3.sh confirmed=2 unsupported=0 skipped=1 

Step 4: Read UNSUPPORTED as information, not failure

An UNSUPPORTED verdict means the command and the claim disagree, and both readings deserve a minute. A model that claims the patch breaks on empty input usually produces a check that also fails on the base commit, which proves the defect is pre-existing rather than introduced. Reclassify it as a separate issue instead of blocking the pull request, then rewrite the check so it distinguishes the two trees.

Two mechanical failure modes show up constantly. The first is a check that cannot fail, such as a command ending in || true or a script with no assertion; a short lint pass over checks/*.sh catches most of them. The second is a check that depends on generated files the base archive does not contain, which produces noise until the setup step moves into the script itself.

Where free model access fits

Generating check scripts is high-volume, low-stakes work: each claim needs a five-line script, and a wrong script is caught by the gate rather than by a human. That shape suits free model access well, and the gate is the reason the free tier stays safe to use here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, and the project currently advertises a free token allowance of 10 million tokens; those terms are set by the project, so confirm the current offer before planning around it. Four rules keep the free lane honest:

• The model returns a check script, never a prose verdict.
• The gate rejects no-op checks (|| true, exit 0, commands under fifteen characters).
• One patch gets at most six checks, deduplicated by the tree each one runs against.
• Confirmation rate is measured on your own repository; a number you did not measure is a rumour.

The free server option matters for the same loop at a larger size. The base-versus-head matrix is plain bash, git, and python3, so it runs wherever those exist, and a free server instance is a reasonable place to keep a long test suite off a laptop.

A routing table for findings

Not every finding deserves the same treatment, and the table below is the one this workflow uses.

Findings that land in the human column are not failures of the process; they are the part of review that was never mechanical. Writing them down as open questions is more honest than dressing them in confident language.

Limitations and who should not use this

• bash "$REPO/$check" executes code from a manifest, which is remote code execution on any untrusted pull request. Run the gate in a container or a throwaway VM.
• Flaky tests generate false UNSUPPORTED verdicts; pin seeds and rerun a check twice before trusting it.
• git archive "$BASE_REF" assumes a short branch; on long-lived branches use git merge-base against the default branch instead.
• A CONFIRMED verdict only means the prediction matched, so revert the patch once and confirm the check flips.
• Skip this workflow for one-line typo fixes, where the ceremony costs more than the review, and for repositories whose suite cannot run outside a proprietary environment.

The loop pays off when patches are frequent and reviewers are scarce, which is the usual situation in small open-source projects. It also gives a model something useful to do: produce falsifiable artefacts instead of paragraphs that merely sound like review.

If you want to try the verification lane without a budget line, MonkeyCode's free model access and free server option are enough to run this loop against a small repository; start with the claim you trust least.