A messy-repo refactor fails when helpers move first. Lock the entrypoint's observable outputs before any edit. Then change one internal function and nothing else. Local confidence is the usual failure Most AI patches target a small function in isolation. That function often looks cleaner after the edit. Callers still depend on files, globals, and print order. A passing helper test does not protect the script. The script is what operators actually run. Treat that script as the contract, not the helper. Cheap model output makes large diffs easy to produce. It does not make those diffs safe to merge. Safety still comes from frozen, replayable observables. Define the mess with a teaching fixture The listing below is a labeled teaching fixture. It is not production code from a live system. It mixes pricing, tax, and invoice file writes. # messy_invoice.py — teaching fixture, not production from pathlib import Path import json import sys TAX = 0.08 OUT = Path("out") _last = {} def load_items(path): rows = [] for line in Path(path).read_text().splitlines(): sku, qty, price = line.split(",") rows.append({"sku": sku, "qty": int(qty), "price": float(price)}) _last["rows"] = rows return rows def subtotal(rows): s = 0.0 for r in rows: s += r["qty"] * r["price"] if r["qty"] >= 10: s -= r["price"] # implicit bulk rule _last["subtotal"] = s return s def tax_on(amount, region): rate = TAX if region == "EU": rate = 0.19 if region == "EXEMPT": rate = 0.0 return round(amount * rate, 2) def write_invoice(rows, region, dest): OUT.mkdir(exist_ok=True) sub = subtotal(rows) tax = tax_on(sub, region) total = round(sub + tax, 2) payload = { "region": region, "lines": len(rows), "subtotal": sub, "tax": tax, "total": total, } Path(dest).write_text(json.dumps(payload, indent=2) + "\n") print(f"WROTE {dest} total={total}") return payload def main(argv): src = argv[1] region = argv[2] dest = argv[3] rows = load_items(src) return write_invoice(rows, region, dest) if __name__ == "__main__": main(sys.argv) Three hazards sit inside that short teaching module. The global _last dict stores implicit process state. The subtotal loop hides a bulk discount rule. Whole-run snapshots beat helper tests Helper tests miss print text and file bytes. They also miss argument order on the CLI. A whole-run snapshot catches those observables in one gate. Build a golden directory from a frozen input corpus. Store stdout, stderr, exit code, and output hashes. Re-run the same command after every internal edit. 1. Freeze the input corpus Keep the input fixtures tiny and committed to git. One CSV is enough for the first gate. Add a second CSV only after the first stays green. # fixtures/items_basic.csv A,1,10.00 B,10,2.50 C,3,4.00 # fixtures/items_eu.csv D,2,40.00 E,10,1.00 2. Compute the expected bytes by hand Hand totals keep the first goldens honest. Do not trust the script to mark its own exam. The US basic case should resolve as follows. Line A contributes 1 * 10.00 = 10.00. Line B contributes 10 * 2.50 = 25.00. Quantity 10 then subtracts 2.50 as bulk credit. Line C contributes 3 * 4.00 = 12.00. Subtotal is 10.00 + 22.50 + 12.00 = 44.50. US tax is round(44.50 * 0.08, 2) = 3.56. Total is round(44.50 + 3.56, 2) = 48.06. Stdout must read WROTE out/inv.json total=48.06. Any later extract must preserve those exact bytes. The EU bulk case should resolve next. Line D contributes 2 * 40.00 = 80.00. Line E contributes 10 * 1.00 - 1.00 = 9.00. Subtotal is 89.00 before region tax. EU tax is round(89.00 * 0.19, 2) = 16.91. Total is round(89.00 + 16.91, 2) = 105.91. 3. Record a command matrix, not a function call Run the entrypoint under a clean working directory. Do not reuse leftover output files between cases. Capture the process, not a Python function call. # char_harness.py — teaching fixture from __future__ import annotations import hashlib import json import os import shutil import subprocess import sys from pathlib import Path ROOT = Path(__file__).resolve().parent GOLDEN = ROOT / "golden" CASES = [ { "name": "us_basic", "args": ["fixtures/items_basic.csv", "US", "out/inv.json"], }, { "name": "eu_bulk", "args": ["fixtures/items_eu.csv", "EU", "out/inv.json"], }, { "name": "exempt_basic", "args": ["fixtures/items_basic.csv", "EXEMPT", "out/inv.json"], }, ] def sha256(path: Path) -> str | None: if not path.is_file(): return None h = hashlib.sha256() h.update(path.read_bytes()) return h.hexdigest() def run_case(case: dict) -> dict: work = ROOT / "work" / case["name"] if work.exists(): shutil.rmtree(work) work.mkdir(parents=True) shutil.copytree(ROOT / "fixtures", work / "fixtures") dest_rel = Path(case["args"][2]) proc = subprocess.run( [sys.executable, str(ROOT / "messy_invoice.py"), *case["args"]], cwd=work, capture_output=True, text=True, env={**os.environ, "PYTHONHASHSEED": "0"}, ) out_file = work / dest_rel return { "name": case["name"], "exit_code": proc.returncode, "stdout": proc.stdout, "stderr": proc.stderr, "out_sha256": sha256(out_file), "out_text": out_file.read_text() if out_file.is_file() else None, } def record() -> None: GOLDEN.mkdir(exist_ok=True) for case in CASES: snap = run_case(case) (GOLDEN / f"{case['name']}.json").write_text( json.dumps(snap, indent=2) + "\n" ) print(f"recorded {case['name']}") def check() -> int: failed = 0 for case in CASES: got = run_case(case) path = GOLDEN / f"{case['name']}.json" want = json.loads(path.read_text()) keys = ["exit_code", "stdout", "stderr", "out_sha256", "out_text"] for key in keys: if got[key] != want[key]: failed += 1 print(f"DRIFT {case['name']} {key}") print(f" want={want[key]!r}") print(f" got ={got[key]!r}") if failed: print(f"{failed} field(s) drifted") return 1 print("snapshot gate green") return 0 if __name__ == "__main__": cmd = sys.argv[1] if len(sys.argv) > 1 else "check" if cmd == "record": record() else: raise SystemExit(check()) Each case records five comparable fields on disk. Those fields are the only pass signal. Pretty logs outside the snapshot are noise. 4. Prove the gate can fail A gate that never fails is not a gate. Break one print on purpose after recording. Confirm that check reports DRIFT on stdout. python char_harness.py record python char_harness.py check Edit the print format, then run check again. Restore the print before any real refactor. The restored run must return exit code 0. A sample drift block looks like this. The numbers below match a one-cent rounding slip. Treat that slip as a failed extract, not noise. DRIFT us_basic stdout want='WROTE out/inv.json total=48.06\n' got ='WROTE out/inv.json total=48.06\n' DRIFT us_basic out_text want='{\n "region": "US",\n "lines": 3,\n "subtotal": 44.5,\n "tax": 3.56,\n "total": 48.06\n}\n' got ='{\n "region": "US",\n "lines": 3,\n "subtotal": 44.5,\n "tax": 3.56,\n "total": 48.05\n}\n' 5. Change one internal function only Do not rename files in the same patch. Do not move CLI flags in the same patch. Do not retune tax rounding in the same patch. The smallest safe change in this module is extraction. Pull the bulk rule out of subtotal. Keep write_invoice output byte-identical after the extract. def bulk_credit(row): if row["qty"] >= 10: return row["price"] return 0.0 def subtotal(rows): s = 0.0 for r in rows: s += r["qty"] * r["price"] s -= bulk_credit(r) _last["subtotal"] = s return s Re-run the snapshot gate after that extract. Green means the entrypoint still writes the same bytes. Red means the extract changed pricing or print text. Smallest-change checklist Edit one function body, or extract from it. Leave CLI argument order unchanged. Leave fixture files byte-identical. Leave golden snapshots byte-identical. Add no library dependency in that patch. Re-run python char_harness.py check once. Stop on the first drifted field. Skip any patch that fails one checklist row. Wide cleanup is not a first-cycle goal. Queue extra extracts for later green cycles. Decision table for snapshot drift Drift field Likely cause Safe action exit_code uncaught exception or new sys.exit stop; inspect traceback stdout print format or call order stop unless format was the goal stderr new warning or log line treat as contract unless documented out_sha256 numeric rounding or key order stop; compare out_text out_text tax, bulk rule, or region mapping revert; split the change Use one row as a stop rule, not a suggestion. Any unexplained drift must block the current patch. Do not rewrite goldens to match a guessed refactor. Invoice totals look like business rules, not formatting. A one-cent drift is a failed extract. Do not round-trip floats through new types in the same patch. Do not switch json.dumps settings during extract. Indent, separators, and key order are contract bytes. Hash equality will fail if those settings move. Control cwd, env, and hash seed The working directory leaks into relative output paths. Set PYTHONHASHSEED to keep hash walks stable. Copy fixtures into a fresh work tree every case. Path separators can drift across operating systems. Keep dest arguments in POSIX form inside cases. Run the gate on one OS, not two, per corpus. Grow new cases only from escaped production bugs. This teaching fixture records only three named cases. Those three cases will still miss many branches. Add a case when a real bug escapes, not before. Each new case must fail once before it is recorded. A case that never failed is an untested assertion. Where a coding model belongs The coding model proposes the internal extract only. The snapshot gate accepts or rejects that extract. Do not let the model refresh golden files. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those two options can host this harness beside the messy module. The model sees the frozen cases and the current function body. It does not get permission to rewrite golden snapshots. Keep the model prompt narrow, mechanical, and single-purpose. The block below is an unexecuted prompt template. It is not a log from a real session. Paste it only after check is already green. Preserve golden/* byte-for-byte. Do not edit fixtures, CLI args, or char_harness.py. Extract bulk_credit from subtotal in messy_invoice.py. Return one patch. Stop if any snapshot field would drift. Ask for one extract that preserves snapshot bytes. Reject patches that touch fixtures, goldens, or CLI args. If you try that workflow, run the harness locally first. Then point a free MonkeyCode session at the same gate. What this gate does not prove It does not prove thread safety or performance. It does not prove unknown regions or empty files. It does not prove tax law, only current bytes. Hash equality is brittle with unstable key order. JSON dumps must keep stable separators and indent. Timestamps inside invoices will break this design. Hidden network calls will also escape this gate. So will clock reads and unordered set iteration. Strip those sources before recording the first corpus. Who should skip this approach Skip this if you still lack a runnable entrypoint. Skip this if outputs include raw secrets or PII. Skip this if the script is nondeterministic by design. Skip this for greenfield modules with no users. Those modules need designed tests, not snapshots. Characterization is for behavior you cannot rewrite from memory. Do not use this as a license for large rewrites. The method allows one internal change per cycle. Wide cleanups belong after many green cycles, not before. Recap Start at the script the operator actually runs. Record stdout, exit code, and output file hashes. Extract one helper only after that gate is green.