A refactor is only as small as its residue.

You can pin golden outputs, run the suite green, merge the patch, and still ship a change that cannot be undone. The suite told you the output matched. It never told you the process was clean.

This post adds one gate after your characterization tests: apply the patch, revert it, and prove the original bytes come back byte-identical. If they do not, the patch is not small. It is hiding state.

The failure this catches

Characterization tests compare artifacts. Reverts compare state.

A patch can pass every golden and still leave residue. A module-level cache gets a new key. An import-time registry gains an entry. A log handler is registered twice. None of that shows in a single-pass assertion.

Revert makes the residue visible. State that survives a revert was never part of the contract. It was a side effect.

Step 1: Give every golden a file boundary

Tests must write artifacts to disk. In-memory assertions cannot be fingerprinted.

Add one fixture. It routes all outputs under an environment variable.

# conftest.py import os import pathlib import pytest @pytest.fixture def out_dir(request): base = pathlib.Path(os.environ["ARTIFACT_DIR"]) d = base / request.node.name d.mkdir(parents=True, exist_ok=True) return d

Then change one assertion per test. Write the rendered report, the exported CSV, the CLI help text.

def test_report_render(out_dir): rendered = render_report(SAMPLE_ROWS) (out_dir / "report.json").write_text(rendered, encoding="utf-8") assert rendered.startswith("{")

That assertion looks weak. It is. It exists to prove the artifact was written. Byte comparison happens in the harness, not the test.

Step 2: Fingerprint three states

Three states matter: before the patch, after the patch, after the revert.

# tools/revert_check.py import hashlib import json import os import shutil import subprocess import sys from pathlib import Path ROOT = Path(__file__).resolve().parents[1] WORK = ROOT / ".revert-check" TARGETS = ["tests/test_report_render.py", "tests/test_csv_export.py"] def fingerprint(tag: str) -> dict[str, str]: out = WORK / tag if out.exists(): shutil.rmtree(out) out.mkdir(parents=True) env = {**os.environ, "ARTIFACT_DIR": str(out)} subprocess.run( [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", *TARGETS], cwd=ROOT, env=env, check=True, ) return { str(p.relative_to(out)): hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(out.rglob("*")) if p.is_file() } def repo_state() -> str: return subprocess.run( ["git", "status", "--porcelain=v1", "-uall"], cwd=ROOT, capture_output=True, text=True, check=True, ).stdout def apply(patch: Path, reverse: bool = False) -> None: cmd = ["git", "apply", "--whitespace=nowarn"] if reverse: cmd.append("-R") subprocess.run([*cmd, str(patch)], cwd=ROOT, check=True)

The -p no:cacheprovider flag matters. Pytest's cache directory is state. Left on, it will pollute your status comparison.

Step 3: Compare revert against head

The driver does three runs and one comparison.

def main(patch: Path) -> int: head_files, head_state = fingerprint("head"), repo_state() apply(patch) try: after_files = fingerprint("after") finally: apply(patch, reverse=True) back_files, back_state = fingerprint("revert") report = { "patch": str(patch), "changed": sorted( k for k in set(head_files) | set(after_files) if head_files.get(k) != after_files.get(k) ), "residue": sorted( k for k in set(head_files) | set(back_files) if head_files.get(k) != back_files.get(k) ), "status_drift": head_state != back_state, } (WORK / "report.json").write_text(json.dumps(report, indent=2)) if report["residue"] or report["status_drift"]: print(json.dumps(report, indent=2)) return 1 return 0 if __name__ == "__main__": sys.exit(main(Path(sys.argv[1])))

Run it against a patch file you exported first.

git diff > patch.diff python tools/revert_check.py patch.diff echo "exit=$?"

Exit 0 means the patch reverts cleanly. Exit 1 means you have residue to read.

changed is informative. residue is the gate. A patch that changes twenty artifacts and reverts to zero residue is smaller than a patch that changes one artifact and leaks a cache file.

Step 4: Read the residue report

Illustrative output shape, with an invented file set:

{ "patch": "patch.diff", "changed": ["report.json", "export.csv"], "residue": ["report.json", "_cache/report.pkl"], "status_drift": true }

Two signals. A changed file did not come back. A new file appeared and stayed.

The _cache/report.pkl entry is the real finding. The render path memoized on disk. Reverting the code did not empty the cache, so the next run reads a stale entry.

Step 5: Shrink until residue is zero

Do not fix residue by adding cleanup. Fix it by removing the state.

Residue signal Likely cause Smallest safe change
Timestamp field differs after revert Clock read inside the render path Inject a fixed clock at the boundary
Extra _cache/*.pkl file Module-level memoization Move the cache onto the instance, clear in fixture
app.log has duplicated lines Handler added at import time Register once, guard by handler name
Registry list is longer Global list appended on call Build the registry per invocation
status_drift: true with no file diff Runtime wrote outside ARTIFACT_DIR Route all writes through one path helper

Three rules keep the patch small.

  1. Touch one function per patch. If residue appears, you know where to look.
  2. Move state, never delete it. Deleting a cache changes behavior and breaks characterization.
  3. Prefer a parameter over a global. Parameters revert; globals do not.

When residue cannot be removed cheaply, split the patch. Land a pure-function extraction first. That patch reverts to zero residue. Then do the stateful rewrite as a second patch with its own check.

Where a free model and a free server fit

Drafting the patch is the slow part of this loop. Diff-shaped prompts help. Ask for a unified diff that changes one function and adds no module-level state.

Return only a unified diff. Change exactly one function. Do not add module-level state, caches, registries, or new imports. Do not reformat untouched lines.

That constraint is what makes the revert check meaningful. An unconstrained draft tends to add a helper module and a cache, which is precisely the residue class this gate rejects. MonkeyCode's free model access is usable here for draft patches, since the harness rejects bad drafts mechanically. Its free server option lets you run the three-pass loop somewhere other than your laptop. Both are operator-supplied availability claims; check current limits yourself before relying on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The point is the gate, not the tool. Any model that emits a clean unified diff works. The harness does the judging.

Limitations

This check assumes deterministic artifacts. Three conditions break it.

  • Goldens containing wall-clock time or random IDs fail revert comparison for the wrong reason. Freeze the clock and seed the RNG first.
  • Tests writing outside ARTIFACT_DIR produce empty fingerprints. The check passes and proves nothing.
  • Patches that touch migrations or schema DDL are not invertible by design. Do not use this gate there.

git apply -R also fails on patches applied with fuzz. Keep the diff exact, unedited, and applied to a clean tree. Check git status before every run.

Who should not use this

Skip it if your outputs are not reproducible byte-for-byte. Skip it for forward-only work like data backfills and migrations. Skip it if your suite takes longer than a few minutes per pass; three passes will make the loop unusable.

Use it when you are about to refactor a messy module and want one number that says the change is reversible. Pair it with your characterization tests. They prove the output matched. This proves the process came back.

One last habit: keep report.json as a build artifact. A growing list of zero-residue patches is a better refactor record than any changelog line.