Prove Your Characterization Tests Can Fail: A Mutation Preflight for Messy Repos

A passing characterization suite proves the code runs. It does not prove the suite can fail.

Most messy-repo refactors die in that gap. You capture behavior, tests go green, then you make the smallest safe change and nothing notices a regression.

This article adds one gate before the refactor: a mutation preflight. You break the code on purpose, in the exact region you plan to touch, and confirm the suite screams.

The failure mode: green tests that assert nothing

Captured suites are usually written for speed, not detection. They call a function and check it does not throw.

A golden file regenerated on mismatch, an assert result is not None, a bare try/except: pass — all green forever. None of them are a gate.

Coverage will not rescue you. A line can be executed by a test that never inspects the value it produced.

The preflight asks one question per region: if behavior changes, does at least one test go red?

Three gates, one loop

The harness runs three checks in a fixed order.

1. Baseline gate. The suite must pass on the untouched tree. A red baseline means the capture is wrong — stop, fix the suite, do nothing else.
2. Apply gate. Each mutant must match exactly one site in one file. Zero matches means the code drifted; two matches means the rewrite is ambiguous.
3. Kill gate. With the mutant applied, the suite must fail. If it still passes, the mutant survived — that is a hole in your gate.

Survivors are the deliverable. Each one names a behavior your refactor could silently change.

Step 1: Write the mutant list as JSON

Keep the list beside your notes. Do not put it in the source tree.

Mutants are plain text substitutions. They are deliberately dumb: no AST tooling, no plugins, nothing to debug at 2 a.m.

[ { "label": "invoice_total: drop rounding", "file": "billing/invoice.py", "find": "return round(total, 2)", "replace": "return total" }, { "label": "invoice_total: skip discount branch", "file": "billing/invoice.py", "find": "if discount and tier == \"gold\":", "replace": "if False:" }, { "label": "invoice_total: off-by-one on quantity", "file": "billing/invoice.py", "find": "qty * unit_price", "replace": "qty * unit_price + 0.01" } ] 

Three rules keep this honest.

• One behavior per mutant: off-by-one, branch removal, return None, flipped comparison.
• Scope to the files in your planned diff. Mutating the whole repo measures the whole repo.
• Exactly one match per file. The harness enforces it and refuses to guess.

Step 2: The harness

Save this as mutation_preflight.py. Standard library only, no install.

from __future__ import annotations import json import shutil import subprocess import sys import tempfile from dataclasses import dataclass from pathlib import Path ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve() MANIFEST = Path(sys.argv[2] if len(sys.argv) > 2 else "mutants.json").resolve() # Point this at the suite that guards the region you are about to refactor. TEST_CMD = ["python", "-m", "pytest", "-x", "-q", "tests/characterization"] TIMEOUT = 300 IGNORE = [".git", ".venv", "node_modules", "__pycache__", ".pytest_cache"] @dataclass class Mutant: label: str file: str find: str replace: str def run_tests(workdir: Path) -> tuple[str, str]: """Return (verdict, tail); verdict is 'pass', 'fail' or 'timeout'.""" try: proc = subprocess.run( TEST_CMD, cwd=workdir, capture_output=True, text=True, timeout=TIMEOUT ) except subprocess.TimeoutExpired: return "timeout", "" verdict = "pass" if proc.returncode == 0 else "fail" return verdict, (proc.stdout + proc.stderr)[-300:] def load_mutants(path: Path) -> list[Mutant]: return [Mutant(**raw) for raw in json.loads(path.read_text())] def copy_tree(dest: Path) -> None: shutil.copytree( ROOT, dest, ignore=shutil.ignore_patterns(*IGNORE), dirs_exist_ok=True ) def apply_mutant(workdir: Path, m: Mutant) -> str: target = workdir / m.file if not target.exists(): return "missing_file" text = target.read_text() hits = text.count(m.find) if hits != 1: return f"matches_{hits}" target.write_text(text.replace(m.find, m.replace)) return "ok" def main() -> int: verdict, tail = run_tests(ROOT) if verdict != "pass": print(f"BASELINE {verdict}: fix the suite before mutating") print(tail) return 2 results = [] for m in load_mutants(MANIFEST): with tempfile.TemporaryDirectory() as tmp: workdir = Path(tmp) / "repo" copy_tree(workdir) applied = apply_mutant(workdir, m) if applied != "ok": results.append({"label": m.label, "status": applied}) continue verdict, _ = run_tests(workdir) status = {"fail": "killed", "pass": "survived", "timeout": "timeout"}[verdict] results.append({"label": m.label, "status": status}) Path("preflight_report.json").write_text(json.dumps(results, indent=2)) survivors = [r["label"] for r in results if r["status"] == "survived"] print(f"{len(results)} mutants, {len(survivors)} survived") for label in survivors: print(f" SURVIVED {label}") return 1 if survivors else 0 if __name__ == "__main__": sys.exit(main()) 

Run it from the repo root.

python mutation_preflight.py . mutants.json 

One practical gotcha: the copy skips .venv, so the child process uses whichever interpreter resolves python. If your suite needs an activated environment, replace "python" in TEST_CMD with an absolute interpreter path. Otherwise you will get import errors that look like kills.

Step 3: Read the report

Expected output format, shown here as an illustration rather than a measurement from a specific repository:

3 mutants, 1 survived SURVIVED invoice_total: skip discount branch 

Map every status to a decision. Do not average them into a score.

A survivor on a branch you plan to delete is fine. A survivor on a function you plan to extract is a blocker.

Step 4: Use a model to propose mutants, not to judge them

The bottleneck is writing candidate mutants, not running them. That is where free model access is genuinely useful in this workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A cheap loop works like this: paste one function into the model, ask for five single-line behavioral mutations as JSON, then let the harness decide. The apply gate rejects anything that does not match exactly one site. The kill gate rejects nothing — it only reports.

The model proposes. The runner and your review dispose. Never let generated mutants skip the baseline gate.

Because the loop is embarrassingly parallel and mostly waiting on pytest, running it on a free server option is a reasonable fit when your laptop is otherwise occupied. Both the free model access and the free server option are things the operator states are available; verify current terms yourself before depending on them for a deadline.

Limitations and who should skip this

Text substitution is brittle. It breaks on reformatted code, and it cannot express structural mutants like moving a call across a boundary.

Equivalent mutants are permanent noise. A mutation that changes syntax but not behavior will survive forever and waste triage time. Human review is mandatory.

Runtime scales linearly. Ten mutants against a four-minute suite is forty minutes of wall clock. Please do not run this on the full repo.

Skip the preflight entirely if your baseline is not green. Skip it if you cannot afford one suite run per mutant. Skip it if the region you plan to change has no tests at all — write the capture first, then come back.

This is a gate, not a guarantee. It tells you the suite can fail in the region you are about to edit. It does not tell you the refactor is correct.

If you try it, start with five mutants scoped to one file, and add an assertion for every survivor before you open the diff.