Characterize First, Then Refactor: A Golden-Corpus Harness With a Mutation Gate

Most refactors break behavior that no test ever covered. Adding more hand-written tests is slow and biased toward what you already suspect. A pinned behavior snapshot is faster, and a mutation gate tells you whether that snapshot can actually fail.

The rule: pin first, prove the pin bites, then change the smallest thing. If the pin never fails on a deliberate bug, you have documentation, not a safety net.

Step 0: Pick one function, not one file

Choose a target that is boring and load-bearing. Small input surface, no hidden globals, callable from a script. Write the choice down in the commit message.

If the function reads the clock, calls the network, or draws randomness, stop. You need a seam before a snapshot. Pin a wrapper with the bad inputs injected instead.

Step 1: Freeze an input corpus

Build a JSONL file of real-shaped inputs. One case per line, stable ids, keyword arguments. No customer data, no tokens, no emails.

{"id":"c001","kwargs":{"tier":"gold","qty":1,"coupon":null}} {"id":"c002","kwargs":{"tier":"gold","qty":0,"coupon":"SAVE10"}} {"id":"c003","kwargs":{"tier":"free","qty":-1,"coupon":null}} 

Three sources work well. Existing fixture files, call sites in the repo, and log payloads you have already stripped. Twenty to forty cases covers most small functions.

Include the ugly ones: empty string, zero, negative, missing key, duplicate id. The golden file should pin the error paths too, not just the happy path.

Step 2: Normalize the output

Raw output rarely compares cleanly. Dict ordering drifts, floats repeat, timestamps move. Normalize before hashing.

# char_harness.py (part 1) def normalize(value): if isinstance(value, dict): return {str(k): normalize(v) for k, v in sorted(value.items(), key=lambda kv: str(kv[0]))} if isinstance(value, (list, tuple)): return [normalize(v) for v in value] if isinstance(value, (set, frozenset)): return sorted(normalize(v) for v in value) if isinstance(value, float): return round(value, 9) if isinstance(value, bool) or value is None or isinstance(value, (int, str)): return value try: attrs = normalize(vars(value)) except TypeError: attrs = {} return {"__type__": type(value).__name__, "__attrs__": attrs} 

Normalization is a tradeoff, not a free win. Rounding floats hides small drift. Falling back to type name hides everything inside slot-only objects. Log both decisions in the harness header.

Step 3: Pin a hash per case

The harness fingerprints each case and stores a sha256. Exceptions are pinned by class name and message.

# char_harness.py (part 2) import argparse, hashlib, importlib.util, json from pathlib import Path def load_target(path): spec = importlib.util.spec_from_file_location("target", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod def fingerprint(mod, fn_name, case): fn = getattr(mod, fn_name) try: payload = {"ok": True, "out": normalize(fn(**case["kwargs"]))} except Exception as exc: payload = {"ok": False, "err": type(exc).__name__, "msg": str(exc)} blob = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256(blob.encode()).hexdigest(), payload def main(): ap = argparse.ArgumentParser() ap.add_argument("--target", required=True) ap.add_argument("--function", required=True) ap.add_argument("--corpus", default="corpus.jsonl") ap.add_argument("--golden", default="golden.json") ap.add_argument("--mode", choices=["write", "check"], default="check") args = ap.parse_args() mod = load_target(args.target) lines = Path(args.corpus).read_text().splitlines() cases = [json.loads(l) for l in lines if l.strip()] results = {c["id"]: dict(zip(("sha256", "payload"), fingerprint(mod, args.function, c))) for c in cases} if args.mode == "write": Path(args.golden).write_text(json.dumps(results, indent=2, sort_keys=True)) print(f"pinned {len(results)} cases -> {args.golden}") return 0 golden = json.loads(Path(args.golden).read_text()) drift = [cid for cid in golden if results.get(cid, {}).get("sha256") != golden[cid]["sha256"]] missing = [cid for cid in golden if cid not in results] if drift or missing: print(json.dumps({"drift": drift, "missing": missing}, indent=2)) return 1 print(f"ok: {len(golden)} cases unchanged") return 0 if __name__ == "__main__": raise SystemExit(main()) 

Run it twice. Once to write, once to confirm the harness is stable. A pin that drifts on a second run is not a pin.

python char_harness.py --target ./src/pricing.py --function normalize_offer \ --corpus corpus.jsonl --golden golden.json --mode write python char_harness.py --target ./src/pricing.py --function normalize_offer --mode check 

Step 4: Prove the harness can fail

This is the step most people skip. A snapshot nobody has ever seen fail is unverified. So break the code on purpose, one behavior at a time.

Keep each mutant as a tiny patch file. Each one must change real behavior, not formatting.

# mutants/01-boundary.patch --- target.py +++ target.py @@ - if qty >= 1: + if qty > 1: 

# mutant probe: a failed check is a pass here for m in mutants/*.patch; do cp target.py /tmp/target.bak if ! patch -s -p0 < "$m"; then echo "SKIP $m (does not apply)"; continue; fi if ! python -c "import target" 2>/dev/null; then echo "SKIP $m (mutant does not import)"; cp /tmp/target.bak target.py; continue fi if python char_harness.py --target target.py --function normalize_offer \ --mode check >/dev/null 2>&1; then echo "MISSED $m <-- corpus is blind here" else echo "CAUGHT $m" fi cp /tmp/target.bak target.py done 

A MISSED line is the useful output. It names a behavior your corpus never exercises. Add one case, re-pin, re-probe. Repeat until every survivor is a mutant you can prove is equivalent to the original.

A SKIP line proves nothing. A patch that fails to apply, or a mutant that fails to import, is not evidence of coverage.

Step 5: Change the smallest thing, under zero drift

Now refactor. Rename, extract, inline, move. Re-run the check after every commit.

Example output after a clean extract-method commit: ok: 34 cases unchanged. If a hash moves, the patch was not behavior-preserving. Revert it, split it, try again.

git diff --numstat | awk '{a+=$1; d+=$2} END {print "changed:", a+d; exit ((a+d) > 40)}' 

One extra rule keeps this honest. Any commit that updates golden.json must also carry a BEHAVIOR: line in the message. That line explains why the change is intended. A golden diff without that line is a refactor that leaked.

Step 6: Re-run the probe

After the refactor, run the mutant loop again. Every mutant that was CAUGHT must still be CAUGHT. Coverage can survive a move while silently degrading.

Illustrative output shape for a 34-case corpus and 9 mutants, not a measured benchmark:

Where a free model and a free server fit

Two parts of this loop are tedious to write by hand. First, the normalization layer, which needs to survive real shapes you have not seen. Second, the mutant set, which needs genuinely different behaviors rather than nine cosmetic edits.

MonkeyCode's free model access and free server option are relevant here, both as operator-supplied availability claims rather than measured benchmarks. A free model is good at drafting normalize branches for shapes you forgot. It is also good at proposing candidate mutants across comparison operators, defaults, branches, and validation paths. The free server option matters because the mutant loop is a matrix of full test runs. Running that matrix in a sandbox keeps your local credentials and your laptop's patience out of the loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use model output as a proposal, never as a gate. Every suggested mutant needs a human check that it compiles and actually changes behavior. In my experience with generated mutants, the failure mode is equivalent mutants and non-applying patches, not missing ideas. Reject them loudly; a mutant set padded with noise makes the probe look green while proving nothing.

When pinning is the wrong tool

Limitations worth stating out loud

• Golden files ossify bugs. Anything pinned as-is stays pinned, including mistakes.
• Normalization hides drift by design. Rounding and type-only fallbacks are blind spots.
• Exception messages can embed variable data. Pin the class, and normalize the message.
• Slot-only objects lose attributes. The __type__ fallback is weaker than it looks.
• A large corpus makes the check slow and the diffs noisy. Keep it near the function, not the repo.
• Mutation survival is evidence of gaps, not proof of equivalence. You still decide.

Who should not use this

Skip this workflow if you have no reproducible input source. Skipped if the function cannot be imported without side effects, or if the output is dominated by data you cannot legally freeze. Skip it when the real risk is security or concurrency, where a pinned snapshot is silent. And skip it if nobody will maintain the mutant set. A stale probe is worse than none, because it looks like evidence.

The most useful file you will produce is the MISSED list. Save it, and let it drive the next corpus case.