A large refactor diff asks reviewers for trust. A dual-run harness asks the code for proof. Run the old and new paths on the same frozen inputs, diff the outputs, then flip the default.
Characterization tests pin the old path. That part is necessary. It is not the whole move. The new path never executes until a human eyeballs the diff or a user hits it.
So stop reviewing big diffs alone. Ship two paths, one flag, one differ, one deletion step.
The shape of the move
- Cut one seam.
- Freeze a corpus of real inputs.
- Add inputs the traffic never sends.
- Put the new code behind one flag.
- Run both paths and diff normalized outputs.
- Flip the default, hold, then delete the old branch.
Six steps. One release each, if the seam is small.
Step 0: cut one seam, not one layer
A seam is one function boundary with a name. "Order pricing" is a seam. "The billing module" is a wish.
Write the seam down before touching code:
seam: pricing.compute_totals(cart, discount_rows) -> dict call sites: 4 direct, 1 via getattr in pricing_shim.py side effects: none (pure) That last line decides the method. Pure seams are cheap to dual-run. Impure seams need stubs first.
Step 1: freeze a corpus of real inputs
Record inputs at the live call site. Store them as JSONL. Hash each input so duplicates collapse.
# pseudocode, not executed for this article import hashlib, json, pathlib CORPUS = pathlib.Path("corpus/pricing.jsonl") def record(call, payload): out = call(payload) digest = hashlib.sha256( json.dumps(payload, sort_keys=True).encode() ).hexdigest()[:16] row = {"id": digest, "input": payload, "output": out} with CORPUS.open("a") as fh: fh.write(json.dumps(row, sort_keys=True) + "\n") Two rules. Redact before you write. Keep the corpus small and boring.
A few hundred rows beats a million. You are hunting shape, not traffic volume.
Step 2: add inputs the traffic never sends
Recorded traffic covers the happy path. It rarely covers empty carts, duplicate discount rows, or unicode in a name field.
Write the mutation list by hand first. Then generate variants from it. Keep the list in the repo next to the corpus.
This is the one generative step where free model access is genuinely useful. Hand the model the function signature and the corpus schema. Ask for inputs only, never for fixes. Keep the cases that parse and reject the rest.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you have access to the free server option, run the mutation generator and the parity runner there. Keep production credentials off that box. Treat it as a scratch bench, not capacity you depend on, and move the runner in-house before it gates a release.
Step 3: put the new path behind one flag
One flag. Read once at startup. No per-call evaluation.
NEW_PRICING = os.environ.get("PRICING_NEW", "0") == "1" def compute_totals(cart, discount_rows): if NEW_PRICING: return new_compute_totals(cart, discount_rows) return old_compute_totals(cart, discount_rows) The old function stays. Do not delete it during the parity window.
Step 4: dual-run and diff
The runner calls both paths. It never trusts the flag.
# pseudocode, not executed for this article def run_pair(row): a = old_compute_totals(row["input"]["cart"], row["input"]["rows"]) b = new_compute_totals(row["input"]["cart"], row["input"]["rows"]) return row["id"], diff(normalize(a), normalize(b)) def diff(a, b): keys = set(a) | set(b) return [k for k in sorted(keys) if a.get(k) != b.get(k)] python -m parity.run --corpus corpus/pricing.jsonl --fail-fast Illustrative sample output:
rows=412 equal=409 differing=3 pricing.jsonl#9f2c1a total_cents 4120 != 4125 pricing.jsonl#41bb07 discount_rows 2 != 1 pricing.jsonl#c8de10 rounding "half_up" != "bankers" Three diffs in 412 rows. Two usually turn out to be real bugs. One usually turns out to be a normalizer gap. Sort them before you flip anything.
Step 5: normalize in exactly one function
Volatile fields break exact equality. Sort keys. Drop generated ids. Round floats once, in one place.
Do not normalize anything else. Every field you drop is a field you stop testing.
If a diff survives normalization, decide: expected change, or bug. Write that decision into the corpus row as a comment field. Undocumented diffs become permanent.
Step 6: flip, hold, delete
Flip the flag in one environment. Run the differ again on the same corpus. Zero diffs, or a written reason per diff.
Then hold for one release. Then delete the old function and the flag together. A grep check proves the branch is gone.
rg "old_compute_totals|PRICING_NEW" src/ || echo "flag removed" An old branch with no test that fails on its return is not removed. It is just hidden.
When dual-run is not worth it
| Signal | Dual-run helps | Cheaper alternative |
|---|---|---|
| Pure function, dict output | Yes | none needed |
| Writes to a database | Only with stubs | transaction rollback test |
| Sends email or charges cards | No | canary at one call site |
| Output is rendered HTML | Partly | snapshot diff on markup |
| Only latency changes | No | load test |
Limitations
- Corpus coverage is the ceiling. Parity on 412 rows proves 412 rows.
- Side effects double. Stub them or you will charge a card twice.
- Over-normalization hides drift. Review the normalizer like production code.
- Rare branches need hand-written inputs, not recorded traffic.
- Free-tier access is convenience, not a guarantee. Keep your own runner.
Do not use this on payment capture, irreversible migrations, or anything you cannot replay.
Who should skip this
Teams without recorded inputs should start at the corpus step, not the harness. Teams on a refactor that changes latency only get nothing from output diffing. Teams without a rollback path should fix that first.
Close
Small seam, frozen inputs, one differ, one flag, one deletion grep. That is the whole method.
If the corpus-mutation step is the tedious part for you, that is the step worth testing with free model access first.
