Characterize the Failure Path First: A Fault Matrix Before Your Smallest Refactor

Output snapshots tell you what legacy code returns. They rarely tell you what it does before it fails. That gap is where small refactors break production. This post builds a fault matrix first, then makes one change.

The claim is narrow. Before you touch a messy function, record its ordered side effects under injected boundary failures. Then refactor, re-run, and count drifting snapshots. Zero drift means the failure contract held.

The harness below is a template. I have not run it on your repo. Treat every number as something you measure locally.

Why output-only characterization leaks

A golden-output test passes if the return value matches. It says nothing about whether debit fired before save. It says nothing about how many retries happened after a timeout.

Legacy code accumulates order-dependent behavior. Some of that order is load-bearing. Some is accidental. You cannot tell them apart without a trace.

So capture two things per case: the outcome, and the ordered list of calls that happened before the outcome. The second list is the actual contract.

Step 1 — Pick one boundary, not one function

Do not characterize the whole module. Choose the single call path you plan to change. Find the outer boundary it crosses: a client, a repository, a mailer.

Watch three to five calls on that boundary. Writes first. Reads second. Notification calls last.

If you cannot name the boundary in one sentence, the slice is too big. Split it.

Step 2 — Write five faults, not fifty

Five faults cover most of the failure surface. Each one pins a different branch.

Keep the list in code, not prose. Faults drift when they live in a wiki.

# tests/characterization/cases.py from checkout import ledger, mailer WATCH = [(ledger, "debit"), (ledger, "save"), (mailer, "send")] FAULTS = [ {"name": "dependency_raises", "target": "checkout.ledger.fetch_balance", "exc": TimeoutError("db timeout")}, {"name": "returns_empty", "target": "checkout.ledger.fetch_balance", "exc": None, "ret": []}, {"name": "returns_none", "target": "checkout.ledger.fetch_balance", "exc": None, "ret": None}, {"name": "returns_duplicate", "target": "checkout.ledger.fetch_balance", "exc": None, "ret": [{"id": 7, "amt": 5}, {"id": 7, "amt": 5}]}, ] CASE = { "name": "place_order_happy", "watch": WATCH, "entry": lambda: ledger.place_order(cart_id=42, user_id=9), } 

Step 3 — Record outcome plus call order

One recorder handles all cases. It spies on watched calls, injects the fault, and returns a JSON record.

# tests/characterization/charfault.py import json, pathlib from unittest import mock SNAP = pathlib.Path("tests/characterization/snapshots") def spy(obj, name, calls): real = getattr(obj, name) def wrapper(*args, **kwargs): calls.append([name, repr(args)[:120], repr(sorted(kwargs))]) return real(*args, **kwargs) return mock.patch.object(obj, name, wrapper) def run(case, fault): calls, patchers = [], [] for obj, name in case["watch"]: patchers.append(spy(obj, name, calls)) if fault: side = fault.get("exc") if side is None: patchers.append(mock.patch(fault["target"], return_value=fault["ret"])) else: patchers.append(mock.patch(fault["target"], side_effect=side)) for p in patchers: p.start() try: value = case["entry"]() outcome = {"status": "ok", "value": repr(value)[:200]} except Exception as exc: outcome = {"status": "raised", "type": type(exc).__name__} finally: for p in reversed(patchers): p.stop() return { "case": case["name"], "fault": fault["name"] if fault else "nofault", "outcome": outcome, "calls": calls, } def check(records): drift = [] for rec in records: path = SNAP / f"{rec['case']}__{rec['fault']}.json" text = json.dumps(rec, indent=2, sort_keys=True) + "\n" if not path.exists(): path.parent.mkdir(parents=True, exist_ok=True) path.write_text(text) elif path.read_text() != text: drift.append(path.name) return drift 

Note the nofault run. It is a control. It proves the harness spies without changing behavior.

Step 4 — Prove the harness is deterministic

Run the matrix twice on untouched code. Compare the two output sets.

python -m tests.characterization.record --out /tmp/a python -m tests.characterization.record --out /tmp/b diff -r /tmp/a /tmp/b && echo IDEMPOTENT 

If that diff is not empty, your snapshots contain noise: random ids, real timestamps, dict ordering from sets. Normalize those fields now.

A non-deterministic matrix cannot gate anything. It only produces arguments.

Commit the snapshots. They are the pre-refactor contract.

Step 5 — Make one change, then count drift

Refactor the smallest thing that changes behavior you intend to change. Nothing else.

Re-run the matrix. The gate is a number, not a feeling.

git checkout -b refactor/place-order git switch -c refactor/place-order python -m tests.characterization.record --check 

Two is a budget, not a law. Set it before you start, not after you see the number.

Step 6 — Re-baseline with a reason code

Sometimes drift is the point. A refactor that removes a duplicate send call should change a snapshot.

Re-baseline explicitly. Name the reason in the commit message.

refactor(checkout): drop duplicate mailer.send on empty cart Drift: place_order_happy__returns_empty.json Reason: send fired twice; second call had no order id 

Silent snapshot regeneration erases the evidence. Always diff before you accept.

Where a free model and a free server fit

Two parts of this loop are mechanical. Drafting the fault list from boundary signatures is one. Running the matrix in a clean environment is the other.

MonkeyCode's free model access is useful for the first part: paste the boundary signatures, ask for a fault table, then delete anything you cannot tie to a real branch. Its free server option is useful for the second: run the matrix in a container that has no local env vars and no stale sqlite file. Snapshots should not depend on your laptop's state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat model output as a draft, not a contract. A generated fault that no branch can reach is noise. It costs seconds per run and earns nothing.

Availability, quotas, and model names change without notice. I am not quoting any of them here. Check the current offering yourself before you build a habit on it.

When this is worth the hour

Limitations

• Spies add overhead. Keep the watched list short.
• Mocking the boundary hides real driver behavior. Run one integration test separately.
• Snapshots encode repr() output. Change a __repr__ and every snapshot drifts.
• Fault injection at the wrong layer tests your mocks, not your code.
• This records what code does, not what it should do. Preserve a bug knowingly or fix it in its own patch.

Who should skip this

Skip it if your refactor has no side effects to order. Skip it if you cannot make the run deterministic within one sitting. Skip it if your team will regenerate snapshots without reading the diff.

That last one is the real risk. A gate nobody reads is a ritual.

Start with one case and five faults. If the first matrix run shows zero drift and you learned nothing, delete it and move on.

If you do try it on one legacy function this week, tell me which fault exposed a branch you did not know existed.