The Patch Edited the Fixture, Not the Code: Catching Fixture Laundering in Agent Patches

The most common way an agent patch defeats a test suite is not deleting an assertion. It regenerates the fixture that the assertion reads. The test stays green, the test count is unchanged, and the assertion silently starts measuring a different world.

This post is about making that move visible and expensive. It describes a small gate that classifies every fixture change by its bytes, forces the patch to declare the same class in a commit trailer, and re-runs a pinned probe set with base fixtures against head code so behavior drift cannot hide behind a fixture refresh.

Why fixture edits survive review

Source diffs get read line by line. A 400-line fixture diff gets read as "file changed" and approved, because the reviewer's question for fixtures is "did you regenerate?" and regeneration is exactly what an agent under pressure to go green will do.

There is a second, quieter problem. Most fixture changes are legitimate. Vendor formats move, rounding rules change, an upstream schema adds a field. A gate that forbids fixture edits gets switched off within a week. The gate has to sort legitimate drift from laundering, not block both.

Four classes, four different risks

Classify by what changed in the data, not by which file changed.

The right-hand column is the whole policy. REMOVED blocks because a deleted fixture is a deleted test in disguise. SEMANTIC_REWRITE is the laundering signature — same fields, different strings or categories, so every assert x == "premium" now compares against something else. TOLERANCE_DRIFT is usually honest rounding, which is why it only needs a reason, not a debate.

Step 1 — Hash the fixture tree on both trees

You need two checked-out trees: the merge base and the patch head. Hashing both gives you the change set without trusting the diff.

git worktree add --detach /tmp/base "$(git merge-base origin/main HEAD)" git worktree add --detach /tmp/head HEAD 

Step 2 — Classify each changed file by content

The classifier below is stdlib-only and runnable as written. It walks JSON fixtures structurally; anything binary or unparseable falls back to SEMANTIC_REWRITE, which is the conservative answer.

#!/usr/bin/env python3 """fixture_gate.py — classify fixture drift between two trees.""" from __future__ import annotations import argparse, hashlib, json, math, re, sys from pathlib import Path TRAILER = re.compile( r"^Fixture-Change:\s+(?P<cls>[A-Z_]+)\s+(?P<path>\S+)\s+(?P<reason>.+)$", re.M ) def digest(p: Path) -> str: return hashlib.sha256(p.read_bytes()).hexdigest()[:16] def tree_hashes(root: Path) -> dict[str, str]: if not root.exists(): return {} return {p.relative_to(root).as_posix(): digest(p) for p in sorted(root.rglob("*")) if p.is_file()} def classify_json(old, new, eps: float = 1e-6) -> str: shape = drifted = False def cmp(x, y) -> None: nonlocal shape, drifted if isinstance(x, dict): if set(x) != set(y): shape = True for k in set(x) & set(y): cmp(x[k], y[k]) elif isinstance(x, list): if len(x) != len(y): shape = True for a, b in zip(x, y): cmp(a, b) elif isinstance(x, bool) or x is None or isinstance(x, str): if x != y: drifted = True elif isinstance(x, (int, float)) and isinstance(y, (int, float)): if not math.isclose(x, y, rel_tol=eps, abs_tol=eps): drifted = True elif type(x) is not type(y): shape = True elif x != y: drifted = True cmp(old, new) if shape: return "SHAPE_CHANGE" return "SEMANTIC_REWRITE" if drifted else "TOLERANCE_DRIFT" def classify_pair(old: Path, new: Path) -> str: if old.suffix == ".json" and new.suffix == ".json": try: return classify_json(json.loads(old.read_text()), json.loads(new.read_text())) except (json.JSONDecodeError, UnicodeDecodeError): pass return "SEMANTIC_REWRITE" def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--base-tree", required=True) ap.add_argument("--head-tree", required=True) ap.add_argument("--fixtures", default="tests/fixtures") ap.add_argument("--message-file", required=True) ap.add_argument("--report", default="fixture-report.json") args = ap.parse_args() base = tree_hashes(Path(args.base_tree) / args.fixtures) head = tree_hashes(Path(args.head_tree) / args.fixtures) claims = {m["path"]: m for m in (g.groupdict() for g in TRAILER.finditer( Path(args.message_file).read_text(encoding="utf-8")))} findings, blocking = [], [] for rel in sorted(set(base) | set(head)): if rel not in base: cls = "ADDED" elif rel not in head: cls = "REMOVED" elif base[rel] == head[rel]: continue else: cls = classify_pair(Path(args.base_tree) / args.fixtures / rel, Path(args.head_tree) / args.fixtures / rel) claim = claims.get(rel) declared = bool(claim and claim["cls"] == cls and claim["reason"].strip()) verdict = "accept" if cls == "REMOVED": verdict = "block" elif cls != "ADDED" and not declared: verdict = "block" findings.append({"path": rel, "class": cls, "declared": declared, "verdict": verdict}) if verdict == "block": blocking.append(rel) Path(args.report).write_text(json.dumps( {"fixtures": args.fixtures, "findings": findings, "blocking": blocking}, indent=2) + "\n", encoding="utf-8") for f in findings: print(f"{f['class']:18} {f['verdict']:6} {f['path']}") return 1 if blocking else 0 if __name__ == "__main__": sys.exit(main()) 

Step 3 — Make the trailer match the classifier

The trailer is the accountability hook. A human or agent writing the patch has to name the class before the gate computes it, and the gate rejects a mismatch in both directions.

Fix FX rounding in rate loader Fixture-Change: TOLERANCE_DRIFT tests/fixtures/rates.json vendor spec v3 rounds to 6dp Fixture-Change: SHAPE_CHANGE tests/fixtures/accounts.json upstream added `region` 

Two properties matter here. First, the class is written by the patch author, so a mismatch is a signal that someone did not look. Second, the reason is prose, so a SEMANTIC_REWRITE on a fixture full of expected-status strings cannot be waved through with the word "update".

Run it:

python fixture_gate.py \ --base-tree /tmp/base --head-tree /tmp/head \ --fixtures tests/fixtures \ --message-file /tmp/head/.git/COMMIT_EDITMSG \ --report fixture-report.json echo "exit=$?" # 1 when any fixture change is undeclared or a deletion 

A synthetic patch producing one accepted drift and one blocked rewrite reports like this:

TOLERANCE_DRIFT accept tests/fixtures/rates.json SEMANTIC_REWRITE block tests/fixtures/accounts.json 

Step 4 — Pin fixtures to base, run probes against head code

The classifier catches dishonest fixture edits. It does not catch the case where the fixture edit is required because the code changed behavior — which is precisely the case that should go through review, not through a green run.

The probe step separates those. Overwrite the head tree's fixtures with the base revisions, keep the patch's source, and run a probe suite that reads only fixtures.

BASE=$(git rev-parse origin/main) git archive "$BASE" tests/fixtures | tar -x -C /tmp/head python -m pytest probes/ -q -p no:cacheprovider --maxfail=1 

Probes are cheap by construction because they only read data. Three kinds cover most fixture contracts: uniqueness, shape floor, and range/sign constraints.

# probes/test_rate_fixture_contract.py import json, pathlib import pytest FIX = pathlib.Path("tests/fixtures/rates.json") @pytest.fixture(scope="module") @pytest.mark.usefixtures() def rates(): return json.loads(FIX.read_text(encoding="utf-8"))["rates"] def test_pair_keys_unique(rates): pairs = [r["pair"] for r in rates] assert len(pairs) == len(set(pairs)) def test_record_fields_do_not_shrink(rates): assert all(set(r) >= {"pair", "value", "as_of"} for r in rates) def test_values_positive(rates): assert all(isinstance(r["value"], (int, float)) and r["value"] > 0 for r in rates) 

A probe failure on pinned fixtures is the useful outcome. It means the patch changed observable behavior and rewrote the data to match, so the PR needs a Behavior-Change note and a reviewer who understands the domain, not another regeneration cycle.

Flaky tests: key the freeze to the fixture hash

Freezes rot when they are keyed to a date, because nobody remembers why the entry exists. Key them to evidence instead: (test_id, fixture_hash).

That single change gives the ledger a useful property. When the fixture changes, the hash no longer matches, the freeze entry expires automatically, and the test must be re-justified against the new data before it can be quarantined again. Unfreezing is then a replay condition — capture a fixed number of distinct failing seeds, replay them deterministically, and drop the freeze when the replay passes — rather than a calendar event someone has to police.

Where a hosted model and server actually help

This gate is tedious to write and cheap to review, which is the right shape for delegation. I drafted the classifier's edge-case handling and the probe skeletons with MonkeyCode's free model access, and used the free server option to run the probe loop on a throwaway workspace while the shell wiring was still wrong — availability as supplied by the operator, not a benchmark claim.

The division of labor that held up: the model proposes fixture classes and probe invariants, a human decides which invariants are contractual, and CI enforces what survives. Nothing in that loop needs a paid seat to start.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations

• Binary fixtures hash-compare only, so every change lands in SEMANTIC_REWRITE and blocks. Keep them out of --fixtures and review them by hand.
• Non-deterministic fixtures (embedded timestamps, generated UUIDs) will churn the report every run. Normalize them before hashing or the gate becomes noise.
• The classifier proves consistency between the trailer and the bytes, not intent. An author who writes a matching trailer for a bad rewrite still passes; that is what step 4 and human review are for.
• Golden-file suites, where the fixture is the expected output, defeat the whole model. There, a fixture edit is an assertion edit, and it should route to a separate lane with mandatory human approval.
• Hashing a fixture directory with tens of thousands of files costs real seconds. Sample by directory, or restrict to the directories the patch touched.

Who should not use this

Skip it if your fixtures are generated at test time and never committed — there is nothing to audit. Skip it for single-file scripts and prototypes. Skip it if you cannot produce a stable base commit, because "the fixture at the time of branching" is the only reference point that makes comparison meaningful.

The short version

Fixture changes are assertions in disguise, so treat them like assertions: classify them, make someone name the class, and prove the class survives a probe run against the previous data. If you want to try the drafting loop before wiring the gate into CI, MonkeyCode's free tier is a reasonable place to prototype the classifier and probes without committing budget.