Your Fixtures Are the Spec: Fixture Mutation as the Missing Gate for Agent Patches

An agent patch can change what a fixture means without touching a single assertion. The suite stays green. Your code-mutation job stays green too, because it mutates the code under test — not the hand-built data your tests were quietly calibrated against.

That gap is the one I keep seeing in review: the patch is fine, the tests are fine, and the contract between them drifted. The fix is not more tests. It is one more gate that fails when a fixture stops being load-bearing.

Below is a three-gate harness you can wire into CI this week: fixture mutation, property checks with a hard case budget, and a flake quarantine keyed on failure signature instead of test name. All three emit a single JSON receipt your pipeline can read.

1. The blind spot, concretely

The example below is illustrative, not a benchmark — but it is the shape of the failure.

# src/config.py, before the patch def load(raw: dict) -> dict: return {"db": raw["db"], "cache": raw["cache"]} 

# tests/test_config.py def test_db_host(config): assert config["db"]["host"] == "localhost" 

The agent patch changes load to return a dataclass. The fixture still builds a plain dict in the new shape, because the fixture was written by hand, not by calling load. The test passes for a reason that has nothing to do with the patch.

Code mutation never finds this. Deleting a line in load may not even be reachable from a test that never calls it.

2. Gate 1 — Mutate the fixture, not the code

Start from the assumption that each fixture is a specification. Then ask the mutation question about the specification.

Step 1: declare mutants next to the fixture. One dict, one callable per mutant, no cleverness.

# gate/fixture_mutants.py from typing import Any, Callable Mutator = Callable[[Any], Any] MUTANTS: dict[str, dict[str, Mutator]] = { "invoice": { "lines-empty": lambda fx: {**fx, "lines": []}, "qty-zero": lambda fx: {**fx, "lines": [{**l, "qty": 0} for l in fx["lines"]]}, "total-negative": lambda fx: {**fx, "total_cents": -abs(fx["total_cents"])}, "currency-null": lambda fx: {**fx, "currency": None}, "created-utc": lambda fx: {**fx, "created_at": fx["created_at"].replace(tzinfo=None)}, }, } 

Step 2: apply the active mutant in conftest.py, and fail loudly on typos.

# conftest.py import os import pytest from gate.fixture_mutants import MUTANTS @pytest.fixture def invoice(base_invoice): name = os.environ.get("FIXTURE_MUTANT") if not name: return base_invoice table = MUTANTS["invoice"] if name not in table: pytest.fail(f"unknown fixture mutant: {name}") return table[name](base_invoice) 

Step 3: run each mutant in its own process. Shared state between mutants manufactures false kills.

# gate/run_fixture_mutation.py import json, os, subprocess, sys from gate.fixture_mutants import MUTANTS def run(node_ids, mutant): env = {**os.environ, "FIXTURE_MUTANT": mutant} return subprocess.run( [sys.executable, "-m", "pytest", "-q", "-x", *node_ids], env=env, capture_output=True, text=True, ).returncode def main(): node_ids = sys.argv[1:] if not node_ids: sys.exit("usage: run_fixture_mutation.py <pytest node ids>") report = {"killed": [], "survived": []} for mutant in MUTANTS["invoice"]: bucket = "killed" if run(node_ids, mutant) != 0 else "survived" report[bucket].append(mutant) report["verdict"] = "pass" if not report["survived"] else "gap" print(json.dumps(report, indent=2)) sys.exit(0 if report["verdict"] == "pass" else 1) 

Step 4: triage survivors, do not silence them. A survivor is one of three things: a real gap in the tests, an equivalent mutant that cannot be observed, or a fixture that should not exist. Record which, and why, in the receipt. An unlabeled survivor is a lie you tell your CI.

Step 5: seat the gate on touched fixtures only. Derive the fixture list from the diff; run the full matrix nightly. A five-mutant matrix over one module costs seconds. Over eighty modules it does not.

3. Gate 2 — Property checks with a budget and a shrink log

Fixture mutation proves your examples are load-bearing. It does not prove the input space is covered. Keep property checks narrow and bounded.

from hypothesis import given, settings, strategies as st @given(st.lists(st.integers(min_value=0, max_value=10_000), min_size=1, max_size=8)) @settings(max_examples=200, deadline=None, derandomize=True) def test_total_is_never_negative(qty_and_price): ... 

Three rules that keep this gate honest:

1. derandomize=True plus a committed seed profile, so a red run is reproducible on the reviewer's laptop.
2. A fixed max_examples. Unbounded property runs are flaky tests wearing a lab coat.
3. The shrunk counterexample goes into the receipt verbatim. If the report says "property failed" and nothing else, the gate will be disabled within a month.

4. Gate 3 — Quarantine by signature, not by name

Flaky-test freezes rot because they freeze a name. A name says nothing about whether the failure you froze is the failure you are seeing now.

Key the freeze on a normalized failure signature instead:

{ "node": "tests/test_invoice.py::test_rounding", "signature": "AssertionError|0.01 != 0.010000000000000009|test_invoice.py:88", "first_seen": "2026-11-02", "expires": "2026-11-16", "owner": "@finley", "repro": "18/200", "reason": "float accumulation in legacy rounding path" } 

The gate fails when: the signature differs from what the run produced, the entry is past expires, the owner is empty, or repro is absent. Four conditions, all checkable mechanically.

5. One receipt, three verdicts

{ "fixture_mutation": { "verdict": "gap", "survived": ["total-negative"] }, "properties": { "verdict": "pass", "cases": 200, "shrunk": null }, "quarantine": { "verdict": "pass", "active": 2, "expired": 0 } } 

Emit it as an artifact on every run. Reviewers read the diff and the receipt together; the receipt answers "what did this patch have to survive?" without anyone re-reading the suite.

6. Which gate is mandatory for which patch

If a patch class is missing from the table, default to all three and argue your way down in review.

7. Where a free model tier actually helps

Two steps in this harness are cheap, bounded, and tedious: proposing fixture mutants from a diff, and drafting property generators for a new invariant. Both are good fits for a coding agent, and both fail badly if you hand them an unbounded budget.

This is where MonkeyCode's free model access fits in my loop: I paste the diff and the touched fixtures, ask for at most five mutants per fixture, and treat the output as a proposal list that a human filters before it enters fixture_mutants.py. The surviving-mutant triage stays manual — distinguishing "real gap" from "equivalent mutant" needs the judgment the model does not have. The free server option is useful for the nightly full matrix, which is CPU-bound, long, and does not need to run inside the main CI queue.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Check the project page for current limits, quotas, and server terms — those change, and this article deliberately does not restate them.

8. Limitations, and who should skip this

• Cost scales with mutants times suite time. Ten mutants over a four-minute targeted suite is fine. Over a forty-minute suite it is a nightly job, not a PR gate.
• Equivalent mutants are the tax. Some fixture changes are unobservable by design. Budget time for triage or the gate becomes noise.
• Weak generators pass quietly. A property test over st.integers() proves nothing about your timezone handling. Property checks need adversarial generators, not default ones.
• Signature normalization is runner-sensitive. Normalize paths and float formatting before hashing, or you will quarantine the same flake twice.
• Quarantine hides bugs if left unowned. An entry without an owner and an expiry is a permanent mute button.
• Skip Gate 1 entirely if your tests build data inline per test and never share a fixture — you have already solved the problem structurally.
• Do not treat this as a replacement for code mutation. It is an additional gate over the specification layer, not a better version of the existing one.

Start with one fixture

Pick the fixture your last three agent patches touched. Write five mutants. Run the runner above against the tests that use it.

Most suites I have looked at this way survive at least one mutant they should have killed. That single survivor is more useful than another hundred lines of tests written after the fact — and it costs one afternoon.

If you run the loop and want a second read on a survivor you cannot classify, the receipt format above makes that conversation short.