Core claim: A green run after an agent patch is not validation. It is a zero exit code from a suite the model was allowed to edit. Route every failing nodeid into one of three planes—property, fixture, or freeze—before any generator may touch a test file. Reject the patch if a case changes planes, even when CI is green. That rule is dull. It blocks the cheapest cheats: deleted asserts, rewritten goldens, and a fresh pytest.skip. Green is the wrong oracle Agent patches optimize the signal you reward. Reward a zero exit code, and the cheapest diffs are skips, looser matchers, and committed fixtures that match the new bug. Line coverage does not catch this. A mutation score on production code does not catch it either. The suite is the surface the model can mutate. A practical countermeasure is not a larger model. It is a failure taxonomy the model cannot rewrite. Three planes, one membership Treat each collected test as belonging to exactly one plane. Unclassified is not “the agent may proceed.” Unclassified means stop. Plane Question the test answers Agent may edit production code? Agent may edit this test? Failure policy Property Does an invariant hold across many inputs? Yes No Fail the patch Fixture Does a known input still match locked bytes? Yes No (hash lock) Fail on hash or byte change Freeze Is this case known-noisy for a dated reason? Yes, but not to “fix” the flake No Fail if skip/xfail is added in-tree The table is the contract. The scripts below are a proposed harness, not a measured production run. 1. Inventory nodeids, then stop inventing names Collect once, on a runner the agent cannot write. pytest --collect-only -q tests/ | tee /var/gate/collect.txt python3 tools/plane_inventory.py /var/gate/collect.txt \ --manifest tests/planes.json \ --out /var/gate/inventory.json planes.json is human-reviewed. New tests do not default to writable. { "property": [ "tests/test_invariants.py::test_parse_roundtrip", "tests/test_invariants.py::test_idempotent_normalize" ], "fixture": [ "tests/test_corpus.py::test_golden_csv" ], "freeze": [ "tests/test_net.py::test_retry_window" ] } If collect.txt contains a nodeid missing from all three lists, the inventory command exits 2. Drift is a failed gate, not a suggestion. Proposed inventory checker: # tools/plane_inventory.py — proposed from __future__ import annotations import argparse, json, sys from pathlib import Path def nodeids(collect_txt: str) -> list[str]: rows = [] for line in Path(collect_txt).read_text().splitlines(): line = line.strip() if line.endswith(".py") or not line or line.startswith("="): continue if "::" in line: rows.append(line) return rows def main() -> int: p = argparse.ArgumentParser() p.add_argument("collect_txt") p.add_argument("--manifest", required=True) p.add_argument("--out", required=True) args = p.parse_args() man = json.loads(Path(args.manifest).read_text()) assigned = set() for plane in ("property", "fixture", "freeze"): for nid in man.get(plane, []): if nid in assigned: print(f"duplicate plane membership: {nid}") return 2 assigned.add(nid) missing = [n for n in nodeids(args.collect_txt) if n not in assigned] extra = sorted(assigned - set(nodeids(args.collect_txt))) Path(args.out).write_text(json.dumps({ "missing_from_planes": missing, "planes_without_collect": extra, }, indent=2) + "\n") if missing or extra: print(Path(args.out).read_text()) return 2 return 0 if __name__ == "__main__": raise SystemExit(main()) 2. Property oracles live where the workspace cannot write Property tests should not call helpers the patch is allowed to rewrite if those helpers are the oracle. Keep oracles in oracles/, owned by CI. Proposed example (unexecuted): # oracles/parse_roundtrip.py from hypothesis import given, settings, strategies as st from mypkg.parse import parse, dump from mypkg.normalize import normalize PRINTABLE = st.text(min_size=0, max_size=64) @settings(max_examples=80, deadline=None) @given(PRINTABLE) def test_parse_roundtrip(s: str) -> None: again = dump(parse(s)) assert parse(again) == parse(s) @given(st.lists(st.integers(min_value=0, max_value=255), max_size=40)) def test_idempotent_normalize(raw: list[int]) -> None: b = bytes(raw) assert normalize(normalize(b)) == normalize(b) Eighty examples is a budget, not a proof. Raise it when the invariant is cheap. Drop Hypothesis if the domain is an explicit corpus; the plane still applies. The gate accepts a patch only if this directory’s tree hash is unchanged. find oracles -type f -print0 | sort -z | xargs -0 sha256sum \ > /var/gate/oracles.sha256 cmp /var/gate/oracles.sha256 /var/gate/oracles.sha256.expected 3. Fixture bytes get a manifest, not a comment Golden files are the second cheapest thing a model will rewrite. Hash them outside the working tree. # tools/fixture_lock.py — proposed from __future__ import annotations import hashlib, json, sys from pathlib import Path def digest(p: Path) -> str: h = hashlib.sha256() h.update(p.read_bytes()) return h.hexdigest() def main(manifest_path: str) -> int: man = json.loads(Path(manifest_path).read_text()) errors = [] for rel, expected in man["files"].items(): p = Path(rel) if not p.is_file(): errors.append(f"missing {rel}") continue got = digest(p) if got != expected: errors.append(f"{rel}: expected {expected[:12]} got {got[:12]}") if errors: print("\n".join(errors)) return 3 return 0 if __name__ == "__main__": raise SystemExit(main(sys.argv[1])) Manifest shape: { "files": { "tests/fixtures/orders.v1.csv": "c6b0c1a9d0e1", "tests/fixtures/empty.xml": "e3b0c44298fc" } } A patch may change production code that reads these files. It may not change the files, the manifest, or a test that swapped assert actual == expected for assert actual is not None. Enforce the path deny-list on the merge-base diff. # tools/forbid_paths.py — proposed from __future__ import annotations import argparse, re, sys def main() -> int: p = argparse.ArgumentParser() p.add_argument("--deny", action="append", default=[]) args = p.parse_args() names = [ln.strip() for ln in sys.stdin if ln.strip()] bad = [] for name in names: for pat in args.deny: if re.search(pat, name): bad.append(name) break if bad: print("forbidden paths in patch:") print("\n".join(bad)) return 3 return 0 if __name__ == "__main__": raise SystemExit(main()) git diff --name-only origin/main...HEAD | python3 tools/forbid_paths.py \ --deny '^tests/fixtures/' \ --deny '^tests/planes.json$' \ --deny '^oracles/' \ --deny '^freeze_ledger.jsonl$' 4. Freeze flakes in a ledger, never with pytest.skip pytest.skip, xfail, and deleted sleeps are high-probability agent edits. They also erase the only record of why the case was noisy. Use an append-only JSONL ledger the runner reads. Mount it read-only in the agent workspace. {"nodeid":"tests/test_net.py::test_retry_window","reason":"clock-skew-on-shared-runner","owner":"platform","expires":"2026-09-19"} Proposed expiry check: # tools/freeze_gate.py — proposed from __future__ import annotations import json, os, sys from datetime import date from pathlib import Path def load_ledger(path: Path) -> dict[str, dict]: rows = {} for line in path.read_text().splitlines(): if not line.strip(): continue row = json.loads(line) rows[row["nodeid"]] = row return rows def main() -> int: ledger = load_ledger(Path("freeze_ledger.jsonl")) today = date.fromisoformat(os.environ.get("GATE_DATE", date.today().isoformat())) stale = [n for n, row in ledger.items() if date.fromisoformat(row["expires"]) < today] if stale: print("expired freeze entries:") print("\n".join(stale)) return 4 Path("/var/gate/freeze_allow.txt").write_text("\n".join(ledger) + "\n") return 0 if __name__ == "__main__": raise SystemExit(main()) Deselect frozen nodeids. Do not skip them in source. The ledger expiry is the only thaw. python3 tools/freeze_gate.py mapfile -t FROZEN < /var/gate/freeze_allow.txt deselect=() for n in "${FROZEN[@]}"; do deselect+=(--deselect "$n"); done pytest tests oracles "${deselect[@]}" Fourteen days is a policy, not a law. Pick a window someone actually reviews. A freeze that never expires is a skip with extra steps. Tripwire the diff for skip/xfail/pass hunks in tests/. Regex is not a proof. It is a cheap closed door. # tools/scan_test_hunks.py — proposed from __future__ import annotations import re, sys BANNED = ( re.compile(r"^\+\s*(pytest\.mark\.skip|pytest\.skip\(|pytest\.xfail)"), re.compile(r"^\+\s*@pytest\.mark\.(skip|xfail)"), re.compile(r"^\+\s*pass\s*$"), re.compile(r"^\-\s*assert\s+"), ) def main() -> int: hits = [] in_tests = False for i, line in enumerate(sys.stdin, 1): if line.startswith("diff --git") and " tests/" in line: in_tests = True elif line.startswith("diff --git"): in_tests = False if not in_tests: continue for pat in BANNED: if pat.search(line): hits.append(f"L{i}: {line.rstrip()}") break if hits: print("test hunk policy violations:") print("\n".join(hits)) return 5 return 0 if __name__ == "__main__": raise SystemExit(main()) git diff origin/main...HEAD | python3 tools/scan_test_hunks.py 5. Sequence the gate so the model never sees unclassified red Order matters. Do not invert it. Inventory nodeids against planes.json. Exit 2 on drift. Verify oracle tree hash and fixture manifest. Exit 3 on mismatch. Enforce freeze ledger expiry. Exit 4 if stale. Deselect the rest. Run property and fixture planes only. Apply the candidate patch on a throwaway worktree. Re-run steps 1–4. Fail if plane membership changed, forbidden paths changed, or properties/fixtures failed. Only then inspect residual unclassified failures. Humans classify them. The model does not. A cheap generation loop helps because the expensive part is the gate, not the proposal. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access can emit the candidate patch; the free server option can host /var/gate so the ledger, fixture hashes, and oracle tree stay off the writable workspace. Remove both and the same sequence still holds on any CI runner the agent cannot push to. Residual failures: classify, do not skip the row Symptom Likely plane Action Do not Fails on many random seeds, oracle unchanged Property Keep failing the patch Shrink example count to force a pass Fails on one file, bytes differ Fixture Diff the bytes; version a new corpus in review Let the agent overwrite the golden Passes locally, fails on shared CPU/clock/net Freeze Add a dated ledger row with owner Add skip in the module Assertion deleted in the diff Taxonomy violation Reject Re-run until green New test file with no plane Unclassified Human adds a plane row Default to fixture Limitations This workflow assumes at least one invariant worth encoding and a location the agent cannot write. It does not make flaky browser tests deterministic. It does not replace review of production diffs. Hash locks will fight legitimate corpus updates; those updates must take the same human inventory path. Expiry on the ledger will rot if nobody owns the calendar. Hypothesis budgets, SHA-256 manifests, and --deselect are tools. They will not detect a wrong invariant written by a human. Who should not use this Do not install a three-plane gate on a prototype whose tests are scaffolding you expect to throw away this week. Do not use it as a substitute for quarantining a shared mutable staging database. Do not point the agent at the default branch and hope file permissions hold. Mount /var/gate read-only, or keep it on another host. If the suite is entirely time-boxed UI flows with no extractable property, freezing everything teaches nothing. Extract one pure function first. The method is for teams that already let a generator touch production code and have watched the suite get quieter for the wrong reasons. Start with planes.json and a single freeze row. Wire a model only after the inventory command fails closed on drift.