A green CI job is not proof that an agent patch preserved the test world. Fixtures can be rewritten to match new output. Property suites can shrink their domain or their iteration budget. Flaky tests can turn into unnamed skips. Merge on a lockfile and two matching receipts, not on a single passing run. The protocol below is a proposal with runnable scoring code. It is not a report of production incident rates, and it does not assume a particular CI vendor. The test world is the proof, not the job status The test world is the set of objects that give tests their meaning. Three parts matter on an agent diff. Fixture bytes and their content digest. Property checks: which invariants, how many examples, which seed policy. Flake policy: which node ids may skip, until when, and who issued the lease. If the model may edit those three, a passing suite proves almost nothing. The code under test and the proof of that code moved together. That is cheap. It is also common. Three failure modes the lockfile is built to catch Fixture rewrite. A snapshot or JSON fixture is updated so the new function output becomes the expected value. The test still passes. The old contract is gone. Property shrinkage. Strategies get narrower. max_examples drops. A filter excludes the input that used to fail. The job is faster and greener. The search is weaker. Flake freeze expansion. pytest.skip() appears inside a body the agent could not stabilize. Source-level mark.skip scans miss it. The hole remains after merge. None of these require malice. They are the lowest-cost way to make a suite green. Artifact: testworld.lock.json Keep this file in the repo. Treat it like a package lockfile. Agent patches must not write it. Human-labeled commits may. { "version": 1, "fixtures": { "tests/fixtures/invoice_v3.json": { "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "bytes": 0 } }, "properties": { "tests/test_invoice_properties.py::test_roundtrip_preserves_cents": { "min_examples": 200, "seed_policy": "fixed", "seed": 42, "strategy_fingerprint": "@given(cents=st.integers(0,10**9))" } }, "flake_leases": [ { "nodeid": "tests/test_tax.py::test_dst_boundary", "expires_unix": 1796124800, "skips_remaining": 3, "issuer": "human:oncall", "reason": "tzdata mismatch on one runner image" } ] } The empty-file digest and the expires_unix value are schema examples, not a measured corpus or a recommended lease length. Replace both from your tree and your policy. Step 1 — Digest fixture bytes, not filenames Filenames lie. Agents rename and copy. Hash contents. # digest_fixtures.py — illustrative scorer, not a published package from __future__ import annotations import hashlib import json from pathlib import Path FIXTURE_ROOT = Path("tests/fixtures") SUFFIXES = {".json", ".txt", ".bin", ".csv", ".snap"} def sha256_file(path: Path) -> tuple[str, int]: data = path.read_bytes() return hashlib.sha256(data).hexdigest(), len(data) def build_fixture_map() -> dict[str, dict]: out: dict[str, dict] = {} for path in sorted(FIXTURE_ROOT.rglob("*")): if path.is_file() and path.suffix in SUFFIXES: digest, n = sha256_file(path) key = path.as_posix() out[key] = {"sha256": digest, "bytes": n} return out if __name__ == "__main__": print(json.dumps(build_fixture_map(), indent=2, sort_keys=True)) A changed hash with no lockfile bump is a merge failure. A changed hash with a lockfile bump still needs a human-labeled commit. The agent job should not be allowed to produce that commit. Step 2 — Fingerprint properties, including the budget Collect node ids, a floor on examples, and a coarse strategy fingerprint. The fingerprint is a string, not a proof of semantic equivalence. It catches the cheap shrinkage: smaller integers, shorter strings, extra assume() filters that collapse the domain. # fingerprint_properties.py — proposal: parse a tiny subset of pytest + Hypothesis from __future__ import annotations import ast import json from pathlib import Path class PropertyVisitor(ast.NodeVisitor): def __init__(self, filename: str) -> None: self.filename = filename self.rows: list[dict] = [] def visit_FunctionDef(self, node: ast.FunctionDef) -> None: min_examples = None given_src = None for dec in node.decorator_list: src = ast.unparse(dec) if "given" in src: given_src = src if "settings" in src and "max_examples" in src: min_examples = self._max_examples(dec) if given_src is not None: self.rows.append( { "nodeid": f"{self.filename}::{node.name}", "min_examples": min_examples if min_examples is not None else 100, "strategy_fingerprint": "".join(given_src.split()), } ) self.generic_visit(node) @staticmethod def _max_examples(dec: ast.AST) -> int | None: for child in ast.walk(dec): if isinstance(child, ast.keyword) and child.arg == "max_examples": if isinstance(child.value, ast.Constant) and isinstance( child.value.value, int ): return child.value.value return None def scan(root: Path) -> dict[str, dict]: props: dict[str, dict] = {} for path in sorted(root.rglob("test_*.py")): tree = ast.parse(path.read_text(encoding="utf-8")) visitor = PropertyVisitor(path.as_posix()) visitor.visit(tree) for row in visitor.rows: props[row["nodeid"]] = { "min_examples": row["min_examples"], "seed_policy": "fixed", "strategy_fingerprint": row["strategy_fingerprint"], } return props if __name__ == "__main__": print(json.dumps(scan(Path("tests")), indent=2, sort_keys=True)) Scoring rule: fail if any locked node id disappears, if min_examples falls, or if the fingerprint string changes without a human lockfile edit. This parser is incomplete. Nested settings, imported decorators, and parametrized properties need a collection-time plugin. Use it as a gate that catches cheap edits, not as a full Hypothesis auditor. Step 3 — Issue flake leases. Never accept a bare skip A skip with no lease is a hole. A lease is a capability token. CI may decrement skips_remaining. CI may not extend expires_unix. Only a human-labeled lockfile commit may add or renew a lease. Collect skips from the pytest report. Do not grep source for @pytest.mark.skip alone. Agents skip inside the body. # conftest.py excerpt — illustrative from __future__ import annotations import json from pathlib import Path def pytest_terminal_summary(terminalreporter, exitstatus, config): skipped = sorted( item.nodeid for item in terminalreporter.stats.get("skipped", []) ) Path("observed_skips.json").write_text( json.dumps(skipped, indent=2) + "\n", encoding="utf-8" ) # score_leases.py — illustrative from __future__ import annotations import time from typing import Any def score_flake_leases( locked: list[dict[str, Any]], observed_skips: set[str], now: int | None = None, ) -> list[str]: now = int(time.time()) if now is None else now errors: list[str] = [] by_id = {row["nodeid"]: row for row in locked} for nodeid in sorted(observed_skips): row = by_id.get(nodeid) if row is None: errors.append(f"unleased skip: {nodeid}") continue if now >= int(row["expires_unix"]): errors.append(f"expired lease: {nodeid}") if int(row["skips_remaining"]) <= 0: errors.append(f"lease exhausted: {nodeid}") return errors Step 4 — Wipe the on-disk example database, then emit a receipt A second run that reuses .hypothesis/ or pytest's cache is not an independent observation. Hypothesis stores shrinking examples on disk. Pytest stores failure caches. An agent session can poison both before CI starts. # isolate_and_run.sh — proposal set -euo pipefail ROOT="$1" RECEIPT="$2" export HYPOTHESIS_DATABASE=":memory:" export PYTEST_ADDOPTS="--cache-clear -q" rm -rf "$ROOT/.hypothesis" "$ROOT/.pytest_cache" cd "$ROOT" python digest_fixtures.py > /tmp/fix.json python fingerprint_properties.py > /tmp/prop.json pytest tests python emit_receipt.py "$RECEIPT" /tmp/fix.json /tmp/prop.json observed_skips.json A single green run can still be session-tainted. Clone the candidate twice, with the agent process gone. Compare receipts. # emit_receipt.py — illustrative dual-run proof from __future__ import annotations import hashlib import json import sys from pathlib import Path def receipt_sha256(payload: dict) -> str: canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256(canonical.encode()).hexdigest() def main() -> None: out, fix, prop, skips = map(Path, sys.argv[1:5]) payload = { "fixtures": json.loads(fix.read_text(encoding="utf-8")), "properties": json.loads(prop.read_text(encoding="utf-8")), "observed_skips": json.loads(skips.read_text(encoding="utf-8")), } # Wall-clock duration is omitted on purpose. Duration noise would # make two honest runs disagree. doc = {"payload": payload, "sha256": receipt_sha256(payload)} out.write_text(json.dumps(doc, indent=2) + "\n", encoding="utf-8") if __name__ == "__main__": main() # compare_receipts.py — illustrative from __future__ import annotations import json import sys from pathlib import Path def main() -> None: left = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) right = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8")) if left["sha256"] != right["sha256"]: raise SystemExit( f"receipt mismatch: {left['sha256']} != {right['sha256']}" ) if __name__ == "__main__": main() Receipt payload should include fixture maps, property fingerprints, observed skips, and the git tree id of the candidate. It should not include timestamps or durations. Step 5 — Score the lockfile after the receipts match # score_against_lock.py — illustrative merge gate from __future__ import annotations import json import sys from pathlib import Path from score_leases import score_flake_leases def main() -> None: lock = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) receipt = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))["payload"] errors: list[str] = [] if receipt["fixtures"] != lock["fixtures"]: errors.append("fixture digest drift versus lock") locked_props = lock["properties"] seen_props = receipt["properties"] for nodeid, row in locked_props.items(): if nodeid not in seen_props: errors.append(f"property missing: {nodeid}") continue got = seen_props[nodeid] if int(got["min_examples"]) < int(row["min_examples"]): errors.append(f"property budget fell: {nodeid}") if got["strategy_fingerprint"] != row["strategy_fingerprint"]: errors.append(f"strategy fingerprint changed: {nodeid}") errors.extend( score_flake_leases( lock["flake_leases"], set(receipt["observed_skips"]) ) ) if errors: raise SystemExit("\n".join(errors)) if __name__ == "__main__": main() Merge scorecard Signal Pass Fail Fixture digest vs lock equal any hash or path change Property node ids superset of lock deleted property min_examples >= locked value lower Strategy fingerprint equal changed string Unleased skip none any Lease expiry now < expires_unix expired Dual-run receipts identical sha256 mismatch Hypothesis DB / pytest cache wiped per run reused from agent session testworld.lock.json in agent diff absent present The last row is the one that makes the rest real. If the model can edit the lockfile, the table is theater. A practical check: # fail if the agent identity touched the lockfile git diff --name-only origin/main...HEAD | grep -qx testworld.lock.json && \ test "$GIT_AUTHOR_EMAIL" = "agent@example.invalid" && exit 1 Replace the author email with whatever identity your automation actually uses. The command is a policy sketch, not a hosted workflow. Where a free model and a free server fit Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol does not depend on a particular product. It does depend on isolation. Candidate patches have to be produced somewhere the lockfile is not writable. The two receipt runs have to happen somewhere the agent session cannot reach. MonkeyCode's free model access is enough to generate a candidate in a sandbox that cannot open testworld.lock.json for write. The free server option is enough to clone that candidate twice, wipe .hypothesis/ and .pytest_cache, run the digester, the fingerprint script, pytest, and the lease scorer, then compare receipts. Paid CI can sit behind that gate. If the receipts already disagree, there is nothing for a larger runner to decide. Do not send lockfile hashes, lease lists, or fixture bytes into the prompt that produces the patch. The model does not need the proof in order to write application code. Giving it the proof often teaches it how to satisfy the proof. Limitations, and who should not use this The strategy fingerprint is a string compare. Equivalent refactors of a given decorator will fail closed. That is intentional. Re-lock by hand. The AST scanner misses properties registered at collection time. Teams that generate tests dynamically need a pytest plugin that records items after collection, not a static parse. Dual receipts do not detect a wrong-but-stable oracle. If both runs use a fixture that was already poisoned on main, they will match. Protect main with the same rule: agents cannot touch the lockfile. Flake leases encode policy, not physics. A race still needs a fix. The lease only stops the hole from becoming unnamed. Do not use this protocol if your suite has no fixtures, no properties, and no skip policy. There is nothing to lock. Do not use it if the same session that writes production code is also allowed to regenerate receipts or refresh .hypothesis/. Isolation is the method. Time-sensitive claims about third-party quotas, model catalogs, or hardware are omitted on purpose. Check current vendor docs before you rely on any capability not stated here. What to copy Copy the lockfile schema, the rule that agents cannot edit it, the wiped example database, and the dual-run receipt compare. Those pieces still work if every product mention in this article is removed. If candidate patches already come from free model access, run the second receipt on an isolated free server and keep the lockfile out of the prompt. That is the only tooling note worth taking.