Selecting Tests for Agent Patch Queues: A Precision Budget With a Hard Floor

Conclusion first: when model-generated patches arrive faster than your suite can run, do not choose between "run everything" and "run something random". Build a test selector, give it a hard floor of always-run tests, and track a single number — how many real failures the selector lets escape. The selector is only trustworthy after you have measured that number on your own replay history.

This post is about the selection layer, not the gate. Gating logic decides whether a patch is acceptable. Selection decides which evidence you pay for. They fail differently, and the second failure mode is quieter.

1. The cost asymmetry that forces a selection policy

An agent patch queue changes the arithmetic of CI. A human opens a PR every hour or two; a queue of model-generated candidates can produce dozens per hour, each a plausible-looking diff.

When the candidates are cheap to produce, the expensive step becomes verification. If your suite takes twelve minutes and you have thirty candidates, you are looking at six hours of runner time for a batch a reviewer wants to see this afternoon. Running everything is not a virtue; it is a scheduling decision that silently drops the queue.

This is where MonkeyCode's free model access and free server option fit the workflow described here: free model access is one way to produce a candidate queue at the volume that makes selection necessary, and the free server option is one place to run the selected subset instead of the full suite. Both are operator-supplied availability claims, not guaranteed capacity — measure your own durations before relying on either in a pipeline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

2. Step 1 — Build a dependency manifest from a real run, not from imports

Static import graphs lie. Tests reach into conftest files, fixtures, module-level registries, and data loaders. Record what a test actually touched during a run.

The hook below writes a manifest entry per test: repository files imported during that test, the fixtures it requested, and its last observed duration.

# conftest.py — approximate dependency manifest, recorded per test import json, os, pathlib, sys, time ROOT = pathlib.Path(__file__).resolve().parent OUT = pathlib.Path(os.environ.get("TEST_MANIFEST", "manifest.json")) _records, _manifest = {}, {} def repo_files(): files = set() for mod in list(sys.modules.values()): f = getattr(mod, "__file__", None) if not f: continue try: files.add(str(pathlib.Path(f).resolve().relative_to(ROOT))) except ValueError: continue return files def pytest_runtest_setup(item): _records[item.nodeid] = {"before": repo_files(), "t0": time.perf_counter()} def pytest_runtest_teardown(item, nextitem): rec = _records.pop(item.nodeid, None) if rec is None: return _manifest[item.nodeid] = { "files": sorted(repo_files() - rec["before"]), "fixtures": list(getattr(item, "fixturenames", [])), "mutable_fixtures": [m.args[0] for m in item.iter_markers("mutable_fixture")], "seconds": round(time.perf_counter() - rec["t0"], 3), } def pytest_sessionfinish(session, exitstatus): OUT.write_text(json.dumps(_manifest, indent=2, sort_keys=True)) 

Mark the fixtures that mutate shared state so the runner knows which tests cannot be parallelised with each other:

@pytest.mark.mutable_fixture("billing_db") def test_invoice_roundtrip(billing_db): ... 

Known approximation: modules already imported before a test starts never appear in its delta. Tests whose dependencies are entirely imported at session start look dependency-free. That is exactly why the floor in step 2 is not optional.

3. Step 2 — Selection with a non-negotiable floor

The selector maps changed paths to tests, removes frozen tests, unions in the floor, and refuses silently huge selections.

#!/usr/bin/env python3 """select.py — choose a test subset for one agent patch, with a hard floor.""" import argparse, json, pathlib, sys def read_json(path, default): p = pathlib.Path(path) return json.loads(p.read_text()) if p.exists() else default def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--manifest", default="manifest.json") ap.add_argument("--changed", required=True) ap.add_argument("--freeze", default="freeze.json") ap.add_argument("--floor", default="floor.json") ap.add_argument("--budget-seconds", type=float, default=None) ap.add_argument("--out", default="selection.json") a = ap.parse_args() manifest = read_json(a.manifest, {}) freeze, floor = set(read_json(a.freeze, [])), set(read_json(a.floor, [])) changed = [l.strip() for l in pathlib.Path(a.changed).read_text().splitlines() if l.strip()] index = {} for test_id, meta in manifest.items(): for f in meta.get("files", []): index.setdefault(f, set()).add(test_id) touched = set() for path in changed: touched |= index.get(path, set()) selected = sorted((touched | floor) - freeze) seconds = sum(manifest.get(t, {}).get("seconds", 0.0) for t in selected) owners = {} for t in selected: for fx in manifest.get(t, {}).get("mutable_fixtures", []): owners.setdefault(fx, []).append(t) serialize = [{"fixture": fx, "tests": ts} for fx, ts in owners.items() if len(ts) > 1] report = { "selected": selected, "floor": sorted(floor - freeze), "skipped_frozen": sorted(touched & freeze), "gap_floor_frozen": sorted(floor & freeze), "serialize": serialize, "estimate_seconds": round(seconds, 2), } pathlib.Path(a.out).write_text(json.dumps(report, indent=2)) if a.budget_seconds and seconds > a.budget_seconds: print(f"~{seconds:.1f}s exceeds budget {a.budget_seconds:.1f}s", file=sys.stderr) print("escalate: floor-only run, or ask for the patch to be split", file=sys.stderr) return 2 print(json.dumps(report, indent=2)) return 0 if __name__ == "__main__": raise SystemExit(main()) 

estimate_seconds is the sum of last observed durations. It is a planning aid, not a prediction — tests get slower after a patch touches them.

4. Step 3 — Treat the freeze list as an input, and its intersection with the floor as a defect

The freeze file is just a list of node ids:

["tests/test_export.py::test_csv_export_roundtrip"] 

When a patch touches export.py, the test that would have covered it is frozen. That is now a coverage gap, printed as skipped_frozen. Two rules keep it honest:

1. A frozen test may never also be a floor test. If gap_floor_frozen is non-empty, the selector's output is not trustworthy — replace the floor entry or unfreeze.
2. Every skipped_frozen entry must map to a named owner in the freeze record. A frozen test with no owner is an unreviewed skip.

Notice what this design does not do: it does not try to resurrect the frozen test. Freezes expire on their own schedule. Selection's job is to make the gap visible in the output artifact, not to hide it behind a green batch.

5. Step 4 — The escape audit is the only number that matters

Replay historical patches. For each, run the selector, run the full suite, and compare.

#!/usr/bin/env python3 """audit_selection.py — replay patches, measure escaped failures.""" import json, pathlib, sys def failures(path): text = pathlib.Path(path).read_text().splitlines() return {l.split(" ", 1)[1].strip() for l in text if l.startswith("FAILED ")} def main(history_dir): rows, escaped_total = [], 0 for case in sorted(pathlib.Path(history_dir).iterdir()): if not (case / "selection.json").exists(): continue selected = set(json.loads((case / "selection.json").read_text())["selected"]) failed = failures(case / "full_suite.txt") escaped = sorted(failed - selected) escaped_total += len(escaped) rows.append({ "case": case.name, "failed": len(failed), "caught": len(failed & selected), "escaped": escaped, "recall": round(len(failed & selected) / len(failed), 3) if failed else None, }) print(json.dumps(rows, indent=2)) print(f"escaped failures across {len(rows)} replayed patches: {escaped_total}") return 1 if escaped_total else 0 if __name__ == "__main__": raise SystemExit(main(sys.argv[1])) 

Run it until you can state a recall figure for your suite, and keep the failing case directories that produced escapes. Those cases are your floor candidates — they were missed for a structural reason (session-level import, frozen dependency, dynamic fixture), not a random one.

6. Decision table

7. Limitations and who should not use this

• Small suites. Under roughly a hundred fast tests, the selector is overhead with no payoff. Run everything.
• Suites with session-scoped imports of the code under test. The dependency capture will systematically under-report. Fix the test layout first, or keep the floor large enough to cover the gap.
• Hard latency or availability SLOs. A free server option is not a capacity guarantee. If your pipeline has an SLA, measure and budget accordingly, and keep a fallback runner.
• Any team without replay history. A selector with no escape audit is a guess wearing a JSON schema.
• Model or quota assumptions. Free access terms change. Treat availability as a current observation, re-check it, and keep the pipeline working when the queue volume drops to zero.

The method is deliberately boring: record dependencies from a real run, select, subtract freezes, add the floor, escalate over budget, and audit escapes on a schedule. The interesting part is the audit output, because it is the only thing that tells you whether the cheap path is still honest.

If you run the replay audit on your own queue, the escaped-failure count is the number worth publishing back to your team — it converts a policy argument into a measurement.