A property check that passes every wrong implementation you hand it is not a verification gate. It is a comment with a stack trace.
When an agent writes the patch and the tests together, both come from the same mental model. CI goes green on the first run, coverage looks respectable, and nothing in the report distinguishes a correct patch from a plausible one. The fix is not more tests. It is a cheaper, adversarial question asked of each property: which wrong implementation does this property actually kill?
The artifact below answers that with a fixed table — a degenerate-oracle battery. Six deliberately broken implementations of the same contract, every property run against all six, and two independent gates on the resulting matrix.
This is not fixture mutation. Fixture mutation asks what inputs a test needs. The battery asks what a property can distinguish, using the implementation as the adversary.
Step 1 — Freeze the contract surface
The battery only pays off on functions with a checkable contract. Require all three before enrolling a target:
- Pure or effectively pure — no clock, network, or global state in the return value.
- Boundary-level — the agent patch calls it through one named entry point, not through three internal helpers.
- Specifiable without the code — you can state the property in one sentence without reading the diff.
If a function fails (1), the battery will produce noise, not signal. Timing-sensitive and stateful code is out of scope; a clock-dependent function has no stable oracle to be wrong about.
Step 2 — Build the degenerate battery
Six adversaries cover most pure-function contracts: empty, constant, identity, partial, sort-only, and off-by-one. Each is a few lines and each represents a real class of agent bug — the patch that returns the input untouched, or merges but never sorts, or sorts but never merges.
# killmatrix.py — degenerate adversaries for merge_intervals(list[(start, end)]) Interval = tuple[int, int] def degenerate_null(intervals: list[Interval]) -> list[Interval]: return [] # plausible: 'no overlaps to merge' def degenerate_const(intervals: list[Interval]) -> list[Interval]: return [(0, 0)] # plausible: hard-coded seed def degenerate_identity(intervals: list[Interval]) -> list[Interval]: return list(intervals) # plausible: forgot to normalise def degenerate_sorted_only(intervals: list[Interval]) -> list[Interval]: return sorted(intervals) # plausible: sorted, still overlapping def degenerate_first_only(intervals: list[Interval]) -> list[Interval]: return list(intervals[:1]) # plausible: early return in the loop def degenerate_off_by_one(intervals: list[Interval]) -> list[Interval]: ordered = sorted(intervals) return [(lo, hi + 1) if i % 2 else (lo, hi) for i, (lo, hi) in enumerate(ordered)] DEGENERATES = { 'null': degenerate_null, 'const': degenerate_const, 'identity': degenerate_identity, 'sort1': degenerate_sorted_only, 'first': degenerate_first_only, 'obo': degenerate_off_by_one, } Keep the set small and stable. A battery that grows every week stops being comparable against last month's baseline.
Step 3 — Write properties that can fail
Use fixed input cases, not randomised search. The battery needs determinism more than it needs reach: the same property must produce the same verdict across machines and across reruns, or the kill matrix is not a gate.
PROPERTIES = {} def prop(name: str, guard: bool = False): def deco(fn): fn.guard = guard PROPERTIES[name] = fn return fn return deco CASE = [(5, 8), (1, 3), (2, 4)] @prop('sorted_and_non_overlapping') def _shape(merge): out = merge(CASE) assert out == sorted(out), out assert all(a[1] < b[0] for a, b in zip(out, out[1:])), out @prop('coverage_preserved') def _coverage(merge): out = merge(CASE) assert sum(h - l for l, h in out) == sum(h - l for l, h in CASE), out @prop('idempotent') def _idem(merge): once = merge(CASE) assert merge(once) == once, once @prop('empty_is_empty', guard=True) def _empty(merge): assert merge([]) == [], 'empty input' The guard flag matters. Some assertions are cheap sanity checks that will never discriminate much; forcing them through the same threshold produces busywork and teaches reviewers to ignore the gate.
Step 4 — Run the kill matrix
Two axes, two gates. Per property: how many adversaries does it kill. Per adversary: is it killed by at least one property.
def run_matrix(properties, degenerates): rows, coverage = {}, {name: 0 for name in degenerates} for pname, prop in properties.items(): kills = [] for dname, impl in degenerates.items(): try: prop(impl) except AssertionError: kills.append((dname, 'K')) coverage[dname] += 1 except Exception: kills.append((dname, 'E')) # crash counts as a kill, logged coverage[dname] += 1 else: kills.append((dname, '.')) rows[pname] = kills return rows, coverage An illustrative run of the harness above, on the four properties in Step 3:
property null const ident sort1 first obo kills sorted_and_non_overlapping . . K K . K 3 coverage_preserved K K . . K K 4 idempotent . . . . . K 1 empty_is_empty [guard] . K . . . . 1 killed by >=1 property: 1 2 1 1 1 3 The verdict: idempotent fails the gate at one kill. That is the useful part of the report — merge is idempotent under four of six broken implementations, so idempotence alone proves almost nothing about this patch. Adversary coverage is healthy; no blind spot in the battery itself.
Step 5 — Route each outcome
The gate should produce a task, not a mood. Route by cell, not by overall pass rate:
| Property kills | Adversary coverage | Verdict |
|---|---|---|
| 0 | any | Property is decoration — delete or rewrite against the spec |
| 1 | any | Too weak — widen the input case or merge with a sibling property |
| ≥2 | all adversaries killed somewhere | Acceptable, report as baseline |
| ≥2 | some adversary survives everywhere | Battery blind spot — write a property, do not add another example test |
Exit non-zero on rows 1 and 2, and on any unkilled adversary. Keep the threshold at two kills as an explicit constant in the file, so a later patch cannot quietly lower it.
Step 6 — Keep flaky freezes inside the battery's scope
A frozen test still runs in the nightly job, and freezing does not exempt it from review. Two rules keep freezes from hiding non-discriminating checks:
- A freeze may only be lifted together with a replacement property that passes the battery — otherwise the flake returns, now with a green badge.
- The battery runs on the frozen test's module too. A quarantine that removes coverage should not also remove the evidence that the remaining properties discriminate.
Runtime stays bounded: the cost is properties × adversaries fixed-input calls, not a randomised search. For a handful of contracts this is a sub-second job, which is why it is safe to run on every patch.
Where free model access and a free server fit
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The expensive step is Step 2 — authoring plausible-wrong implementations for a new signature. That is drafting work, and free model access makes it cheap: ask for six wrong implementations of the stated contract, each with a one-line rationale explaining which bug it stands for, then prune anything that is wrong for the same reason as another. The nightly matrix itself is a batch job with no database, no browser, and no GPU, so the free server option is sufficient for it.
Neither step is a substitute for the human judgement in Step 1. If the contract cannot be stated without the diff, no amount of generation will fix the battery.
Limitations and who should not use this
- The adversaries are bounded. Passing all six is evidence of robustness against six specific bugs, not correctness.
- Fixed input cases trade reach for determinism. Numeric edge cases and boundary arithmetic need a separate seeded property run.
- Correlated blind spots are real: if the same model drafts the patch, the properties, and the degenerates, weaknesses line up. Draft degenerates from the stated contract, never from the diff.
- Not suitable for latency or performance contracts, UI behaviour, or anything whose oracle is a human reviewer.
- Do not adopt it for a codebase with no pure boundary functions; the setup cost will exceed the return.
Reproduce it
python killmatrix.py --min-kills 2 --fail-on-blindspot python killmatrix.py --json report.json # baseline for the next diff Commit the JSON report alongside the patch. The next reviewer compares matrices, not opinions — and a property that survived every enemy is visible before merge instead of after the revert.
If you want to try the drafting half without a billing decision, MonkeyCode's free model access and free server option cover exactly this workflow; the harness above needs nothing else.
