Trusted Coverage: Stop Counting Tests You've Already Quarantined

Your suite reports 84% coverage. Remove the eleven tests sitting in the freeze registry and the honest number is 71%. That 13-point gap is the part of your test suite you have already decided not to trust, and almost no pipeline reports it.

Conclusion first: track a second number alongside coverage — trusted coverage, computed only from tests that actually run, on the current tree, with no active freeze. Then gate agent patches on the delta of that number rather than on total coverage. A patch that raises coverage by adding skips should fail the same way a patch that breaks a test fails.

This is a workflow, not a product pitch. Everything below runs with standard tools: pytest, coverage.py, a YAML file in the repo, and a CI job.

Why the usual numbers drift

Raw coverage is blind to intent. A skipped test, an xfail, a --deselect in CI config, and a node ID parked in a quarantine list all look identical to the coverage tool: no executed lines. The tool is correct. The reporting is what misleads.

Agents make this worse in a specific way. When a patch turns a red test green by freezing a neighbouring flaky test, the diff looks small, the CI is green, and total coverage barely moves. The trustworthiness of the suite moved a lot.

So measure it directly. Three artifacts:

1. A version-controlled freeze registry.
2. A CI step that computes coverage twice — with and without frozen tests.
3. A gate on the difference, plus an expiry check.

Step 1 — Put every freeze in a registry, not in a skip decorator

A @pytest.mark.skip inside a test file is invisible in review and has no owner. Move the decision into data.

# tests/freeze-registry.yaml frozen: - nodeid: tests/test_billing.py::test_proration_roundtrip owner: '@finley' frozen_at: 2026-10-02 expires_at: 2026-12-01 reason: 'seed-dependent rounding; 3/50 failures on main since 2026-09-28' reentry_property: 'charged_total(p) <= plan_price(p) for every proration input' - nodeid: tests/test_search.py::test_ranking_is_stable owner: '@finley' frozen_at: 2026-10-19 expires_at: 2026-11-19 reason: 'tie-breaking depends on dict order in fixture; fix pending' reentry_property: 'rank(a) == rank(b) whenever score(a) == score(b)' 

Two fields carry the weight. expires_at makes the freeze a dated decision instead of a permanent state. reentry_property states what the test was supposed to prove, in a form a property test can later encode.

If you cannot fill in reentry_property, you have learned something useful: the test was asserting an implementation detail, and freezing it costs you nothing measurable. That is a fine outcome — but write it down as such.

Step 2 — Derive the deselect list from the registry

Never hand-maintain two lists. This script is the single source of truth for both the CI flags and the expiry gate.

#!/usr/bin/env python3 # tools/freeze_registry.py import datetime as dt import pathlib import sys import yaml REGISTRY = pathlib.Path('tests/freeze-registry.yaml') def load(): data = yaml.safe_load(REGISTRY.read_text()) or {} return data.get('frozen', []) def main() -> int: mode = sys.argv[1] if len(sys.argv) > 1 else 'deselect' entries = load() today = dt.date.today() flags, expired = [], [] for e in entries: due = dt.date.fromisoformat(e['expires_at']) if due < today: expired.append(f"{e['nodeid']} (owner {e['owner']}, due {due})") continue # expired freezes stay selected: they run again  if mode == 'deselect': flags += ['--deselect', e['nodeid']] if mode == 'check-expiry': for line in expired: print(f'EXPIRED FREEZE: {line}', file=sys.stderr) return 1 if expired else 0 print(' '.join(flags)) return 0 if __name__ == '__main__': raise SystemExit(main()) 

The continue on an expired entry is deliberate. A freeze that has run out stops being honoured, so the test rejoins the run and either passes (delete the entry) or fails (the debt is now visible). Both outcomes are better than a silent skip.

Step 3 — Run coverage twice and intersect against a fixed universe

The trap here is measurement drift. If you compute the universe from the trusted run only, files that no trusted test imports vanish from the denominator and the ratio goes up after you quarantine something.

Fix it by taking the statement universe from the full run and only the hits from the trusted run.

#!/usr/bin/env python3 # tools/trusted_coverage.py import json import pathlib import sys def read(path): return json.loads(pathlib.Path(path).read_text())['files'] def universe(full): return {f: set(i['executed_lines']) | set(i['missing_lines']) for f, i in full.items()} def hits(path): return {f: set(i['executed_lines']) for f, i in read(path).items()} def ratio(covered, space): hit = sum(len(covered.get(f, set()) & lines) for f, lines in space.items()) total = sum(len(lines) for lines in space.values()) return hit / total if total else 0.0 space = universe(read('cov-full.json')) raw = ratio(hits('cov-full.json'), space) trusted = ratio(hits('cov-trusted.json'), space) print(f'raw={raw:.4f} trusted={trusted:.4f} gap={raw - trusted:.4f}') if '--baseline' in sys.argv: base = float(sys.argv[sys.argv.index('--baseline') + 1]) gate = float(sys.argv[sys.argv.index('--gate') + 1]) if '--gate' in sys.argv else 0.005 delta = trusted - base print(f'trusted_delta={delta:+.4f} gate=-{gate:.4f}') if delta < -gate: print('FAIL: network adds untrusted surface', file=sys.stderr) raise SystemExit(1) 

Note the --baseline argument: pass the trusted ratio you measured on the base branch for the same file universe. Comparing head-to-base on the same universe is what turns this into an attribution signal instead of a vibe.

Step 4 — Wire it into CI, and mind the data file collision

coverage run overwrites .coverage by default. Two runs in one job will clobber each other and your second JSON will silently be the first one. Set COVERAGE_FILE.

# .github/workflows/verify.yml (excerpt) - run: pip install -r requirements-test.txt - run: python tools/freeze_registry.py check-expiry - run: COVERAGE_FILE=.coverage.full coverage run -m pytest -q - run: COVERAGE_FILE=.coverage.full coverage json -o cov-full.json - run: COVERAGE_FILE=.coverage.trusted coverage run -m pytest -q $(python tools/freeze_registry.py deselect) - run: COVERAGE_FILE=.coverage.trusted coverage json -o cov-trusted.json - run: python tools/trusted_coverage.py --baseline 0.7100 --gate 0.005 

Keep --cov-branch on if you already use branch coverage, but parse it consistently. Mixing line-mode JSON from one run with branch-mode JSON from another produces a ratio that means nothing.

Step 5 — Make the property the re-entry condition

A frozen test comes back when its property exists and is falsifiable. Hypothesis is enough for most cases.

# tests/properties/test_proration.py (sketch, not yet run) from hypothesis import given, settings, strategies as st from billing import Charge, charged_total, plan_price @settings(max_examples=300, deadline=None) @given(st.lists(st.integers(min_value=0, max_value=10_000), min_size=1, max_size=50)) def test_charged_total_never_exceeds_plan_price(charges): plan = st.integers(min_value=0, max_value=50_000).example() cs = [Charge(amount=c) for c in charges] assert charged_total(cs) <= plan_price(plan) 

Two checks before you delete the registry entry:

1. The property passes on head.
2. The property fails when you revert the fix that head introduced.

Step 2 is the one people skip. A property that passes in both trees proves nothing about the patch; it is a tautology with a nicer name. Revert the fix locally, run the property, restore. If it stays green, keep the test frozen and rewrite the property.

The decision table

Illustrative values from a synthetic repo, not from your codebase. Read the shape of the rows, then calibrate thresholds against your own CI history.

Row four matters most in practice. Painting every failure onto the incoming patch trains reviewers to ignore the gate. Separate this patch added untrusted surface from this repo has a debt problem and route them to different owners.

Limitations, stated plainly

Trusted coverage is a proxy. It measures executed-and-unfrozen, which is not the same as correct. A green test with a vacuous assertion contributes fully to both numbers.

The line-based universe is coarse. Files never imported by any test disappear from the denominator; branch-mode parsing changes the ratio's meaning. Pick one mode and stay there. Small suites produce jumpy ratios, so a 0.5% gate on a 200-test repo will fire on noise — the threshold is a property of your history, not a constant.

And the ledger is gameable by anyone allowed to edit it. Ownership and expiry are the only defences worth having; both are social controls with a script attached, not guarantees.

Who should not use this: solo repos where you are the only reviewer; teams with no CI history to calibrate a baseline against; and any project where coverage is already advisory, because a second advisory number changes nothing.

Where a free tier actually helps

The expensive parts of this workflow are not the gate — they are the two jobs you want to run often: nightly replays of frozen node IDs to accumulate flake evidence, and drafting candidate properties for each reentry_property field. Both are bursty and both are exactly the work you do not want competing with merge-path CI minutes.

MonkeyCode is an open-source project that, per its operator, offers free model access and a free server option. I have not verified quotas, model names, hardware, or limits, so treat the following as a proposal, not a benchmark. Two narrow uses fit this workflow: point a free model tier at the frozen test bodies to generate candidate properties for human review (expect most drafts to be weak, and never auto-merge them), and run the nightly replay job on the free server so the evidence keeps accruing without touching your main CI budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you have access to the free tier, start with one nightly job that replays only the registry's node IDs and appends pass/fail counts to a log. Get that stable for two weeks before putting any model anywhere near the merge path.

The number worth defending is not the one at the top of your coverage report. It is the one you can still stand behind after subtracting everything you have quietly stopped trusting.