Triage Eval Regressions With a Four-Probe Ladder

Most eval "regressions" are not regressions. In a harness that re-runs a model against stored golden cases, a red nightly run usually originates in one of four places: the plumbing that assembles prompts, the grader that scores answers, the sampling noise of the model itself, or a genuine behaviour change. The cheapest way to find out is to stop asking the model first and interrogate the harness instead. A four-probe ladder, executed cheapest-first, closes the majority of alerts before you pay for a second model call.

Order probes by cost, not by intuition

After a failing nightly run, the reflex is to open the red case and read the model output. That reflex is expensive, because reading output is a human task while the first three probes are mechanical checks that finish in milliseconds. A replay cassette compares the recorded output against the current grader, which tells you whether the scoring path drifted underneath a stable answer. A repeat probe re-samples the same prompt several times and measures the flip rate, which separates noise from a deterministic change. A grader self-test feeds a hand-labelled gold answer and a deliberately mutated answer through the same scorer, which catches graders that became either too lenient or too strict.

The ladder only works if you stop at the first probe that explains the alert. Running all four probes on every failure is comforting but wasteful, and it makes the log harder to read because you cannot tell which check actually resolved the question. Treat the probes as a decision tree with an early exit, not as a checklist to complete.

# triage.py — probe ladder for eval regressions (illustrative; wire adapters to your stack) import json from collections import Counter from dataclasses import dataclass, field @dataclass class Verdict: case_id: str probe: str outcome: str # grader | flaky | change | recovered | *_ok | skip  evidence: dict = field(default_factory=dict) def probe_replay(case, cassettes): """Probe 1: does the recorded output still earn the recorded verdict?""" rec = cassettes.get(case.id) # output + verdict captured on the last green run  if rec is None: return Verdict(case.id, "replay", "skip", {"reason": "no cassette"}) again = case.grader(rec.output, case) if again.passed != rec.passed: return Verdict(case.id, "replay", "grader", {"before": rec.passed, "after": again.passed}) return Verdict(case.id, "replay", "replay_ok", {}) def probe_repeat(case, call_model, k=5, flip_budget=0.2): """Probe 2: is the new failure stable, or is the case just noisy?""" outcomes = [case.grader(call_model(case.prompt), case).passed for _ in range(k)] rate = sum(outcomes) / k if min(rate, 1 - rate) > flip_budget: # neither side wins consistently  return Verdict(case.id, "repeat", "flaky", {"pass_rate": rate, "k": k}) label = "change" if rate < 0.5 else "recovered" return Verdict(case.id, "repeat", label, {"pass_rate": rate, "k": k}) def probe_grader_self_test(case, gold, mutant): """Probe 3: gold must pass, mutant must fail. Anything else is a grader bug.""" gold_passed = case.grader(gold, case).passed mutant_passed = case.grader(mutant, case).passed if gold_passed and not mutant_passed: return Verdict(case.id, "grader_self_test", "grader_ok", {}) return Verdict(case.id, "grader_self_test", "grader", {"gold_passed": gold_passed, "mutant_passed": mutant_passed}) def probe_control(case, call_model): """Probe 4: a known-wrong control answer must still be rejected.""" if case.grader(call_model(case.control_prompt), case).passed: return Verdict(case.id, "control", "grader", {"control_pass": True}) return Verdict(case.id, "control", "control_ok", {}) def triage(case, cassettes, call_model, gold, mutant): trail = [probe_replay(case, cassettes)] if trail[-1].outcome in {"grader", "skip"}: return trail trail.append(probe_repeat(case, call_model)) if trail[-1].outcome in {"flaky", "change", "recovered"}: return trail trail.append(probe_grader_self_test(case, gold, mutant)) trail.append(probe_control(case, call_model)) return trail 

Read the outcome, not the diff

The value of the ladder is that each probe terminates in a small vocabulary of outcomes, and every outcome maps to a different person and a different fix. A grader outcome means the scoring code, prompt template, or label changed since the last green run, and nobody needs to look at model behaviour at all. A flaky outcome means the case specification is under-determined, so raise the sample count or tighten the assertion before raising an incident. Only the change outcome is a model problem, and it is the one outcome that should page someone.

Instrumenting outcomes is what turns the ladder from a script into an operating habit. Append one JSON line per probe verdict to a log, then count the first outcome seen per case, which gives you the distribution of where alerts actually die.

LOG = "triage.jsonl" def record(trail): with open(LOG, "a") as fh: for v in trail: fh.write(json.dumps(v.__dict__) + "\n") def where_alerts_die(path=LOG): first_seen = {} for line in open(path): v = json.loads(line) first_seen.setdefault(v["case_id"], v["outcome"]) return Counter(first_seen.values()) 

Run that counter weekly and you get a tuning signal rather than a vanity metric. If change dominates, your cases are stable and the model genuinely moved. If flaky dominates, your golden set is noisy and every future regression report is suspect until you fix the sampling budget. If skip dominates, the ladder is mostly decorative because most cases have no baseline to compare against.

Where cheap sampling and a spare box help

Probes two and four are the only ones that call a model, and they are also the ones you run most often, nightly and across every case in the set. That is exactly the tier where a second, cheaper backend is useful: sampling re-runs and control projections do not gate a release, so their job is to characterise noise rather than to certify quality. MonkeyCode offers free model access and a free server option, and the operator states a free allowance of 10 million tokens; I use that tier as the sampling backend for non-gating probes and host the nightly ladder on the free server so it stops competing with CI runners for build minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important boundary is that the free tier should never run the gating pass. Release decisions should use the exact production model configuration you ship, because a probe that passes on a substitute backend tells you about the substitute, not about your release. Keep the ladder's model calls behind a single adapter function so the sampling backend can be swapped without touching the graders.

# adapters.py — one seam, two backends # call_model must return plain text for the grader; keep provider details out of cases.  BACKENDS = {"gate": call_production_model, "probe": call_probe_model} 

Limitations and who should skip this

The ladder assumes deterministic graders, so if your scorer is itself a model judge, you need a calibration step for that judge before any of the outcomes mean anything. Five samples per case is enough to detect gross flakiness but not enough to estimate small shifts in pass rate, so do not quote k=5 pass rates as measurements. Cassettes go stale, and a stale cassette will happily report a grader outcome that is really a template change nobody recorded, so expire them on a schedule. Finally, teams with strict latency or data-residency requirements should not run sampling on an external backend at all, and teams shipping weekly may find the whole ladder heavier than simply re-reading five failing cases.

Start with probe one only, run it for a week, and add the remaining rungs once you know how many alerts never needed a model call. If you want a second sampling backend for the non-gating rungs without adding to your CI bill, MonkeyCode's free tier is one reasonable place to start.