I changed one line of a system prompt last month. Three examples looked better. A refund path broke two days later.

Was that bad luck? Partly. Three examples are not a test suite. They are a vibe with a timestamp.

Everyone nods at that in the retro. Then the next review meeting starts. Someone says "same prompt, same answer, so we're fine." Here are five claims like that. Each one is wrong in a fixable way. At the end there is a small gate you can run offline today.

Myth 1: "Same prompt, same answer means it passed"

One run is a sample of size one. Would you ship a function after one assertion?

Evidence is cheap here. Run the same input five times. Count how often the output changes. If it never changes, you learned something small. If it flips, you learned the thing that matters.

Corrected model: a test is a distribution of outcomes, not an outcome. Ask "how often does this item pass?" Do not ask "did it pass?"

Myth 2: "A bigger golden set is a better eval"

Volume is the easy knob. Coverage is the one that catches regressions.

  • Forty items run five times beats two hundred items run once.
  • One item per failure mode beats ten paraphrases of the same happy path.
  • Keep a slice you never tune prompts against. That slice is your witness.

Corrected model: the golden set is a map of failure modes. Size is a side effect. If two items test the same behavior, one of them is decoration.

Myth 3: "A pass rate is a score"

A 91% pass rate hides two very different systems. One system has nine flaky items. The other has one dead item.

Both round to the same number. Only one of them is safe to ship.

Corrected model: aggregate at the item, then at the change. A change ships when every gated item passes every run. A 91% aggregate is not a verdict. It is a hint that you are aggregating too early.

Myth 4: "The judge model is objective"

LLM judges have known biases. Position bias, verbosity bias, and self-preference are all documented in the literature. Jailbroken judges are worse.

Here is the cheap check I use. Label thirty outputs by hand. Run your judge on them twice. Compare judge-to-judge agreement, then judge-to-human agreement.

If judge copy A and judge copy B disagree on 20% of pairs, your judge is not a ruler. It is a second opinion.

Corrected model: a judge is a noisy instrument. Measure its noise before you gate on it. For anything you can check exactly, check exactly.

Myth 5: "Evals are too expensive to run on every change"

This one was true when every pass cost real money. It is less true now, and it was never true for smoke evals.

Split the work. Two different jobs get conflated constantly.

  1. Regression eval. Small set, many runs, exact checks where possible. Target: finish in minutes.
  2. Capability eval. Large set, few runs, judge-heavy. Target: run weekly, not per commit.

Most teams gate merges on the second one while starving the first. Flip that. The regression set is what protects your users.

The artifact: a gate that refuses to round up

This script reads attempts from JSONL. It blocks any item that is under-sampled or flaky. It needs no model call, so it runs everywhere and it runs fast.

#!/usr/bin/env python3 """gate.py - block a prompt change unless every item passes every run. Reads JSONL rows: {"item": "refund-1", "run": 0, "ok": true} Exit 0 = ship, 1 = block. """ import json import sys from collections import defaultdict MIN_RUNS = 3 def load(path): seen = defaultdict(list) with open(path) as fh: for n, line in enumerate(fh, 1): line = line.strip() if not line: continue try: row = json.loads(line) except json.JSONDecodeError: sys.exit(f"bad json on line {n}") seen[row["item"]].append((row["run"], bool(row["ok"]))) return seen def verdict(seen): blocked, flaky = [], [] for item, pairs in sorted(seen.items()): runs = {r for r, _ in pairs} oks = [ok for _, ok in pairs] if len(runs) < MIN_RUNS: blocked.append((item, f"under-sampled {len(runs)}/{MIN_RUNS}")) elif not all(oks): flaky.append((item, f"{sum(oks)}/{len(oks)} pass")) return blocked, flaky def main(): path = sys.argv[1] if len(sys.argv) > 1 else "results.jsonl" blocked, flaky = verdict(load(path)) for item, why in blocked: print(f"BLOCK {item}: {why}") for item, why in flaky: print(f"FLAKY {item}: {why}") if blocked: print(f"\n{len(blocked)} item(s) under-sampled. Run more.") return 1 if flaky: print(f"\n{len(flaky)} item(s) flaky. Treat as failures.") return 1 print("gate: pass") return 0 if __name__ == "__main__": raise SystemExit(main())

Now a fixture that should fail. Two of three runs pass for one item.

printf '%s\n' \ '{"item":"refund-1","run":0,"ok":true}' \ '{"item":"refund-1","run":1,"ok":false}' \ '{"item":"refund-1","run":2,"ok":true}' \ '{"item":"cancel-2","run":0,"ok":true}' \ '{"item":"cancel-2","run":1,"ok":true}' \ '{"item":"cancel-2","run":2,"ok":true}' > results.jsonl python3 gate.py results.jsonl; echo "exit=$?"

Expected output is one FLAKY line, a count of one flaky item, and exit=1. Swap the middle false for true and the gate passes. That is the whole contract.

The producer side stays thin. This sketch is a template, not a library. Wire your own transport and log one row per attempt.

# template - not run here; adapt to your client for run in range(3): for item in golden_set: out = transport(item.prompt) # your code  ok = item.check(out) if item.check else judge(item, out) log({"item": item.id, "run": run, "ok": ok})

Where to run the loop

Repeated passes get expensive in time before they get expensive in money. Three runs over sixty items is one hundred eighty calls. That is a boring CI job, not a research project.

This is the part where free access actually helps. MonkeyCode's free model access keeps the cost of a smoke pass near zero, and the free server option can host the harness if the run size fits. Check the current limits before you depend on either one for merges. Do not assume a free tier stays free forever, and do not assume a server size fits an overnight matrix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations I hit

  • Runtimes add up. Three runs across a big set can exceed a short CI budget. Cap the gated set.
  • Free capacity is shared. Treat peak-hour latency as a scheduling problem, not a product flaw. Move the matrix to a slow lane.
  • all(oks) is strict. Some teams want a per-item budget instead. Change one line if so.
  • Judge items still need human labels. No gate fixes a judge you never checked.
  • The script does not dedupe repeated run values. Log cleanly or you will learn nothing.

Who should not use this

Skip this gate if you need statistical guarantees for a compliance sign-off. Use a proper design of experiments. Skip it if your outputs are long-form prose with no checkable properties. Your real problem is labels, not gating. Skip it if you are measuring capability rather than preventing regressions. Wrong tool.

What to do Monday

Ask your team one question. "How many runs did we do before the last prompt change?" If the answer is one, you already know the bug.

Pick ten items from your last three incidents. Write exact checks where you can. Run each item three times. Put gate.py in CI and let it say no for a week.

If you want to try the loop without a budget conversation, wire one transport adapter to a free endpoint and run the gate against your own fixtures first. The gate is the part that keeps you honest.