A prompt edit is a deploy. Treat it like one: run the new prompt against a fixed case file three times per case, compare pass rates, and block the release when the rate drops. The harness below is about 120 lines of plain Python, needs no framework, and is cheap enough to run on a free model tier before every push.
For a solo founder the goal is not a benchmark suite that impresses anyone. The goal is to catch the one case that silently broke while you were fixing something else — the refund answer that stopped returning JSON, the summary that started inventing a policy number. Twenty minutes of setup buys that.
Why "it looked fine in the playground" fails
A single manual check proves one sample of one case at one moment. Model endpoints are nondeterministic, and free tiers add queueing and cold starts on top. So the interesting number is not "did it pass" but "how often does it pass", measured across repeated runs of the same input.
That second number is the flake rate, and it is the one that hides regressions. A case that passes 2 out of 3 runs will look green on the day you spot-check it and red on the day a customer hits it.
Step 1 — Write the cases as data
Keep cases in JSONL so a non-engineer can add one without touching code. Each line is one test: a prompt plus the cheapest assertions that can be checked by a machine.
# cases/support.jsonl {"id": "refund-json", "prompt": "Extract the refund window in days from: 'You may return items within 30 days of delivery.' Reply as JSON with key days.", "json": true, "must_contain": ["30"], "max_words": 20} {"id": "refund-no-invention", "prompt": "A customer asks about a 45-day-old order. State the return window. Do not invent policy numbers.", "must_not_contain": ["policy #", "policy number"], "must_contain": ["30"]} {"id": "tone-skip", "prompt": "Summarize this cancellation request in one sentence: 'Please cancel order 8812, I ordered the wrong size.'", "max_words": 40} Three assertion types cover most solo-project needs: exact substring present, banned substring absent, and JSON parseable. Resist adding an LLM judge on day one; it doubles the moving parts before you know whether the basic gate works.
Step 2 — One provider-agnostic call function
Isolate the HTTP call. Everything else in the harness should be reusable when you switch endpoints.
# gate.py import argparse, datetime, json, os, pathlib, sys, urllib.request API_URL = os.environ["MC_BASE_URL"].rstrip("/") + "/chat/completions" API_KEY = os.environ["MC_API_KEY"] MODEL = os.environ["MC_MODEL"] def call_model(prompt, timeout=60): body = json.dumps({ "model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0, }).encode() req = urllib.request.Request(API_URL, data=body, headers={ "Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}", }) with urllib.request.urlopen(req, timeout=timeout) as r: payload = json.loads(r.read()) text = payload["choices"][0]["message"]["content"] return {"text": text, "tokens": payload.get("usage", {}).get("total_tokens", 0)} Adjust the request body and the response path to your provider's schema; the shape above is the common chat-completions form and is the only line you should need to touch. Read the key from an environment variable, never from the repo.
Step 3 — The sweep
Each case runs N times. Raw outputs are written to disk so any failure can be reproduced without guessing.
def fails_for(case, text): bad = [] for s in case.get("must_contain", []): if s.lower() not in text.lower(): bad.append(f"missing:{s}") for s in case.get("must_not_contain", []): if s.lower() in text.lower(): bad.append(f"banned:{s}") if case.get("json"): try: json.loads(text) except json.JSONDecodeError: bad.append("invalid_json") limit = case.get("max_words") if limit and len(text.split()) > limit: bad.append(f"over:{limit}w") return bad def main(): ap = argparse.ArgumentParser() ap.add_argument("--runs", type=int, default=3) ap.add_argument("--min-pass", type=float, default=0.9) ap.add_argument("--cases", default="cases") args = ap.parse_args() cases = [] for f in sorted(pathlib.Path(args.cases).glob("*.jsonl")): for line in f.read_text().splitlines(): if line.strip(): cases.append(json.loads(line)) stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") outdir = pathlib.Path("runs") / stamp outdir.mkdir(parents=True, exist_ok=True) rows, tokens = [], 0 for case in cases: ok = 0 for i in range(args.runs): res = call_model(case["prompt"]) tokens += res["tokens"] (outdir / f'{case["id"]}-{i}.json').write_text(json.dumps(res, indent=2)) if not fails_for(case, res["text"]): ok += 1 rows.append((case["id"], ok, args.runs)) passed = sum(ok for _, ok, _ in rows) total = len(rows) * args.runs print("| case | passed | runs |") print("|---|---|---|") for cid, ok, n in rows: print(f"| {cid} | {ok} | {n}{'' if ok == n else ' FLAKY'} |") rate = passed / total print(f"\npass rate: {rate:.2%} ({passed}/{total}) tokens: {tokens}") sys.exit(0 if rate >= args.min_pass else 1) if __name__ == "__main__": main() Run it with python gate.py --runs 3 --min-pass 0.9. Output is a Markdown table you can paste into the pull request body:
| case | passed | runs | |---|---|---| | refund-json | 3 | 3 | | refund-no-invention | 2 | 3 FLAKY | | tone-skip | 3 | 3 | pass rate: 88.89% (8/9) tokens: 1043 That FLAKY marker is the whole point. Nineteen of twenty cases looked perfect; one is a coin flip, and it is the case about inventing policy numbers.
Step 4 — Make it blocking, locally
Run the gate in a pre-push hook rather than only in CI. A solo founder feels the cost of a failing gate before the push, not six minutes after it.
# .git/hooks/pre-push #!/usr/bin/env bash set -e python gate.py --runs 3 --min-pass 0.9 || { echo "Prompt regression. Fix or raise the case file before pushing." exit 1 } Then chmod +x .git/hooks/pre-push. Keep a --runs 1 mode for a fast loop while drafting and reserve the three-run sweep for the moment before the push, because the flake information only matters at the decision point.
Step 5 — Keep the cost envelope visible
The loop prints a token count for exactly this reason. With 40 cases at three runs each, a sweep is 120 calls; at a rough 800 tokens per call that is about 96,000 tokens per sweep. Under the operator's advertised free allowance of 10 million tokens, that arithmetic works out to roughly a hundred sweeps — enough for months of prompt work, assuming your prompts are actually that short. Treat the number as a vendor statement with a shelf life, confirm the current quota in the console, and never hard-code it into the script.
This is where free access changes behavior rather than just the bill. When each sweep feels free, you run it on a Tuesday afternoon instead of saving it for release week, and small regressions get caught while the diff is still one line.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator also advertises a free server option, which is useful for one specific job here: hosting the sweep on a schedule so the case file is exercised even in weeks when you do not push. The harness is a single process with no database, so a small host is sufficient; anything heavier is waste.
A decision table for what to automate
| Check | Automate in the gate | Human review |
|---|---|---|
| Required substring / number present | Yes | No |
| Banned phrase or invented identifier | Yes | No |
| JSON parseable, field names correct | Yes | No |
| Response length bound | Yes | No |
| Tone, helpfulness, brand voice | No | Yes, on a sample |
| Factual correctness of free text | No | Yes |
| Latency and token cost per call | Record only | Interpret later |
| Flake rate per case | Yes | Investigate when nonzero |
The right-hand column is not laziness. A machine-checkable gate is only worth building where the check is deterministic; stretching it to cover judgment calls produces a green light you stop believing.
Limitations and who should skip this
- Free endpoints are noisy. Queueing and cold starts can turn a good prompt into a flaky case. Log the raw outputs and check the failure text before blaming the model; a timeout looks identical to a wrong answer in the table.
- The gate cannot judge quality. It catches breakage, not mediocrity. A prompt that is consistently, fluently wrong will pass every run.
- Three runs is a smoke test, not statistics. It is enough to spot a coin-flip case and not enough to claim a percentage. If you need confidence intervals, this is the wrong tool.
- Skip it if you operate under a formal SLA, if your case file would contain regulated customer data, or if you cannot store raw outputs under your data policy. Also skip it if nobody will act on a red gate — a blocking check that gets bypassed by habit is worse than none.
Start with five cases, not forty. The first sweep tells you which assertions you actually wrote wrong, and that lesson is worth more than the coverage.
If you want to try the sweep against a free endpoint, begin by writing the case file — the signup page can wait until the gate has something to check.
