For two nights my nightly classifier wrote records that validated against my schema and still broke the dashboard. The model was not lying to me; my gate was just asking a question so weak that any answer counted as correct. I spent the next 48 hours measuring output shape instead of trusting a single green run, and these are the notes from that weekend.
The part that broke was my definition of "valid"
My job reads a batch of log lines, asks a hosted model to label each batch, and writes one small JSON record per batch. I hand-rolled a schema in ten minutes, wired it into the writer, and it passed every run I watched. What I never checked was whether the values inside that shape were the values my downstream code could actually handle.
Here is the schema I started with, reduced to its essentials:
# schema-v0.py — the version that lied to me SPEC = { "severity": str, # too loose "line_count": int, # accepts True "tags": list, # accepts [1, null] } Three separate holes lived in those three lines, and each one needed a different fix. Have you ever written a validator that checks existence and type but never checks meaning?
Hole 1: "it is a string" says nothing about the enum
The model returned "high", "HIGH", and once "sev-3", and every one of those is a valid Python str. My code did if record["severity"] == "high" and silently took the else branch on the other two. The schema was satisfied, the chart was blank, and nobody saw an error.
Hole 2: True is an int in Python
Record once arrived with "line_count": true. My check isinstance(v, int) returned True, because in Python bool is a subclass of int. That is not a quirk of the model; it is a documented property of the language, and my validator was wrong before the request ever left the machine.
Hole 3: json.loads accepts NaN and Infinity by default
Python's standard decoder parses the bare tokens NaN, Infinity, and -Infinity into floats unless you intervene. A malformed numeric field therefore parses cleanly, and any later comparison against it produces confusing results instead of an error you can trace. I only found this because a count field arrived as NaN in a fixtures file I was editing by hand.
What I built over the next 48 hours
Three small pieces, each one written only after the previous piece caught something real. None of them required a new dependency, which mattered because I wanted the same harness to run unchanged in two very different places.
- A fixtures file of deliberately boring cases, one per invariant.
- A gate that fails closed and names the exact field path.
- A replay loop that runs the fixtures N times and reports drift as a table.
1. Fixtures that store the raw text, not just the parsed dict
I keep one JSON line per case, storing both the prompt and the raw response text I want to replay. Storing raw text means I can re-parse old fixtures under a stricter gate without re-calling anything.
{"id": "c01", "prompt": "Label this batch: ...", "raw": "{\"severity\":\"high\",\"line_count\":3,\"tags\":[\"auth\"]}"} {"id": "c02", "prompt": "Label this batch: ...", "raw": "{\"severity\":\"HIGH\",\"line_count\":true,\"tags\":[1]}"} Twelve cases were enough to catch every hole above, and I wrote them by hand in under an hour. Note that fixtures are examples I authored, not measurements of any specific model.
2. A gate that fails closed and says where
# shape_gate.py import json def enum(*allowed): def check(v): return isinstance(v, str) and v in allowed check.label = "enum(" + "|".join(allowed) + ")" return check def integer(v): return isinstance(v, int) and not isinstance(v, bool) integer.label = "int" def str_list(v): return isinstance(v, list) and all(isinstance(x, str) for x in v) str_list.label = "list[str]" SPEC = { "severity": enum("low", "medium", "high"), "line_count": integer, "tags": str_list, } def gate(record, spec=SPEC): if not isinstance(record, dict): return ["$: not an object"], set() problems = [] for key, pred in spec.items(): if key not in record: problems.append(f"$.{key}: missing") elif not pred(record[key]): problems.append(f"$.{key}: {record[key]!r} is not {pred.label}") extras = set(record) - set(spec) return problems, extras Extras are reported but not fatal, because a new field is often a legitimate model improvement. The gate returns strings instead of raising, so the caller can decide between rejecting and quarantining. That decision is the interesting one, and I will get to it.
3. A replay loop that measures drift instead of guessing
# replay.py (trimmed to the parts that mattered) import argparse, json, pathlib from shape_gate import gate def _reject_constant(name): raise ValueError(f"non-finite constant {name}") def run_once(case, transport): raw = transport(case["prompt"], case.get("raw")) try: record = json.loads(raw, parse_constant=_reject_constant) except ValueError as exc: return {"problems": [f"$: unparseable ({exc})"]} problems, extras = gate(record) return {"problems": problems, "extras": sorted(extras)} def main(): ap = argparse.ArgumentParser() ap.add_argument("--cases", default="fixtures/cases.jsonl") ap.add_argument("--passes", type=int, default=3) ap.add_argument("--out", required=True) ap.add_argument("--fake", action="store_true") args = ap.parse_args() out = pathlib.Path(args.out); out.mkdir(parents=True, exist_ok=True) transport = fake_transport if args.fake else live_transport for line in pathlib.Path(args.cases).read_text().splitlines(): if not line.strip(): continue case = json.loads(line) for i in range(args.passes): path = out / f"{case['id']}-{i}.json" if path.exists(): # makes a restarted run resumable continue result = run_once(case, transport) result.update(case=case["id"], attempt=i) path.write_text(json.dumps(result, indent=2)) The path.exists() skip is the whole reason this survives an interrupted run. Writing one file per attempt means a crash costs one case, not the batch, and the report is a glob away.
python3 replay.py --fake --passes 3 --out runs/fake-$(date -u +%Y%m%dT%H%M%SZ) python3 replay.py --passes 5 --out runs/live-$(date -u +%Y%m%dT%H%M%SZ) The report groups problems by field path, so $.severity and $.line_count stay separate instead of collapsing into one useless failure count.
Where the free model access and free server actually fit
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I did this harness work against MonkeyCode's free model access, and I ran the replay loop on its free server option, because a loop that fires a few hundred small calls is exactly the kind of thing I do not want on my laptop at 03:00. The transport is the only provider-specific line in the codebase, which kept the gate and the report honest and portable.
# transport.py — the only file that knows about a provider def live_transport(prompt, _cached_raw=None): # Point this at your provider's client; return the raw response text. raise NotImplementedError("wire up your client here") def fake_transport(prompt, cached_raw=None): return cached_raw Treat free access and free server availability as conveniences that can change, not as a capacity guarantee. If my nightly job needed a stated uptime or a fixed quota, I would not build it on any free tier, including this one.
The decision table I use before rejecting anything
| Symptom | Action | Why |
| --- | --- |
| Unknown key in output | Log and accept | Usually a legitimate new field |
| Known key, wrong type | Reject | Downstream code cannot branch on it |
| Known key, value outside enum | Quarantine | Might be a real category I forgot to model |
| true where an int belongs | Reject | Almost always a model formatting slip |
| Unparseable JSON | Reject and keep raw text | The raw text is the only useful artifact |
Quarantine means writing the raw text plus the gate's complaints to a separate directory that a human reviews. It is slower than coercion, and it is the reason I found "sev-3" at all instead of mapping it to medium and moving on.
What I would repeat, and what I would skip
I would repeat the fixtures-first order, because writing the twelve cases forced me to say out loud what each field was allowed to contain. I would also repeat the resumable per-attempt file writes, which turned two interrupted runs into a minor annoyance rather than a rerun.
I would skip the part where I ran the loop five times before the gate existed. Five passes of an unvalidated parser produced five directories of logs and zero information, because nothing in them was comparable. Measurement without a fixed definition of failure is just expensive typing.
What this harness cannot tell you
Twelve cases catch gross drift like a renamed enum or a boolean in a numeric field, and they will not catch a one-in-a-thousand tail. The gate proves that a record matches your declared invariants, which is not the same as proving the label is correct, so semantic accuracy still needs a separate review. And none of this measures latency, cost, or throughput, so please do not read the report as a benchmark of anything.
Who should not copy this
Skip it if the output feeds a decision with real consequences, because a passing gate is not a human review and should never be presented as one. Skip it if your model output is free-form prose, where a field-level gate has nothing stable to grip. And skip it if your team needs a published accuracy number, because a twelve-case replay loop is a smoke test, not evidence.
So what would your gate have caught on the third run instead of the thirtieth? If you want to point the replay loop at a hosted model without spending anything, MonkeyCode's free model access is one place to start, and I would genuinely like to know which field path your gate shouted about first.
