Tape the Model, Test the Code

How many pull requests went red this month for reasons that had nothing to do with the diff inside them? I started counting after the third one, and the answer embarrassed me: every single flake traced back to a model call, and none of them traced back to my code. The endpoint had a slow afternoon, a response came back without the field I expected, and my test suite reported it as a bug in an invoice parser that had not changed in six weeks.

That is a bad trade. A test suite that depends on somebody else's Tuesday is not a test suite, it is a weather report. So I spent a weekend building the thing I should have built first: a recording layer that keeps the model out of the inner loop entirely, and a deliberately boring script that goes out once a night to find out whether the shape of the responses has moved.

The red build that had nothing to do with my diff

Here is what actually happened, because the details matter more than my irritation. My extraction test sent a synthetic invoice to a model and asserted that the JSON came back with vendor and total_cents present. On a good day it passed in two seconds. On the bad day it passed, then failed, then passed again on a rerun, which is the signature of a test whose inputs are fine and whose dependency is not.

I could have added retries and called it resilience. But a retry inside a unit test is just a way of hiding the fact that the test is talking to the internet, and it makes the suite slower and the failures rarer rather than clearer. What I wanted was the opposite: a test that never touches the network, plus a separate job whose entire purpose is to touch the network and report on what changed.

A cassette is a contract with one response

The idea is stolen straight from HTTP record-and-replay tooling, and it transfers cleanly to model calls. You compute a stable key from the request, store the response next to that key, and from then on the test reads from disk. The trick is deciding what belongs in the key, because a request body contains plenty of noise that would produce a fresh cassette every run.

My rule is that the key covers the things that should change behaviour: the model identifier, the messages, the response schema, and the temperature. It ignores identifiers, timestamps, and anything the provider adds for its own tracing. Sort the keys, hash the result, and you get a short filename that is identical on my laptop and in CI, which is the whole point.

# tests/cassette.py from __future__ import annotations import hashlib import json import os from pathlib import Path from typing import Any CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "tests/cassettes")) MODE = os.environ.get("CASSETTE_MODE", "replay") # replay | record | auto  VOLATILE = ("id", "created", "created_at", "request_id", "system_fingerprint") def strip(obj: Any) -> Any: """Drop fields that change on every call but never change meaning.""" if isinstance(obj, dict): return {k: strip(v) for k, v in sorted(obj.items()) if k not in VOLATILE} if isinstance(obj, list): return [strip(v) for v in obj] return obj def key_for(model: str, messages: list, schema: dict | None, temperature: float) -> str: body = {"model": model, "messages": messages, "schema": schema, "temperature": temperature} blob = json.dumps(strip(body), sort_keys=True, separators=(",", ":")) return hashlib.sha256(blob.encode()).hexdigest()[:16] def call(client, *, model, messages, schema=None, temperature=0.0) -> dict: key = key_for(model, messages, schema, temperature) path = CASSETTE_DIR / f"{key}.json" if MODE in ("replay", "auto") and path.exists(): return json.loads(path.read_text())["response"] if MODE == "replay": raise FileNotFoundError(f"no cassette {key}; re-run with CASSETTE_MODE=record") response = client.complete(model=model, messages=messages, schema=schema, temperature=temperature) record = { "key": key, "request": {"model": model, "messages": strip(messages), "temperature": temperature}, "response": strip(response), } CASSETTE_DIR.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(record, indent=2, sort_keys=True)) return record["response"] 

The client here is deliberately thin, a single complete method, so I can point it at any OpenAI-compatible base URL by changing one setting. I am not hardcoding a model name anywhere in this article, because the identifier you should use is whatever the provider currently publishes; write it in conftest.py and let the cassette key absorb it.

With that in place the test becomes almost boring, which is the highest compliment I can pay a test.

# tests/test_extract.py from cassette import call SCHEMA = { "type": "object", "required": ["vendor", "total_cents"], "properties": {"vendor": {"type": "string"}, "total_cents": {"type": "integer"}}, } def test_invoice_fields(model_client): out = call( model_client, model=model_client.model, messages=[{"role": "user", "content": "Synthetic invoice: Acme Ltd, 4280 cents, due 2026-12-01"}], schema=SCHEMA, temperature=0.0, ) assert set(SCHEMA["required"]) <= out.keys() assert isinstance(out["total_cents"], int) 

CASSETTE_MODE=record pytest tests/test_extract.py -q # once, on a good day CASSETTE_MODE=replay pytest -q # from now on, offline 

Note the assertion style. I check that required keys exist and that types hold, and I never assert on the exact sentence the model produced. Exact-text assertions turn a recording into a fossil that fails the moment anyone improves a prompt, and they teach you nothing about the contract you actually care about.

Re-recording on purpose, and diffing the shape

Cassettes freeze a moment, and a frozen moment rots. The answer is not to re-record constantly, because then you are back to depending on someone else's Tuesday, but to re-record deliberately and then compare structure rather than prose. I keep a separate job that writes fresh cassettes into a scratch directory, and a small differ that reduces both sides to types and keys before comparing them.

# tools/drift.py import json import sys from pathlib import Path def shape(value): if isinstance(value, dict): return {k: shape(v) for k, v in sorted(value.items())} if isinstance(value, list): return [shape(value[0])] if value else [] return type(value).__name__ def walk(old, new, path="$", out=None): out = [] if out is None else out if type(old) is not type(new): return out + [("type", path, type(old).__name__, type(new).__name__)] if isinstance(old, dict): out += [("missing", f"{path}.{k}", None, None) for k in old.keys() - new.keys()] out += [("new", f"{path}.{k}", None, None) for k in new.keys() - old.keys()] for k in old.keys() & new.keys(): walk(old[k], new[k], f"{path}.{k}", out) elif isinstance(old, list) and old and new: walk(old[0], new[0], f"{path}[0]", out) return out def main(committed: Path, fresh: Path) -> int: findings = [] for cassette in sorted(committed.glob("*.json")): twin = fresh / cassette.name if not twin.exists(): findings.append(("unreplayed", cassette.name, None, None)) continue old = json.loads(cassette.read_text())["response"] new = json.loads(twin.read_text())["response"] findings += walk(shape(old), shape(new), f"${cassette.stem}") for kind, path, before, after in findings: print(f"{kind:<10} {path:<28} {before} -> {after}") return 1 if any(f[0] in ("type", "missing") for f in findings) else 0 if __name__ == "__main__": sys.exit(main(Path(sys.argv[1]), Path(sys.argv[2]))) 

CASSETTE_MODE=record CASSETTE_DIR=fresh pytest tests/test_extract.py -q python tools/drift.py tests/cassettes fresh; echo "drift_exit=$?" 

Reading the drift report without panicking

A report full of arrows looks alarming and usually is not, so I classify findings before I react to them. The table below is the entire policy, and I keep it in the repo next to the tool so nobody has to remember my mood on a Monday.

The distinction that matters is between a change I can absorb and a change that invalidates an assumption in my parser. Structured, type-level comparison gives me that line, while diffing raw text would bury me in noise that looks like signal.

Where the free tier earns its place in this loop

Here is the part I did not expect. The nightly re-record job is the least valuable place in my stack to spend money, because it is low volume, runs at 3 a.m., and does not care about latency as long as it finishes before standup. That is exactly the profile of workload a free tier suits, and it is how I use MonkeyCode's free models alongside its free server option: the probe service lives there, wakes up on a timer, records a handful of cassettes, and writes the drift report as an artifact for the morning.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would keep the recorder and the differ either way, because they exist to make a dependency inspectable rather than to make any particular provider look good. What I would check before copying this is unglamorous: read the operator's published limits and read the server option's actual specifications rather than assuming them, and treat the free tier as a place where a probe may be relocated at short notice.

The deployment stays small on purpose, since anything bigger turns the probe into infrastructure that needs its own runbook.

What tapes cannot tell you

A cassette proves that a shape was true once. It cannot prove the model got better or worse at the actual task, because the moment you freeze one response you stop sampling the distribution that made it. If your worry is quality rather than structure, you need a small human-reviewed eval set with synthetic inputs, scored weekly, and that is a different tool with a different failure mode.

Streaming is awkward too, since you have to assemble a stream into a final object before it can be recorded, and error paths deserve their own cassettes rather than being treated as noise. And please, record with synthetic text only; a cassette is a committed file, and a committed file is a place production documents should never end up.

Some teams should skip the whole apparatus. If your tool is one script you run by hand, a cassette is more machinery than the problem deserves. If you already hold a contract with a provider that includes schema guarantees, your vendor's change log is the better source of truth than my differ. And if nobody is willing to re-record once a month, the tapes will quietly become a museum of a system that no longer exists, which is worse than having no tests at all.

Start with one test, one cassette, and one scheduled probe, and let the red builds tell you whether it was worth it. If you try it on a free endpoint and find a drift class I have not listed, I would like to hear what it looked like.