Cassette the Tool Layer and Replay Agent Runs Offline

You cannot replay a model. You can replay everything the model was handed, and that turns out to be enough for most agent bugs. The trick is to move the recording boundary down one layer: not the prompts, not the completions, but the tool calls and their results.

Most teams instrument the wrong surface. They save full transcripts, then discover that a transcript is a story about tokens, while the bug lives in what a tool returned at step seven. Re-run the same task and the model writes a different story, so the diff is unreadable and the failure is unreproducible. Tool-layer replay removes that noise by making the environment deterministic while leaving the model free to vary.

Record at the boundary, not at the top

A cassette is an append-only JSONL file of (tool, normalized args) -> result records, plus the ordered key sequence of the run. The key is a hash, so reordering JSON keys or reformatting a path does not break a match.

# cassette.py - record and replay the tool layer only import hashlib, json from pathlib import Path def key(tool: str, args: dict, schema_version: str = "v1") -> str: blob = json.dumps( {"v": schema_version, "tool": tool, "args": args}, sort_keys=True, separators=(",", ":"), ) return hashlib.sha256(blob.encode()).hexdigest()[:16] class Cassette: def __init__(self, path: str, mode: str = "record"): assert mode in {"record", "replay"} self.path, self.mode = Path(path), mode self.calls: list[str] = [] self._seen: dict[str, dict] = {} if mode == "replay": for line in self.path.read_text().splitlines(): rec = json.loads(line) self._seen[rec["key"]] = rec["result"] def call(self, tool: str, args: dict, fn, schema_version: str = "v1"): k = key(tool, args, schema_version) self.calls.append(k) if self.mode == "replay": if k not in self._seen: raise KeyError(f"cassette miss: {tool} {k}") return self._seen[k] result = fn(**args) with self.path.open("a") as f: f.write(json.dumps({"key": k, "tool": tool, "result": result}) + "\n") return result 

Two details matter more than they look. First, schema_version is part of the key, so renaming a field or narrowing a parameter fails loudly at replay instead of silently serving a stale answer. Second, drop wall-clock fields from stored results before writing. Keep ordering; discard absolute time. A created_at inside a cassette guarantees a noisy diff on every run and teaches reviewers to ignore diffs.

Redaction happens on the way in, not on the way out. Hash or truncate anything credential-shaped before the record is written, because cassettes get committed, copied to laptops, and pasted into issues.

The debug loop

Record once against live tools. Replay as many times as you want with the network unplugged.

# 1. capture a baseline run (real tools, real side effects - use a sandbox) python -m agent.run --cassette runs/base.jsonl --task task-07 > /dev/null # 2. replay it offline; any cassette miss is a behavior change python -m agent.run --cassette runs/base.jsonl --replay --task task-07 # 3. compare call paths, not prose diff <(jq -r .key runs/base.jsonl) <(jq -r .key runs/candidate.jsonl) 

The third command is the useful one. It prints a sequence of opaque hashes, which sounds useless until a step shifts and you can see exactly where: identical for nine calls, then a divergence, then a call that never appears in the baseline. That is a call-path diff, and it is stable in a way that text diffs are not. You are comparing decisions to act, not sentences about acting.

A cassette miss is not an error to swallow. Treat KeyError as the signal: the agent asked a question the baseline never asked, which is either a real behavior change or a gap in your recording. Both are worth a human look, and both are cheap to review because the key points at one tool and one argument set.

What repeats and what does not

Text transcripts answer "what did the model say". Cassettes answer "what did the run depend on". The second question generalizes: the same recording can be replayed by a rewritten prompt, a different model, or a refactored loop, and the call path tells you whether the behavior moved.

That table is the honest boundary. Replay is a regression and reproduction tool, not an evaluation harness, and it will lie to you about performance because every replay is fast.

Where the harness runs

A replay proxy is only useful if it is always reachable and free to hit while you debug, which is the unglamorous reason I run this harness on MonkeyCode's free server option and call models through its free model access, both supplied by the operator. The server hosts the cassette endpoint and the tiny diff service; the model access covers the recording runs and the occasional live check when a cassette miss needs a fresh answer. That keeps the loop usable on a laptop with no API budget and no key juggling in CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I did not benchmark the free tier here, and quotas, hardware, and longevity are things you should confirm with the provider yourself before depending on them. If you already have paid infrastructure and a locked-down network boundary, the value is the cassette format and the diff, not the hosting.

Limitations to accept up front

Cassettes rot. A tool whose output legitimately changes every day - prices, timestamps, generated IDs - needs normalization before it goes into the store, or the recording is a one-shot artifact. Side effects still happen on the record pass, so record inside a sandbox with disposable credentials and a throwaway namespace; replay is safe precisely because it never calls out.

Also accept that a cassette cannot prove a run was correct. It proves the run took the same shape as a previous run. Pair it with an assertion about the final artifact, or you will happily certify a regression for its consistency.

Start with one task, record it, and break one tool on purpose to watch the miss fire. If the diff reads cleanly on that toy failure, you have a debug loop worth keeping.