The Cassette Take-Home: Grade Whether an Agent Can Be Re-Run Without the Model

The most useful signal in an agent take-home is not the quality of the demo output; it is whether a reviewer can reproduce that output offline, without an API key, six weeks later. This task grades that property directly. Candidates make one recording pass against a live model endpoint, commit the recording, and from then on the suite must pass with the network disabled. The exercise takes roughly ninety minutes and leaves behind a small harness the hiring team reuses on every subsequent candidate.

The prompt handed to candidates

Build a ticket-triage agent that classifies a support message into an intent and a human-escalation flag. Record one live pass against a model endpoint. Commit the recording. The full suite must then pass in replay mode with no network access and no credentials present in the environment.

Constraints shipped alongside the prompt:

• One model endpoint is used for the recording pass; a free tier is acceptable and no paid key is provided by the interviewer.
• No credentials may appear in the repository, inside cassettes, or in log output.
• Recording mode may be executed once by the candidate; reviewers only ever execute replay mode.
• The repository must expose make record, make test, and make demo.
• A short REPLAY.md must state which request fields were normalized and why.

Prompt quality, latency tuning, and any hosted interface beyond an optional demo page are explicitly out of scope. The graders care about one thing: can a stranger re-run this thing after the endpoint has moved on.

Why a recording beats "the reviewer can just run it live"

Four problems show up whenever a take-home is graded with live model calls, and each one is predictable rather than accidental.

• A live run hands the reviewer a credential problem on day one, and half of them stop there.
• Model endpoints drift, so identical prompts can return materially different text months apart.
• Flaky live calls train reviewers to ignore red builds, which defeats the point of a test suite.
• Replay runs are cheap enough to execute on every commit, including on a modest free server.

The last point matters more than it looks. A suite that costs nothing to run is a suite that actually gets run, and a candidate who understands that distinction usually also understands caching boundaries and cache invalidation risks.

The artifact: a cassette layer of about forty lines

The harness sits at the transport boundary instead of inside the agent, so candidate logic never learns that replay exists.

# cassette.py import hashlib, json, os, pathlib, time VOLATILE = {"user", "request_id", "id", "created"} def canonical(payload: dict) -> str: clean = {k: v for k, v in payload.items() if k not in VOLATILE} return json.dumps(clean, sort_keys=True, separators=(",", ":"), ensure_ascii=False) def key_for(payload: dict) -> str: return hashlib.sha256(canonical(payload).encode("utf-8")).hexdigest()[:16] class CassetteMiss(RuntimeError): pass class Cassette: def __init__(self, path, mode=None): self.dir = pathlib.Path(path) self.dir.mkdir(parents=True, exist_ok=True) self.mode = mode or os.environ.get("CASSETTE_MODE", "replay") if self.mode not in {"record", "replay", "passthrough"}: raise ValueError(f"unknown CASSETTE_MODE: {self.mode}") def call(self, transport, payload): key = key_for(payload) target = self.dir / f"{key}.json" if self.mode == "replay": if not target.exists(): raise CassetteMiss( f"no cassette {key}; run `make record` and commit the result" ) return json.loads(target.read_text())["response"] response = transport(payload) if self.mode == "record": target.write_text(json.dumps({ "key": key, "request": json.loads(canonical(payload)), "response": response, "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), }, indent=2, ensure_ascii=False) + "\n") return response 

Normalization rules the rubric checks

The volatile set is deliberately short, because every dropped field is a place where a real difference can hide.

• Request identifiers, creation timestamps, and end-user tags are stripped before hashing.
• Key ordering and separator whitespace are canonicalized through sort_keys and explicit separators.
• Sampling parameters stay in the key, since a temperature change is a genuine request change.
• The model identifier is stored in cassette metadata rather than hashed, so a model swap is visible in review.

Proving the network is actually off

A replay suite that quietly falls back to the live endpoint passes on the candidate's laptop and fails in review. An autouse fixture closes that gap in six lines.

# conftest.py import socket import pytest @pytest.fixture(autouse=True) def no_network(monkeypatch): def guard(*args, **kwargs): raise RuntimeError("network access attempted during replay") monkeypatch.setattr(socket.socket, "connect", guard) 

# tests/test_triage_offline.py import os os.environ["CASSETTE_MODE"] = "replay" from agent import triage def test_damaged_order_is_a_refund(): out = triage("Order 8812 arrived damaged; the customer wants a refund.") assert out["intent"] == "refund" assert out["needs_human"] is False 

The candidate then demonstrates the workflow with two commands and one size check.

CASSETTE_MODE=record make test # single live pass, free or local endpoint CASSETTE_MODE=replay make test # reviewer path, no credentials, no network du -sh cassettes/ && python scripts/prune_cassettes.py --keep 1 

The rubric

Each row is scored 0 to 3, weighted, and reported with the evidence line that justified the score.

A score of 3 requires the reviewer to reproduce the behavior without asking the candidate anything. That single rule removes most of the ambiguity that interview rubrics usually carry.

The sample solution, step by step

1. The agent exposes a pure function that takes a message and returns a dictionary, with the transport injected at construction time.
2. The transport is wrapped by Cassette.call, so recording and replaying share one code path.
3. A single recording pass produces one cassette file per distinct request, usually between ten and twenty files for this task.
4. The pruning script keeps the newest cassette per key and drops duplicates created by reruns.
5. REPLAY.md records the endpoint family, the recording date, and every field omitted from the hash.

The resulting repository is small enough to read in ten minutes, which is itself part of the signal being measured.

Failure modes seen in review

Seven patterns account for nearly every failure, and each one points at a specific habit.

1. Raw request dumps committed with authorization headers intact, which is a security finding rather than a style issue.
2. Hashing the full prompt string without canonicalization, so a trailing newline invalidates every cassette.
3. A silent fallback to the live endpoint on a cassette miss, which turns reproducibility into an illusion.
4. Replaying an entire conversation as one blob, so no reviewer can tell which turn diverged.
5. Hundreds of megabytes of unpruned cassettes committed, which makes review slow and diffs useless.
6. Claiming a replayed pass proves model quality, when it only proves determinism of the surrounding code.
7. Missing model metadata, so nobody can tell later which endpoint produced the recording.

Failure mode three deserves special attention. A candidate who writes except CassetteMiss: return transport(payload) has optimized for a green build instead of a trustworthy one, and that instinct is expensive to correct after hiring.

Choosing between live calls, stubs, and cassettes

The table is worth including in the prompt itself, because it tests whether a candidate can state where a technique stops being appropriate.

Where free model access and a free server change the economics

A recording pass is a one-time cost, but the review pass runs hundreds of times across candidates and reviewers. At the time of writing, the operator describes MonkeyCode as offering free model access with a ten-million-token allowance plus a free server option, so the recording pass does not need a procurement conversation and the replay demo can be hosted without a budget line. Confirm the current terms on the project site before designing a course or a hiring loop around them, since availability claims age quickly.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The useful part is not the allowance itself but the failure behavior it exposes. Free endpoints are sometimes slow or briefly unavailable, and a candidate who has built the cassette layer correctly simply re-runs the suite offline while everyone else waits for a retry.

Limitations and who should skip this format

• Streaming responses and multi-turn tool calls need extra cassette semantics, and a ninety-minute task may not leave room for them.
• Time-sensitive behavior, such as cache expiry or prompt-version rollout, cannot be captured in a static recording.
• Candidates who have never written a test double will spend most of the budget learning the harness instead of the task.
• Teams hiring for pure product sense should not use this format, because it measures engineering discipline rather than taste.
• Recordings go stale, so the hiring team needs an owner who refreshes them when the endpoint changes.

A recording is evidence about a codebase, never evidence about a model, and reviewers should keep that boundary explicit in their written feedback.

If the harness looks reusable, the cheapest way to validate it is a single recording pass against MonkeyCode's free endpoint and a reviewer-only replay page on the free server; the code above depends on neither, which is rather the point.