Agent Replay Lab: a local-first VCR for TypeScript agent tool I/O
I open-sourced Agent Replay Lab, a local-first TypeScript toolkit that records, mock-replays, diffs, and CI-tests the tool I/O of tool-using agents.
- Project page: https://jarvisosorio.com/agent-replay-lab/
- Source: https://github.com/JarvisOsorio/agent-replay-lab
It answers a narrower question than observability products, which usually re-run live: does this agent still take the same tool path (name / order / args / responses) against the same recorded tool world, offline, in CI? That is behavioral regression for tool trajectories — not proof that the LLM would make the same decisions again.
Local-first. SQLite plus portable JSON. It is not an observability dashboard, eval leaderboard, agent framework, or a full-trajectory “deterministic agent” engine.
Honest modes in v0.1
| Mode | What it does | In v0.1? |
|---|---|---|
| offline / mock | Stub tools from the cassette; fail closed if a mock is missing | Yes (default) |
| structural compare | Diff tool name / order / args / response | Yes |
| live-tools | Call real tools | No (0.2) |
| live-model | Call a real model | No (0.2) |
Offline mock stubs tools only. It does not re-execute or pin LLM requests, responses, or tool-selection decisions; agent state machines, retries, or parallel scheduling; or clock, randomness, or other nondeterministic inputs. You can optionally record model.requested / model.responded events for inspection, but agent-replay replay does not play them back as a live model.
The cassette category is crowded (Cassette, Agentic Stash, Agent VCR, StoneTape, TracePact, agent-retrace, AgentTape, and others). Fail-closed offline replay, redaction, and CI diffs are not unique inventions — the bar for this repo is a small, honest TypeScript/SQLite knife.
Install (not on npm yet)
v0.1.0, Apache-2.0. Requires Node.js 22+ and pnpm (the repo pins pnpm@10.33.4). Clone the monorepo:
git clone https://github.com/JarvisOsorio/agent-replay-lab.git cd agent-replay-lab pnpm install pnpm build pnpm test # 5-minute proof pnpm --filter @agent-replay-lab/example-basic-agent demo Packages: @agent-replay-lab/core and @agent-replay-lab/cli (agent-replay commands: init, list, inspect, replay, diff, fixture, test, export, import, doctor). fixture create pins tool sequence, args, and responses. test golden --run fails with exit 1 on structural drift. Replay tests use in-memory or local cassette data and make no network calls.
Capture seam is wrapTool from @agent-replay-lab/core. Known secret key names and common token shapes are redacted before persist — best-effort, not a PII product. Pair with agentscrub for after-the-fact store remediation.
Not in v0.1: MCP adapter, OpenTelemetry adapter, flexible matching, semantic compare, dashboard, live auto-replay, telemetry, or behavioral-policy enforcement.
I'm Jarvis Osorio; I built this. Happy to answer questions about the fail-closed mock path or the structural CI gate.
