Letter to Pre-CI-Me: Make the Token Ledger Fail the Build

Three weeks ago I edited one sentence in a system prompt. It read cleaner. It also doubled the steps the agent took per task. No test failed. No alarm fired. The allowance drained quietly over two days.

I keep writing these letters to the version of me who starts the next harness. This one covers the gap I keep falling into. I measured agent cost. I never enforced it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What Went Wrong, in Order

1. A ledger nobody reads. I logged tokens per run into a file. The file grew. Nothing compared today to last week.
2. Tests against the live model. Every check called a real endpoint. Same prompt, different step count, no way to tell drift from noise.
3. No per-task ceiling. One tool-loop bug ran 41 steps before I noticed. The loop was legal, so nothing stopped it.

All three have the same fix. Record a run once, commit it as a fixture, then fail the build when the next run drifts.

Step 1 — Record Once, Replay Forever

The gate must never touch the network. That keeps it fast, deterministic, and free. You record fixtures on a real model, then commit the numbers.

// recorder.mjs — sketch. I have not executed this exact file. // Run once per case against a live model, then commit the JSON it prints. import { writeFileSync } from "node:fs"; const CASES = ["summarize-issue", "rename-symbol"]; const out = { version: 1, cases: {} }; for (const name of CASES) { const run = await runAgentCase(name); // your existing harness call out.cases[name] = { steps: run.steps, tokens: run.promptTokens + run.completionTokens }; } writeFileSync("run.json", JSON.stringify(out, null, 2)); 

Two fixtures matter: a baseline you trust and a current run. Keep them in the same shape or the gate cannot diff them.

Step 2 — Commit the Baseline

{ "version": 1, "cases": { "summarize-issue": { "steps": 4, "tokens": 1500 }, "rename-symbol": { "steps": 2, "tokens": 640 } } } 

Commit baseline.json with the same care as a lockfile. Update it in the same pull request that intentionally changes prompt or tool schema. Never update it in a fix-up commit.

Step 3 — Write the Gate

Zero dependencies. Node 18 and up.

// gate.mjs import { readFileSync } from "node:fs"; const load = (p) => JSON.parse(readFileSync(p, "utf8")); const baseline = load(process.argv[2] ?? "baseline.json"); const current = load(process.argv[3] ?? "run.json"); const TOKEN_GROWTH_LIMIT = 1.2; // allow 20% drift const STEP_GROWTH_LIMIT = 1; // allow one extra step const failures = []; for (const [name, base] of Object.entries(baseline.cases)) { const now = current.cases[name]; if (!now) { failures.push(`${name}: missing from run`); continue; } if (now.tokens > Math.ceil(base.tokens * TOKEN_GROWTH_LIMIT)) { failures.push(`${name}: tokens ${base.tokens} -> ${now.tokens}`); } if (now.steps > base.steps + STEP_GROWTH_LIMIT) { failures.push(`${name}: steps ${base.steps} -> ${now.steps}`); } } if (failures.length) { console.error("cost gate failed:\n" + failures.map((f) => " - " + f).join("\n")); process.exit(1); } console.log(`cost gate passed for ${Object.keys(baseline.cases).length} cases`); 

Run it locally before you trust it:

node gate.mjs baseline.json run.json # exit 0 when clean, exit 1 with a per-case diff when it drifts 

The thresholds are the whole argument. Twenty percent absorbs normal sampling variance. One extra step catches tool-loop regressions without flagging every model update.

Step 4 — Wire It Into CI

name: cost-gate on: [pull_request] jobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: { node-version: 20 } - run: node gate.mjs baseline.json run.json 

Now a prompt edit that doubles steps fails review. That is the only change I actually needed three weeks ago.

Step 5 — Record on a Disposable Box

The recording step is the only part that costs quota or needs a machine. That is where I now use MonkeyCode's free model access and free server option: record once on a throwaway box, commit the fixtures, delete the box. Check the project page for the current allowance and limits before you plan around them, since availability can change.

Open-source MonkeyCode is worth a look if you want to run this harness without a paid key on day one. The gate itself stays provider-neutral. That is deliberate, because the fixture format outlives any single endpoint.

When This Gate Pays Off

Limits and Who Should Skip It

This gate is a regression alarm, not a cost model. Fixtures go stale when the upstream model changes behavior. Token counts from a recorder are estimates of your own pipeline, not provider billing. And it says nothing about whether the output is still correct, so keep your assertion tests beside it.

Skip it if you run fewer than a dozen agent calls a day. Skip it if prompts live in a spreadsheet nobody versions. Skip it if you need latency percentiles, because a replay with no network cannot give you those.

What I Would Tell Myself

Do not build another dashboard. Build one file that exits non-zero. The ledger was never the useful artifact. The refusal was.

If you want to try the recording step without setup overhead, the MonkeyCode project is a reasonable place to start, and the gate above works the same either way.