Meter Cost per Accepted Answer, Not per Call
Last Tuesday a merged pull request fired the "Draft changelog" button in our internal release dashboard, and the model answered in about four seconds with JSON that read fine to a human. It failed one layer later, inside the worker's schema check, because body came back as an array of strings instead of a string. My retry loop then did exactly what I had told it to do: it appended the validation error to the same conversation and asked again, which is a reliable way to make attempt two cost more than attempt one. My dashboard said everything was healthy, because it measured tokens per call.
I want to argue a specific, unhedged position here: the billable unit of an AI feature is the accepted result, and per-call metrics are a form of self-deception that happens to be the easiest thing to build. Every rejected attempt is a small number that dissolves into a large average, so the chart stays flat while the thing you actually pay for quietly gets worse. A free token grant makes this failure mode more likely rather than less, because nothing hurts while you are forming the habit, and the habit you form at zero cost is the habit you keep when the cost is real. Would you rather learn this in month one at zero dollars, or in month three on an invoice you have to explain?
So I built the smallest object that tells the truth: a ledger keyed by task id that counts every attempt and reports tokens per accepted task. It has no dependencies, it fits in about sixty lines, and the test below runs offline against a fake transport. I have not benchmarked any provider with this code, and every number in this post is arithmetic rather than measurement, so please read it as a method you can reproduce instead of a result you should trust.
The ledger
// meter.js export function createMeter({ now = () => Date.now() } = {}) { const tasks = new Map(); const row = (taskId) => { if (!tasks.has(taskId)) { tasks.set(taskId, { attempts: [], accepted: null }); } return tasks.get(taskId); }; return { attempt(taskId, usage, outcome = 'returned') { row(taskId).attempts.push({ usage, outcome, at: now() }); }, accept(taskId, artifact) { const t = row(taskId); if (t.accepted) throw new Error(`task ${taskId} already accepted`); t.accepted = { artifact, at: now() }; }, report() { const rows = [...tasks].map(([taskId, t]) => { const tokens = t.attempts.reduce((n, a) => n + a.usage.total_tokens, 0); return { taskId, attempts: t.attempts.length, tokens, accepted: Boolean(t.accepted), tokensPerAccepted: t.accepted ? tokens : null, }; }); const done = rows.filter((r) => r.accepted); const acceptedTokens = done.reduce((n, r) => n + r.tokens, 0); return { rows, acceptedCount: done.length, tokensPerAcceptedTask: done.length ? Math.round(acceptedTokens / done.length) : null, abandonedTokens: rows.filter((r) => !r.accepted).reduce((n, r) => n + r.tokens, 0), }; }, }; } The important design decision is that the meter never learns about a success from the model call, only from the acceptance rule. That inversion is what kills the flattering average, because there is no code path where a rejected attempt can be counted as a completed unit of work. Everything else here is bookkeeping, and bookkeeping you can rewrite in an afternoon.
The retry wrapper
// client.js export async function generateWithRetry({ meter, taskId, transport, validate, prompt, maxAttempts = 3 }) { const messages = [{ role: 'user', content: prompt }]; let lastError; for (let i = 1; i <= maxAttempts; i++) { const res = await transport({ messages }); meter.attempt(taskId, res.usage, i === 1 ? 'first' : 'retry'); let parsed; try { parsed = validate(JSON.parse(res.content)); } catch (err) { lastError = err; // The correction you append is billed on the next attempt. Measure it. messages.push({ role: 'user', content: `Previous answer failed validation: ${err.message}. Return JSON only.` }); continue; } meter.accept(taskId, parsed); return parsed; } throw lastError; } Notice the line that appends the validation error, because that line is the whole problem in miniature. Attempt two carries attempt one's prompt plus a correction, so a "cheap" retry is strictly more expensive than the call it replaces. If your only instrument is tokens per call, the second attempt looks like one more ordinary data point and the growth is invisible.
The test that fails on a naive meter
// test/meter.test.js import test from 'node:test'; import assert from 'node:assert/strict'; import { createMeter } from '../meter.js'; import { generateWithRetry } from '../client.js'; const chat = (content, total) => ({ content, usage: { total_tokens: total } }); const validate = (o) => { if (typeof o.body !== 'string') throw new Error('body must be a string'); return o; }; test('counts every attempt behind an accepted answer', async () => { const meter = createMeter(); const replies = [ chat('{"title":"Fix login redirect","body":["a","b"]}', 1500), // rejected chat('{"title":"Fix login redirect","body":"Redirects now keep the return path."}', 1730), ]; const transport = async () => replies.shift(); const out = await generateWithRetry({ meter, taskId: 'pr-4821', transport, validate, prompt: 'draft a changelog entry', }); assert.equal(out.title, 'Fix login redirect'); const report = meter.report(); assert.equal(report.rows[0].attempts, 2); assert.equal(report.tokensPerAcceptedTask, 3230); assert.equal(report.abandonedTokens, 0); }); test('separates abandoned work from accepted work', async () => { const meter = createMeter(); const transport = async () => chat('{"title":"x"}', 900); await assert.rejects( generateWithRetry({ meter, taskId: 'pr-4822', transport, validate, prompt: 'draft', maxAttempts: 2 }), ); const report = meter.report(); assert.equal(report.tokensPerAcceptedTask, null); assert.equal(report.abandonedTokens, 1800); }); Run it with node --test test/meter.test.js and you get a red bar against any meter that remembers only the last attempt. That test is the artifact I care about, because it locks the opinion into CI instead of leaving it in a document nobody rereads.
The arithmetic your dashboard hides
With the numbers above, tokens per call average out to 1615 and tokens per accepted changelog is 3230, a factor of two that no per-call chart will ever show you. Now imagine the acceptance rule rejects the second attempt too, which happens constantly when a validator is strict about types. The third attempt carries two corrections, the task finally lands, and your per-call average still looks reasonable while the real unit cost has tripled. Is that a model problem, a prompt problem, or a measurement problem? From where I sit it is almost always the third one, and it is the only one of the three you can fix without guessing.
Let the acceptance rule own the retry
The architectural consequence is small and it changed how I structure workers: the validator owns the retry decision, the meter owns the ledger, and the task id owns idempotency. The model call becomes a dumb function that takes messages and returns text plus usage, and it stops being the place where policy accumulates. Once those three responsibilities are separate, a network retry can no longer silently double-charge your ledger, because the ledger is keyed by task id and refuses to accept the same task twice. That guard has saved me more debugging time than any prompt rewrite I have done this year.
Running the loop where it is not your laptop
A meter that only runs during local testing measures your enthusiasm, not your feature. You need the worker, the ledger write, and the schedule living on a machine that keeps going after you close the lid, which is why MonkeyCode's free model access and free server option are relevant to this workflow at all. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The framing I would defend is that the free grant is not really about free text generation, it is permission to run an embarrassing worker on a schedule against a real endpoint while the validator, the retry path and the ledger are all still wrong. Point the transport at a free model, deploy the worker to the free server, and let the acceptance rate and the abandoned-token number accumulate for a week before you touch a single prompt.
The specific grant is advertised as ten million free tokens at the time of writing, but check the current terms yourself, because free-tier conditions are set by the provider and they move. What you are buying with them is the observation window, and an observation window you never use is worth exactly what you paid for it.
What this does not tell you
Client-side usage numbers can disagree with provider billing, and when they do, the provider wins and your ledger needs an import step. Acceptance rate is a property of the validator you wrote, so a loose rule will make the meter look great while accepting garbage, and a strict rule can make a good model look expensive. One task type is not a cost model, and a week of changelog entries says nothing about a support summarizer with a longer prompt and a worse schema. Skip this approach entirely if you already get per-task billing from your provider, if you are building a demo that will never see a second user, or if your validator accepts everything by construction, in which case the meter is just a per-call chart with extra code.
A checklist you can steal
- Key every attempt by task id, and let only the acceptance rule mark a task done.
- Assert tokens per accepted task in a test with a fake transport before you point anything at a real endpoint.
- Log the correction you append on retry, because that text is billed on the next attempt.
- Track abandoned tokens separately, since work with no artifact is the cheapest thing to ignore and the most expensive to carry.
- Run the loop on a schedule somewhere that stays up, and compare client-side totals to the provider's bill weekly.
Which handoff is least stable in your stack right now: the validator deciding to retry, the meter writing the ledger, or the worker surviving a restart mid-task? Send me a concrete failure state or response code and I will fold it into the next iteration of this meter, because the version I am running today is only the second one I have trusted.
