48-Hour Field Notes: I Built a Token Ledger Before Pointing a Free LLM Tier at a Loop

The free allowance is not the interesting part; the accounting is

Every time a provider hands out a generous free allowance, someone points a while True at it and discovers what "generous" means at three in the morning. Have you ever watched a nightly job eat a month of quota in eleven minutes because a retry loop never checked whether the previous call actually succeeded? The number on a pricing page is not a budget, and a budget without a ledger is just a wish with better typography.

So I spent 48 hours this week doing the boring thing first: putting every model call behind a ledger before letting any batch job near a free endpoint. The endpoint I used was MonkeyCode's free model access, which the project's outreach describes as an open-source project with a free tier and a free server option; I treated both as things to measure rather than things to trust, because availability claims age faster than code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I actually ran, in three phases

I did not start with the batch. I started with one call, then a fixed suite, then a timer. That order matters more than any tool choice below.

1. Phase 1 — one call, logged by hand. A single prompt, one response, and a CSV row I typed myself. If you cannot account for one call, you cannot account for four thousand.
2. Phase 2 — a fixed 12-case suite, run three times. Same prompts, temperature=0, same model string. The goal was not quality scoring; it was finding out whether two token counters ever agree.
3. Phase 3 — a timed batch behind a stop rule. This is where a free allowance becomes dangerous, because the failure mode is not an error, it is silence plus a shrinking balance.

Why estimated and reported tokens never quite match

Before the code, here is the thing that surprised me least and irritated me most. Three separate effects push your local estimate away from the provider's reported usage, and none of them are bugs:

• Different tokenizers. Your cl100k_base guess is not necessarily the tokenizer behind the endpoint, so the gap scales with prompt length.
• Hidden overhead. Chat templates, system scaffolding, and tool schemas are billed even though you never typed them.
• Streaming without usage. Some streaming paths omit the usage block entirely, so you get a completion and no receipt.

If your ledger only stores one number, you have no way to tell a cheap run from a misconfigured one. Store both, always.

The harness: one file, no vendor lock-in

Everything below talks to any OpenAI-compatible endpoint. I deliberately did not hardcode a base URL, because those change and a stale URL in a tutorial is worse than no URL at all.

# ledger.py — endpoint-agnostic token ledger for a fixed eval suite import csv, hashlib, json, os, time from dataclasses import dataclass, asdict from openai import OpenAI client = OpenAI( base_url=os.environ["LLM_BASE_URL"], # whatever the provider's docs show today  api_key=os.environ["LLM_API_KEY"], timeout=45.0, max_retries=1, # retries are a budget decision, not a default ) REQUIRED = {"title": str, "risk": str, "steps": list} def estimate_tokens(text: str) -> int: try: import tiktoken return len(tiktoken.get_encoding("cl100k_base").encode(text)) except Exception: return max(1, len(text) // 4) # crude fallback, label it in your report  def valid_schema(text: str) -> bool: try: obj = json.loads(text) except json.JSONDecodeError: return False return isinstance(obj, dict) and all( isinstance(obj.get(k), t) for k, t in REQUIRED.items() ) 

The row shape is the artifact I care about most, because it is the part you will still be reading in six months.

@dataclass class Row: case_id: str prompt_sha: str attempt: int latency_ms: int prompt_tokens_reported: int | None completion_tokens_reported: int | None prompt_tokens_estimated: int completion_tokens_estimated: int schema_ok: bool error: str FIELDS = list(Row.__dataclass_fields__) def run_case(case: dict, attempt: int) -> Row: started, text, err = time.monotonic(), "", "" reported_in = reported_out = None try: resp = client.chat.completions.create( model=os.environ["LLM_MODEL"], messages=[{"role": "user", "content": case["prompt"]}], temperature=0, ) text = resp.choices[0].message.content or "" usage = getattr(resp, "usage", None) if usage is not None: reported_in, reported_out = usage.prompt_tokens, usage.completion_tokens except Exception as exc: err = f"{type(exc).__name__}: {exc}" return Row( case_id=case["id"], prompt_sha=hashlib.sha256(case["prompt"].encode()).hexdigest()[:12], attempt=attempt, latency_ms=int((time.monotonic() - started) * 1000), prompt_tokens_reported=reported_in, completion_tokens_reported=reported_out, prompt_tokens_estimated=estimate_tokens(case["prompt"]), completion_tokens_estimated=estimate_tokens(text), schema_ok=valid_schema(text), error=err, ) 

The stop rule is the whole point

A ledger that only records is a diary. A ledger that refuses to continue is a circuit breaker, and the breaker is what actually protects a free allowance from your own retry logic.

class BudgetGuard: def __init__(self, allowance: int, stop_at: float = 0.20): self.allowance, self.stop_at, self.spent = allowance, stop_at, 0 def charge(self, row: Row) -> None: used = (row.prompt_tokens_reported or row.prompt_tokens_estimated) + \ (row.completion_tokens_reported or row.completion_tokens_estimated) self.spent += used if self.spent >= self.allowance * self.stop_at: raise SystemExit( f"stop rule hit at {self.spent}/{self.allowance} " f"({self.spent / self.allowance:.1%}) — read the ledger first" ) 

Note the 20% default. Stopping at a fifth of the allowance feels paranoid until the first run where you discover a prompt template that quietly doubled in size.

Wiring it together, with crash-safe writes

An append-and-flush write per row is unglamorous and it is the difference between a partial ledger and no ledger when a run dies mid-batch.

allowance = int(os.environ.get("TOKEN_ALLOWANCE", "10000000")) # ^ set this to the number your account page shows today; the outreach material # cited 10,000,000 at time of writing, and free-tier numbers do change.  guard = BudgetGuard(allowance) for case in json.load(open("cases.json")): row = run_case(case, attempt=1) with open("ledger.csv", "a", newline="") as fh: writer = csv.DictWriter(fh, fieldnames=FIELDS) if fh.tell() == 0: writer.writeheader() writer.writerow(asdict(row)) fh.flush() os.fsync(fh.fileno()) guard.charge(row) 

Reconciling afterwards takes ten lines and answers the only question that matters: is my estimate drifting away from reality?

python - <<'PY' import csv rows = list(csv.DictReader(open("ledger.csv"))) num = lambda v: int(v) if v not in ("", "None") else None rep = sum((num(r["prompt_tokens_reported"]) or 0) + (num(r["completion_tokens_reported"]) or 0) for r in rows) est = sum(int(r["prompt_tokens_estimated"]) + int(r["completion_tokens_estimated"]) for r in rows) ok = sum(r["schema_ok"] == "True" for r in rows) print(f"calls={len(rows)} schema_ok={ok}/{len(rows)} reported={rep} estimated={est}") print("est/reported ratio:", round(est / rep, 3) if rep else "n/a — no usage block seen") PY 

What broke, and what I would repeat

Three failure modes showed up fast, and I would bet on all three showing up for you too.

• Missing usage on success. A 200 with no usage block is not an exception, so a naive ledger records zero spend and reports a beautiful, fictional ratio.
• Retries silently double-charging. My first max_retries=3 turned one logical case into four billed calls; setting it to 1 and retrying from my own loop made the accounting legible.
• Partial writes. Logging after the loop means a crash deletes the evidence you needed, which is exactly when you need it most.

What I would repeat without hesitation: run the fixed suite three times before scaling, and make the stop rule the first thing you write, not the last.

Laptop, free server, or paid endpoint?

The free server option is genuinely useful for one job shape and wasteful for another. Here is the table I used, and you should rewrite it for your own constraints.

One caveat I will not bury: if you move the batch to a remote box, check its clock before you trust any timestamp in your ledger. I have been bitten by that before, and a ledger with the wrong clock is a ledger you cannot audit.

Who should not do any of this

Be honest about the fit before you copy the code. Skip this workflow if your calls are interactive and one-at-a-time, because the harness costs more attention than it saves. Skip it if your provider already gives you hard spend caps and per-key budgets, since you would be rebuilding a solved problem. And skip the free server option entirely if the data cannot leave your machine, no matter how convenient the allowance looks.

Closing note

A free allowance is a loan against your future debugging time, and the interest is charged in ambiguity. Whether you run this against a free tier or a paid one, the ledger is what turns "it seemed fine" into a number you can argue with. If you want somewhere costless to try the harness first, the project's free model access and free server option are a reasonable place to start — just bring the stop rule with you.

Verify current limits, model names, and server terms on the project's own documentation before you depend on them; free-tier details are exactly the kind of thing that changes between the day an article is written and the day you read it.