The Budget Guard Refunded Twice, So the Batch Outlived Its Cap

The short version: my token budget guard refunded the same reservation on every retry. The counter drifted upward, the batch ran past its cap, and half the shards died at the worst moment. The fix was one rule: a reservation can be closed exactly once.

One note before we start. The log shapes below are rebuilt from a synthetic repro, not a customer run. The pattern is real. The numbers are illustrative.

The alert said 78%, the provider said 100%

The batch stopped at 04:10. Three of twelve shards failed on quota errors.

My local counter showed 78% of the budget consumed. The provider dashboard showed 100%.

Both numbers came from the same run. Only one of them was real.

So which one do you trust when the tool you built is the thing lying to you?

Not the counter. Never the counter. Trust the ledger.

Step 1: Stop counting, start logging

A single integer cannot be debugged. It has no history and no owner.

So I replaced the counter with an append-only event log. Every state change writes one line.

import json, time def log(path, ev, rid, amount): with open(path, "a") as f: f.write(json.dumps({"ev": ev, "rid": rid, "amount": amount, "t": time.time()}) + "\n") 

Three event types. reserve when a call starts. settle when real usage is known. refund when nothing was consumed.

The rid field matters most. It is the reservation identity, and it is where the bug was hiding.

Step 2: Replay the ledger instead of rereading the code

Reading your own code makes you defend it. Replaying events makes the code confess.

I wrote 20 lines that fold the log into a final number.

#!/usr/bin/env python3 import json, sys, collections events = [json.loads(l) for l in open(sys.argv[1])] closes = collections.Counter(e["rid"] for e in events if e["ev"] in ("refund", "settle")) print("reserves", sum(1 for e in events if e["ev"] == "reserve")) print("refunds ", sum(1 for e in events if e["ev"] == "refund")) print("settles ", sum(1 for e in events if e["ev"] == "settle")) print("net ", sum(e["amount"] for e in events)) print("multi-closed rids", {r: c for r, c in closes.items() if c > 1}) 

The replay printed 412 reserves and 449 refunds. More refunds than reserves.

No accounting system can refund more often than it reserves. That is the entire bug, visible in one line of output.

python replay.py run.jsonl | tail -3 # refunds 449 # settles 0 # multi-closed rids: 37 entries 

Zero settles. Interesting. The happy path never reconciled at all.

Step 3: Find both refund call sites

Here is the retry wrapper as it shipped.

def call_with_retry(client, prompt, est, budget): for attempt in range(3): try: return client.complete(prompt) except TransientError: budget.refund(est) # refund A: per failed attempt  time.sleep(2 ** attempt) raise QuotaExceeded("gave up") 

And here is the caller.

def guarded_call(client, prompt, budget): est = estimate(prompt) + MAX_OUTPUT budget.reserve(est) try: return call_with_retry(client, prompt, est, budget) except Exception: budget.refund(est) # refund B: on final failure  raise 

Two refunds for one reservation. A transient timeout refunds. The outer handler refunds again.

Two retries plus a final failure equals three refunds against one reserve. The budget grows by two estimates.

Is that a retry bug or an accounting bug? It is an ownership bug. Nobody owned the reservation.

Step 4: Make close idempotent

The fix is small. A reservation can be closed exactly once, and the closer must present the rid.

import uuid class Budget: def __init__(self, limit, path): self.limit, self.spent, self.path = limit, 0, path self.closed = set() def reserve(self, est): rid = uuid.uuid4().hex[:12] self.spent += est log(self.path, "reserve", rid, est) return rid def _close(self, rid, ev, amount): if rid in self.closed: return False # second close is a no-op  self.closed.add(rid) self.spent += amount log(self.path, ev, rid, amount) return True def refund(self, rid, est): return self._close(rid, "refund", -est) def settle(self, rid, est, actual): return self._close(rid, "settle", actual - est) 

Now the caller owns the reservation, and the retry loop does not touch it.

def guarded_call(client, prompt, budget): est = estimate(prompt) + MAX_OUTPUT rid = budget.reserve(est) try: resp = call_with_retry(client, prompt) except Exception: budget.refund(rid, est) raise budget.settle(rid, est, resp.usage.total_tokens) return resp 

Notice what changed in the retry path. A timed-out attempt keeps its reservation open.

Why? Because a timeout does not mean zero tokens. The server may have generated the full completion.

Treating that as free is how you get a surprise. Write the assumption down in a comment.

Step 5: Pin the invariant with a test

Every ledger needs one assertion that must never fail.

def test_counter_matches_ledger(tmp_path): b = Budget(limit=10_000, path=tmp_path / "l.jsonl") rid = b.reserve(800) assert b.refund(rid, 800) is True assert b.refund(rid, 800) is False # idempotent  events = [json.loads(l) for l in open(b.path)] assert b.spent == sum(e["amount"] for e in events) == 0 

Run it on every change to the retry layer. That layer is where accounting goes to die.

pytest -q tests/test_budget.py -k ledger 

Step 6: Resume from checkpoints, not from zero

A quota stop is not a crash. It is a pause. Treat it like one.

Each shard writes its cursor after every completed item.

import json, os def save(path, state): tmp = path + ".tmp" open(tmp, "w").write(json.dumps(state)) os.replace(tmp, path) # atomic on POSIX 

On restart, read the cursor and skip finished work. The rerun cost 4% of the original budget instead of 100%.

Running the replay harness somewhere cheap

The replay is light on CPU and heavy on output. I stopped running it on my laptop.

That is where MonkeyCode fit in. It is an open-source project with free model access and a free server option, which is enough to host a nightly replay job and a log summarizer. The operator states the free tier includes a 10M-token allowance; confirm the current terms in the console before you plan around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I used the model for was boring and useful. It turns a 400-line ledger into a five-line summary of which rids closed twice. That is a good fit for a free tier. It is a terrible fit for anything needing a hard spend guarantee.

Where this approach breaks

• A client-side ledger cannot cap provider-side billing. Only a provider quota can. Use both.
• If your retry policy treats timeouts as free, your numbers will drift. Document the assumption.
• Ledgers grow fast. Rotate per run and keep about 14 days.
• If you make fewer than 100 calls a day, skip all of this. A spreadsheet is enough.

Should you build the ledger?

The checklist I reuse

1. Replace the counter with an append-only log.
2. Give every reservation an ID.
3. Replay the log before you read the code.
4. Count closes per ID. More than one is your bug.
5. Assert the invariant in a test.
6. Keep reservations open across retries.
7. Reconcile against provider usage once per run.
8. Checkpoint work so a pause is cheap.

The counter was never the problem. The missing owner was.

If you run this harness on a free tier, grep your ledger for multi-closed IDs first. That one line told me more than two hours of reading the retry code.