Ever since AI has taken over the internet, I've been wondering how we can optimize the full ability of AI in our life. Therefore I began building a small gateway that sits between your code and the OpenAI, Anthropic and Gemini APIs. It has one job: if a request would push today's spend past a daily cap you set, refuse it before it goes out.

Why bother? Because the stories are always the same. An agent gets stuck in a loop, keeps retrying, every retry sends a longer conversation, and nobody notices until the bill shows up. Logs and alerts tell you what happened. I wanted something that says no before it happens.

(To be fair to OpenAI: they now have hard spend limits, and you should turn them on. But they're monthly, they're OpenAI only, and their own docs say enforcement isn't instant, so spend can go slightly over. I wanted a daily cap that works across providers.)

The idea is simple: today's spend + whatever is still running + an estimate for this call. Over the cap? Refuse.

The estimate is where I kept getting it wrong. Here are six places, with real numbers from my own test account.

1. You can't know the output, so you reserve the worst case

Before a request goes out you know the prompt. You don't know the answer. So the only honest number is the ceiling: max_tokens × the output price. No max_tokens? I assume 4,096. Above 10,000? I clamp it down, so one request can't reserve something silly.

To test it I set my own daily cap to $0.001 (yes, a tenth of a cent) and sent gpt-6-luna with max_tokens: 10000. The output reserve alone is 10,000 × $0.50 per million = $0.005, five times the cap. The gateway said no with a 402. OpenAI never saw the request. Cost: $0.

What surprised me: once a cap sits in front, max_tokens stops being a length setting and becomes a budget setting. Plenty of clients send a huge number by default. Under a small cap, they get refused.

2. Reasoning models can spend the whole budget thinking

This one confused me for a minute. To keep a test cheap I sent gpt-6-luna a short prompt with max_tokens: 20. I got a 200 back... and an empty answer. finish_reason: "length". Usage said 7 in, 20 out.

All 20 output tokens went to reasoning. Nothing was left for the reply.

On OpenAI's reasoning models the output limit covers the thinking plus the visible text. For the budget math that's fine, since the reserve is still the ceiling. For whoever gets the empty reply, not so fine. If you use reasoning models, leave real headroom in max_tokens, or you'll pay for a call that returns nothing.

3. My input count comes in low

I count input with the o200k tokenizer over the message text. Same short test: I estimated 1 input token. OpenAI billed 7.

The gap is the formatting the chat format wraps around each message, which I wasn't counting.

In money it's nothing ($0.0000101 estimated, $0.0000107 actual). But it always misses in the same direction, and it grows with every message. Agent conversations have a lot of messages. I haven't fixed it yet. It's next on my list, and I'd rather say that than pretend it's done.

4. Don't guess a price for a model you don't know

My first version had what I thought was a clever fallback: unknown model? Price it like the most expensive one I know and let it through. Safe, right?

Not really.

  • You can't just strip the date to find the price. gpt-4o-2024-05-13 was $5 / $15 per million tokens; gpt-4o is $2.50 / $10.
  • Models get retired. OpenAI shut down o4-mini, o3-mini and gpt-4.1-nano on October 23, 2026, and gpt-5, gpt-5-mini, gpt-5-nano, o3 and o3-pro go on December 11 (deprecations).

So now it's strict. If the model isn't in my price table, the request gets a 400 UNKNOWN_MODEL before it's sent, with a note to use the name without a date. My rule now: if I can't price it, I can't cap it. The downside is a chore: every time I add a model, I check the provider's deprecation page first.

5. Images are a guessing game

Version one: if a request had any image in it, I counted the whole thing as 4,000 input tokens, text included. So a big image was under-counted and the text was ignored completely. Not great.

Now I count the text normally and reserve each image at that model's worst case, from the provider's own rules. That's up to 36,000 tokens per image on some newer OpenAI models, and 4,784 on newer Claude models. One image on gpt-6-luna reserves about $0.0036.

Safe, but the real charge is usually much lower. With a small cap, image-heavy traffic runs out of room because of reservations, not because of real spend. The reserve is swapped for real usage once the call finishes, but until then it takes up space.

6. Caching makes the check stop too early

Long agent runs re-read the same context again and again, and providers charge much less for cached input. My estimate charges all input at full price.

Someone on the OpenAI forum posted a 13-hour, two-agent run where 95.4% of the tokens were cached re-reads (thread). On a run like that, my check would hit the cap long before the real bill does. A reply in that thread put it better than I can: price everything at full rate and you stop too early; assume the cache and you get surprised when it misses.

I picked "stop too early". The better fix is probably to look at each session's recent cache-hit rate and estimate from that. I haven't built it.

If you're building one yourself

Everything above leans toward refusing too early. A refused call costs nothing; a missed one costs money. That's basically the whole philosophy.

It's still an estimate, not a billing guarantee. Input can come in low (see #3), and the cap only sees traffic that actually goes through it.

If I had to give one piece of advice: decide what your gate does when it can't price a call, before anything else.

I'm curious how others handle this. If you've solved the per-message overhead or cache-aware estimates, how did you do it?