My denial agent's config said Sonnet. I had 94 eval runs and I was proud of them. Then, the week I was handing the system over, I read the logs line by line. All 94 came from Haiku. Sonnet had been returning 429s the whole time, and the fallback had caught every one of them without a word. So I had good numbers for a model I never picked.
What actually happened
Daniel is the agent I built to read insurance denials for a medical billing operation. It reads the denial, works out why the claim was rejected, and says what to do next. Like most production LLM code, it has a fallback: if the main model fails, try a cheaper one. That sounds responsible. It's what you're supposed to do.
The trouble is what counts as failing. A 429 doesn't mean the model is broken. It means you hit your rate limit. My dev key was on a small plan, so Sonnet threw 429s often enough that nearly every assessment went to Haiku. The config said one thing and the logs said another. I was reading the config.
- 94 / 94 eval runs that were really Haiku
- 0 people who chose Haiku
- 1 field it took to catch it
I'd like to say I caught it early. I didn't. I caught it at handover, which is the worst time to find something like this, and also the most common.
The part that scared me
The numbers weren't the scary part. Haiku did fine on the evals. What scared me was what would happen next.
The billing client would run Daniel on their own paid key, with much higher limits. On their account Sonnet would almost never return a 429. So their real traffic would go to Sonnet, a model with no eval runs behind it at all. Different model, same config file. All my testing described a product they would never actually run.
What broke: Every eval I ran described Haiku. The customer with the bigger account would have quietly gotten Sonnet, so upgrading their plan would have changed the product without anyone touching the code.
Think about that for a second. A billing decision on their side would change what the agent says about a denied claim. Nobody would see it. Nobody would get paged. The answers would just drift, a little at first, in work where a small drift means money left on the table or an appeal filed that shouldn't be.
Your eval numbers belong to whichever model actually answered, and your quota decides that, not your config.
It isn't a healthcare problem
What bothers me more than the bug is that I see the same shape everywhere I work. I run a denial agent for medical billing, voice agents that call insurance payers, and a quoting engine for a contracting business. Those have almost nothing in common. Except this: each one has a file that says what's supposed to happen, and a runtime that does whatever it can get away with.
A fallback is one way the two drift apart. A retry is another, and so is a cached answer, or a default that kicks in when an env var goes missing. Each of those is a quiet place where what you tested and what you shipped stop being the same thing. Nothing errors, so nobody looks.
The fix is boring. I pin the exact model I measured. If a fallback fires, it logs loudly and stamps the output with the model that really answered. And until the logs prove otherwise, I treat the fallback as the real primary, because for a lot of traffic it is.
Key insight: Until your logs tell you which model answered, assume the fallback is your real primary, and test that.
The result: Daniel now runs on one pinned model, the one with 94 eval runs behind it. Every response records which model produced it, so a mismatch shows up in a query instead of a customer complaint.
What I'd tell someone building this
- Read the logs, not the config. The config only tells you what you meant to happen.
- Don't let a rate limit switch models. A 429 means wait and retry, not swap in a different brain.
- Stamp every output with the model that produced it. It costs you one field.
- Test on the customer's quota, or one like it. Your dev key is running a different product.
- If you keep a fallback, test it like a primary, because some days it will be one.
I used to think of a fallback as a safety net. Now I think of it as a second product that ships whenever the first one is busy. Usually nobody tests it, and it ends up talking to the customers anyway.
When you test an agent, whose quota do you test it on: yours, or your customer's?
Originally published at nabbilkhan.com.
