A serving bug can break a few percent of a model's answers while every eval score you track stays flat. Counting the answers that fail a simple check catches it with thirty times less traffic.
Engineer's view. This is one bad host behind a load balancer. One server in thirty returns garbage, and average latency barely moves. You find it by counting errors, not by averaging. A hosted model breaks the same way. A bug on some of its servers hides inside your average eval score while users see broken replies.
TL;DR. To detect an LLM API quality drop, count the responses that fail a cheap check instead of averaging an eval score. How rarely that check fires on healthy replies decides how fast you see the drop.
Explain Like I'm 5
You run a lemonade stand, and a helper pours every cup. One day the helper starts adding salt to one cup in thirty. You taste-test one cup a day and give it a score out of ten.
The scores barely change. Most cups are fine, and your scores already wobble from day to day.
Now try something else. Ask every customer one question: "Was it salty?" Nobody ever says yes on a normal day. So two yeses in a morning tell you something broke.
The Problem
You have shipped this bug before, and it had nothing to do with AI.
One host behind your load balancer went bad. It returned errors on maybe one request in thirty. Your dashboard showed average latency and average success, and both looked normal. Users complained for days before someone split the error count by host.
A hosted model is that fleet of hosts. You call one model name, and the provider serves it from thousands of machines. If some of those machines are misconfigured, a small share of your replies come back broken. The rest are fine.
This is not hypothetical. In August 2025, three serving bugs at Anthropic degraded Claude's answers for about a month. One sent some requests to servers set up for a different context length. It touched 0.8% of one model's requests at first, and 16% in its worst hour. Another made the model sometimes pick tokens it should almost never pick, like Thai characters in an English reply.
Anthropic wrote that its evaluations "simply didn't capture the degradation users were reporting." The model often recovers from one bad token, so a quality score barely drops. If the company that runs the model missed it, your weekly benchmark run will too.
The fix: count the replies that fail a cheap check, compare that count to its normal rate, and size your traffic so a real drop can show.
Read the full post on The AI Commit
How to Tell Whether the Model Behind Your API Got Worse carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.