Most speech-to-text comparisons end with one number, the word error rate. It's a useful number, and I still check it first when I look at a new model. But when I'm building a voice agent that books appointments or takes order details over the phone, WER keeps telling me things are fine on calls that actually failed.
This post is about why that happens and how I'd score the small set of words that decide whether a call succeeds: digits, names, dates and codes.
A small example from this week
Voice Arena released its Dictation Bench this week. They played 761 recordings from 87 speakers into 10 dictation apps on a Mac, and then five LLM judges checked whether each draft could be sent as it was. One example in their results stuck with me. A speaker read a locker code as "twelve, thirty, eight". Only Gemini typed 12-30-8. The other apps wrote things like 12:38, 1238 or 12-38.
You can see what happened. The model heard "thirty" followed by "eight" and merged them into "thirty-eight", which is how most people say that pair of words. The pause and the comma in the speaker's voice were the only signals that these were separate numbers, and most systems ignored them.
In a dictation app this is a small edit. In a voice agent, the agent now has the wrong code, and the caller doesn't find out until the door doesn't open.
Why WER hides this
WER counts substitutions, deletions and insertions, and divides by the number of words in the reference transcript:
WER = (S + D + I) / N Every word has the same weight. Say the full utterance was "my locker code is twelve thirty eight". After normalization the reference becomes my locker code is 12 30 8 and the hypothesis becomes my locker code is 12 38. That's one substitution (30 became 38) and one deletion (the 8 is gone), so two errors out of seven words.
That sounds bad, but nobody scores one sentence. Over a full call of about 200 words, those same two errors are 1% WER, and a model at 1% looks excellent on a dashboard. The call still failed, because the only words the backend needed were the wrong ones.
The opposite also happens. A model can get "um", "okay" and "so yeah" wrong all day and push WER up without hurting a single booking. So WER can be high on a call that worked and low on a call that didn't.
Normalization makes this harder to see. Most WER scripts normalize both sides before comparing, so "twelve" and "12" count as the same word. That's the right choice for fair comparisons between models, but it also means the scoring step may quietly turn "12:38" and "12 38" into the same tokens. If you only look at the final number, you won't notice that formatting was where the real damage was.
Score the slots separately
What I'd do instead is keep WER as a general health number, and add a second score for the values the agent actually acts on. I call these slots, because that's what they are in the conversation: the phone number, the order ID, the date, the customer's name.
For each test call you need the expected slot values, written down by a person who listened to the audio. Then you run the call through your pipeline and compare what came out.
For digits, I check two things. First, did we get the right digits in the right order? Second, did we keep the grouping the speaker used? The grouping matters for codes like 12-30-8, and it matters less for something like a phone number, where the backend usually only needs the digit string.
type DigitScore = { sameDigits: boolean; sameGroups: boolean }; function digitGroups(text: string): string[] { return text.match(/\d+/g) ?? []; } function scoreDigits(expected: string, heard: string): DigitScore { const exp = digitGroups(expected); const got = digitGroups(heard); const sameDigits = exp.join("") === got.join(""); const sameGroups = sameDigits && exp.length === got.length && exp.every((group, i) => group === got[i]); return { sameDigits, sameGroups }; } scoreDigits("12-30-8", "12:38"); // { sameDigits: false, sameGroups: false } scoreDigits("12-30-8", "12 30 8"); // { sameDigits: true, sameGroups: true } Notice that "12:38" doesn't even pass the first check. The digit string is 1238, not 12308, because the zero from "thirty" disappeared when the two numbers were merged. A WER script that splits "12:38" into "12" and "38" would still give some credit for the "12". The slot check gives none, which matches what the caller experiences.
Names and emails need a different check. I compare them after lowercasing and removing spaces and punctuation, and I log near misses separately. "Aqib" heard as "a keep" is a real failure, but it's useful to know whether the model was close, because that tells you whether a custom vocabulary list would fix it.
Report it per slot type
Once you have slot scores, report them as a small table instead of folding them into one number:
| Slot type | Calls | Exact match |
|---|---|---|
| Phone number | 120 | ? |
| Order ID | 80 | ? |
| Date | 95 | ? |
| Name | 110 | ? |
The question marks are on purpose. These numbers come from your own test set, and they'll be different for every product. What matters is that each row is something you can act on. If dates are weak, you look at how your prompt or parser handles "the third" versus "3rd". If order IDs are weak, you look at letter and digit confusion, like "B" and "D" or "eight" and "A".
This is also where the test set itself matters. Read speech in a quiet room is easy. The cases that break are callers reading a code off a card while walking, saying a number twice because they corrected themselves, or pausing in the middle of a phone number. I'd rather have 50 calls like that than 500 clean ones.
Fixes that follow from the score
When a slot type keeps failing, there are a few fixes I'd try, roughly in this order.
The first is to tell the STT what to expect. Many providers let you pass a list of key terms or boost certain words. If the agent knows it's about to ask for an order ID, it can pass the expected format or known product names for that turn.
The second is to stop trusting the formatted transcript for structured values. Ask for the raw words or the word-level output with timestamps if the provider gives it, and parse the digits yourself. The gap between "thirty" and "eight" is often visible in word timestamps even when the formatted text merged them.
The third is to read the value back. "I have your code as one two, three zero, eight. Is that right?" costs a couple of seconds, and it turns a silent failure into one the caller can fix right there. For anything the backend can't undo, like a payment amount or a booking date, I think the read-back is worth it even when the model is usually right.
The last one is to make the score part of every change. When you swap an STT model, change the endpointing settings or edit the prompt, rerun the slot checks. Turn detection changes are a common cause of slot failures, because cutting the caller off after "twelve thirty" produces a clean transcript with a missing digit.
Wrapping up
WER tells you how well a model transcribes speech in general. A voice agent needs to know something narrower: did it get the handful of values it's about to act on? Scoring those slots separately, with checks for digits, grouping and names, gives you a number that moves when calls actually fail. It also points you at the fix, which a single WER figure rarely does.
