Quick question. If you run a voice agent, do you know your prompt cache hit rate on turn 1? Not the average across the call. Just turn 1, when the caller has said hello and is sitting in silence waiting for your agent to speak. We thought ours was fine. We pre-warm before every call, like everyone tells you to. Then we measured it properly, and our outbound calls were doing less than half as well as our inbound calls. Same code, same provider, same prompt template. It took us a while to find out why let jump on to it how we find out Quick recap: prefix caching When you send a prompt, the provider processes it into an internal representation (the KV cache). Prefix caching keeps that around for a few minutes. If your next request starts with the exact same text, the provider skips reprocessing that part and only works on what is new. Request 1: [ 3,000 tokens of instructions ][ user says hi ] └─ processed from scratch ─────┘ Request 2: [ 3,000 tokens of instructions ][ user asks something else ] └─ reused from cache ──────────┘ └─ only this is new ─┘ Every big provider does this now, and the numbers look roughly the same everywhere: Typical behaviour Minimum cacheable prefix ~1,024 tokens Cache read price ~10% of normal input price How long it lives Minutes of inactivity, sometimes ~30 Do you enable it? Usually automatic For a voice agent this should be free money. The system prompt, tools and guardrails are the same on every turn of every call, and they are most of your tokens. The cost saving is nice. The latency saving is the real prize, because you are cutting hundreds of milliseconds off time to first token while someone waits on the phone. Three rules nobody tells you Caching is automatic, but it is not unconditional. Three things have to be true. 1. The prefix must match exactly. Not mostly. Not semantically. Byte for byte, starting from the very first token. One character different and the match stops right there. 2. It has to be long enough. Below the minimum, usually 1,024 tokens, you get nothing cached. Not a partial hit. Zero. 3. It has to land on the same machine. The cache lives in the memory of one server. Providers route you based on a hash of the start of your prompt, but under load they spread traffic around. Land on a new machine and the cache is cold no matter how clean your prompt is. Rule 1 is the one that quietly kills hit rates. That is the one we broke. What everybody already does The standard trick for voice agents is pre-warming. Before the call connects, you fire a throwaway request with the same system prompt so the provider caches it. By the time the caller says hello, turn 1 has something to reuse, and turns 2, 3, 4 ride on top of it. We do this too. It works. But look at what it actually optimises: one call. Turn 2 reuses turn 1 of the same call. Nobody talks about the other half, which is reusing the cache across calls. That is where our numbers were bleeding. The number that made no sense We started tracking cache hit rate properly: cached tokens divided by total prompt tokens, per call. Overall it was around 40%, and about a third of eligible calls got zero cache even though our prompt is several times bigger than the minimum. Then we split it by direction: Call type Cache hit rate Inbound 70-75% Outbound 20-30% Same infrastructure. Same pre-warm. Fifty point gap. So we broke outbound down by how many LLM turns each call made: LLM turns in the call Cache hit rate 1 ~24% 2 ~36% 3 ~46% 4+ ~50% A clean upward line, which is exactly what you would expect. Caching was working inside a conversation. Turn 2 reused turn 1, turn 3 reused turn 2, and so on. Once a call has made a couple of requests and keeps landing on the same machine, it builds its own cache and the later turns are fine. They fix themselves. Turn 1 has nothing. There is no earlier turn in that call to reuse. And here is the killer: most of our outbound calls never make it past turn 1 or 2. Voicemail, instant hangup, a two sentence "not interested". A big chunk of our traffic lives in the top row of that table, at 24%. Making turn 4 better does nothing for a call that ended on turn 1. So from this point on, forget the average. The only number worth improving is turn 1. The smoking gun We pulled the raw per call token counts from our logs and looked at how many tokens were actually being cached. The most common value, by a wide margin, was exactly 1,024. Not around a thousand. Exactly 1,024, the provider's hard floor, the smallest amount it is willing to cache. Hundreds of calls landing on that same number. That number is a message. It means the provider walked through our prompt looking for the longest matching prefix, hit a mismatch just past the 1,024 mark, and cached the bare minimum it was allowed to. So we opened the system prompt and counted. About a thousand tokens in, there it was: the customer's name. A bit further down, a timestamp. ┌─────────────────────────────────┐ │ static instructions │ ← ~1,000 tokens, same every call ├─────────────────────────────────┤ │ Hello {customer_name} │ ← different every call ├─────────────────────────────────┤ │ the other 6,000+ tokens │ ← unreachable. always full price. │ of static instructions │ └─────────────────────────────────┘ Everything above the name was identical across every call. Everything below it, thousands of tokens of instructions, examples and tool descriptions, was being reprocessed at full price on every single request. Forever. Because one variable sat in the way. This also explained something we had been staring at for days. Single turn calls were hitting ~14%, which looked like our pre-warm was only working one time in six. It was not broken at all. Our prompt is about 7,200 tokens, and 1,024 out of 7,200 is 14%. The warm-up worked perfectly. It just could not warm more than the prefix allowed. Before we got here we burned time on two theories that went nowhere. We thought we were overloading a single cache key with our batch dialling, so we bucketed calls by concurrency and found no correlation at all. We also nearly reached for explicit cache keys, which control which machine you land on, when our actual ceiling was how much was reusable once you got there. The lesson: a bad hit rate has two separate causes, depth (how much of your prompt is reusable) and coverage (how often you reach a warm machine). In an aggregate percentage they look identical. Fix the wrong one and you get nothing. Why inbound was fine all along The inbound prompt had almost no dynamic variables in it. We do not know who is calling until they tell us, so there is nothing to personalise at the top. That means every inbound call sent a byte identical prefix. One call warmed the machine, the next twenty reused it. That 70-75% was not something clever we built. It is just what the number looks like when the prefix is not broken. Once we saw that, inbound stopped being a mystery and became the target. We shared the finding with the team and treated 70% as the bar outbound should be able to hit. Optimising across calls, not just inside one Here is the shift. Everyone tunes the cache for a single call: pre-warm, then let later turns reuse the earlier ones. That already works, and it is why turns 3 and 4 looked okay. But it can never help turn 1 of a fresh call, because there is nothing from that call to reuse yet. The only thing that can help turn 1 is a cache some other call left behind. And we are set up perfectly for that. We dial in bulk, hundreds of calls in the same window, and a prefix cache lives for something like 30 minutes depending on the provider. Call 1 pays for the cold start and calls 2 through 50 should ride on it, starting from their very first turn. That was not happening, because the customer's name a thousand tokens in made every call's prompt unique. Call A's cache entry was useless to call B. BEFORE — every call warms its own private prefix call 1 ──> machine A (warms something only call 1 can use) call 2 ──> machine B (warms something only call 2 can use) call 3 ──> machine A (still a miss, different prompt) AFTER — every call warms the prefix for every other call call 1 ──> machine A (cold: writes the shared prefix) call 2 ──> machine B (cold: writes the shared prefix) call 3 ──> machine A (HIT, call 1 warmed it) call 4 ──> machine B (HIT, call 2 warmed it) call 5 ──> machine C (cold: writes it) call 6+ ──> A/B/C (HIT, HIT, HIT...) The fix Move every dynamic value out of the top of the prompt. Static stuff first: instructions, tool definitions, examples, guardrails, anything identical on every request. Personalisation, timestamps, session IDs and user context all go at the end, after the reusable block. BEFORE AFTER ┌──────────────────┐ ┌──────────────────┐ │ instructions │ │ instructions │ │ {customer_name} │ ✗ break │ tool definitions │ │ tool definitions │ │ examples │ │ examples │ │ guardrails │ │ {timestamp} │ ├──────────────────┤ │ guardrails │ │ {customer_name} │ ✓ └──────────────────┘ │ {timestamp} │ └──────────────────┘ It is a reorder, not a rewrite. The model sees the same information. But now the reusable prefix runs the whole length of the static block instead of stopping at the first placeholder. What changed We deployed and compared the same metrics across the boundary. Outbound only, since inbound was already healthy and we did not touch it. LLM turns Before After 1 ~24% 62% 2 ~36% 69% 3 ~46% 65% 4 ~50% 60% Turn 1 went from 24% to 62%, and that is the row carrying most of our traffic. Be clear about what this fix is and is not. It is a turn 1 fix. Turns 3 and 4 barely moved, and that is fine, because they were never the problem. A call that survives a few turns and keeps landing on the same machine builds its own cache and gets there on its own. Turn 1 was the only turn with no path to a hit, and now it has one: the calls that went out before it. The curve flattening is the real signal. It no longer matters much how long a call runs, because the first turn already starts warm. One more thing worth being precise about. The fix made hits deeper, not more frequent. The share of requests that reach a warm machine barely moved. What changed is that when a request does reach a warm machine, it reuses the whole static prompt instead of a 1,024 token stub. And bulk dialling flipped from a problem into an advantage. Before, every call warmed a private prefix nobody else could use, so more concurrency just meant more cold starts. Now the first few calls to each machine seed a prefix that everything after reuses. With enough volume you saturate the pool of machines your traffic touches, and a steady stream of calls is what keeps the cache alive. The honest caveat: this is per machine and often per region. Split your traffic across regions and each one warms separately. Cold machines do not disappear either, they just become a smaller share as volume grows. Takeaway Pre-warming is table stakes and it only buys you one call's worth of cache. Later turns will sort themselves out. Turn 1 will not, unless the calls before it left something behind that this call can actually use, and that only happens if they all share the same prefix. Which costs nothing except keeping your variables at the bottom of the prompt. If your hit rate is stuck, go look at the distribution of cached tokens. If there is a spike at exactly 1,024, you already know where your first variable is.