Retries feel like the responsible thing to add to an LLM call. The provider hiccups, you wait half a second, you try again, the user never notices.
That holds right up until the provider is actually down. Then every retry is one more request that will also fail, sent at the exact moment the provider can least take it. In a drill against an offline simulator, I watched 1,000 prompts turn into 16,000 failed provider requests, and all of it came from my own "helpful" code.
Here are the five mistakes that got me there, roughly in the order I made them, plus the skeleton of the fix.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder. The complete, tested retry-budget project from this post is in there.
Building a paid, multi-tenant AI product? My AI SaaS Starter Kit for .NET 10 meters AI usage per organization and returns 429 with Retry-After from per-plan rate limits.
Mistake 1: Retrying in two layers without knowing it
The OpenAI .NET SDK has its own retry policy, and by default it retries a failed call 3 times. If your HttpClient also has a standard resilience handler with 3 retries, those multiply: 4 attempts × 4 attempts = 16 requests per prompt. Nobody wrote that number down. It just emerges.
The rule I follow now: exactly one layer owns retries, and it's the HttpClient layer, because that's the one that sees every request from every caller.
var options = new OpenAIClientOptions { Transport = new HttpClientPipelineTransport(resilientHttpClient), RetryPolicy = new ClientRetryPolicy(0), // SDK retries off, the handler owns them // endpoint, timeouts ... in the complete project }; I also like a startup check that refuses to boot when "SDK attempts × handler attempts" gets above 4. It's a few lines, and it stops someone from quietly re-enabling the second layer later.
Mistake 2: Thinking "one layer of retries" is the fix
Removing the duplicate layer took my outage drill from 16,000 requests to 4,000. Better, but that's still 4x normal traffic aimed at a provider that's already failing. Now multiply by every replica you run, all retrying at once. That's a retry storm, and it can keep a provider down after the original problem is fixed.
The question isn't "how many times should one call retry?" It's "how much extra load am I willing to add while things are failing?"
Mistake 3: Reaching for a circuit breaker as the only safety valve
A circuit breaker is a switch. Once failures cross a threshold, it blocks everything for a while, including requests that would have succeeded.
What I actually wanted was a dimmer: let retries flow when they're rare, and cut them off when they become a big share of traffic. That's a retry budget. The idea is old (the Google SRE book describes keeping retries under about 10% of a client's requests), and Microsoft.Extensions.Http.Resilience 11.0 now ships one, still targeting net10.0.
The shape of it on the standard handler:
services.AddHttpClient("llm") .AddStandardResilienceHandler() .Configure(o => { o.Retry.MaxRetryAttempts = 3; o.Retry.RetryBudget = new HttpRetryBudgetOptions { RetryRatio = 0.1, // retries may add ~10% load SamplingDuration = TimeSpan.FromSeconds(10), MinimumThroughput = 10, }; // attempt/total timeouts and breaker window for slow LLM calls: // full settings in the complete project }); The budget tracks requests and retries in a sliding window. Once the window has enough traffic, a retry is allowed only while retries stay under the ratio. When it says no, the error goes straight back to the caller instead of being retried.
One honest flag: these APIs are marked [Experimental("EXTEXP0001")] in 11.0.x. The package is stable, but the API shape may still change.
Custom pipeline? If you build your own with AddResilienceHandler, plain Polly AddRetry can't apply a budget, and setting one throws a ValidationException. Use AddRetryWithBudget(...) instead, which adds both the request-counting strategy and the retry.
Mistake 4: Adding a "just in case" retry loop in the app layer
This one is sneaky. The HTTP layer is budgeted, then someone wraps the use case in a for loop "for resilience", and the storm is back, one layer up.
My application handler never retries. It maps the outcome and tells the client when to come back:
return outcome switch { ModelCallOutcome.Answered a => /* 200 with the answer */, ModelCallOutcome.Busy b => /* 503 + clamped Retry-After */, ModelCallOutcome.Rejected r => /* upstream rejected, no retry */, _ => throw new InvalidOperationException() }; // mapping + Retry-After clamping in the complete project Failing fast is a feature here. In my outage drill, failed calls came back in about 4.6 ms (p50) with the budget, against about 88 ms with plain retries, because nobody sat through three backoffs first.
Mistake 5: Turning it on and never measuring it
A budget you can't see is a budget someone will "temporarily" disable during an incident. Two things fixed that for me:
- Count denials. Each denied retry raises a Polly telemetry event called
OnRetryBudgetExhausted. A smallTelemetryListenerthat counts it, next toOnRetry, gives you the number that proves the budget is protecting the provider.
public override void Write<TResult, TArgs>(in TelemetryEventArguments<TResult, TArgs> args) { if (args.Event.EventName == "OnRetryBudgetExhausted") Interlocked.Increment(ref _denied); // retries counter, snapshot endpoint ... in the complete project } - Drill it. I run the same calls through four setups (no retries, plain retries, budgeted retries, SDK + handler stacked) against an OpenAI-compatible simulator that fails an exact share of attempts. Same input, same counts every run.
These are my local numbers (1,000 calls, concurrency 16, 3 retries, ratio 0.1). Run your own drill and you'll get your own:
| Scenario | Plain retries | Retry budget (0.1) |
|---|---|---|
| Full outage, provider requests | 4,000 | 1,100 |
| 10% failing, success / load | 100% / 1.11x | 99.0% / 1.10x |
| 30% failing, success / load | 99.2% / 1.42x | 77.0% / 1.10x |
That last row is the honest cost. At 30% failures, a tight budget gives up success to protect the provider. Raising the ratio buys it back: 0.5 gave 98.9% at 1.41x in my drill. Pick the ratio on purpose, don't inherit it.
Quick checklist before your next deploy
- [ ] Only one layer retries (SDK retries set to 0 if the handler owns them)
- [ ] A startup check fails if total attempts per call go above your limit
- [ ] The retry layer has a budget, not just a max attempt count
- [ ] Custom pipelines use
AddRetryWithBudget, notAddRetry - [ ] The app layer has no retry loop, and busy maps to 503 +
Retry-After - [ ] Budget denials are counted and visible on a dashboard
- [ ] A drill in CI compares retry modes on identical traffic
Get the complete project
The snippets above are the skeleton. The full, tested .NET 10 project, with Clean Architecture layers, the retry-stack startup check, the denial counters and stats endpoint, the drill CLI with the ratio sweep and CI gate, and a long PDF walkthrough with architecture and request-flow diagrams, is in Tech Skill Builder. It's $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.
And if you're turning AI features into a paid product, the AI SaaS Starter Kit for .NET 10 handles per-organization AI metering and per-plan rate limits for you.
How many retry layers does your LLM call actually go through? I'd honestly like to know if anyone else found a hidden one.