There is a proverb I grew up with: don’t trouble the trouble, otherwise trouble will trouble you. It’s usually offered as advice about people. It’s even better advice for dealing with distributed systems.
Every engineer shares the same instinct. A call fails. Try it again. It’s such a reasonable response that most frameworks now handle it for you silently, baking in a sensible-looking default of three attempts. You don’t write the retry; you inherit it. For a single client, that’s correct. Networks drop packets. Instances get recycled mid-flight. Trying again is often all it takes. The trouble starts when there are ten thousand clients, and they all have that exact same instinct at the exact same moment.
The Arithmetic Nobody Does
Imagine a service that comfortably handles 1,000 requests per second with healthy headroom. It can manage about 1,200 before it starts queueing.
Now, something downstream degrades. Not catastrophically—maybe a database failover, an evicted cache node, or a dependency whose p99 latency drifts past the timeout. Responses slow down. Calls start hitting the one-second timeout.
Every caller does what it was configured to do: it retries. Twice.
- The service is now being offered 3,000 requests per second.
- Its capacity is 1,200 requests per second.
Every request now times out, meaning every request gets retried again. The load no longer depends on the original fault that started it. The dependency recovers; the service stays down.
| Strategy | Attempt 1 | Retries / Action | Total Load Offered | System Capacity | Result |
| Blind Retry | 1,000 rps | 2,000 rps (after timeouts) | 3,000 rps | 1,200 rps | System Down (Self-sustaining queue collapse) |
| Retry Budget (10%) | 1,000 rps | 100 rps (capped / circuit breaker open) | 1,100 rps | 1,200 rps | Stable (Queue drains, fast failure) |
Notice what happens to the cause. The original fault—the slow dependency—is no longer what’s generating the traffic. The load is now self-sustaining, feeding entirely on its own response to the fault. You can heal the dependency completely, and the service will stay down because the retry traffic has taken over.
Network engineers gave this a name decades ago: congestion collapse. We tend to rediscover it every time a new layer of the stack gets a default retry policy.
The Client Cannot Tell Slow from Dead
The deeper problem is that a retry is a decision made with insufficient information.
When a caller times out, it knows precisely one thing: no response arrived within the deadline. It doesn't know whether the request was lost in transit, whether it was received and dropped, whether it’s still executing, or whether it actually succeeded while the response is sitting stuck in a buffer somewhere. A timeout isn't a failure signal; it's the absence of a signal—which is an entirely different thing.
So the client guesses. And the guess it makes ("probably transient, try again") is the correct one when the system is healthy, and the exact wrong one when the system is overloaded. Retries are most aggressive precisely when the system can least afford them.
Doing Less, Deliberately
What’s striking about the fixes for this pattern is that none of them involve adding capacity. They are all exercises in restraint:
- Cap retries as a fraction of traffic, not per request. Instead of "each request may be attempted three times," the rule becomes "retries may not exceed 10% of total requests." Under normal conditions, nothing changes. Under failure, amplification is hard-bounded by construction.
- Back off exponentially and add jitter. Without jitter, every client that failed at the same moment waits the exact same interval and slams the service back in a synchronized wave. Randomizing the delay is what actually diffuses the traffic.
- Open a circuit breaker. After consecutive failures, stop calling altogether and fail fast. It feels counterintuitive—you are refusing to try something that might work—but a fast, honest failure gives the downstream service room to drain its queue. Nothing recovers while it's being hammered.
- Shed load at the door. If you can't service a request within its deadline, drop it immediately rather than accepting it into a queue where the caller has already given up. Admission control separates a degraded service from a dead one.
Agents Made This Much More Expensive
This dynamic has always mattered, but it matters considerably more now because of what a retry actually costs.
In a conventional service, a wasted retry burns a few milliseconds of CPU and some socket connections. Annoying, but cheap. In an agentic system, a retried step is a full model round trip: tokens in, tokens out, GPU seconds, and the key-value cache held resident for the duration. The amplification factor is identical, but the unit cost has multiplied by orders of magnitude.
It’s worse because agents retry at multiple levels. The orchestration framework retries the failed tool call. Then the model, seeing an error returned in its context window, often decides on its own to try an alternative approach—triggering a secondary retry loop your configuration knows nothing about and cannot bound.
A three-attempt policy wrapped around a model that independently reasons its way into three more attempts is a nine-attempt policy that nobody designed.
If you are running agents in production and haven’t looked at what happens to your token spend during a minor dependency blip, it’s worth an afternoon. The cloud bill is frequently more alarming than the downtime.
The General Shape of It
The instinct to retry is the instinct to do something. It feels like diligence. When a system struggles, doing something is how we prove we are paying attention.
But a loaded system doesn’t need more attention; it needs less demand. The useful question during a degradation event is almost never "what else can we try?"—it is "what can we stop asking for?"
Shed the traffic, open the breaker, bound the retries, and let the queue drain. More often than not, the system will recover on its own. Leave the trouble alone, and it tends to pass. Poke at it three times a second from ten thousand machines, and it will absolute trouble you back.
I have written about the operational side of this at more length—circuit breakers, backpressure, and the finding that a large share of serious outages begin as a dependency failure that propagates rather than as an internal fault—in Operational Excellence in Real-Time AI Systems, European Journal of Computer Science and Information Technology 13(47), 2025.

