Our stock lookup service answers in about twelve milliseconds at the ninety ninth percentile, and the shop calls it with a timeout of two hundred and fifty. It runs as eight pods behind an Application Load Balancer that sends traffic straight to the pods. For a year, every deploy produced a small burst of timeouts in the shop: a few hundred failed lookups, products shown as unavailable for a minute, then nothing. We had filed it under deploys being a bit noisy.

In October we started deploying several times a day, and the bursts became a visible share of the shop's errors, so I looked properly. Every timeout came from a new pod, in its first forty to sixty seconds.

A JVM starts by interpreting bytecode. Methods that run often get compiled, quickly by the C1 compiler first and later, after many more invocations, into optimised code by C2. Until the hot paths are compiled, a request costs many times what it will cost a minute later, and class loading and the first database connections add to it. Our readiness probe said ready as soon as Spring had started, so the load balancer gave each new pod its full share of traffic immediately. The pod was correct, only slow, and slow was enough to cross the shop's timeout.

The rollout replaced pods two at a time, so at every step a quarter of the traffic was going to cold code.

The service now warms itself before it reports ready. At start it replays a few thousand recorded stock lookups through the real code path against the real database, and readiness turns true only when that is done and the last batch has a p99 within twice the normal figure. The target group has slow start set to ninety seconds, so the load balancer ramps a new pod's share of requests up over that time instead of handing it everything at once. The deploy bursts have gone, and each rollout takes two minutes longer, which nobody has noticed.

Ready meant the process was up. It needed to mean the process was fast.

– Sergey Shinder