We scaled our pricing service to twelve pods and the first four kept every request

Our checkout service calls the pricing service over gRPC, through an ordinary ClusterIP Service on port 50051. Pricing has a horizontal pod autoscaler targeting seventy percent CPU, and on a normal day it runs four pods.

On the Friday of our autumn sale, traffic rose and the autoscaler did what we asked. It took pricing from four pods to twelve within ten minutes. Checkout latency kept climbing anyway. The four original pods were at full CPU and the eight new ones were close to idle. The autoscaler looked at the average, about thirty five percent, decided we had capacity to spare and began scaling back down.

A ClusterIP Service is balanced by kube-proxy, and kube-proxy chooses a backend when a TCP connection opens. After that every packet on the connection goes to the same pod. gRPC runs over HTTP/2, which carries every call on one long lived connection, and the Go client's default policy, pick_first, keeps that one connection for as long as it works. Each of our six checkout pods had connected once, at startup, to one of the four pricing pods that existed then. Nothing closed those connections, so nothing ever reached the eight pods added later. Our earlier load test had started all its clients after the servers were scaled, which spread them evenly.

That afternoon we restarted checkout, which at least moved its six connections onto six different pods, and latency eased. Then we fixed it properly. Pricing now has a headless Service as well, so DNS returns the address of each ready pod. Checkout dials it with the dns resolver and the round_robin policy in its service config, which opens a connection to every address and spreads calls across them. Pricing sets MaxConnectionAge to five minutes in its keepalive parameters, so clients reconnect regularly and the resolver looks again, which is how new pods join the rotation. We also export calls per pod, and alert when the busiest pod handles more than twice the median.

Kubernetes balanced every connection we opened. We opened six of them on the first morning and then sent everything through them for the rest of the day.

– Sergey Shinder