Our quote API runs as eight pods behind a Service, and the Service sends traffic to a named port:

ports:
- port: 80
targetPort: http

A named targetPort is resolved separately for every pod. The endpoints controller looks in each pod's spec for a container port with that name and uses its number, which is what lets different pods use different numbers behind one Service. If a pod has no port with that name, it is simply left out of the Service's endpoints. Nothing marks the pod as failed, and its readiness is not affected.

In September we were tidying our manifests to follow a naming convention for a service mesh trial, and a pull request renamed the container port in the Deployment from http to http-api. The Service lived in a different file and was not touched. The change was reviewed, merged and rolled out at ten past two on a Thursday.

The rollout did what rollouts do. New pods started and passed their readiness probes, and the old pods were terminated two at a time. Every new pod was ready and none of them was an endpoint. The Service shrank from eight addresses to six, then four, then two, and when the last old pod went it had none. ingress-nginx answered every quote request with 503. kubectl rollout status reported that the deployment had successfully rolled out, because by every measure the Deployment knows about, it had.

The engineer on call saw eight healthy pods and spent fifteen minutes on the ingress before running kubectl get endpointslices and finding it empty. A rollback brought the old port name back and the Service filled again within seconds. We could not quote for nineteen minutes.

The deploy pipeline now waits after every rollout until the number of ready endpoints behind each Service in the release equals the number of ready pods it selects, and fails if it does not. A policy check in CI rejects any Service whose named targetPort does not exist in the pod template it selects. We alert on any Service with no ready endpoints that had some in the last hour. And each Service now lives in the same file as the Deployment it fronts.

A pod being ready and a pod receiving traffic are two separate facts, and Kubernetes keeps them in two separate places.

– Sergey Shinder