A liveness probe is meant to recover one stuck container. Configured badly, it restarts every healthy pod you have at the same moment, and a 30 second blip becomes an outage.
Here is how that happens, and how to set the three probes so it does not.
Three probes, three jobs
Kubernetes gives a container three probes. They look alike in YAML and do very different things when they fail.
- Startup probe: has the app finished starting? Until it passes, the other two probes do not run.
- Readiness probe: can this pod take traffic right now? If it fails, the pod is removed from the Service endpoints. It is not restarted.
- Liveness probe: is this process stuck beyond recovery? If it fails, the kubelet restarts the container.
The difference between "stop sending traffic" and "kill it" is the whole story.
How a blip becomes an outage
A common setup: one /health endpoint that checks the database, used for both readiness and liveness.
- The database has a 30 second failover.
-
/healthstarts failing on every pod, because every pod shares that database. - Liveness fails everywhere, so every container is restarted at roughly the same time.
- The database comes back. Now all pods are cold-starting together, reconnecting together, and warming caches together.
- The startup load slows the app down, probes time out again, and you get a restart loop.
Nothing was wrong with the pods. Restarting them could not fix the database. The probe turned a dependency problem into a self-inflicted one.
What each probe should check
Liveness: only the process itself. Can the event loop or a worker thread answer a trivial request? No database, no cache, no downstream API. If a restart would not fix the failure, liveness should not be testing for it.
Readiness: what this pod needs to serve a request. It is reasonable for readiness to notice that the pod has lost something it cannot work without. Be careful with shared dependencies though: if every pod fails readiness at once, the Service has no endpoints and clients get connection errors instead of your app's own error handling. For a shared dependency, serving a clear error is often better than vanishing.
Startup: slow boots. If the app takes a while to start (migrations, cache warm-up, JIT), use a startup probe with a generous failureThreshold instead of a long initialDelaySeconds on liveness. You get a long allowance at boot and a quick liveness check afterwards.
A starting point
startupProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 5 failureThreshold: 30 # up to 150s to start livenessProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 readinessProbe: httpGet: { path: /readyz, port: 8080 } periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 2 /livez returns 200 if the process can respond at all. /readyz checks what this pod needs before it takes traffic. Treat the numbers as a starting point and tune them against how your app really behaves under load.
Things that catch people out
- Timeouts under load. The default
timeoutSecondsis 1. A busy but healthy pod can miss that, fail liveness and be restarted exactly when you need it most. - Tight CPU limits. A throttled container answers probes slowly. If probes start failing during traffic peaks, look at throttling before you blame the app.
- Liveness on a shared thread pool. If the probe handler queues behind real requests, saturation looks like a hang. That may be what you want, or it may restart pods that were just busy. Decide on purpose.
- Same endpoint for both probes. It is the shortest path to the outage described above.
The short version
Readiness answers "should this pod get traffic?". Liveness answers "would a restart help?". If the honest answer to the second question is no, keep that check out of the liveness probe.