Should a failed health check restart the service automatically, or page a human first? Let's argue about it

Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — upordown.online

Your health check fails at 03:10. Something is going to happen next. Either the platform restarts the service on its own (Kubernetes liveness probes, systemd Restart=on-failure, an autoscaling group replacing the instance, a supervisor process), or a person gets paged and decides what to do.

Most teams never make this choice deliberately. They inherit whatever the platform's defaults are, and find out what they chose during an incident. So let's make the argument out loud.

The case for restarting automatically

Most failures are transient, and a restart fixes them. A memory leak that takes six days to exhaust the heap, a connection pool stuck after a database failover, a deadlock that happens once a week. A restart clears all of these in seconds. A human would do the same thing, just 15 minutes later and grumpier.

Humans are slow at 3 a.m. Time to acknowledge a page, open a laptop, connect to the VPN and look at a dashboard is rarely under ten minutes. If a restart takes 20 seconds, the automated path has 9.5 minutes of advantage on every incident.

It scales. With 40 services and 200 instances, paging a human for every unhealthy instance means paging constantly. Self-healing at the instance level is the only way to run a fleet without a 24-hour operations room.

You can still alert on the pattern. Restart automatically, and page when restarts happen more than N times per hour. The human gets involved for the recurring problem, not for every blip.

The case for paging first

A restart destroys evidence. The heap state, the stuck threads, the open connections, the exact request that hung the worker: all gone when the process restarts. If you never look at a failing instance before it's recycled, you'll never find out why it failed. You'll restart it every Tuesday for two years.

Restarts can make things worse. If the health check fails because a dependency is down, restarting every instance at once means they all come back cold, all reconnect to the struggling dependency at the same moment, and all fail again. That's a restart loop, and it turns a degraded service into a dead one. Health checks that include dependencies are particularly prone to this.

Auto-restart hides slow degradation. A service that's restarted 30 times a day by the platform looks green on every dashboard that shows current state. Nobody notices until the restarts happen faster than the startup time and the whole thing falls over.

Some failures need judgement. Restarting a stateless web server is safe. Restarting a database primary, a job runner in the middle of a migration, or a process holding a lock can cause data problems that are far worse than the original failure.

Need a second opinion before the restart fires? upordown.online checks the same endpoint from outside your network, so you can tell a local flake from a real outage.

Where I've landed

I don't think there's one answer for every service, but there's a reasonable default:

Auto-restart stateless things, with a budget. Web servers, API workers and other replaceable processes restart automatically, but with backoff (Kubernetes' CrashLoopBackOff does this; systemd has StartLimitBurst and RestartSec) so a restart loop doesn't hammer everything else.
Never restart everything at once. Whatever does the restarting should restart one instance at a time, or a small percentage. If all instances fail health checks simultaneously, that's almost never a per-instance problem, and restarting them all is the worst possible response.
Capture evidence before the restart. A pre-stop hook that dumps a thread dump, heap summary or the last 500 log lines to somewhere durable costs a few seconds and turns "it was restarted, no idea why" into a real investigation.
Page on rates, not events. One restart is a log line. Five restarts of the same service in an hour is a page. Restarts across many instances at once is a page immediately.
Page first for stateful things. Databases, queues and anything holding a lock or partway through a long operation get a human before an automated action.
Watch from outside too. Self-healing can make internal dashboards look fine while users see errors during each restart. An external check like upordown.online shows whether the restarts are actually invisible to users or just invisible to you.

The part I'm least sure about is the threshold. Five restarts an hour? Three? Is one restart of a service that "never restarts" already worth a page?

The edge cases that start arguments

Liveness probes that check dependencies. If the database is slow and every pod's liveness probe fails, Kubernetes restarts every pod. Should liveness ever depend on anything outside the process? (I'd say no, but I've heard good arguments for "only the local disk".)
Restarting during a deploy. If the new version fails health checks, should the platform restart it, roll it back, or stop and wait for someone?
Small teams. For a team of two with one service, is paging on every failure fine because it happens twice a year?

What the default looks like in config

For a systemd service, "restart with a budget" is a few lines:

[Unit] StartLimitIntervalSec=600 StartLimitBurst=5 [Service] Restart=on-failure RestartSec=10

Five restarts in ten minutes, then systemd gives up and the unit stays failed, which is exactly the moment a human should hear about it. In Kubernetes, keep the liveness probe about the process only (an endpoint that returns 200 if the event loop is responsive), put dependency checks in the readiness probe, and alert on kube_pod_container_status_restarts_total increasing faster than your threshold.