Our rates API runs on twelve virtual machines as a systemd service. The deploy script is older than most of our team. For each host in turn it copies the new jar, runs systemctl restart over ssh and moves on to the next host if the command succeeded. A failure stops the loop, which is meant to leave eleven healthy servers when the first one goes wrong.
On a Tuesday in October a release added a required setting for a new currency feed. The setting had been added to staging's configuration and not to production's. The deploy took three minutes, every restart succeeded, the script printed done, and every server was down. Quotes failed for sixteen minutes until we redeployed the previous version.
The unit was Type=simple, the default when a unit has an ExecStart line and no Type or BusName. For a simple service, systemd considers the unit started as soon as it has forked the process, so systemctl restart returns success before the program has read a line of its configuration. The systemd documentation warns that with simple, systemctl start will succeed even if the binary cannot be run at all. Type=exec waits only until the binary has been executed. Neither waits for the application to be ready, and neither notices it exiting eight seconds later.
Our application threw on the missing setting during startup and exited with status 1. Restart=on-failure brought it back five seconds later, and it failed again. By then the script had already moved on to the next host. The safety check in the loop was asking a question that systemd answers before the answer is known.
The script now waits after each restart for the service's readiness endpoint to return 200, for up to ninety seconds, and stops the whole deploy if it does not. We looked at Type=notify, where the service tells systemd when it is ready, but a readiness check over HTTP tests the path our callers use anyway.
The configuration for production is now checked against the application's list of required settings in the pipeline, before any jar is copied.
systemctl restart tells you that systemd did what you asked. Whether the application did what you hoped is another question, and it has to be asked over the application's own interface.
– Sergey Shinder
