Our stock feed service runs under systemd on three integration servers, and Ansible owns its unit file. The supplier's endpoint lives in the unit itself, as an Environment line, because that is where it was put in 2021 and nobody moved it. In September the supplier gave us a new endpoint and a date, three weeks away, when the old one would stop answering.
We changed the variable, the role templated the new unit file, the template task reported changed, and its handler restarted the service. The play was green on all three servers. Three weeks later, at nine in the morning, the feed failed on every server with connection refused from the old address.
The on call engineer ran systemctl cat on the service and saw the new endpoint, which made the next half hour confusing. systemctl cat reads the files on disk. systemctl show -p Environment reports what the running manager has loaded, and it showed the old one.
systemd reads unit files when it starts and when it is told to reload them, and at no other time. Writing a new file changes nothing until someone runs systemctl daemon-reload. A restart before that restarts the service from the definition already in memory, which is exactly what our handler did. systemctl does warn: the unit file changed on disk, run daemon-reload. The warning goes to standard error, the restart exits with success, and Ansible's service module reported the restart as done. systemctl status had been printing the same warning for three weeks to anyone who looked, and nobody had.
That morning we ran daemon-reload and restarted, and the feed came back in a minute. Then we searched the fleet for units in the same state, using the NeedDaemonReload property that systemctl show exposes. Eleven units on six servers had changed files that had never been loaded. Two of them had been waiting for a reboot to apply changes made in the spring.
Every role that templates a unit file now notifies a handler that uses the systemd module with daemon_reload set, before any restart. A final task in each play fails if any unit on the host reports NeedDaemonReload as yes. The endpoint has moved out of the unit into an environment file the service reads at start.
Ansible checked the file. Nothing had checked what systemd was actually running.
– Sergey Shinder