I run an automated system on a remote machine with nobody watching it. When I started, I described it to people as passive. Over two days it needed five separate interventions. None of them were exotic, and that's the point of this post. The gap between "automated" and "passive" is made of very ordinary things. Here's what actually broke, and the patterns I use now. The five failures, named Two copies of the same process running side by side for ten hours. Both were writing. Neither complained. The watchdog that was supposed to prevent exactly this compared strings after formatting instead of before, so the duplicate didn't look like a duplicate to it. A lost login that nobody noticed. The service kept running, kept polling, kept reporting healthy. It just wasn't authenticated any more. A component quietly detached from its host. Still installed, still configured, no longer attached to the thing it was supposed to be attached to. Nothing logged it. A dead event that never fired. A file was written for a consumer that had stopped reading it months earlier. Correct, current, and read by nobody. A config that silently reverted to defaults after a restart. The system came back up with different behaviour than it had before, and the only sign was that the numbers looked slightly different. I'd expected the dramatic failures: crashes, disk full, network down. All five were quiet ones. Not one produced an error message a human would see. Pattern 1: restart is a state change, not a no-op Two of those five happened only because something restarted. We tend to think of a restart as returning to a known good state. In practice it's a transition with its own failure modes. Parameters reset to defaults. Identifiers regenerate. Connections re-establish in a different order than they did the first time. I hit a third version of this elsewhere: a startup sequence read a value from a remote service before the connection to that service was authorized. The read returned a clean, plausible zero. The system then used that zero as a measurement and configured itself three hours away from correct. The journal shows it by the second: component loaded at :29, value read at :36, connection authorized at :39. The measurement happened three seconds before the thing it measured existed. A measurement taken before its input exists isn't a measurement. It's a default value wearing a measurement's clothes, and if your priority rules trust measurements over configuration, it will win every time. Anything that must survive a restart has to be designed to survive it, and anything read at startup needs to answer "is the thing I'm reading from actually available yet". Pattern 2: scheduled jobs fail in complete silence I moved a directory. The wrapper script coped fine, because it resolved its own location at runtime. The registered scheduled tasks did not: they held absolute paths recorded at registration time. They failed. Nothing on the machine said so. This is worth sitting with, because it's the quietest failure mode a system has. A crashed process leaves a trace. A failing request returns a status. A scheduled job that stops running produces nothing at all — and "nothing" is exactly what a working system that had nothing to do also produces. What I do now: every scheduled job writes a heartbeat on every run, including runs where it decided there was nothing to do absence of heartbeat for longer than the interval is itself an alert anything path-dependent resolves its own location rather than trusting a value stored elsewhere The general rule: if the only evidence of health is the absence of errors, you have no evidence of health. Pattern 3: an error is not "nothing new" My message loop checked whether a response contained items. If it didn't, the loop went back to sleep. Token revoked returns an error, not items. A second instance stealing the connection returns an error. Rate limiting returns an error. All three looked identical to "nothing new happened", and the process stayed alive, the watchdog stayed green, and the system was completely deaf. Now: status gets checked before contents, an unexpected status is loud, and no network call goes out without a client-side timeout. "Waiting forever" is another way to look alive while doing nothing. Pattern 4: confirm delivery before you mark it done My notification pipeline deleted each event from disk before confirming it had been sent. Send fails, event is already gone. No retry, no record, nothing to recover. I'd built at-most-once delivery by accident, in the one place where at-least-once was the entire point. A duplicate notification is annoying. A missing one is invisible, and invisible is worse. The order is the whole fix: confirm, then mark done. Never the reverse. Pattern 5: updates on a machine you can't babysit This is the part that took longest to get right, so I'll describe the shape I landed on. Pull, not push. The machine checks a manifest on a schedule and decides for itself. Nothing is pushed at it from outside. Verify before applying. The manifest carries a hash of every artifact. Hash mismatch means refuse and alert, not "try anyway". Refuse during work. If the system is mid-operation, the update waits for the next window. An update that interrupts work is worse than an update that's a day late. Back up, swap, then self-verify. After applying, the system proves it's running the new version by reading it back from the running process, not from the file on disk. If verification fails, it rolls back automatically and alerts. A canary ring of one. New versions go to a single machine first, and that machine is always mine. Whoever writes the update should be the first person it breaks. The honest summary Automation moves work, it doesn't remove it. The work changes shape: instead of doing the thing, you maintain the thing that does the thing, and you build the instruments that tell you whether it's still doing it. That second part is most of the cost, and it's the part nobody estimates. The question I ask before automating anything now isn't "can this run without me". It's "which failure am I prepared to discover late" — because that's the one I'll get.