On 28 August 2026 I couldn't log into my AI agent's dashboard. While I was looking into it, the Proxmox node it runs on patched itself and rebooted. Which turned out to be handy, because it showed me the actual problem, and it wasn't the dashboard.
What happened
At 09:05:37 the node ran apt-get dist-upgrade. That's my deliberate patching path, not an accident. It took Proxmox VE from 9.2.2 to 9.2.11 and pulled in a new kernel. At 09:09 the node rebooted. Nothing had scheduled that reboot in a timer, it came out of the patch run.
Then the agent's dashboard took about 15 minutes to serve a single request. For the whole of that, Caddy in front of it answered 502.
The disk
Here's where every guest on that node lives:
One 931.5G spinning drive, 5400rpm. The fast Optane is only 13G, so it just holds the Proxmox root.
After the reboot, five containers cold-booted at once on that one spindle. The heavy ones were:
- ClickHouse, inside my Langfuse container
- MariaDB plus a live Django
python manage.py migrate, inside a 3D-printer monitoring container - Nextcloud
That drove /proc/pressure/io to full avg60=81, load average 23, and 87% iowait.
I checked whether the storage itself was unwell. It wasn't. The thin pool was 7.8% used, no IO errors, no device-mapper or ext4 faults. It was just seek contention: lots of things wanting different bits of one slow disk at the same time. There's no tuning knob for that. A spinning disk can only be in one place.
The signature to recognise
This is the bit I actually wanted to write down:
-
systemctl is-activefor the agent's web service says active - nothing is listening on its port
- the process is in
Dstate (uninterruptible sleep), with its wchan showingwait_on_buffer,d_alloc_parallelordm_thin_find_block
"Active" plus "not listening" looks like a hung service. The obvious move is to restart it. Don't. D state with those wait channels means it's waiting on the disk. It hasn't crashed. Restarting it only queues the same IO again, behind everyone else's.
What actually shortened it
I stopped the Langfuse container. It's pure tracing, nothing depends on it being up, and ClickHouse is by far the heaviest thing starting on that box.
The dashboard came up 21 seconds later.
I started Langfuse again once full avg60 in /proc/pressure/io had dropped under 25. That took 260 seconds.
The one I deliberately didn't stop is the printer-monitoring container. It runs a Django database migration at boot, and killing a migration halfway through is a real way to break something. Pick the guest that's heavy and doesn't matter, not just the heaviest one.
Also worth knowing: every guest has onboot: 1 and they do all come back on their own. A couple of them just lag. So "stopped" a minute after boot isn't a fault. Look again before you start poking them.
Two things that broke because the boot was slow
A slow boot shows up the stuff that only worked because boots used to be quick.
1. The gateway watchdog was dead exactly when I needed it. I have a systemd timer that restarts the agent's gateway if it falls over. It fires 2 minutes after boot (OnBootSec=2min). On this boot, 2 minutes wasn't enough for root's user manager to have created /run/user/0, so the watchdog's systemctl --user call died with "Failed to connect to bus". So the thing that revives the gateway failed on the one boot where it mattered.
The fix: the watchdog script now waits up to 60 seconds for the user bus (and starts user@0.service itself if it has to), and exits 0 instead of failing. Plus a drop-in with After= and Wants=user@0.service.
2. systemd-journal-flush.service timed out inside the container after 90 seconds and left it degraded. That one was just collateral. It had worked on every boot before. systemctl restart systemd-journal-flush once the IO had calmed down cleared it.
A side lesson about writing that fix
I first wrote the new watchdog script with a heredoc through ssh into the node, then pct exec into the container, then bash -lc. Three layers of quoting. The file ended up with a literal \$(seq ...) in it, a syntax error, and the unit failed.
What works: write the file locally, base64 -w0 it, pipe that in and decode it on the other side. Then run bash -n on it before you start the unit.
What I'd do differently
- Don't put all your guests on one spinning disk. I know. The real fix is an SSD, and nothing in this post replaces that.
- Know which guest is your release valve before the storm. For me it's the tracing box. Heavy, and nobody misses it for five minutes.
- Never restart a
Dstate process to "unstick" it. Check/proc/<pid>/wchanfirst. - Test your watchdogs on a slow boot. A watchdog that only works when everything else is fine isn't much of a watchdog.
- Patch reboots are boots too. Whatever happens on a cold start happens every time you patch the host.
There's a follow-up to this. A month later the same node rebooted again, and a slow plugin load during the storm took my Discord bot offline for a week without anything noticing. I've written that one up separately.
What's the slowest disk you're still running guests on? Mine's still that 5400rpm one, for now.
🤖 Drafted with AI assistance from my own homelab notes, logs and repos, and fact-checked against them before publishing.

