A postmortem of the 2026-11-14 outage of our home server "D" — written from the machine's own journal, with a second review of the same record and the limits of the evidence kept attached.
TL;DR
On the afternoon of 2026-11-14, our main development server lost its network at 15:24 PST and stayed off it until a power cycle at 17:03 restored connectivity; that episode's trigger remains unisolated. That evening a second failure appeared: across eight recorded boot starts, a small service we had installed ourselves triggered a kernel fault about five seconds into boot, and the wired network never came up. The machine was not dead — all eight starts reached the OS. Disabling the service removed the observed boot-time EEE call, and a later boot had working Ethernet — though the record does not isolate that single change as the sole fix for both phases of the outage.
Field warning: if you run an RTL8125 on the r8169 driver, check your journal
This is not only our story — the trigger combination is worth checking on any similar machine, so here is the part a stranger can use.
A Debian bug report (#1059149) shows a closely similar call path on a different kernel and machine. It was closed as old without a demonstrated resolution; it supports an analogy, not proof of an identical defect here. Because it was closed without a demonstrated fix, the combination is worth checking yourself: on a Realtek RTL8125 driven by the in-kernel r8169 driver, a boot-time ethtool --set-eee <iface> eee off — exactly what a well-meaning "disable Energy-Efficient Ethernet" oneshot service does — triggered this crash on all eight of our recorded failed boots. The same unit also ran without crashing on other boots, which is part of why it survived as long as it did.
You can check your own machine with one grep of a boot's journal:
journalctl -b 0 | grep -E "phy_read_mmd|rtl8169_set_eee|ethnl_set_eee" What you are looking for is a kernel BUG: NULL pointer dereference faulting in phy_read_mmd, with rtl8169_set_eee in the call trace, called from ethnl_set_eee — the Ethernet settings path — about five seconds into boot, followed by the network service timing out so no interface ever comes up. If your journal shows that trace after a boot-time ethtool --set-eee, you are looking at the same failure signature we hit, and the first suspect is not your hardware: it is the boot-time EEE call. What fixed our evening boots, as far as the record shows, was disabling the service that made that call — a later boot had working Ethernet and no repeat of the Oops.
What happened
Server D is a desktop-class machine (Lenovo Legion T5) that runs our telephony stack, an Android emulator test stand, and the agent sessions that build our product. On 2026-11-14 at 15:24 PST — mid-afternoon — it fell off the network: the Ethernet interface lost its IPv4 address and routes in one second, while the OS kept running. The 15:24 network loss persisted until a 17:03 PST power cycle restored connectivity. Its trigger remains unisolated; a LAN-side fault is not excluded. Earlier, separate route-deletion episodes on this machine had self-recovered; this one did not.
A second failure appeared that evening. The 17:03 PST power cycle restored the daytime connection, but a later sequence of failed network startups began at 18:10 PST. From the outside, the machine looked powered off or hung. We worked through power cycles and investigated hardware possibilities, and replaced the PSU the next day — because a machine that doesn't answer looks like a hardware problem.
It wasn't — at least, the evening failure wasn't. When we finally read the system journal boot by boot, the evening looked completely different:
- The machine started eight times between 18:10 and 19:36 PST; the last journal record is at 19:42 (we had been power-cycling it, trying to revive it).
- All eight starts reached the OS and logged the same kernel Oops. About five seconds into each boot, the kernel faulted in
phy_read_mmdwhilertl8169_set_eeewas on the call path, called fromethnl_set_eee. After the Oops,systemd-networkdtimed out and the wired network did not come up in the failed boots. The exact mechanism of the timeout remains unproven. The journal does not establish that every boot reached a login screen, and the Oops did not itself power the machine off. - The caller was
paragon-eee-off.service— a small oneshot unit we wrote ourselves, which ranethtool --set-eee enp5s0 eee offat boot to disable Energy-Efficient Ethernet on the Realtek RTL8125 NIC. It had originally been introduced as a stability measure.
A unit we added as a stability measure triggered this Oops on all eight recorded failed evening boots.
How we got back in is part of the story. The next day the owner powered the machine up with a monitor attached and gave it internet through his phone, tethered over USB. The tether is a separate network interface on a different driver, and the service only ever touched the wired port — so the machine came onto the network through the phone while the wired path was still broken. From inside, over that link, we read the journal boot by boot, identified the caller, and disabled the unit (2026-11-15, about 18:04 PST). Then came the controlled check: tether removed, the Ethernet cable plugged back into the same port, on the same router, and a cold boot at 18:28 PST — the wired network came up clean, with no repeat of the Oops. The cable, the router and the NIC had never been the problem; the tether hadn't fixed the wired path, it had walked around it and given us the view from inside.
The method: three checks, three different limits
The journal, a second review of it, and a later boot observation have different evidentiary limits — so here is each, with its limit attached.
- The journal. Eight boot starts, the same crash signature, the same caller. The daytime 15:24 drop had a different shape — a route/address loss with the OS alive — and remains a separate episode whose trigger is unisolated; a LAN-side fault is not excluded.
- A second review. Another reviewer, working from the same journal, read it the same way — a second interpretation of one record, not an independent experiment. It also kept two honest qualifications beside the causal account: the exact mechanism behind the network service's timeout is unproven, and the Oops did not power the machine off — the boots ended by power-key shutdowns, not by the crash.
- A later boot. A later Owner-reported cold boot, at about 18:28 PST on November 15, came up with Ethernet and DNS. A read-only check found carrier and synchronized time; the reviewed boot journal had no repeat of the Oops. That later successful boot is consistent with removing the service trigger, but it was not a one-variable test: hardware and cable state also changed.
We disabled the unit on 2026-11-15 at about 18:04 PST, working from inside over the phone-tether link described above; an independent read of the machine minutes later found the unit disabled. The controlled cold boot that followed — tether removed, Ethernet restored — had working Ethernet and no repeat of this Oops; longer-term stability was not established by this check. One honest footnote: during the same weekend we also replaced the original PSU and the RAM. We have no evidence that the PSU was faulty or that the swap fixed the network failure. The unit-triggered Oops was associated with repeated loss of network startup; it did not power the host off.
The same check also caught a documentation debt: the incident card in our operations documentation still described this unit as a stability measure. The card has since been corrected, so the prohibition is now documented where the next operator will see it. The disable was real first; the paperwork caught up after.
Lessons we wrote down
- A boot-time "optimization" is production code. A small oneshot unit, live-checked when installed but never validated across subsequent reboots, took the wired network down on all eight recorded failed boots. It deserves the same review and rollback discipline as any deploy.
- "Machine is dead" and "machine has no network" are different incidents. From outside they look identical. The journal knew the truth the whole evening; we read it after the hardware checks instead of before. Read the journal first.
- Check the call trace against known bug reports before inventing a theory. A filed Debian report showed a closely similar call path on a different machine. Earlier comparison with that trace might have shortened the diagnosis.
- Change one variable, then test. The hardware changes we made along the way did not isolate the cause — too much changed at once. The later healthy boot supports the service hypothesis, but does not uniquely prove it.
- Write the prohibition where the next operator will see it. We disabled the unit — and then corrected the incident card that still described it as a stability measure, because a prohibition counts as documented only when the correction lands in the card itself. A lesson that lives only in a postmortem is a lesson half-kept.
The uncomfortable part is also the valuable part: the boot unit we added caused a repeatable failure in the eight recorded evening boots — a component we added ourselves, for a sensible-sounding reason, live-checked at install but never given a subsequent reboot test. That's exactly the class of failure a postmortem exists to catch — and exactly why we keep operations documentation of what we changed, why, and what it taught us, and why that documentation has to be corrected when it is wrong.
