The dashboard stopped updating. The ESP32 kept measuring temperatures. The Raspberry Pi was still running.
So where did the readings go?
I put together a small MQTT debugging scenario using four log files:
- ESP32 firmware log
- Raspberry Pi system journal
- Mosquitto broker log
- Dashboard application log
The interesting part wasn't finding the disconnect message. It was reconstructing what happened before, during, and after the outage.
By combining the logs into a single timeline, I could trace the broker shutdown, compare the clients' reconnection behavior, and identify four measurements that were taken but never published.
I documented the investigation step by step, including screenshots and reusable highlighting tokens:
I wish I’d had a tool like Emlyzer sooner. Here’s how I debug an MQTT outage with it.
The temperature on my dashboard is stuck at 22.40 °C. The ESP32 is still taking readings. The Raspberry Pi is still on the network. And the dashboard has one clue: MQTT disconnected.
That gives me a place to start. I want to know what broke first, what happened to the readings in between, and when the dashboard finally caught up.
I start by following a single reading
I open four logs in Emlyzer: the ESP32 firmware log, the Pi’s system journal, the Mosquitto log, and the dashboard log. To see how a temperature reading gets from the ESP32 to the dashboard, I select seq=1724. The sequence number ties the firmware and dashboard entries together, and the Mosquitto entries in between show the broker receiving and forwarding a message on the temperature topic. There’s also a lab-health entry from journal.log among the surrounding lines.
The dashboard warning gives me my first highlight token, disconnected. It also picks out MQTT_EVENT_DISCONNECTED on the ESP32, 45 milliseconds after the dashboard warning.
I look just before those two entries in the Pi’s journal. Mosquitto was already stopping. I add tokens for Stopping mosquitto.service and Started mosquitto.service to mark the interruption. A different color for New client connected makes it easy to spot when each client comes back.
The broker comes back before the readings do
In dashboard.log, I find the last reading before the gap, seq=1841. At 10:15:02.010, systemd begins shutting down Mosquitto, and the dashboard and ESP32 lose their connections. Once the broker is back up, the dashboard subscribes again and the sensor reconnects. The first reading to reach the dashboard after the gap is seq=1846.
The Pi journal puts the client errors in order. The service was already stopping when the dashboard lost its connection, and both retry failures occurred before the service came back. The two clients then recovered on different schedules. The dashboard subscribed again almost two seconds before the sensor reconnected.
The dashboard goes ten seconds between readings. Four measurements are missing. Where did they go? The firmware log shows that the ESP32 took readings 1842 through 1845 but skipped publishing them while its MQTT connection was unavailable.
Then comes 1846. I add seq=1846 as a token and follow it from the firmware queue at 10:15:10.020 to the dashboard 35 milliseconds later. Mosquitto logs a publish arriving and being forwarded on the temperature topic in between. The sequence number ties the two ends together, and the broker’s topic and timestamps show the handoff.
The service journal shows the stop and subsequent start. In this demo the restart is intentional. In a field capture I’d look for the trigger in operator, deployment, or audit records before attributing the restart to a person or a job.
The next outage starts with a saved token set
Before closing the logs, I save the tokens I built during the investigation as MQTT Connection Issue.etk. When the dashboard freezes again, I can open the new logs and load that file. The searches and colors are already there. I can go straight to the service events, see which client reconnected first, and check whether readings resumed. A colleague investigating a similar issue can start with the same file.
Some clues travel better than others. The service and connection tokens apply to another capture with the same log wording. The token seq=1846 belongs to this run. Next time I’d replace it with the sequence number I’m following. If a new failure turns up a useful search, such as reconnect failed, I can save it in the set for the next investigation.
Want to investigate it yourself?
I've made the example logs and token files available through the Emlyzer GitHub repository.
I'm curious how others approach this: when debugging communication issues across embedded devices, brokers, and applications, do you usually inspect each log separately, or do you combine them into a common timeline?
Originally published on emlyzer.com.
