Picture a small device you built, a sensor inside a machine at a customer's site a few hundred kilometres away. One morning the customer tells you it restarted during the night. What do you know about why?
On your desk, a debugger would answer that: a cable to the chip and a tool that stops the program and shows where it is. In the field there is no cable, no screen and nobody watching. The device is the only witness, and as we'll see in a moment, by default it forgets everything when it restarts.
ferrite exists for this situation: a device in the field has crashed, and there is no stack trace, no log and nothing that says what happened.
We'll follow one crash twice. First we ask what it leaves behind with nothing in place. Then we let ferrite, an open-source Rust library for this problem, save a small record of the crash, and we turn that record back into the line of code that crashed. Everything runs in QEMU, a program that emulates a computer, here a board with an Arm Cortex-M3 processor, so you can repeat it without hardware. The crash is planted on purpose: this post shows how a crash gets recorded, not how to find a real bug.
What happens when a microcontroller crashes?
A microcontroller is a small computer on one chip, with a processor, memory and peripherals, running one program, the firmware, with no operating system underneath. When the processor meets something it cannot do, such as an instruction it does not know or a memory access that fails, it stops what it was doing and calls a fault handler: a function the program provides for exactly this case. On Cortex-M chips the catch-all one is called HardFault.
Here is the function that will crash for us:
/// The bug we are going to find. `udf` is an instruction the processor /// refuses to execute, so this function crashes every time. #[inline(never)] fn handle_message() { unsafe { core::arch::asm!("udf #0") }; } udf means "undefined": an instruction reserved to be invalid, so running it always faults. #[inline(never)] keeps handle_message a function of its own, so the crash has a name we can find later.
What does the fault handler do if we don't write one? cortex-m-rt, the startup crate this program is built on, provides a default that loops forever (its source). The device freezes. Many products have a watchdog, a hardware timer that resets the chip unless the program checks in regularly, so a frozen device restarts a moment later. I ran this program without ferrite in QEMU, and it printed its first line and then nothing, until I stopped QEMU.
So after the restart, what is left? Before main runs, the startup code gives every global variable its initial value: it zeroes some memory and copies initial values into the rest. Whatever the crashed run knew is overwritten, and the new run starts exactly like the first one. That is why all the customer can tell you is "it restarted".
What would we want the device to tell us?
Suppose the device could leave us a note. What should it say? Three things get us from "it restarted" to a line of code:
- That it crashed, and the processor's reason. Cortex-M chips keep this in a fault status register, a few bits that tell an undefined instruction apart from, say, a bad memory access.
- Where it was: the program counter, the address of the instruction that was running.
- Which build it was running, because an address only means something in the exact program that produced it.
The note also has to travel from the device to someone who can read it. That part comes in later posts. Here the device prints its note.
A note that survives the reset
ferrite brings its own HardFault handler. It copies the registers that matter into a small region of RAM and then resets the chip itself. What makes this work is where it writes. That region is a small block, at most 256 bytes (192 in this build), in a part of RAM the build sets aside under the name .uninit.ferrite. It lies outside the memory the startup code initialises, so the next run's startup leaves it alone. A reset restarts the program, but it does not erase RAM, so the note is still there afterwards. Post 2 looks at this region more closely, including why it does not survive a power cut.
Here is main. Look for two things: init, which tells ferrite about the device, and last_fault, which returns the note if a previous run left one.
#[entry] fn main() -> ! { ferrite_sdk::init(SdkConfig { device_id: "qemu-demo-01", firmware_version: "0.1.0", build_id: 0, ticks_fn: || 0, ram_regions: &[RamRegion { start: 0x2000_0000, end: 0x2000_FC00 }], }); if let Some(fault) = last_fault() { hprintln!("boot: the previous run crashed"); hprintln!(" pc = {:#010x}", fault.frame.pc); hprintln!(" cfsr = {:#010x}", fault.cfsr); debug::exit(debug::EXIT_SUCCESS); } hprintln!("boot: no crash recorded, handling a message"); handle_message(); hprintln!("message handled"); debug::exit(debug::EXIT_SUCCESS); loop {} } hprintln! prints to your terminal through a debug channel QEMU provides, and debug::exit ends QEMU. ram_regions tells the fault handler which addresses are safe to read while it collects its note. pc is the program counter, and cfsr is the fault status register.
Before we run it, what should it print? On the first run there is no note, so we should see "no crash recorded", then the crash in handle_message. The handler saves its note and resets, and the program starts again from the top. This time last_fault finds the note, so we should see a second "boot" line from the same QEMU session, and never "message handled".
Running it in QEMU
The project is the program above plus a Cargo.toml and a few build files for this QEMU board, in the ferrite-sdk repository as examples/qemu-fault at commit b51bfed. With QEMU and Rust's thumbv7m-none-eabi target installed, run this in that directory:
cargo build --release qemu-system-arm -cpu cortex-m3 -machine lm3s6965evb -nographic \ -semihosting-config enable=on,target=native \ -kernel target/thumbv7m-none-eabi/release/qemu-fault I ran it from a fresh checkout of commit b51bfed, with rustc 1.97.0 and QEMU 7.2.22 (Debian 12's package), and it printed:
Timer with period zero, disabling boot: no crash recorded, handling a message boot: the previous run crashed pc = 0x00000158 cfsr = 0x00010000 The first line is QEMU's own message about a timer on the emulated board, not our program's. The rest matches the prediction. When the handler asked for a reset, QEMU restarted the emulated chip and loaded the program again, and like the real chip it left the rest of RAM as it was, so the note survived.
What about cfsr = 0x00010000? Only bit 16 is set, and in Arm's documentation for this register bit 16 is called UNDEFINSTR: the processor tried to run an undefined instruction. That is our udf. So the note already tells us why. Where is the next question.
From an address to a line of code
0x00000158 is only a number. To turn it into a line, we need the file the compiler produced, the ELF file. Besides the program, it can carry debug information: a table from addresses to source lines. Our Cargo.toml asks for it in release builds with debug = 2. The tool addr2line from GNU binutils (in the arm-none-eabi build) looks an address up in that table:
arm-none-eabi-addr2line -f -C -s -e target/thumbv7m-none-eabi/release/qemu-fault 0x00000158 -f asks for the function name, -C turns Rust's encoded symbol names back into readable ones, and -s leaves out the directories. It prints:
qemu_fault::handle_message main.rs:39 Line 39 of main.rs is the udf line.
There is a condition hiding in that step. We looked the address up in the same file that ran. Change one line above handle_message, rebuild, and the code moves to different addresses: the same 0x00000158 would then point somewhere else, and addr2line would print that other line without complaint. That is why the build belongs in the note. ferrite sends the firmware_version from init with every report, so whoever reads a report knows which release's ELF to look it up in. A version string only identifies a build if it changes with every build, though: our example says "0.1.0" whatever we change, so after that one-line edit its note would still claim 0.1.0. The field meant for the exact build is build_id, which the SDK documents as the first 8 bytes of the ELF's build ID. We left it at 0 here; post 4 is about matching reports to builds.
What a real device does differently
Our program stops after printing so that QEMU exits. A device in the field would carry on with its work after the reset. When it next has a connection, ferrite sends the note, together with the device name and firmware version, to ferrite's server. The server stores the ELF files you upload and runs the addr2line step itself. Making sure each report is looked up in the ELF of the build that produced it is the subject of post 4. After a successful upload, the device clears the note, ready for the next crash.
Let's take stock. With a small block of RAM the startup code does not touch, the program counter and the fault status survived the reset, and the ELF of the same build turned the address into main.rs:39. That rests on one claim we haven't examined: that RAM keeps its contents through a reset. Why does it, how does the program tell a real note from leftover noise, and what happens when the power goes? That's the next post.

![Motion Planning and Navigation — Ch.04: Optimization Based Planning [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqyrjjn7tc2h8vbe5rduk.png)
![Motion Planning and Navigation — Ch.05: Local Planners [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0tuhfqde1o6zcyskw7f4.png)
![Motion Planning and Navigation — Ch.07: Multi Robot Navigation [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwj9k0uan25q7slmyqpoo.png)
![Motion Planning and Navigation — Ch.10: Navigation2 Stack [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6asnejhpxvcrd68qhrfg.png)