Stop Guessing Why Your Crash Left No Core: Practical systemd-coredump on Linux

At 03:12 a service segfaults. systemctl status is unambiguous:

Main process exited, code=dumped, status=11/SEGV 

At 09:00 you run coredumpctl list and get nothing. The natural conclusion is that the core was never written, or that the disk was full, or that someone deleted it. Usually none of those is true. The core passed some gates and failed a later one, and the tool that lists cores is not the tool that stores them.

This is a practical map of the path a core takes on a systemd host: the six places it can disappear, the knobs that control each one, and the commands that prove what your machine is actually doing.

The path a core takes

A core dump is not one event. It is a chain:

Each gate has a different symptom. A core that fails at gate 1 leaves no journal entry at all. A core that fails at gate 6 leaves a journal entry pointing at a file that no longer exists — which is exactly the confusing missing status in coredumpctl list.

Gate 1: Is a handler even installed?

Start with the kernel's instruction for what to do when a process dumps core:

cat /proc/sys/kernel/core_pattern 

On a host with the standard systemd handler, the output starts with a pipe and looks like this:

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e 

The exact path varies by distribution (/lib/systemd/... on some Debian releases, for example). What matters is the leading |: the kernel will pipe the core to that program instead of writing ./core in the crashing process's working directory.

If you see core, you are in legacy mode and cores land in the current directory of the crashing process — often /, often unwritable. If you see |/bin/false, coredump handling is disabled. systemd writes that value during very early boot and late shutdown, when storage and the handler are not available; if it stays there for the whole uptime, the handler package is probably missing.

Check that the handler units exist and are active:

systemctl status systemd-coredump.socket --no-pager coredumpctl --version 

On many distributions systemd-coredump is part of the main systemd package; on Debian-derived systems it is often split into a separate package. Minimal images frequently omit it. That is the first thing to check when coredumpctl itself is "command not found".

Gate 2: RLIMIT_CORE is checked twice, by two different layers

This is the part almost every guide gets half right.

The kernel does not enforce RLIMIT_CORE when core dumps are piped to a program. core(5) says it plainly: "The RLIMIT_CORE limit is not enforced for core dumps that are piped to a program via this mechanism." So a pipe handler receives the core even when the crashing process has a zero core limit.

But systemd-coredump checks the limit itself and respects it. The systemd coredump documentation states that individual services, processes, or users can opt out of collection by setting RLIMIT_CORE to 0, and that the handler will respect limits set by processes.

Both facts matter:

• ulimit -c 0 in a shell stops cores for processes started from that shell, because the handler honors the inherited limit.
• It does not affect anything else on the system.
• PID 1 raises its own RLIMIT_CORE to infinity at boot, so services inherit an unlimited core limit by default. "No core by default" is not the systemd behaviour.

Check what a unit actually gets:

systemctl show -p LimitCORE --value myapp.service 

and what the running process inherited:

pid=$(systemctl show -p MainPID --value myapp.service) grep 'Max core file size' /proc/$pid/limits 

The property usually reads infinity, and /proc/<pid>/limits renders the same thing as unlimited. If you want a specific service to never write a core, set it deliberately:

# /etc/systemd/system/myapp.service.d/20-no-core.conf [Service] LimitCORE=0 

sudo systemctl daemon-reload sudo systemctl restart myapp.service 

Do not reach for a global ulimit -c 0 in /etc/profile: it only covers login shells, and it is the wrong layer for a daemon.

Gate 3: Is the process dumpable?

Even with a handler and an unlimited core limit, the kernel refuses to dump some processes:

• A set-user-ID or set-group-ID program owned by a different user than the real UID.
• A program with file capabilities.
• An executable that is not readable by the crashing process.

This is a security feature: a core is a memory image, and a setuid binary's memory image may contain credentials. The exception is controlled by /proc/sys/fs/suid_dumpable. Its default is 0 ("disabled"). Setting it to 1 ("debug") allows dumps but is unsafe on a multi-user host. The value 2 ("suidsafe") allows dumps only when core_pattern is an absolute path or a pipe — which is the case on a systemd host.

Check the process rather than guessing:

cat /proc/sys/fs/suid_dumpable getcap /usr/local/bin/myapp ls -l /usr/local/bin/myapp 

A service that drops privileges is usually fine. A wrapper that is setuid root is not, and no amount of coredump.conf tuning will change that.

Gate 4: Which pages are in the core?

The kernel does not dump the entire address space. /proc/<pid>/coredump_filter is a bitmask of mapping types:

The default mask is displayed in hex as 33. Bits 0, 1, 4, and 5 are set. That includes the stack and the heap, both of which are anonymous private mappings — which is why a default core is useful for debugging and also why it can contain secrets.

Two ways to change it:

Per process, before it crashes. The value is inherited across fork() and preserved across execve():

echo 0x11 > /proc/self/coredump_filter exec ./myapp 

0x11 keeps anonymous private memory and ELF headers, and drops anonymous shared and huge pages.

Per systemd unit, since systemd 246. CoredumpFilter= accepts whitespace-separated names or numbers (base 16), and repeated assignments are ORed:

# /etc/systemd/system/myapp.service.d/30-core-filter.conf [Service] CoredumpFilter=private-anonymous elf-headers 

Valid names are private-anonymous, shared-anonymous, private-file-backed, shared-file-backed, elf-headers, private-huge, shared-huge, private-dax, shared-dax, plus the special values all and default. If the option is unset or empty, the inherited value is left alone.

Verify the effective mask on the running process:

systemctl show -p CoredumpFilter myapp.service cat /proc/$(systemctl show -p MainPID --value myapp.service)/coredump_filter 

Now the honest caveat. People often try CoredumpFilter=private-anonymous to remove passwords from the heap. That also removes the stack, because both are anonymous private mappings. You cannot separate heap from stack with this knob. The filter reduces the size and the set of mappings; it does not sanitize memory. If a buffer must never reach disk, mark it with madvise(MADV_DONTDUMP) and clear it after use. Treat every core as sensitive data, restrict access to /var/lib/systemd/coredump, and never attach one to a public ticket without reviewing it.

Gate 5: Storage and size policy

coredump.conf decides what happens after the core reaches the handler. The defaults are reasonable, but the interactions surprise people:

Two details that are easy to miss:

• With Storage=journal, the core is still written to disk first so a stack trace can be generated. Storage= only controls whether the file is left there afterwards.
• Storage=none plus ProcessSizeMax=0 disables all coredump handling except a log entry. That is the documented way to keep the crash record but never store memory.

A practical policy for a server that needs cores for a week but must not fill the disk:

# /etc/systemd/coredump.conf.d/60-local.conf [Coredump] Storage=external Compress=yes ProcessSizeMax=2G ExternalSizeMax=4G MaxUse=1G KeepFree=5G 

Validate the merged configuration, not just the file you wrote:

systemd-analyze cat-config systemd/coredump.conf 

On classic systemd, each crash is handled by a fresh systemd-coredump@.service instance, so the next crash picks up the new configuration. Newer systemd (262 and later, with kernel 6.19 or later) can use a long-running systemd-coredumpd.service; in that mode the configuration is read once at start, and the documented way to apply changes is:

sudo systemctl reload systemd-coredumpd.service 

Gate 6: Concurrency, retention, and the two clocks

/proc/sys/kernel/core_pipe_limit limits how many crashing processes may be piped to a handler in parallel. The default 0 is special: unlimited parallelism, but the kernel does not wait for the handler, so the handler is not guaranteed access to /proc/<pid> of the crashing process. A non-zero value makes the kernel wait, which is safer for metadata collection, but crashing processes above the limit are logged and their cores are skipped. If you set it, keep the handler fast.

Retention is where the "missing core" mystery usually lives. Two independent mechanisms delete things:

1. Time-based cleanup by systemd-tmpfiles. Check the shipped rule on your host:

 grep coredump /usr/lib/tmpfiles.d/systemd.conf 

1. Size-based cleanup by MaxUse= and KeepFree=, which remove old cores as new ones arrive.

The journal entry and the core file have separate lifecycles. Journal rotation can remove the metadata while the file remains; tmpfiles or MaxUse= can remove the file while the journal entry remains. coredumpctl queries the journal, so an empty list does not prove the directory is empty:

sudo ls -lt /var/lib/systemd/coredump | head coredumpctl list --since "2 days ago" 

That is also why coredumpctl list shows missing in the COREFILE column: the metadata survived, the file did not.

Hands-on: make a core, find it, read it

Use a disposable process. Sending SIGSEGV to a shell is the example from the coredumpctl manual page:

bash -c 'kill -SEGV $$' 

Then query the journal:

coredumpctl list --since "10 min ago" coredumpctl info -1 

info gives PID, UID/GID, signal, command line, cgroup, unit, the storage path, size on disk, and a stack trace when the handler could generate one. If the list is empty, walk gates 1 through 5 before assuming anything was deleted.

Open the core in a debugger:

coredumpctl debug -1 

By default this invokes gdb. To use another debugger, pass --debugger= or set SYSTEMD_DEBUGGER. For a non-interactive register dump, the manual page's form is:

coredumpctl debug -1 --debugger-arguments="-batch -ex 'info all-registers'" 

Extract the core and keep it independent of the journal:

coredumpctl dump -1 --output /tmp/bash.core gdb -q "$(command -v bash)" /tmp/bash.core 

Useful frames need debug symbols. Install the debug package for the crashing binary (-dbgsym on Debian/Ubuntu, -debuginfo on Fedora), or point gdb at a debuginfod server if your distribution runs one.

If coredumpctl is unavailable or you are scripting, the same record is a normal journal entry with a fixed message ID:

journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -o verbose -n 1 

That prints COREDUMP_PID=, COREDUMP_SIGNAL=, COREDUMP_RLIMIT=, COREDUMP_FILENAME=, and the rest of the metadata directly.

Cleanup and rollback

Remove old files according to the tmpfiles rules, then confirm:

sudo systemd-tmpfiles --clean sudo ls -lh /var/lib/systemd/coredump 

To delete one specific core, match the exact filename from coredumpctl info rather than guessing a glob:

sudo rm -f /var/lib/systemd/coredump/core.bash.1000.<timestamp>.zst 

To roll back a storage change, remove the drop-in and confirm the merged config:

sudo rm /etc/systemd/coredump.conf.d/60-local.conf systemd-analyze cat-config systemd/coredump.conf 

For a unit-level filter or limit, remove the drop-in, then reload and restart:

sudo rm /etc/systemd/system/myapp.service.d/30-core-filter.conf sudo systemctl daemon-reload sudo systemctl restart myapp.service 

What this does not cover

systemd-coredump handles userspace crashes: a process receives a fatal signal and the kernel produces a core. A kernel panic is a different pipeline entirely — kdump/kexec or pstore/ramoops, depending on firmware and configuration. Tuning coredump.conf will not capture a panic, and a panic will not appear in coredumpctl. Keep the two runbooks separate.

The short version

When a crash leaves no core, do not start by deleting files or clearing the journal. Walk the chain:

1. cat /proc/sys/kernel/core_pattern — is a handler installed?
2. systemctl show -p LimitCORE and /proc/<pid>/limits — is the limit zero?
3. Check setuid, file capabilities, and executable permissions.
4. Read /proc/<pid>/coredump_filter and any CoredumpFilter= drop-in.
5. Read the merged coredump.conf — storage, size caps, and cleanup limits.
6. Check core_pipe_limit, tmpfiles age, and journal rotation.

A core is a memory image with a short, fragile life. The system tells you at every gate what it did; the mistake is assuming the list command is the whole story.

References

• core(5) — core file naming, pipe handlers, RLIMIT_CORE, coredump_filter, core_pipe_limit
• coredumpctl(1) — list, info, dump, debug, matching and options
• systemd-coredump(8) — handler units, journal fields, storage behaviour
• coredump.conf(5) — Storage=, size limits, MaxUse=, KeepFree=
• systemd.exec(5) — CoredumpFilter= and LimitCORE=
• systemd Coredump Handling — PID 1 limit setup and handler semantics
• journalctl(1) — message-ID filtering and field output