"The disk is full, I already deleted the big log file – but not a single byte was freed." A classic. The file is gone from the directory, but its content still takes up space.

Why deleted files still use space

On Linux, rm only removes the directory entry, the name. The file system releases the data only when no name points to the file and no process has it open any more.

If a service keeps a file open – typically a web server, a database or an application with its log file – the space stays allocated. Often the process even keeps writing to it. The file just can't be reached through a path any more.

That explains the disagreement between two tools: df asks the file system how much space is used and includes the file. du walks the directories and no longer finds it.

Finding deleted but still-open files

First confirm the gap:

df -h /var du -xsh /var 

If the numbers are far apart, lsof shows the suspects. The +L1 option lists open files with a link count below one, i.e. already deleted:

sudo lsof -nP +L1 

The interesting columns are COMMAND, PID, FD, SIZE/OFF and the path marked (deleted). To limit the output to one file system, append the mount point:

sudo lsof -nP +L1 /var 

If lsof isn't installed, /proc helps:

sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null 

Freeing the space

Reload or restart the service. Usually the cleanest way. Many services reopen their log files on reload, for example:

sudo systemctl reload nginx sudo systemctl restart servicename 

A reload only helps if the service reopens its logs while doing it. Otherwise a restart is needed – on production systems, with care and at a suitable time.

Truncate the file through /proc. If a restart isn't possible right now, the deleted file can be truncated to zero bytes through its file descriptor. PID and FD number come from the lsof output (for FD 4w, the number is 4):

sudo sh -c ': > /proc/1234/fd/4' 

This is a stopgap: the content is gone afterwards, so be sure it really is a log file you can do without.

Why it happens in the first place

  • Manually deleting a live log file with rm. Better: empty it instead of deleting it, for example with truncate -s 0 file.log.
  • Log rotation without reopening. logrotate renames the file or removes old rotations, but the service keeps writing to the old one. Fix: reload the service in the postrotate block, or use copytruncate for services that can't reopen their logs.
  • Hung or forgotten processes, such as an old worker that was never stopped.

Not every gap between df and du has this cause. Other suspects: files hidden under a mount point mounted on top of them later, and on ext4 the blocks reserved for root. And if df -h shows free space but nothing can be written, check df -i – the inodes may be exhausted.

How to check

  • Compared df -h and du -xsh for the same file system?
  • Ran lsof -nP +L1 and found processes with (deleted) files?
  • Reloaded or restarted the service, then checked df -h again?
  • Reviewed the logrotate configuration for the service (postrotate or copytruncate)?
  • Ruled out inodes with df -i and hidden mount points?
  • Set up monitoring for disk usage?

A full disk is often just the symptom. The cause is frequently a log nobody rotates – a good example of why the second look pays off.

Originally published at daniele-michelin.com. More guides, free checks (SPF, TLS, headers) and an error-message lexicon: daniele-michelin.com/en/.