I broke a nightly Postgres backup script on purpose to see which failures I'd actually hear about. Pointed at a closed port, it exited 0 and left a 20-byte .gz file. A truncated dump got past pg_restore --list. And if cron never starts the job, there's no exit code to check at all.

1. The job fails, and the exit code says it worked

I started with this line:

pg_dump "$DATABASE_URL" | gzip > "backup-$(date +%F).sql.gz" 

I pointed it at a port with no database behind it:

$ ./nightly-naive.sh; echo "exit $?" pg_dump: error: connection to server at "127.0.0.1", port 5999 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? exit 0 $ ls -l backup-*.sql.gz -rw-r--r-- 1 me wheel 20 Nov 19 10:31 backup-2026-11-19.sql.gz 

By default, bash gives a pipeline the exit status of its last command. Here gzip compressed an empty input into a 20-byte file and returned 0, so the script returned 0 too. If cron mail is set up, the pg_dump error lands there, which helps only if someone reads that mailbox.

set -o pipefail fixes the exit code (the same script then exits 1), but the 20-byte file is still there, named like every other night's backup. So the script has to clean up after itself too.

The simpler fix is to drop the pipe. pg_dump --format=custom --file=... writes a compressed archive by itself, so there's no second command hiding pg_dump's exit code. Check that exit code before keeping the file.

2. The job doesn't run at all

A failed run can at least report something. A run that never starts can't: the server was replaced and the crontab didn't come along, someone commented the line out while debugging, the machine was off at 2 a.m. Every alert that depends on the job reporting its own failure stays quiet.

For this you need something outside the server that expects to hear from the job and complains when it doesn't: a dead man's switch. Healthchecks.io is the one I use (the free plan covers 20 checks, and it's open source if you'd rather run it yourself). You give a check the job's schedule, run the job once by hand so the check goes from "new" to "up" (a check that has never been pinged won't alert), and from then on a success ping that's late past the grace time raises an alert. The script below also sends /start and /fail pings. When a /fail gets through, Healthchecks marks the check down right away.

Here's the script I ended up with. It needs DATABASE_URL and PING_URL (the check's ping URL) in its environment, and the cron user has to be able to write to BACKUP_DIR, which defaults to /var/backups/pg:

#!/bin/bash # Nightly pg_dump that reports to a dead man's switch. # cron: 15 2 * * * /opt/backup/nightly-backup.sh >> /var/log/pg-backup.log 2>&1 set -uo pipefail BACKUP_DIR="${BACKUP_DIR:-/var/backups/pg}" out="$BACKUP_DIR/app-$(date -u +%Y-%m-%dT%H%M%SZ).dump" tmp="$out.partial" ping() { curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL$1" --data-raw "${2:-}" || true; } fail() { echo "backup failed: $1" >&2 rm -f "$tmp" ping /fail "$1" exit 1 } ping /start # Custom format is compressed already, so there's no pipe to swallow pg_dump's exit code. pg_dump --format=custom --file="$tmp" "$DATABASE_URL" || fail "pg_dump exited $?" # Reading the archive end to end catches a dump that stopped partway. pg_restore --file=/dev/null "$tmp" || fail "pg_restore could not read the archive" mv "$tmp" "$out" || fail "could not move the dump into place" ping "" "$(du -h "$out" | cut -f1) $(basename "$out")" 

A few choices in there:

  • A failed ping doesn't stop the backup. Each curl attempt has a 10-second limit and up to three retries, and || true lets the script carry on if they all fail. A missed success ping can cause a false alarm. If /fail doesn't get through, the alert comes later: the run already sent /start, so Healthchecks raises it once the run goes past the grace time without a success ping.
  • The failure message is short, like pg_dump exited 1 or pg_restore could not read the archive. pg_dump's own errors include hostnames, and sometimes more. The full error goes to the local log; the alert only needs to tell you to go and look.
  • Set the grace time to your slowest run, plus a margin. With a start ping, Healthchecks also uses the grace time as the longest a run may take, so a dump that hangs halfway raises an alert too.

To test it without a real check, I pointed PING_URL at a small local server that logs each request, then ran the script once against a test database (50,000 rows) and once against a port with nothing on it:

POST /ping/abc/start '' POST /ping/abc '604K app-2026-11-19T023906Z.dump' POST /ping/abc/start '' POST /ping/abc/fail 'pg_dump exited 1' 

The second run exited 1 and left no .partial file behind.

If your dumps end up in S3, also check the bucket and prefix you'd restore from. A renamed project or a wrong env var can send backups somewhere else while the job keeps reporting success. For snapshots written by the backupdrill CLI, a daily Lambda that checks the bucket for an overdue snapshot looks for the newest one with a manifest.json, which the CLI writes last. For plain uploads, check the timestamp of the newest .dump under the expected prefix.

3. The job runs, and the file is no good

The pg_restore --file=/dev/null line is there because of this one. I cut a good 619 KB dump down to its first 300 KB and checked it two ways:

$ pg_restore --list half.dump > /dev/null; echo "exit $?" exit 0 $ pg_restore --file=/dev/null half.dump; echo "exit $?" pg_restore: error: could not read from input file: end of file exit 1 

--list could still read the table of contents at the start of my truncated file. pg_restore --file=/dev/null reads the archived data as well, turning it into SQL and throwing the output away, and on the truncated file it stopped with an end-of-file error. You can end up with a file like this by piping pg_dump straight into an upload: if pg_dump dies partway, the uploader can still save the bytes it got. There's a version that reads the dump back before uploading in this write-up on pg_dump to S3 with checks for truncated dumps.

A file that reads end to end can still be the wrong backup, though: a dump of a replica that stopped replicating last Tuesday, or of the staging database because someone copied an env file. To see what actually made it in, restore it into a throwaway Postgres and look at the data. Run select max(created_at) on a busy table and compare it with the newest row you'd expect that night's dump to have, then look for a row you know exists only in production. A fifteen-minute restore test walks through the restore for a Supabase database and what to check afterwards. If pg_restore prints errors along the way, a short deck on Postgres restore drills shows which ones to expect in a throwaway database.

The short version

  • Write the dump with pg_dump --format=custom --file=... instead of piping it into gzip. If you keep the pipe, set -o pipefail and delete the output on failure.
  • Have the job ping a dead man's switch, and run it once by hand so the check is live. Then a run that never starts still raises an alert.
  • Read the dump end to end before you call it done, and restore one now and then to check the data.

I work on BackupDrill, a backup service for Supabase. On paid plans it restores backups into a throwaway Postgres on a schedule and compares row counts against the dump for regular tables. The script and the manual restore test above work without it.

I used AI to help draft this post.