Nobody discovers their backups are broken at a convenient moment. Most backup monitoring verifies the least important thing: that the script exited zero tells you a process finished, not whether the file contains your data.
Layer one: did it run at all
A backup script that is not running produces no output and no alert. A heartbeat covers this: give the job a ping URL, set the interval to the schedule, and chain the ping to a successful exit.
#!/bin/bash
set -euo pipefail
BACKUP_FILE="/backups/db-$(date +%F-%H%M).sql.gz"
pg_dump "$DATABASE_URL" | gzip > "$BACKUP_FILE"
curl -fsS --retry 3 "$PING_URL"
Set the grace period generously — a dump that took eight minutes last year may take twenty-five now.
Layer two: is the output plausible
This catches the backup that succeeded and produced nothing useful. Add sanity checks before the ping: a size floor (a fraction of the expected size, revisited as the database grows), an archive integrity test, and a header content check.
SIZE=$(stat -c%s "$BACKUP_FILE")
if [ "$SIZE" -lt $((50*1024*1024)) ]; then echo "Backup too small" >&2; exit 1; fi
if ! gzip -t "$BACKUP_FILE"; then echo "Corrupt archive" >&2; exit 1; fi
curl -fsS --retry 3 "$PING_URL"
Layer three: can you restore it
The only layer that actually proves anything, and the one most teams never build. On a schedule, restore your most recent backup into a scratch database, run assertions, then throw it away, with its own heartbeat. Beyond row counts, check data freshness — a backup taken from a replica that stopped replicating three weeks ago restores perfectly and is three weeks stale.
Where to send the alerts
A failed nightly backup is a same-day alert, not a 3am page — you still have yesterday's. A backup that failed two nights running is a real page. A failed restore test is a page regardless of when it happens, because it means your backups have been useless for an unknown length of time.
The measurement that matters
Once all three layers exist, you can answer the question that matters in a crisis: how much data would we lose, and how long would recovery take? Before restore testing, both answers are guesses. After, they are measurements.