Here's an uncomfortable exercise: think about the last time an uptime alert reached you for a real outage, at a time you weren't already at your desk.
If you can't remember, you don't know whether your alerting works. You know your monitors work, because you've seen them go green. That's not the same thing.
A monitoring system has two halves:
- Detection: noticing the site is broken.
- Delivery: getting that fact to a human who can do something about it.
Teams test the first half all the time and almost never test the second. This checklist is for the second half. Go through it once, then repeat a short version every quarter.
Delivery path
☐ 1. Every monitor has an alert destination
It sounds obvious. But monitors get created in a hurry ("I'll set up notifications later"), and some never get a contact attached. List every monitor and its destination. Anything with "none" is decoration.
☐ 2. The destination is a person or rota, not an archive
Alerts that go to a shared mailbox, a mailing list nobody reads, or a chat channel with 400 unread messages are delivered in the technical sense only. For anything that matters, at least one destination should be a phone that makes a noise.
☐ 3. The alert bypasses Do Not Disturb
Most phones silence notifications at night unless you allow specific apps or contacts through. Check that your alerting app or SMS sender is on the allowed list. This is the most common reason a "working" alert fails at 3 a.m.
- iOS: Focus → choose the mode → allowed apps/people. Some alerting apps support "critical alerts" that override silent mode. Enable it if offered.
- Android: Do Not Disturb → exceptions / apps that can interrupt.
☐ 4. Email alerts aren't being filtered
Send a test alert by email and check where it lands. Inbox, Promotions, Spam and "Other" are four very different places. Add the sender to your contacts or create a filter that marks it important.
☐ 5. Chat alerts trigger a real notification
If alerts go to Slack, Discord or Teams, make sure the channel isn't muted for the people who need it. Consider mentions (@oncall) for critical alerts so they notify even in busy channels.
Coverage
☐ 6. Someone is covered at every hour
Write down who gets alerted on a Saturday at 04:00, during a public holiday, and while the usual person is on a plane. If the answer is "the same person, always", that's a single point of failure with a pulse.
☐ 7. There's an escalation step
If the first person doesn't acknowledge within N minutes, who's next? Even on a two-person team, a simple "if not acked in 15 minutes, alert the other person" doubles your odds.
☐ 8. Alert contacts survive people leaving
When someone leaves or changes phone numbers, their alerts don't follow them. Add "update alert contacts" to your offboarding checklist. Check now for phone numbers and emails belonging to former teammates.
Content
☐ 9. The alert says what broke, in the first line
Compare:
❌ Monitor #4471 state change: DOWN ✅ DOWN: Checkout API (https://example.com/api/checkout) returning 503 from 3 regions At 3 a.m., on a lock screen, only the first ~80 characters get read. Make them count.
☐ 10. The alert links to what to do next
A link to the runbook, the dashboard, or at minimum the status page admin. The person woken up shouldn't have to remember where things are.
The actual test
☐ 11. Run a fire drill
Pick a quiet time, tell the team it's a drill, and cause a real alert. Point a test monitor at a URL that will fail, or temporarily break a test endpoint:
# a tiny local endpoint you can switch between healthy and broken python3 - <<'PY' from http.server import BaseHTTPRequestHandler, HTTPServer import os class H(BaseHTTPRequestHandler): def do_GET(self): broken = os.path.exists("/tmp/break-drill") self.send_response(503 if broken else 200); self.end_headers() self.wfile.write(b"drill: broken\n" if broken else b"drill: ok\n") HTTPServer(("0.0.0.0", 8080), H).serve_forever() PY # in another terminal: touch /tmp/break-drill (break) rm /tmp/break-drill (recover) Expose it publicly for the drill, point a monitor at it, then break it. Measure:
- Time to detect: from breaking to the monitor marking it down.
- Time to notify: from detection to the phone buzzing.
- Time to acknowledge: from buzz to a human responding.
Write the numbers down. If any step surprises you, you've just found the bug that would have hurt you in a real outage.
☐ 12. Test the recovery notification too
When you fix the test endpoint, does a "recovered" message arrive? Without it, people waste time checking an incident that's already over, or worse, assume a real incident is resolved when it isn't.
Make it a habit
- Quarterly: a 15-minute drill (items 11–12) and a contact review (items 6–8).
- After any team change: items 6–8.
- After any phone or OS upgrade: item 3. OS updates have a habit of resetting notification settings.
Whatever you use for detection, whether a cron job and curl, an open-source tool, or a hosted monitor like upordown.online, the delivery path is yours to test. No tool can check whether your phone is on silent.
The short version
| Check | Question |
|---|---|
| Destinations | Does every monitor alert a human? |
| Phone | Does it break through Do Not Disturb? |
| Coverage | Who's covered at 4 a.m. on a holiday? |
| Escalation | What happens if nobody acks? |
| Content | Does line one say what broke? |
| Drill | When did you last prove it works? |
Related reading
- Stop monitoring your homepage. Monitor the path that makes you money.
- Alert after 1 failed check or 3? Tuning uptime alerts so on-call stops ignoring them
- A free uptime check with GitHub Actions that opens (and closes) an issue for you
- Your health check returns 200 while users get errors: 7 health check mistakes
When did you last run a real alert drill, and what broke? Mine found a phone that silenced the alerting app after an OS update. Share yours in the comments.
