The box rebooted for kernel patches at 01:55 and came back at 02:10. The 02:00 backup didn't run. Plain cron doesn't look back, so nothing complained. The next backup ran at 02:00 the following day, and for 24 hours there was no backup at all.
Another team had the opposite problem. Their scheduler was down for six hours and then "caught up" by firing 72 runs of a five-minute sync job at once, hammering a partner API into rate-limiting them for the rest of the day.
Both teams had a missed-run policy. Neither had chosen it.
TL;DR
- When a scheduled time passes while the scheduler is down, there are three sane behaviours: skip, run once late, or run every missed slot.
- Classic cron skips. systemd timers with
Persistent=trueand anacron run once late. Some job frameworks replay every slot. - Pick per job: daily and weekly jobs usually want "run once late". Frequent sync jobs usually want "skip". Per-period jobs (billing for each day) need "every slot", but run in sequence, not in a burst.
- My opinion: "run once late, within a deadline" is the right default, and plain cron's silent skip is the most dangerous behaviour of the three.
The three policies
| Policy | After 6 h of downtime, a 5-min job... | A daily 02:00 job... | Good for |
|---|---|---|---|
| Skip | resumes at next slot | waits until tomorrow | frequent, stateless syncs |
| Run once late | runs once now, then normal | runs once now | backups, reports, cleanups |
| Run every slot | runs 72 times | runs once (or N for N days) | per-period work like billing |
Classic cron: skip, silently
cron checks every minute whether any entry matches the current time. If the daemon or machine wasn't running at 02:00, that minute never happened as far as cron is concerned. No catch-up, no log line. That's one of the less obvious answers in Cron job not running? A step-by-step checklist.
DST does the same thing on a smaller scale: in many time zones, a 02:30 job doesn't exist on the spring-forward night. Cron job timezone problems covers that one.
systemd timers: run once late
# /etc/systemd/system/backup.timer [Timer] OnCalendar=*-*-* 02:00:00 Persistent=true RandomizedDelaySec=5m [Install] WantedBy=timers.target Persistent=true stores the last trigger time on disk. If a run was missed, the timer fires once shortly after boot. It doesn't replay every missed slot, which is what you want for backups. More on picking between the two in Cron vs systemd timers.
anacron: run once late, for daily-ish jobs
anacron is built for machines that aren't always on. It tracks when each job last ran and runs it once if its period (daily, weekly, monthly) has passed. Many distros already route /etc/cron.daily through it. Its resolution is days, so it's no help for hourly jobs.
Kubernetes CronJob: run once late, with a deadline
apiVersion: batch/v1 kind: CronJob metadata: { name: nightly-report } spec: schedule: "0 2 * * *" timeZone: "Europe/Kyiv" startingDeadlineSeconds: 3600 concurrencyPolicy: Forbid jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: report image: example/report:1.4 startingDeadlineSeconds: 3600 means: if the controller couldn't start the 02:00 run, it may still start it until 03:00, after which that run is skipped. Without it, Kubernetes counts missed schedules and refuses to start the job at all if more than 100 were missed, which surprises people after long outages. concurrencyPolicy: Forbid stops a late run from overlapping with the next one.
Run every slot: do it in order, with a cap
For per-period jobs, skipping is data loss. Billing for 1 November must happen even if you only notice on 3 November. Don't rely on the scheduler to replay. Make the job itself work out what's missing:
from datetime import date, timedelta def run_billing(today=None): today = today or date.today() last_done = db.scalar("SELECT max(period) FROM billing_runs WHERE status='done'") day = last_done + timedelta(days=1) if last_done else today - timedelta(days=1) processed = 0 while day < today and processed < 7: # cap the backlog per run bill_period(day) # idempotent per period db.execute("INSERT INTO billing_runs(period, status) VALUES (%s,'done') " "ON CONFLICT (period) DO UPDATE SET status='done'", (day,)) day += timedelta(days=1) processed += 1 Now the schedule only decides when to check. The job decides what's owed, processes it in order and caps how much backlog it handles at once. Because bill_period is idempotent, an overlapping or repeated run is harmless. The patterns are in the at-least-once post.
Whatever you choose, detect it
A skip policy is only safe if someone knows a run was skipped. Alert when a job's last success is older than its period plus a margin. For HTTP-triggered jobs, whether from cron and curl or a hosted scheduler like delay.run, log the scheduled time and the actual start time separately, so "ran 40 minutes late" is visible rather than inferred. And if jobs get killed mid-run by deploys, that's a different failure with a different fix: see graceful shutdown for scheduled work.
A quick decision checklist
- If this run is skipped, is anything lost forever? Yes: run every slot, inside the job, in order.
- Does running it late still help? Yes: run once late with a deadline.
- Will the next run cover the same ground anyway? Yes: skip.
- Whichever you picked, is a missed or late run visible to someone?
My take
The scheduler's default missed-run behaviour is an accident of its design, not a choice for your job. Write the policy down next to the schedule, one line per job, and configure it explicitly. It takes five minutes and prevents the "we had no backups for a day and didn't know" conversation.
Over to you
Which policy do your most important jobs use, and did you choose it on purpose or discover it during an incident? I'm especially curious how people handle catch-up for jobs that call rate-limited third-party APIs.
