Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job in hours. Somewhere in the controller logs or the CronJob's events there's a message like this:

Cannot determine if job needs to be started: too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew. 

On newer clusters the same situation shows up as a warning event with the reason TooManyMissedTimes. Either way, the message is telling you exactly what to do, but not why it happened. Here's both.

What the controller is counting

Each time the CronJob controller looks at a CronJob, it works out which scheduled times have passed since the job last ran (.status.lastScheduleTime, or the CronJob's creation time if it has never run). Normally that's zero or one.

If the controller wasn't able to start the job for a while, the count grows. For a job scheduled every minute, two hours of not starting means 120 missed times. Above 100, the controller logs this message.

What happens next depends on your Kubernetes version:

  • The original CronJob controller (the default in older clusters, before the rewritten controller became the default around Kubernetes 1.21) stops scheduling the job entirely at that point. It keeps logging the error and creating nothing, until someone intervenes. That's the "my CronJob silently stopped" version of the problem.
  • The rewritten controller used by current versions records a TooManyMissedTimes warning event and still schedules the most recent missed run, so the job usually recovers. The warning is still worth fixing, because it means runs were missed and something caused it.

Step 1: find out how it got there

kubectl describe cronjob my-job -n my-namespace kubectl get events -n my-namespace --field-selector involvedObject.name=my-job kubectl get cronjob my-job -n my-namespace -o jsonpath='{.status.lastScheduleTime}{"\n"}' 

Compare lastScheduleTime with the current time. A large gap means something stopped runs from being created. The usual causes:

The CronJob was suspended, then resumed

suspend: true stops new runs. When you set it back to false days later, the controller counts every scheduled time during the suspension as missed. For an every-minute job, a day of suspension is 1,440 missed times.

The controller wasn't running

If kube-controller-manager was down, restarting, or overloaded (for example during a control plane upgrade, or on a struggling self-managed cluster), no CronJob was evaluated. On managed clusters this is rare but not impossible during upgrades.

Clock skew

If the clock on the control plane node jumped, or differs a lot from the time in lastScheduleTime (recorded earlier, possibly by another controller instance), the controller can compute a large number of missed slots. Check that NTP is running on control plane nodes. On a managed cluster, this one is on your provider.

The schedule is very frequent and runs were blocked

With concurrencyPolicy: Forbid, a run that hangs blocks every later run. If an every-minute job hangs for two hours, the controller counts 120 skipped schedules. Fix the hang too: set activeDeadlineSeconds on the Job template so a stuck run is killed.

Kubernetes CronJobs aren't the only way to get a reliable schedule. delay.run runs HTTP cron jobs with retries and start logs, without depending on a controller's missed-start budget.

Step 2: set startingDeadlineSeconds

The fix the message suggests is the right one. startingDeadlineSeconds tells the controller: "If you're more than this many seconds late for a scheduled run, skip it." With that set, the controller only counts missed times within the deadline window, so the count can't reach 100 unless the window is huge.

apiVersion: batch/v1 kind: CronJob metadata: name: my-job spec: schedule: "* * * * *" startingDeadlineSeconds: 120 # skip runs that are more than 2 minutes late  concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: activeDeadlineSeconds: 300 # kill a run that takes more than 5 minutes  backoffLimit: 2 template: spec: restartPolicy: Never containers: - name: worker image: registry.example.com/worker:1.4.2 args: ["run-once"] 

How to pick the value:

  • It must be comfortably larger than the controller's evaluation interval (around 10 seconds). Very small values, under about 10 seconds, can make the job never run at all.
  • For frequent jobs where only the latest run matters (sync, cleanup, polling), a value close to the schedule interval is fine: a late run is replaced by the next one.
  • For daily or weekly jobs that must not be skipped, use a larger value, such as a few hours, so a short control plane outage still results in a late run rather than a skipped one.

Step 3: if the job is stuck on an old cluster

On a cluster with the legacy controller, adding startingDeadlineSeconds is enough: the next evaluation only looks inside the window, finds at most a couple of missed times, and starts scheduling again. If you need it to run right now, trigger one manually:

kubectl create job --from=cronjob/my-job my-job-manual-$(date +%s) -n my-namespace 

Deleting and recreating the CronJob also resets the count, but you lose its history and status, so prefer the deadline.

Step 4: decide whether missed runs matter

The error is a symptom of runs not happening. Ask whether that's acceptable for this job:

  • If a missed run means missed invoices, reminders or backups, you need monitoring that notices when a job didn't run, independent of the CronJob controller. A heartbeat ("the job pings a URL on success; alert if no ping in N minutes") is the simplest version.
  • If the job must catch up after an outage, make it work out what it missed (from a "last processed" timestamp in your own data) instead of relying on the scheduler to replay every slot.

Schedulers outside the cluster, such as delay.run, avoid the dependency on the control plane for timing, but the heartbeat and the catch-up logic are worth having wherever the schedule lives.

Related reading

Have you hit "too many missed start times"? Was it a resumed suspension, a controller outage, or something stranger?