Our route planning job runs every night and decides which van takes which parcels the next day. In August we gave it its own pool of three nodes, so its memory use could not crowd out anything else, and tainted them with workload=batch:NoSchedule. Only pods with a matching toleration can be scheduled there, which was the point. The job got the toleration. Nothing else did.

In October the job failed at three in the morning and the depots had no routes at six. The pods had already been cleaned up, so we went to our log store. There was nothing from the batch nodes, not that night and not any night since August. Our node dashboards had no data for them either.

Logs and node metrics both come from DaemonSets, one pod per node: Fluent Bit and node exporter. A DaemonSet's pods are scheduled like any others and respect taints. The DaemonSet controller adds tolerations of its own for the taints Kubernetes puts on nodes with problems, not ready, unreachable, disk or memory pressure, unschedulable, but not for taints we invent. Neither DaemonSet tolerated workload=batch, so neither had ever run on those three nodes.

What hid it for two months was the DaemonSet's own status. Desired, current and ready all said nine. The desired count is the number of nodes the DaemonSet is allowed to run on, not the number of nodes in the cluster, so a DaemonSet that excludes three nodes is perfectly healthy by its own measure. Our alert compared ready with desired, and they agreed.

We reran the job by hand and the vans left an hour late. The cause turned out to be a single depot with a malformed postcode, which took ten minutes to find once we had logs. Every agent DaemonSet now carries a toleration with operator Exists and no key, which tolerates every taint, because an agent belongs on every node. An alert compares count(kube_node_info) from kube-state-metrics with each agent's desired number scheduled, and any difference pages.

A DaemonSet's health is measured against the nodes it chose. Ours had quietly chosen to skip the three nodes we cared about that night.

– Sergey Shinder