Kubernetes CronJobs fail in more ways than classic cron, and the platform's own signals are surprisingly weak about it.
What actually goes wrong
The pod is never scheduled — insufficient resources, a node selector that matches nothing, a taint added last week. The pod is evicted mid-run. Concurrency policy silently skips runs: with concurrencyPolicy: Forbid, if the previous run is still going the next is skipped, so a job that gradually slows down starts skipping runs without anything looking broken. Deadline exceeded. In every case the reliable signal is the same: the work did not report in.
The minimal setup
Add a heartbeat ping to the end of the container command, chained to success.
command:
- /bin/sh
- -c
- "/app/export.sh && curl -fsS --retry 3 --max-time 10 $PING_URL"
The ping URL comes from a secret, not the manifest — these tokens are credentials. The --max-time 10 bounds the ping so a monitoring hiccup cannot hang your pod. The && means a failed run and a missing run produce the same alert.
Set the grace period from your real p99
Kubernetes adds scheduling latency classic cron does not have: image pulls on a cold node, admission webhooks, waiting for a scale-up. Look at your actual run durations over the last month, take the worst case, and double it. Do not set grace to zero on Kubernetes.
The NAT problem nobody warns you about
Rate limits on ping endpoints are commonly keyed by source IP. Your cluster almost certainly egresses through a single NAT gateway, so every heartbeat shares one budget. Ninety CronJobs all pinging at exactly 02:00 is ninety requests in one second from one address. Stagger your schedules across minutes, add jitter with a short random sleep before the ping, and retry the ping.
What to do when a heartbeat fires
Because Kubernetes hides the evidence quickly, put the diagnosis commands in the runbook. Start with whether a Job object exists for the expected run at all — if not, the problem is scheduling. Set successfulJobsHistoryLimit and failedJobsHistoryLimit high enough that you can actually look at yesterday's run.