Blog

Blog

How to monitor cron jobs with heartbeats

Scheduled jobs fail silently. Learn how dead-man’s-switch heartbeat monitoring works, how to choose an interval and grace period, and what to alert on.

  • cron
  • heartbeat
  • background jobs

The problem with silent cron failures

Scheduled work — nightly backups, billing runs, queue workers, data syncs — has a failure mode that ordinary uptime checks cannot see. There is no URL to probe, because the job is supposed to reach out, not be reached. When it stops running, nothing errors and nothing pages; the backup simply is not there on the day you need to restore from it.

These are among the most damaging outages precisely because they are invisible until the outcome matters. The fix is to invert the check: instead of asking "is this reachable?", ask "did this run when it was supposed to?".

How heartbeat monitoring works

Heartbeat monitoring is a dead-man's switch. You give your job a unique URL and have it send a short request — a "ping" — every time it finishes successfully. The monitor expects that ping on a schedule, and alerts you when it does not arrive. The alert fires from the absence of a signal, which is exactly what a silently dead job produces.

Because the job reports its own success, you also learn when it starts running late, not just when it stops entirely.

Choosing an interval and grace period

The interval is how often you expect the ping; the grace period is the extra slack you allow before a late ping counts as missed. The grace period absorbs normal scheduling jitter — a job that usually runs in two minutes but occasionally takes six should not page anyone. In PageLantern, if you leave the grace period blank it defaults to a quarter of the interval, with a minimum of 60 seconds and a maximum of one hour; setting an explicit value, including 0, overrides that default.

Match the interval to the job's real schedule and set the grace period to a little more than its worst normal run time. Too tight and you get false alarms; too loose and a genuinely dead job goes unnoticed for too long.

Ping only on success

Send the heartbeat at the very end of the job, after the work has actually succeeded — not at the start. A job that pings on launch and then crashes halfway will look healthy while doing nothing useful. Pinging on success means the monitor is asserting that the work completed, which is the thing you actually care about.

For long or multi-stage jobs, you can treat the final successful step as the signal, so a partial run that fails before the end correctly shows up as a missed heartbeat.

A heartbeat monitoring checklist

Give every critical scheduled job a heartbeat. Ping only after the job succeeds. Set the interval to the job's real cadence and the grace period to a little over its worst normal run time. Route missed-heartbeat alerts to the team that owns the job. Backups, billing, and anything whose failure you would not notice for days are the highest priority to cover first.