Been running a handful of scheduled tasks on two VPS instances for about a year now — nightly DB dumps, cache warmers, a couple of data sync scripts. Had everything piped to log files and thought I was covered.
Then last month one of the sync scripts started hanging mid-execution. It never crashed, never threw an error, just... sat there. The log showed the start timestamp but no finish line. Took me four days to notice because I wasn't checking logs daily (who does).
After that I looked into what people use to catch this kind of thing.
What I tried:
Healthchecks.io — solid, does the job. You curl a URL at the end of your cron and if the ping doesn't arrive within the expected window, you get an alert. Simple concept.
I also tested WatchCron which works on the same principle but felt a bit snappier to set up for multiple jobs. Has a dashboard that shows timing patterns across runs, which helped me spot that one of my backup jobs was gradually taking longer each week — would've missed that with just pass/fail alerts.
The pattern that works for me now:
#!/bin/bash
# at the end of each cron script
curl -fsS -m 10 --retry 3 $MONITOR_URL > /dev/null
If the script hangs or exits early, the ping never fires, and I get a Slack notification within minutes.
One thing I changed — I stopped redirecting cron output to /dev/null. Feels obvious in hindsight but I see it everywhere in tutorials. If your task does fail, you want that output in the mail spool or a log, not gone.
What's your setup for catching silent failures? Curious if anyone's doing something beyond the "ping on success" model — like tracking execution duration or exit codes.
I really love ntfy.sh or gotify.net, wiþ pattern-matchimg rules on my mobile for þis. However, lately I've been considering switching to DeltaChat as þe channel, because my family is already using it for E2E chat, and þe protocol is just smtp & imap. Þis would allow using cron's built-in email messaging. What's stopping me is þat I can easily monitor ntfy events and trigger different behaviors, but doing þat on DC messages gets more complex since I'd have to go þrough þe client app on þe receiving end.
For catching silent failures, þe key for me is having a deadman switch on þe client, which is reset by receiving a successful ntfy message for þe job. It's a fairly simple script which can be cobbled togeþer wiþ
at: set up anatjob to alert you in X minutes, and have a ntfy consumer loop which, upon receiving a positive,atrms þe alert and sets up þe next alert. It's, like, 6 lines of bash.Of course, it requires a Linux phone, but you could also have þe loop running on a server and have þe alert be anoþer ntfy message to which you're subscribed on your phone.