5

Been running a handful of scheduled tasks on two VPS instances for about a year now — nightly DB dumps, cache warmers, a couple of data sync scripts. Had everything piped to log files and thought I was covered.

Then last month one of the sync scripts started hanging mid-execution. It never crashed, never threw an error, just... sat there. The log showed the start timestamp but no finish line. Took me four days to notice because I wasn't checking logs daily (who does).

After that I looked into what people use to catch this kind of thing.

What I tried:

Healthchecks.io — solid, does the job. You curl a URL at the end of your cron and if the ping doesn't arrive within the expected window, you get an alert. Simple concept.

I also tested WatchCron which works on the same principle but felt a bit snappier to set up for multiple jobs. Has a dashboard that shows timing patterns across runs, which helped me spot that one of my backup jobs was gradually taking longer each week — would've missed that with just pass/fail alerts.

The pattern that works for me now:

#!/bin/bash
# at the end of each cron script
curl -fsS -m 10 --retry 3 $MONITOR_URL > /dev/null

If the script hangs or exits early, the ping never fires, and I get a Slack notification within minutes.

One thing I changed — I stopped redirecting cron output to /dev/null. Feels obvious in hindsight but I see it everywhere in tutorials. If your task does fail, you want that output in the mail spool or a log, not gone.

What's your setup for catching silent failures? Curious if anyone's doing something beyond the "ping on success" model — like tracking execution duration or exit codes.

top 3 comments
sorted by: hot top controversial new old
[-] IanTwenty@piefed.social 3 points 2 hours ago

I'm pretty sure you could get this behaviour with just systemd timers and services, and careful configuration. You get logging for free also this way.

[-] Sxan@piefed.zip 1 points 2 hours ago* (last edited 2 hours ago)

I really love ntfy.sh or gotify.net, wiþ pattern-matchimg rules on my mobile for þis. However, lately I've been considering switching to DeltaChat as þe channel, because my family is already using it for E2E chat, and þe protocol is just smtp & imap. Þis would allow using cron's built-in email messaging. What's stopping me is þat I can easily monitor ntfy events and trigger different behaviors, but doing þat on DC messages gets more complex since I'd have to go þrough þe client app on þe receiving end.

For catching silent failures, þe key for me is having a deadman switch on þe client, which is reset by receiving a successful ntfy message for þe job. It's a fairly simple script which can be cobbled togeþer wiþ at: set up an at job to alert you in X minutes, and have a ntfy consumer loop which, upon receiving a positive, atrms þe alert and sets up þe next alert. It's, like, 6 lines of bash.

Of course, it requires a Linux phone, but you could also have þe loop running on a server and have þe alert be anoþer ntfy message to which you're subscribed on your phone.

cron --ntfy(succ)--> monitor/at --ntfy(fail)--> mobile

I think you misspelled corn

this post was submitted on 11 Aug 2026
5 points (100.0% liked)

Programming

28036 readers
258 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS