I love https://ntfy.sh/ for my services running on headless servers - it lets me ping my phone with messages of varying urgency, and even duplicate the notifications to email for particularly information-dense messages.
Ask HN: How do you monitor your systemd services?
21–30 of 61 posts
Re: Ask HN: How do you monitor your systemd services?
#22> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it."
This is really more in the realm of a shell script.
You could do this verbosely:
#!/bin/sh
/path/to/my/backup_job
if [ $? -ne 0 ]
then /path/to/my/failure_alert
fi
#!/bin/sh
/path/to/my/backup_job || /path/to/my/failure_alert
Re: Ask HN: How do you monitor your systemd services?
#23`0 * * * * journalctl --since="61 minutes ago" --priority=warning --quiet` In crontab piped to a bunch of grep -v for the things I want to ignore So basically the email approach, just have to be religious about marking unread if not immediately actioned
What kind of warning (and up) messages do want to keep in your journal - but ignore in your alerts? Ie why adjust your grep patterns - not what you log - to increase signal to noise ratio?
Re: Ask HN: How do you monitor your systemd services?
#24Everyone seems to be talking about production services & headless servers, but my impression is that you meant on the desktop?
I wrote a little script that puts a failed service count in waybar, and throws up a dismissable swaynag message with buttons to 'toggle details', and reset or restart the failed system/user units.
It's a bit noisy at the moment - but I think that's probably just a helpful indication of units I need to sort out/make a bit more robust anyway.
Re: Ask HN: How do you monitor your systemd services?
#25If you are ok with a Saas and if it's just scheduled jobs that you are monitoring, there are a number of monitoring tools where you tell when job completes (with a http request) and a missing ping (after a grace period) means that it failed. I think https://deadmanssnitch.com/ may have been the original service for this. https://healthchecks.io/ has a fairly generous free tier that I use now. There are others that do…
https://cronitor.io/ is another option here that works for me. You can set up rules like "It should run once a day and return after at least this amount of time and also return a number greater than 1" Then just use come curl calls to your scripts at start and end and you are good to go.
Re: Ask HN: How do you monitor your systemd services?
#26I think this is a great question. Consider https://blog.wesleyac.com/posts/how-i-run-my-servers. His unit files do not mention monitoring.
Re: Ask HN: How do you monitor your systemd services?
#27I have a carefully designed alert service for every project, checking various aspects of the system. It periodically checks heartbeats from various systems to make sure everything is in order. It sends alerts to UI via websocket, and to slack channels and makes calls to twilio numbers if things do not self-recover in time. I only check if the alert system is running via cron.
Re: Ask HN: How do you monitor your systemd services?
#28I’d go with sending myself a push notification on the phone through a dedicated service file and then call it with this in my unit file:
OnFailure=send-push-notification.service
Perhaps via a WhatsApp notification or any other instant message [0] or any other service such as matrix as said in another comment.
[0] https://developers.facebook.com/docs/whatsapp/cloud-api/get-...
Re: Ask HN: How do you monitor your systemd services?
#29Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget. Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to s…
> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…
Alternative view point.
Observability is hella expensive. Orgs should consider TCO when making such decisions. Paying a few hundred thousands more for the skills to self run could literally chop tens of millions off vendor bills.
Re: Ask HN: How do you monitor your systemd services?
#30> Notifications are [...] ephemeral, meaning I might forget about it if I don't deal with it immediately.
If you do like the notification method aside from this issue, try passing "--urgency=critical" or "--expire-time=0" to notify-send. Either (or both) of those should make the notifications stay popped up, assuming your notification daemon is doing something reasonable with those hints.
(Disclosure: I'm the author of xfce4-notifyd, which does behave in this way; other daemons may do other things.)