Ask HN: How do you monitor your systemd services?

In general this evolves to a SIEM-like solution in IT or gets added to the tag menagerie in OT.

If you're focused on "notifications are bad" note that notifications are push, and pull solutions are possible. Tail logs (or journalctl) and post significant events to Redis (https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) for example.

@wh33zle For work, well, i have to follow already-established convention (some that others have noted). but for personal machines, i have not rolled out too many comprehensive monitoring solutions or platforms. Rather, i add focus on moitoring specific jobs/tasks, and as such leverage cron to run the job, and use basic, old school sorts of bash scripts to assess success or failure. I'm statrting to look into leveragin more systemd as you noted.

Now, specific to alerting, well, i have rolled out my own solution...Caution: self-promotion coming next...

I stopped relying on email being sent from servers since i've had too many annoyances, constraints in my history. Also, nowadays email is a medium that is slow for me...that is, i treat it like its non-time-sensitive3 messaging (for the majortiy of the time). So, for system alert-style messagings, I use my own little python script that sends messages into a dedicated matrix room. Since, i'm always on matrix, its a place where i can quickly see a new system alert messaage (matrix clients like Element allow you to adjust visibility - i think they call it noise level - of which messages are given higher or lower priority for the client vieew, etc.). And, those messages tend to be ephemeral, since they're just alerts, and such messages do not pollute my email inbox. There are plenty of options in this space of course. Mine is not the only one, but i also wanted to learn how to make apps for matrix ecosystem, etc. Here's a link to my little notification app/script that leverages the matrix network chat ecosystem: https://github.com/mxuribe/howler

Another happy healthchecks.io user here. You can also run it on your own infra: https://healthchecks.io/docs/self_hosted/

I don't monitor services at that level at all, because it means basically nothing. More acutely: the the lack of a notification doesn't tell mean everything is "ok".

I tend to monitor the actual service. If it's a web server, have something checking that a specific URL is working (tip: use something specific, not /). Likewise any other network service is pretty easy to monitor.

For backups, check the date on the most recent file in the backup target location. If that date is older than "x", something is broken. This can apply to most other types of backend apps too -- everything has some kind of output.

It's when these checks fail that you can investigate deeper and start diagnosing systemd or whatever. It's also possible there's a bigger problem -- like DNS got messed up, or the hardware died -- and checking the final outcome will catch all this.

Basically explicitly checking systemd is a lot of extra work for no real added benefit. If your systemd service is failing often enough that knowing that is the problem immediately (at the alert level) IMHO you'd be better off to spend the time fixing the service definition so it doesn't fail.

Only send alerts from the end user perspective. In your case the end user would most likely go into the backups and list them. So I would have a job that lists the backups every day and if something is missing it alerts.

See the difference here is that you don't monitor the systemd backup job, you monitor the backup backend instead. Because systemd can be configured to retry a job, the end result is in the backend.

And in other cases I do have monitoring for individual services, but I only send alerts if the end user experiences an issue. So a web server process/systemd unit is being monitored, but the alert is on a different monitor that checks if the website returns 200, or if it contains a keyword indicating it works.