Live data from Hacker News

Ask HN: How do you monitor your systemd services?

news.ycombinator.com

1–10 of 61 posts

Ask HN: How do you monitor your systemd services?

#1
I am using systemd on my machine and try to configure most things through it. For example, I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it. Over time, I've had multiple solutions for this:

Send a notifcation via notify-send

Add `systemctl --failed` to my shell startup script

Send myself emails

None of these are quite ideal. Notifications are disruptive of the current workflow and ephemeral, meaning I might forget about it if I don't deal with it immediately. Similarly, reading `systemctl --failed` on every new terminal is also disruptive but at least it makes me not forget about it. Both of these are also not really applicable to server systems. Sending myself emails feels a bit wrong but has so far been the best solution.

How are other people solving this? I did some research and I am surprised that there isn't a more rounded solution. I'd expect that pretty much every Linux user must run into this problem.

Re: Ask HN: How do you monitor your systemd services?

#3
This thread is one of those cases where you read something and realize you've been completely missing something. I don't monitor these as much as I should

Servers/services? Definitely - take your pick. Timers/jobs, particularly those on my system? Nothing!

With the right directives laid out ('Wants/Requires/Before/After'), they can be pretty robust/easily forgotten.

I've been lucky in this regard; I check 'systemctl list-timers' just to be sure - but they always run

Re: Ask HN: How do you monitor your systemd services?

#5
Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget.

Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to start from the beginning with an extensible system. For me, Prometheus has been the best option: easy to configure, lightweight, doesn't even need to run in the host, and can monitor multiple systems. You just have to configure which exporters you want it to pull data from. In this case, prometheus_node_exporter has a massive amount of stats about a system (including SystemD), and there are default alarms and dashboards out there that will help you create basic monitoring in a minute.

You can choose to use Grafana for visualization, and then either the integrated Grafana alerting or use the Prometheus alerting + Prometheus Alertmanager. I think in the latest versions Grafana Alerting includes basically an embedded AlertManager so it should have the same features.

Regarding the type of alert itself, I send myself mails for the persistence/reminders + Telegram messages for the instant notifications. I find it the best option tbh.

Re: Ask HN: How do you monitor your systemd services?

#7

I run the Prometheus node_exporter on my servers. That has a systemd collector for the state of services. That reports the state of all systemd services to a central Prometheus and alertmanager cluster, which has various alert rules.

I recommend this approach, as you can also deploy blackbox_exporter to get an "outside" perspective on service status.

Re: Ask HN: How do you monitor your systemd services?

#8
If you are ok with a Saas and if it's just scheduled jobs that you are monitoring, there are a number of monitoring tools where you tell when job completes (with a http request) and a missing ping (after a grace period) means that it failed.

I think https://deadmanssnitch.com/ may have been the original service for this.

https://healthchecks.io/ has a fairly generous free tier that I use now.

There are others that do the same thing Sentry, Uptime Robot, ...

Re: Ask HN: How do you monitor your systemd services?

#10
post #5

Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget. Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to s…

> Short answer: Prometheus + Grafana + Alertmanager.

Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config stanzas (e.g. redis_exporter, postgresql_exporter, etc.), and a log multiplexer for their Loki logging service [which is to logs-based metrics as Prometheus is to regular time-series metrics.])

Why use a managed service? Because, unless your IT department is large enough to have its own softball team, the stability/fault-tolerance of the "monitoring and alerting infra" itself is going to be rather low-priority for the company compared to other things you're being asked to manage; and also will be something you rarely need to touch... until it breaks. Which it will.

You really want some other IT department whose whole job is to just make sure your monitoring and alerting stay up, doing this as their product IT focus rather than their operations IT focus.

(You also want your alerting to be running on separate infra from the thing it watches, for the same reason that your status page should be on a separate domain and infra from the system it reports the status of. Having some other company own it is an easy way to achieve this.)

> Regarding the type of alert itself, I send myself mails for the persistence/reminders + Telegram messages for the instant notifications.

Again, higher-level rec appropriate for SMBs: sign up for PagerDuty, and configure it as the alert "Notification Channel" in Grafana Cloud. If you're an "ops team of one", their free plan will work fine for you.

Why is this better than Telegram messages? Because the PagerDuty app does "critical alerts" — i.e. its notifications pierce your phone's silent/do-not-disturb settings (and you can configure them to be really shrill and annoying.) You don't want people to be able to call you at 2AM — but you do want to be woken up if all your servers are on fire.

---

Also: if you're on a cloud provider like AWS/GCP/etc, it can be tempting to rely on their home-grown metrics + logging + alerting systems. Which works, right up until you grow enough to want to move to a hybrid "elastic load via cloud instances; base load via dedicated hardware leasing" architecture. At which point you suddenly have instances your "home cloud" refuses to allow you to install its monitoring agent on. Better to avoid this problem from the start, if you can sense you'll ever be going that way. (But if you know your systems aren't "scaling forever" and you'll stay in the cloud, the home-grown cloud monitoring + alerting systems are fine for what they do.)

Post reply on HN