We just fire off a Slack message. It does the trick.
Ask HN: How do you monitor your systemd services?
41–50 of 61 posts
Re: Ask HN: How do you monitor your systemd services?
#42Earlier quoted context omitted.
> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…
So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…
If you have a one-off server running nodejs, you've definitely got maintenance
Re: Ask HN: How do you monitor your systemd services?
#43> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it." This is really more in the realm of a shell script. You could do this verbosely: #!/bin/sh /path/to/my/backup_job if [ $? -ne 0 ] then /path/to/my/failure_alert fi ...or, you could do this tersely: #!/bin/sh /path/to/my/backup_job || /path/to/my/failure_alert The wrapper script would go into your…
That's great but isn't the real question about what goes in /path/to/my/failure_alert ?
The original poster hinted that "notifications" and email were options.
For SMS text message notifications, I use an AWK script to send SMTP to an email-SMS gateway. I try to keep these under the 160 character limit, only sent in extraordinary situations (high server room temp, decoy port triggering on the firewall hinting an intrusion, etc). I don't want this blowing up my phone.
For email, I have a MIME pack script that allows me to send a message with an arbitrary number of base64-encoded attachments.
Does that cover what might be in a failure alert script?
Re: Ask HN: How do you monitor your systemd services?
#44> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it." This is really more in the realm of a shell script. You could do this verbosely: #!/bin/sh /path/to/my/backup_job if [ $? -ne 0 ] then /path/to/my/failure_alert fi ...or, you could do this tersely: #!/bin/sh /path/to/my/backup_job || /path/to/my/failure_alert The wrapper script would go into your…
That might be a good first step, but certainly isn't sufficiently robust.
What happens when when the /path/to/my/failure_alert script fails?
What happens when your backup job returns success but didn't generate any output?
What happens when you turn off the systemd timer for a while and forget to turn it back on?
What happens when the server stops running, has a full disk, or has a networking issue?
Ultimately, some of the other answers are better. You should have a separate system monitoring this. And that separate system should track every time a backup happens, either by checking the backup exists at the target location (good), or checking that the backup system sent a "Yes, I did a backup" message (ok, but not as good).
I use Telegraf for data collection, InfluxDB (v1) as a time series database, and Grafana (v7) for graphing and alerts. I'm using an older version of InfluxDB and Grafana because they just work and keep on working. Many other tools will work just as well as these do. I'm just giving them as an example.
Such a system may seem like overkill to just keep track of a few things, but you need something that'll tell you when you get no data. So at a minimum you'll want something on a separate server and you'll want it to send alerts when an expected event doesn't happen.
Re: Ask HN: How do you monitor your systemd services?
#45Earlier quoted context omitted.
So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…
If you have a one-off server running nodejs, you've definitely got maintenance
Why's that?
I think the only time I sshd to that server was last week when I added usb device monitoring and had to docker pull & & docker up -d.
Other than that... Can't remember dealing with the "monitoring stack".
Re: Ask HN: How do you monitor your systemd services?
#46For monitoring and alerts I look to how industrial SCADA does it.
Unfortunately I have no code to share, because... I'm a dev, rather than a sysadmin, and I do backups and such at home with the GUI, and I don't work 9m anything microservicy, so I've only done monitoring of features within one monolithic application.
My preferred way to monitor a backup task would just be to use a backup tool that had it's own monitoring built in, or integrations with a popular monitor solution. I've done DIY backup scripts, it always seems so simple that you might as well just write a few lines... But it's also so common of a use case that there's lots of really nice options.
I've done the systemd --failed thing on every new terminal, and probably should go back to doing so, but it doesn't do much if you're not logging in regularly. Although it does help when you're logging in to see what went wrong.
But the general idea when I have actually implemented monitoring, is that you have state machine alerts. They go from normal, to tripped, to active.
If you acknowledge it, it becomes acknowledged, if it bad condition goes away, it becomes cleared, and returns to normal when acknowledged(Or instantly, if auto-ack is selected).
Every alert has a trip condition, which can be any function on one or more "Tag points"(Think observable variables with lots of extra features).
A tripped alert only becomes active if it remains tripped for N seconds, to filter out irrelevant things caused by normal dropped packets and such, while still logging them.
While an alert is active, it shows in the list on the server's admin page, and can periodically make a noise or do some reminder. Eventually I'd like to find some kind of MQTT dashboard solution that shows everything in one place, and sends messages to an app, but I haven't needed anything like that yet.
Under the hood the model is fairly complex but you don't have to think about it much to use it.
Re: Ask HN: How do you monitor your systemd services?
#47Uptime-Kuma [1] with ntfy [2]. Most of my services expose HTTP so I just have Uptime-Kuma monitor that. But if you have something that is not exposed to the public you can still use a "push" type monitor, and in a cron job on your server(s), send heartbeat to it when everything is working.
[1] https://github.com/louislam/uptime-kuma
[2] https://ntfy.sh/
Re: Ask HN: How do you monitor your systemd services?
#48I have an /etc/inittab entry with
stmd:2345:respawn:/bin/systemdRe: Ask HN: How do you monitor your systemd services?
#49> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it." This is really more in the realm of a shell script. You could do this verbosely: #!/bin/sh /path/to/my/backup_job if [ $? -ne 0 ] then /path/to/my/failure_alert fi ...or, you could do this tersely: #!/bin/sh /path/to/my/backup_job || /path/to/my/failure_alert The wrapper script would go into your…
That might be a good first step, but certainly isn't sufficiently robust. What happens when when the /path/to/my/failure_alert script fails? What happens when your backup job returns success but didn't generate any output? What happens when you turn off the systemd timer for a while and forget to turn it back on? What happens when the server stops running, has a full disk, or has a networking issue? Ultimately, some…
The original poster asked for simple detection of non-zero exit status.
What you speak of is far, far beyond the original question.
I am quite pleased with the reaction to my post, and I do not feel the need to compare technical merit.
Perhaps you would be happier with JCL?
In any case, enjoy your tooling.
Re: Ask HN: How do you monitor your systemd services?
#50Earlier quoted context omitted.
> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…
Alternative view point. Observability is hella expensive. Orgs should consider TCO when making such decisions. Paying a few hundred thousands more for the skills to self run could literally chop tens of millions off vendor bills.
But then you aren't taking into account server and storage costs of self managed monitoring.
Unless it's Datadog. That's expensive.