Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you monitor your systemd services?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Ask HN: How do you monitor your systemd services?

#51

Post by mike_hearn »

rudasn wrote: post #34
derefr wrote: post #10

Earlier quoted context omitted.

> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…

So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…

So .... why were the servers shutting down, and what metric did your own system capture that the others didn't which let you determine that?

Re: Ask HN: How do you monitor your systemd services?

#52

Post by andrewm4894 »

rudasn wrote: post #34
derefr wrote: post #10

Earlier quoted context omitted.

> Short answer: Prometheus + Grafana + Alertmanager. Or, a higher-level recommendation, appropriate for most SMBs: sign up for Grafana Cloud's managed prometheus+grafana (or any equivalent external managed monitoring stack), and then follow their setup instructions to install their grafana-agent monitoring agent package (which sticks together node_exporter, several other optional exporters enable-able with config sta…

So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…

> So I turned to Netdata. A one liner on each server and we had super sexy and fast dashboard for each server. No birds eye view, but fine. I then spent maybe 3-4 days trying to figure out how to get alerting to work (just email, but fine) and get temperature readings (or something like that).

I work in Netdata on ML. Just wanted to mention that as of last release a parent node will show all children in the agent dashboard so if doing again as of today a parent netdata might have got you the birds eye view as a starting point https://github.com/netdata/netdata/releases/tag/v1.41.0#v141...

(of course we also have Netdata Cloud which would have probably worked too but maybe was not as built out 3 years ago as is now - but don't want to go into sales mode and get blasted :) )

Re: Ask HN: How do you monitor your systemd services?

#53

Post by mike_hearn »

Sadly there are lots of basic must-have tasks that Linux distros simply do not support out of the box. It's not so much an OS as a kit for making operating systems. Backup is another.

Here's how I set up email monitoring of systemd services, for anyone who wants it:

https://gist.github.com/mikehearn/f1db694f24eaa05c753e5a7598...

It consists of three parts. Firstly a shell script that will email the unit status colorized to your preferred email address. Secondly, a service file that tells systemd how to call it, and finally, an OnFailure line in each service that you want to monitor. You can use systemd's support for overlays to add this to existing services you didn't write yourself.

You also have to make sure that your server can actually send mail to you. Installing default-mta will get you an SMTP relay that's secure out of the box but your email service will consider it spam. If you use gmail it's typically sufficient to just create a filter that ensures emails from your server are never marked as spam.

Re: Ask HN: How do you monitor your systemd services?

#54

Post by scrapheap »

mcpherrinm wrote: post #4

I run the Prometheus node_exporter on my servers. That has a systemd collector for the state of services. That reports the state of all systemd services to a central Prometheus and alertmanager cluster, which has various alert rules.

You can also restrict the list of systemd services that node_exporter provides metrics for - that way you only store metrics for those services you care about.

Re: Ask HN: How do you monitor your systemd services?

#55

Post by scrapheap »

gjulianm wrote: post #5

Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget. Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to s…

I can definitely vouch for Prometheus for monitoring - once you get your head round the basics then it's so easy to extend it to monitor other things.

Re: Ask HN: How do you monitor your systemd services?

#56

Post by rudasn »

mike_hearn wrote: post #51
rudasn wrote: post #34

Earlier quoted context omitted.

So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…

So .... why were the servers shutting down, and what metric did your own system capture that the others didn't which let you determine that?

Well, at first I was able to gather and correlate enough cpu, temperature, entrypoint data for apparently problematic servers.

The servers were shutting down due to high temperatures caused by persistent high cpu usage.

Knowing that, I installed datadog with APM on just a couple of the servers (because $$) which led me to postgres issues (indexing), weasy pdf generation issues (a python lib), and some bad django code (queryset to list before pagination).

Re: Ask HN: How do you monitor your systemd services?

#57

Post by rudasn »

andrewm4894 wrote: post #52
rudasn wrote: post #34

Earlier quoted context omitted.

So about 3 years ago we had a bunch of on prem servers shutting down around March/April. We had even more servers that weren't shutting down so we had to "move fast" before they all had issues. I must have spent about a week trying to learn just enough about prometheus and grafana (I had used grafana before with influx but for a different purpose) so that we could monitor temperature, memory, cpu, and disk (the bare…

> So I turned to Netdata. A one liner on each server and we had super sexy and fast dashboard for each server. No birds eye view, but fine. I then spent maybe 3-4 days trying to figure out how to get alerting to work (just email, but fine) and get temperature readings (or something like that). I work in Netdata on ML. Just wanted to mention that as of last release a parent node will show all children in the agent das…

Hey! I subscribe to your github releases and was reading about all that the other day (the parent/child node stuff).

When/If I have the time I'll dig into Netdata some more as I like your approach. :)

I'm not a devops/sre/systems guy, I just do it because I have to, so it's a bit difficult for me to find the time to experiment with these tools.

Re: Ask HN: How do you monitor your systemd services?

#58

Post by andrewm4894 »

rudasn wrote: post #57
andrewm4894 wrote: post #52

Earlier quoted context omitted.

> So I turned to Netdata. A one liner on each server and we had super sexy and fast dashboard for each server. No birds eye view, but fine. I then spent maybe 3-4 days trying to figure out how to get alerting to work (just email, but fine) and get temperature readings (or something like that). I work in Netdata on ML. Just wanted to mention that as of last release a parent node will show all children in the agent das…

Hey! I subscribe to your github releases and was reading about all that the other day (the parent/child node stuff). When/If I have the time I'll dig into Netdata some more as I like your approach. :) I'm not a devops/sre/systems guy, I just do it because I have to, so it's a bit difficult for me to find the time to experiment with these tools.

Cool! - we always looking for feedback, feel free to hop into our discord, forum, or GH discussions (links here: https://www.netdata.cloud/community/) to leave any feedback or ask any questions if you run into any issues.

(cheers for the mention here too - always nice to try get some feedback and discussion going on HN as its so candid :0 )

Re: Ask HN: How do you monitor your systemd services?

#59

Post by ownagefool »

happymellon wrote: post #50
ownagefool wrote: post #29

Earlier quoted context omitted.

Alternative view point. Observability is hella expensive. Orgs should consider TCO when making such decisions. Paying a few hundred thousands more for the skills to self run could literally chop tens of millions off vendor bills.

But then you aren't taking into account server and storage costs of self managed monitoring. Unless it's Datadog. That's expensive.

Not in the post, but I think there's still some pretty large savings.

Pretty much anything SaaS based is ridiculous. If you can swing self-hosted ( managed but in your account with there's potential for a discussion, but with many products, it's the actual integration work that's the real work.

Don't get me wrong, there's specific "always going to be small" where it likely makes sense.

Re: Ask HN: How do you monitor your systemd services?

#60

Post by happymellon »

ownagefool wrote: post #59
happymellon wrote: post #50

Earlier quoted context omitted.

But then you aren't taking into account server and storage costs of self managed monitoring. Unless it's Datadog. That's expensive.

Not in the post, but I think there's still some pretty large savings. Pretty much anything SaaS based is ridiculous. If you can swing self-hosted ( managed but in your account with there's potential for a discussion, but with many products, it's the actual integration work that's the real work. Don't get me wrong, there's specific "always going to be small" where it likely makes sense.

I'm just saying that SaaS provides a lot more than just the cost of having an engineer or two.

It may not be cost effective, but if you think that hiring two people will be all you spend when you move everything on prem, you'll be in for a bit of a shock.

Post reply on HN