Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Explaining modern server monitoring stacks for self-hosting

dataswamp.org

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Explaining modern server monitoring stacks for self-hosting

#21

Post by Sin2x »

viraptor wrote: post #18
Sin2x wrote: post #4

Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.

In what way do you think it's an overkill? As in, what's too much / gets in the way? I've gone exactly the opposite way with: I don't want to deal with a highly opinionated and integrated thing like zabbix if I can put a 3 small things together (grafana, influx, telegraf in my case) and have a simple system monitoring that also can handle anything I want to throw at it.

You already need to deal with 3 things as opposed to one and learn PromQL to boot. If you want to have dashboards you also need to learn and use Grafana. I used both Prometheus+Thanos+Grafana stack and Zabbix in production and can say with certainty that the latter is much easier to use and set up for infra monitoring than the former. You need a whole dedicated observability team to use Prometheus effeciently.

Re: Explaining modern server monitoring stacks for self-hosting

#22

Post by evv555 »

staticassertion wrote: post #12
mr-karan wrote: post #6

Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.

At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure). I like this approa…

Part of what parent is describing is pushing configurations down to the agents from a centrally managed admin panel. I think this works well for basic system metrics but not for anything off the beaten path requiring integrations and scripts.

In response to your vector solution:. What I found highly adaptable is having all servers forward to a central vector/fluent-bit agent, have that agent forward to kinesis firehose, then attach a transform lambda to create the final output. You end up with 2 configuration points from which you can control and transform all your logs. Instead of having to push this down to the servers through config management.

Re: Explaining modern server monitoring stacks for self-hosting

#25

Post by doubled112 »

Gigachad wrote: post #10
melvyn2 wrote: post #2

For my single self-hosted server (also NixOS), I use netdata, because it has all the info I need, is efficient enough while keeping a day’s worth of history as well, and most importantly has built-in alerts for services like telegram or slack. Other stacks can do the same, but I haven’t found anything as simple to set up; netdata is just a simple service.

Self hosting in general tends to be overkill. It’s mostly people practicing for enterprise tasks. If you wanted to be practical about things you very likely would just give up self hosting entirely.

I think there's a pretty large middle ground between what I'd consider a home server setup (or self hosting it wherever) and practicing for enterprise tasks.

Much of the crowd around here and certain subreddits IS probably practicing for more, but I have a counterexample. You won't find him around the internet. Not interested.

I have a friend who's recently learned just enough Linux and Docker to get through a Debian install with some prompting, and run a Matrix home server, Nextcloud and Jellyfin because he thought they were cool. Its an old (for him) repurposed desktop.

He's a mechanic. He doesn't care about anything enterprise. He doesn't want monitoring because he'll know if it's down. He doesn't care if backups run because he's sure he's got the important stuff on an old disk somewhere, probably.

Re: Explaining modern server monitoring stacks for self-hosting

#26

Post by infocollector »

lupimiguel wrote: post #23

I find it very difficult to understand how "modern server monitoring" requires 6 or 7 components, when you can just deploy netdata on all your nodes with a 1-line installer and get way higher fidelity monitoring.

Is anyone using netdata in production? Has anyone looked at their data sharing/selling policies? How does it compare with other tools (https://github.com/louislam/uptime-kuma) ? Looks very impressive.

Re: Explaining modern server monitoring stacks for self-hosting

#27

Post by linsomniac »

viraptor wrote: post #17

Since this post was interested in resource usage, I'm using telegraf for pushing system information (similar idea as collectd) and it's at 24kB of res memory. It reads from / writes to pretty much anything you can think of https://docs.influxdata.com/telegraf/v1.23/plugins/

Also a telegraf user here, been using it for ~5 years across 100+ machines, and it's been rock solid.

Re: Explaining modern server monitoring stacks for self-hosting

#28

Post by ssddanbrown »

sccxy wrote: post #16

I have tried multiple monitoring tools, but I have found Uptime Kuma most useful for me. Alerts work well and it is not too complicated for simple things. https://github.com/louislam/uptime-kuma

My monitoring needs are very simple and I've had good success with Uptime Kuma. For checking server metrics i built a simple go service [1] to expose metrics with simple rules applied so uptime kurma will notify me when issues arise (Since the go service will change http status upon alert scenarios).

[1]: https://github.com/ssddanbrown/wirestat

Re: Explaining modern server monitoring stacks for self-hosting

#29

Post by viraptor »

Sin2x wrote: post #21
viraptor wrote: post #18

Earlier quoted context omitted.

In what way do you think it's an overkill? As in, what's too much / gets in the way? I've gone exactly the opposite way with: I don't want to deal with a highly opinionated and integrated thing like zabbix if I can put a 3 small things together (grafana, influx, telegraf in my case) and have a simple system monitoring that also can handle anything I want to throw at it.

You already need to deal with 3 things as opposed to one and learn PromQL to boot. If you want to have dashboards you also need to learn and use Grafana. I used both Prometheus+Thanos+Grafana stack and Zabbix in production and can say with certainty that the latter is much easier to use and set up for infra monitoring than the former. You need a whole dedicated observability team to use Prometheus effeciently.

I'd disagree as someone running grafana+influx+telegraf for a few reasons.

Dealing with 3 things... feels about as much work as zabbix. That one requires server + db + agents. Basically the same components I'm running.

Promql/flux - while I know it, I almost never use it - instead in grafana click the database name, metric name, aggregation and I'm done.

While you need people to use it effectively, it's the same for everything, including mysql for zabbix. You can dump prometheus/influx somewhere with no configuration and survive for quite a long time.

Re: Explaining modern server monitoring stacks for self-hosting

#30

Post by JustFinishedBSG »

infocollector wrote: post #26
lupimiguel wrote: post #23

I find it very difficult to understand how "modern server monitoring" requires 6 or 7 components, when you can just deploy netdata on all your nodes with a 1-line installer and get way higher fidelity monitoring.

Is anyone using netdata in production? Has anyone looked at their data sharing/selling policies? How does it compare with other tools ( https://github.com/louislam/uptime-kuma ) ? Looks very impressive.

They aren't even in the same category though ( except the "monitoring" super-category ). Uptime-kuma monitors uptime only while netdata doesn't even (afaik) monitor uptime, it's mostly machine related metrics.
Post reply on HN