I made my own monitor in 500 lines of code.
It uses a HTTP client to poll all instances every minute which return /proc/stat so that I can see how much CPU they use:
Explaining modern server monitoring stacks for self-hosting
11–20 of 36 posts
Re: Explaining modern server monitoring stacks for self-hosting
#12Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.
At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure).
I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache.
This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.
Re: Explaining modern server monitoring stacks for self-hosting
#13Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.
How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).
Exactly. I feel that monitoring solutions have to be the opposite of backup ones.
For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups.
But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
Re: Explaining modern server monitoring stacks for self-hosting
#14Earlier quoted context omitted.
How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network. This is my main gripe with Prometheus (didn't use it yet).
Exactly. I feel that monitoring solutions have to be the opposite of backup ones. For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups. But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.
Re: Explaining modern server monitoring stacks for self-hosting
#15I always default to Netdata as it is batteries-included and can rarely outgrow it.
Re: Explaining modern server monitoring stacks for self-hosting
#16I have tried multiple monitoring tools, but I have found Uptime Kuma most useful for me.
Alerts work well and it is not too complicated for simple things.
Re: Explaining modern server monitoring stacks for self-hosting
#17Since this post was interested in resource usage, I'm using telegraf for pushing system information (similar idea as collectd) and it's at 24kB of res memory. It reads from / writes to pretty much anything you can think of https://docs.influxdata.com/telegraf/v1.23/plugins/
Re: Explaining modern server monitoring stacks for self-hosting
#18Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.
In what way do you think it's an overkill? As in, what's too much / gets in the way? I've gone exactly the opposite way with: I don't want to deal with a highly opinionated and integrated thing like zabbix if I can put a 3 small things together (grafana, influx, telegraf in my case) and have a simple system monitoring that also can handle anything I want to throw at it.
Re: Explaining modern server monitoring stacks for self-hosting
#19For my single self-hosted server (also NixOS), I use netdata, because it has all the info I need, is efficient enough while keeping a day’s worth of history as well, and most importantly has built-in alerts for services like telegram or slack. Other stacks can do the same, but I haven’t found anything as simple to set up; netdata is just a simple service.
Self hosting in general tends to be overkill. It’s mostly people practicing for enterprise tasks. If you wanted to be practical about things you very likely would just give up self hosting entirely.
based on?
Re: Explaining modern server monitoring stacks for self-hosting
#20Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.
I'm working on a side project that will essentially cover uptime and metrics monitoring for servers, as I find that trying to roll out prometheus monitoring across large numbers of servers across multiple organisations and setups is too much effort. I want to be able to quickly and easily add servers to a dashboard, set a number of alarms, and then not have to worry. Anything happens, an alarm trips, offline etc, and I get an email/app/slack/discord notification, etc.
I looked around but didn't really find anything that fit for me. There are a lot of complicated (albeit powerful) options, but I want simple, easy, lightweight, quick. These days I'm juggling so much, I want to be as efficient as possible with my time.
It's still early days but hoping to be able to onboard people towards the end of the year, for anyone who is interested feel free to join the waitlist: https://serverduty.co
Interestingly, I'm using ServerDuty to monitor ServerDuty as I build ServerDuty. I mean, if that isn't dogfooding, I don't know what is.