Live data from Hacker News

Show HN: Homelab Monitoring Setup with Grafana

blog.randombits.host

51–60 of 84 posts

Re: Show HN: Homelab Monitoring Setup with Grafana

#51
post #47
post #30

Earlier quoted context omitted.

I recently setup basic monitoring using Telegraf + Influx + Grafana. Here are the alert triggers, in order of importance (imo): * ZFS pool errors. Motivator: one of my HDDs failed and it took me a few days to notice. The pool (raidz1) kept chugging along of course. * HDD and SSD SMART errors * High HDD and SSD temperatures * ZFS pool utilization * High CPU temperature. Motivator: one of my case fans failed and it too…

Setting an email address you actually check in /root/.forward would provide most of this, and all of it with the addition of low-tens of lines of shell script and a cron job or two, no? I get that tastes vary, but adding more services to worry about & keep updated to my home server(s) is not my idea of a good time. I doubt the custom pieces required to get all of those alerts via email would take longer than installi…

Oh definitely, but only if you are not interested in the visualization side.

I wanted the ability to quickly see the current & historical state of these and other metrics, not just configure alerts.

I’m also omitting the fact that I have collectors running inside different VMs on the same host. For example, I have Telegraf running on Windows to collect GPU stats.

Re: Show HN: Homelab Monitoring Setup with Grafana

#52
post #51
post #47

Earlier quoted context omitted.

Setting an email address you actually check in /root/.forward would provide most of this, and all of it with the addition of low-tens of lines of shell script and a cron job or two, no? I get that tastes vary, but adding more services to worry about & keep updated to my home server(s) is not my idea of a good time. I doubt the custom pieces required to get all of those alerts via email would take longer than installi…

Oh definitely, but only if you are not interested in the visualization side. I wanted the ability to quickly see the current & historical state of these and other metrics, not just configure alerts. I’m also omitting the fact that I have collectors running inside different VMs on the same host. For example, I have Telegraf running on Windows to collect GPU stats.

Ah, yeah, that probably won't be enough for you then. Need Windows monitoring, and want the graphs—yeah, much bigger pain to get anything like that working via email.

Re: Show HN: Homelab Monitoring Setup with Grafana

#54

I have been using Zabbix to monitor my servers for the last years, since I wanted something simple and this Grafana/Prometheus stack always scared me because, as the OP says, of the amount of “moving parts”. Zabbix has been quite solid and has lots of templates for different servers (linux, windows, etc), triggers and can also monitor docker containers (although i never tried that). The only thing Zabbix cant do well…

I use Zabbix and Grafana. Grafana has a zabbix data source plugin so you can have best of both worlds really.

Re: Show HN: Homelab Monitoring Setup with Grafana

#55
post #29
post #16

Earlier quoted context omitted.

> It's easy to get overwhelmed by all the moving pieces Exactly my thoughts! Isn't there something (open source and as good as Prometheus+Grafana) that doesn't have as many moving parts as the stack used by OP? I can imagine there are many use cases for that: from side projects (homelabs) to small startups that don't have huge distributed systems, but still need monitoring (without relying on third-parties). Ideally,…

I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have. Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup. A couple of months ago I…

Nice experiment.

Side note: it should be possible to tweak some config parameters to optimize the memory usage or cpu usage of quickwit. Ask us on the discord server next time :)

Re: Show HN: Homelab Monitoring Setup with Grafana

#56
I went down the Grafana rabbit hole, and without a doubt, it's a fantastic tool. It can handle just about any kind of data you throw at it, and when it comes to visualizing time series data, it's second to none. That said, it's a slog to set up and configure, but once finished, I had a beautiful dashboard for my home media server, and life was good. Unfortunately, a few months later, I was forced to upgrade and lacked the time to reconfigure Grafana. So, as a stopgap, I installed Netdata... fast-forward two years, and today I still haven't reconfigured Grafana, nor do I plan to.

For my use case, a home media server, Netdata turned out to be way simpler to set up, and, most importantly, way less of a hassle/dink-around. It's a basic plug-and-play operation with auto-discovery. While the dashboard isn't nearly as beautiful or configurable, it gets the job done and provides everything I pretty much need or want. It offers a quick overview, historical metrics (over a year of data) to analyze trends or spot potential issues, and push/email notifications if something goes awry.

If you decide to go down this route, there are two major items:

1. You'll need to configure the dbengine[1] database to save and store historical metric data. However, I found the dbengine configuration documentation to be a bit confusing, so I'll spare you the trouble - just use this Jupyter Notebook[2]. If needed, adjust the input, run it, scroll down, and you'll see a summary of the number of days, the maximum dbengine size, and the yaml config, which you can copy, paste, and voila.

2. If you're hoarding data, you'll probably want to set up smartmontools/smartd[3] in a separate Docker container for better disk monitoring metrics. However, I think you can enable hddtemp[4] with Netdata through the config if you don't want or need the extra hassle. You can have Netdata to query this smartd container, but with a handful of disks, it ends up timing out frequently, so I found it's best to simply set up smartd/smartd.conf to log out the smartd data independently. Then all you need to do is tell Netdata where to find the smartd_log[5], and Netdata handles the rest.

Boom, home media server metrics with historical data, done. It still takes a bit of time to set up, but way less than Grafana. Anywho, hopefully, this saves you from wasting as much time as I did. And if you're looking for a smartd reference, shoot me a reply, and I'll tidy up and share my Docker config/scripts and notes.

[1] https://learn.netdata.cloud/docs/typical-netdata-agent-confi... [2] https://colab.research.google.com/github/andrewm4894/netdata... [3] https://www.smartmontools.org/wiki [4] https://github.com/vitlav/hddtemp [5] https://learn.netdata.cloud/docs/data-collection/storage,-mo...

Re: Show HN: Homelab Monitoring Setup with Grafana

#57

I self host for years about 30 services, out of these 3 are vital (bitwarden, home assistant and pihole). I work in IT, I am a geek so I tried a few monitoring systems and wrote two myself. Then I realized that I have self-sustaining, 24/7 monitoring agents: wife and children. I gave up trying to have the right stack and just wait for them to yell. Seriously: it works great and it made me wonder WHY I am trying to mo…

This is the first time I've heard a parent refer to their currently-living-at-home child as "self-sustaining".

Usually that means they know how to microwave their own hot pockets and open a can of spaghetti-Os.

Re: Show HN: Homelab Monitoring Setup with Grafana

#59
post #31
post #16

Earlier quoted context omitted.

> It's easy to get overwhelmed by all the moving pieces Exactly my thoughts! Isn't there something (open source and as good as Prometheus+Grafana) that doesn't have as many moving parts as the stack used by OP? I can imagine there are many use cases for that: from side projects (homelabs) to small startups that don't have huge distributed systems, but still need monitoring (without relying on third-parties). Ideally,…

I use Telegraf (collector) + Influx (storage) + Grafana (visualization and alerting). Telegraf is amazingly simple to use for collection and has a ton of plugins available.

I also started with that stack, but swapped out InfluxDB for Postgres + TimescaleDB extension, which adds timeseries workflows (transparent partitioning, compression, data retention, continuous aggregates, …).

I found InfluxDB to be lacking in terms of permissions management, flexibility regarding queries (SQL, joins), data retention, ability to debug problems. In Postgres, for example, I can look into the execution plan of a statement, log long running queries, and so on.

Telegraf as an agent is very flexible; it has input plugins for every task I could want, and besides it’s default „pull workflow“ (checks on defined interval) I also like to push new metrics directly to the Telegraf inputs.socket plugin from my scripts (backup stats, …).

Re: Show HN: Homelab Monitoring Setup with Grafana

#60
post #37

Mildly related: can anyone recommend a time series database that supports easy aggregation by week (with the ability to configure the start of the week) and month? I'm looking for something to switch from InfluxDB which I'm currently using. The linked article is using Prometheus which also doesn't appear to support this functionality.

You could take a look at Postgres + TimescaleDB extension, which offers a nice time_bucket() function on its hypertables[1]. You can also materialize using continuous aggregates („self updating“ materialized views).

1: https://docs.timescale.com/api/latest/hyperfunctions/time_bu...

Post reply on HN