Show HN: Homelab Monitoring Setup with Grafana • Hacker News Board

Re: Show HN: Homelab Monitoring Setup with Grafana

#41

Post by oaktowner » Wed, Jun 07, 2023, 6:37 PM UTC

BrandoElFollito wrote: post #35

I self host for years about 30 services, out of these 3 are vital (bitwarden, home assistant and pihole). I work in IT, I am a geek so I tried a few monitoring systems and wrote two myself. Then I realized that I have self-sustaining, 24/7 monitoring agents: wife and children. I gave up trying to have the right stack and just wait for them to yell. Seriously: it works great and it made me wonder WHY I am trying to mo…

Reminds me of the (possibly apocryphal) monitoring that was in place when Healthcare.gov was launched: they had a TV tuned to the news, and the news would tell them whenever the site crashed!

Re: Show HN: Homelab Monitoring Setup with Grafana

#42

Post by gurchik » Wed, Jun 07, 2023, 6:42 PM UTC

sjsdaiuasgdia wrote: post #3

This confirms to me what I suspected when I was trying to determine whether to host my own Grafana stack or use the Grafana Cloud free tier - that I'd end up spending a ton of time fiddling with a constellation of services I didn't actually care about that I could spend on the projects and services I do care about. I've not found it too hard to stay within the limits of the free tier. The 10 dashboards limit is the m…

If you are running Kubernetes in your homelab then, for better or for worse, the Prometheus helm chart abstracts all of this away. The default Helm values worked perfectly for me to gather metrics from my cluster and make a quick dashboard in Grafana. Other than increasing the default size of the Prometheus storage volume and configuring the node exporter for a non-Kubernetes host I wanted metrics from, I didn't have to touch anything.

Re: Show HN: Homelab Monitoring Setup with Grafana

#43

Post by applied_heat » Wed, Jun 07, 2023, 6:53 PM UTC

BrandoElFollito wrote: post #35

I self host for years about 30 services, out of these 3 are vital (bitwarden, home assistant and pihole). I work in IT, I am a geek so I tried a few monitoring systems and wrote two myself. Then I realized that I have self-sustaining, 24/7 monitoring agents: wife and children. I gave up trying to have the right stack and just wait for them to yell. Seriously: it works great and it made me wonder WHY I am trying to mo…

I monitor so that when there is a problem I have some data I can use to troubleshoot the problem and identify possible solutions.

Re: Show HN: Homelab Monitoring Setup with Grafana

#44

Post by applied_heat » Wed, Jun 07, 2023, 7:05 PM UTC

adql wrote: post #11

I've found VictoriaMetrics all-in-one binary to be perfect size for home at the very least for metrics gathering. Supports Prometheus querying and few other formats for ingesting so any knowledge bout "how to get data into prometheus" applies pretty much 1:1 + their own vmagent is pretty advanced. Not related to company in any way, just a happy user. https://victoriametrics.com/

I haven’t started using it yet but i identified Victoria metrics as the first time series database I would try as a replacement for our wonderware historian so I won’t have to use AVEVA’s half baked web dashboard product and can use grafana or something else sane instead

Re: Show HN: Homelab Monitoring Setup with Grafana

#45

Post by cbreshears » Wed, Jun 07, 2023, 7:08 PM UTC

czzzzz wrote: post #39

Shameless plug for AppScope ( https://github.com/criblio/appscope ) which is designed for exactly this. Capturing observability data from processes in your environment without code modification, and shipping the data off to tools like grafana for monitoring.

+1 on AppScope!

Re: Show HN: Homelab Monitoring Setup with Grafana

#46

Post by nullify88 » Wed, Jun 07, 2023, 7:14 PM UTC

esafak wrote: post #40

dracyr wrote: post #29

Earlier quoted context omitted.

I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have. Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup. A couple of months ago I…

Does using vector commit you to DataDog in any way?

Not at all.

Re: Show HN: Homelab Monitoring Setup with Grafana

#47

Post by bamfly » Wed, Jun 07, 2023, 7:16 PM UTC

Cyph0n wrote: post #30

bombcar wrote: post #14

Earlier quoted context omitted.

What does the monitoring actually do for you? I've seen these setups, even setup one for myself a few times (either Grafana or similar such as Netdata, or Linode's Longview) but I've not really seen what it does for me beyond the "your disk is almost full" warnings.

I recently setup basic monitoring using Telegraf + Influx + Grafana. Here are the alert triggers, in order of importance (imo): * ZFS pool errors. Motivator: one of my HDDs failed and it took me a few days to notice. The pool (raidz1) kept chugging along of course. * HDD and SSD SMART errors * High HDD and SSD temperatures * ZFS pool utilization * High CPU temperature. Motivator: one of my case fans failed and it too…

Setting an email address you actually check in /root/.forward would provide most of this, and all of it with the addition of low-tens of lines of shell script and a cron job or two, no? I get that tastes vary, but adding more services to worry about & keep updated to my home server(s) is not my idea of a good time. I doubt the custom pieces required to get all of those alerts via email would take longer than installing and configuring that stack, and then the maintenance is likely to be zero for so long that you'll probably replace the hardware before it needs to be touched again (... and if you scripted your setup, it'll very likely Just Work on the replacement)

Re: Show HN: Homelab Monitoring Setup with Grafana

#48

Post by cocire » Wed, Jun 07, 2023, 7:24 PM UTC

shrx wrote: post #37

Mildly related: can anyone recommend a time series database that supports easy aggregation by week (with the ability to configure the start of the week) and month? I'm looking for something to switch from InfluxDB which I'm currently using. The linked article is using Prometheus which also doesn't appear to support this functionality.

would love an answer to this as well! something with great Python (Flask, maybe even SQLAlchemy) support would be cool too

Re: Show HN: Homelab Monitoring Setup with Grafana

#49

Post by lcnPylGDnU4H9OF » Wed, Jun 07, 2023, 7:26 PM UTC

BrandoElFollito wrote: post #35

I self host for years about 30 services, out of these 3 are vital (bitwarden, home assistant and pihole). I work in IT, I am a geek so I tried a few monitoring systems and wrote two myself. Then I realized that I have self-sustaining, 24/7 monitoring agents: wife and children. I gave up trying to have the right stack and just wait for them to yell. Seriously: it works great and it made me wonder WHY I am trying to mo…

This is the first time I've heard a parent refer to their currently-living-at-home child as "self-sustaining".

Re: Show HN: Homelab Monitoring Setup with Grafana

#50

Post by nullify88 » Wed, Jun 07, 2023, 7:27 PM UTC

danwee wrote: post #16

bovermyer wrote: post #6

I'm in the process of building out a Grafana stack (Prometheus, Loki, Tempo, Mimir, Grafana) for my day job right now. ...and also for one of my side projects, OSRBeyond. It's easy to get overwhelmed by all the moving pieces, but it's also a lot of _fun_ to set up.

> It's easy to get overwhelmed by all the moving pieces Exactly my thoughts! Isn't there something (open source and as good as Prometheus+Grafana) that doesn't have as many moving parts as the stack used by OP? I can imagine there are many use cases for that: from side projects (homelabs) to small startups that don't have huge distributed systems, but still need monitoring (without relying on third-parties). Ideally,…

The closest thing may well be Elasticsearch (and Kibana for visualisation), if you are fine with the Elastic license. As its document format is very flexible, it can be used to store logs, metrics, and traces. It'll be a solution inferior to specialised tools like Prometheus, Mirmir, Tempo, though. And some may be put off by the difficulty of running Elasticsearch.

Alternatives could be other general purpose databases.