Just love https://healthchecks.io
I set it up on all my boxes with these two scripts:
win
https://github.com/reboot81/healthchecks_service_ps/
macos
https://github.com/reboot81/hc_check_maker_macos
linux
https://healthchecks.io/docs/bash/
Monitoring My Homelab, Simply
21–30 of 70 posts
Re: Monitoring My Homelab, Simply
#22I understand your pain. I used to have sensu, but it was a pain to keep updated (and didn't work that well on old rpis) But what I did find was a good alternative was telegraph->some sort of time series (I still really like graphite, influxQL is utter horse shit, and prometheus's fucking pull models is bollocks) Then I could create alert conditions on grafana. At least that was simple. However the alerting on grafana…
Why is the pull model bollocks? I’ve been building monitoring for stuff since nagios and zabbix were the new hot tools; and I can’t really imagine preferring the oldschool ways vs the pretty much industry standard of promstack these days…
Zabbix is bollocks. so is nagios. Having remote root access to all your stuff is utter shite.
Prometheus as a time series DB is great, I even like its QL. What I don't like is pull. Sure there is agent mode or telegraf/grafana agent. But the idea that I need to hold my state and wait for Prometheus to collect it is utterly stupid. The biggest annoyance is that I need to have a webserver somewhere, with a single god instance(s) that can reach out and touch it.
Great if you have just one network, but a bollock ache if you have any kind of network isolation.
This means that we are using influxdb and its shitty flux QL (I know we could upgrade, but thats hard)
Re: Monitoring My Homelab, Simply
#23I understand your pain. I used to have sensu, but it was a pain to keep updated (and didn't work that well on old rpis) But what I did find was a good alternative was telegraph->some sort of time series (I still really like graphite, influxQL is utter horse shit, and prometheus's fucking pull models is bollocks) Then I could create alert conditions on grafana. At least that was simple. However the alerting on grafana…
Why is the pull model bollocks? I’ve been building monitoring for stuff since nagios and zabbix were the new hot tools; and I can’t really imagine preferring the oldschool ways vs the pretty much industry standard of promstack these days…
Both models are totally fine, for their specific use cases.
Re: Monitoring My Homelab, Simply
#24Did he reinvent monit? Even a quick Prometheus + alert manager setup with two docker containers is not difficult to manage - mine just works, I seldom have to touch it (mainly when I need to tweak the alert queries). I use pushover for easy api-driven notifications to my phone, it’s a one-time $7 fee or so and it was money well spent.
I have a similar setup, prometheus and grafana (alertmanager is a separate thing from the normal grafana setup, right? I'm not even using that), and I use discord webhooks for notifications to my phone (I just @ myself or use channel notification settings).
Re: Monitoring My Homelab, Simply
#25Uptime Kuma (https://github.com/louislam/uptime-kuma). With email notifications. So much simpler, and free.
Re: Monitoring My Homelab, Simply
#26[dead]
Re: Monitoring My Homelab, Simply
#27Earlier quoted context omitted.
Why is the pull model bollocks? I’ve been building monitoring for stuff since nagios and zabbix were the new hot tools; and I can’t really imagine preferring the oldschool ways vs the pretty much industry standard of promstack these days…
Zabbix is bollocks. so is nagios. Having remote root access to all your stuff is utter shite. Prometheus as a time series DB is great, I even like its QL. What I don't like is pull. Sure there is agent mode or telegraf/grafana agent. But the idea that I need to hold my state and wait for Prometheus to collect it is utterly stupid. The biggest annoyance is that I need to have a webserver somewhere, with a single god i…
eh in a standard three tier you’re usually okay to pull up and push down aren’t you? Run it in the lower network..
Were all kubernetes these days so i guess i didn’t think about it a lot in recent years.
Re: Monitoring My Homelab, Simply
#28I’ve been facing a similar search for an ultra-simple but ultra-extensible monitoring solution for my homelab. I’ve had the idea to write a Python program where the main script is just responsible for scheduling and executing the checks, logging, and alerting based on set thresholds.
All monitoring would be handled via plugins, which would be extremely easy to write.
It would ship with a few core plugins (ping, http, cert check, maybe snmp), but you could easily write a plugin to monitor anything else — for example, you could use the existing Python Minecraft library and write a plugin to monitor your Minecraft server. Or maybe even the ability to write plugins in any language, not just Python.
I’m not a developer and I’m opposed to vibe coding, so it’ll be slow going :)
Re: Monitoring My Homelab, Simply
#29There's an article-bias towards rejectionism, towards single shot adventures. "I didn't grok so and so and here's the shell scripts I wrote instead". Especially for home cloud, home ops, home labs: that's great! That's awesome that you did for yourself, that you wrote up your experience. But in general I feel like there's a huge missing middle of operations & sys-admin-ery that creates a distorted weird narrative. Th…
This is sort of a ramble, so I apologize in advance.
I love the idea of writing up my ultimately-successful experiences of using open source software. I'm currently working on a big (for me anyway) project for my homelab involving a bunch of stuff I've either never or rarely done before. But... if I were to write about my experiences, a lot of it would be "I'm an idiot and I spent two hours with a valid but bad config because I misunderstood what the documentation was telling me about the syntax and yeah, I learned a bit more about reading the log file for X, but that was fundamentally pointless because it didn't really answer the question." I'd also have to keep track of what I did that didn't work, which adds a lot more work than just keeping track of what did work.
There's also a social aspect there where I don't want to necessarily reveal the precise nature of my idiocy to strangers over the internet. This might be the whole thing here for a lot of people. "Look at this awesome script I made because I'm a rugged and capable individualist" is probably an easier self-sell than "Despite my best efforts, I managed to scrounge together a system that works using pieces made by people smarter than me."
I think I might try. My main concern is whether it will ruin the fun. When I set up Prometheus, I had a lot of fun, even through the mistakes. But, would also trying to write about it make it less fun? Would other people even be interested in a haphazard floundering equivalent to reading about someone's experience with a homework assignment? Would I learn more? Would the frustrating moments be worse or would the process of thinking through things (because I am going to write about it) lead to my mistakes becoming apparent earlier? Will my ego survive people judging my process, conclusions, and writing? I don't know. Maybe it'll be fun to find out.
Re: Monitoring My Homelab, Simply
#30I’m using node exporter to Prometheus and grafana. I also use uptime kuma, and send alerts via pushover.
It’s shockingly easy to setup. I have the monitoring stack living on a GCP host that I have setup for various things and have it connected via tailscale.
It actually paid for itself by alerting me to low voltage events via NUT. I probably would have lost some gear to poor electrical conditions.