[deleted]
Monitoring My Homelab, Simply
31–40 of 70 posts
Re: Monitoring My Homelab, Simply
#32Uptime Kuma ( https://github.com/louislam/uptime-kuma ). With email notifications. So much simpler, and free.
Love this - tried it. The problem as I see it is that these still require hosting - ideally (again, as I see it) self-hosting a script that monitors internal/homelab things also requires its own monitoring.
Short of paying for a service (which somewhat goes against the grain of trying to host all your own stuff), the closest I can come up with is relying on a service outside your network that has access to your network (via a tunnel/vpn).
Given a lot of my own networking set-up (DNS/Domains/Tunnels etc) are already managed via Cloudflare, I'm thinking that using some compute at that layer to provide a monitoring service. Probably something to throw next at my new LLM developer...
Re: Monitoring My Homelab, Simply
#33I appreciate the "How to monitor the monitor?" section. Always need a meta-monitor! :)
Hope you might give us a try at https://heiioncall.com/ and let me know if that fits. (Disclosure: our team is building and operating it as a simple monitoring, alerting, on-call rotations solution.) We have the cron job heartbeats, HTTP healthchecks, SSL certificate expiration, etc etc all in one simple package. With mobile app alerts (critical and non-critical), business hours rules, etc. And a free tier for homelabbers / solo projects / etc. :)
Edit: since you mentioned silencing things in your post, we also have very flexible "silence" buttons, which can set silence at various levels of the hierarchy, and can do so with a predefined timer. So if you know you want things to be silenced because you're fixing them, you can click one button and silence that trigger or group of triggers for 24 hours -- and it'll automatically unsilence at that time -- so you don't have to remember to manually manage silence/unsilence status!
Re: Monitoring My Homelab, Simply
#34> So I have two dead man switch’s.
I am reminded of an aphorism about having a problem and deciding to use regex.
> Historical data: I’m not chasing down grand mysteries that require fleet-wide aggregate metrics.
Everyone believes this .. until it isn't true, and then you find yourself needing logs from the last two weeks.
For home labs, log aggregation is an easy problem to deal with these days, and a secure sink to send all your logs to has (potentially) more than one benefit.
Anecdote - I've just been tracking some unpleasant FLUSH CACHE EXT errors on my aging pre-owned Xeon box, and having an understanding of frequency / distribution of those errors on the hypervisor, but also correlation with different but related errors presenting in the VMs, was a) very useful, b) not something I'd have predicted I'd need before hand.
Re: Monitoring My Homelab, Simply
#35ok.. so your solution is using at minimum a $5/month service. Yikes, I'd prefer something like pushover before that. :/
You can self-host ntfy.sh but then you need to find a place outside of your infra to host it.
I have AWS based services to monitor (servers/websites/etc) and use my homelab system to monitor resources that I am interested in.
I just use a simple script that is run every 60 seconds and a list of resources to check.
Re: Monitoring My Homelab, Simply
#36Honest question, what the hell is everyone monitoring in a home lab that isn't already monitored?
I have an enterprise grade NAS, but if there's any kind of disk or RAID issue it beeps the shit out of me; I call that enough for home use.
I have a Unifi router, if there is a connection issue it fails over to LTE and I get a notification on my phone.
I have a UPS, if there is a power failure, my lights shut off, my NAS and workstation shuts down via NUT, and I can restart them remotely via VPN into my router and sending WOL packets.
Basically everything is already taken care of.
What the hell else do I need for a home? When I'm away I don't exactly have 10 million users trying to access my system, let alone 1.
Re: Monitoring My Homelab, Simply
#37Honest question, what the hell is everyone monitoring in a home lab that isn't already monitored? I have an enterprise grade NAS, but if there's any kind of disk or RAID issue it beeps the shit out of me; I call that enough for home use. I have a Unifi router, if there is a connection issue it fails over to LTE and I get a notification on my phone. I have a UPS, if there is a power failure, my lights shut off, my NAS…
[dead]
Re: Monitoring My Homelab, Simply
#38My solution is to just be OK with http status checking (run a webserver on important machines), and use a service like updown.io which is so cheap it's almost free. e.g. For 1 machine, hourly checking is ~$0.25/year
Do you do regular backups? If your backup system breaks and stop making new backups, what will let you know? What if your RAID is failing, running out of space, remounted read-only after an error?
I have found that "machine is online" is usually not what I need monitoring for, at all. I'll notice if it's down. It's all the mission-critical-but-silently-breakables that I bother to monitor.
Re: Monitoring My Homelab, Simply
#39My solution is to just be OK with http status checking (run a webserver on important machines), and use a service like updown.io which is so cheap it's almost free. e.g. For 1 machine, hourly checking is ~$0.25/year
Do you do regular backups? If your backup system breaks and stop making new backups, what will let you know? What if your RAID is failing, running out of space, remounted read-only after an error? I have found that "machine is online" is usually not what I need monitoring for, at all. I'll notice if it's down. It's all the mission-critical-but-silently-breakables that I bother to monitor.
Not OP, https://healthchecks.io is great for monitoring automated tasks like backup scripts. Also has the option to immediately signal failure and send an alert: https://healthchecks.io/docs/signaling_failures/
Re: Monitoring My Homelab, Simply
#40This resembles how I monitor all infrastructure I run. One of them has 150 small independent VMs, for which I had to build a custom micro service monitoring open source tool that I still use to this day: https://github.com/valeriansaliou/vigil
There’s no certificate expiration monitoring just yet, but everything else is there: poll probes (active ICMP or TCP probes), push probes (reporting HTTP API for apps), and local probes (reporting HTTP API for sub-Vigil for firewalled infrastructure parts).