Live data from Hacker News

I don't maintain my homelab, it maintains itself

cleberg.net

51–60 of 85 posts

Re: I don't maintain my homelab, it maintains itself

#51
This is a fantastic article! I completely agree with the author's philosophy. Simple automation can reduce maintenance to nearly zero, and it's incredible how much can be achieved with just a few well-crafted scripts.

I use a nearly identical alias for docker pull to keep my containers updated. To ensure everything stays running smoothly, I've built a lightweight watchdog (a mix of bash scripting and Uptime Kuma/Beszel) that monitors my services and containers and restarts them if they crash. This way, I rarely need to intervene manually.

For critical services (DNS, VPN, git, web search, crawler and mail, etc.), I add an extra layer of redundancy by running them on multiple servers across different locations. If one server fails, the others seamlessly take over. I also use DNS round-robin as a simple but effective way to handle load balancing and failover; no HaProxy, K8, expensive IP Takeover (ARP Spoofing) or BGP Anycast and VRRP/CARP, Proxmox or fancy orchestration tools required. If a node goes down, another watchdog script temporarily removes it from DNS, and traffic shifts to the remaining servers. Most often the services are self-healing. The best part? My deployment and monitoring are fully self-scripted (no Terraform, Ansible or BundleWrap). Moving services to a new server is as easy as running some scripts over SSH. Everything sets itself up automatically. Currently I run my services on 2 Pi's, 2 stratum 1 servers (from centerclick), and 8 VPSs that cost me around $40/month. It's a great example of how a little automation and redundancy can go a long way in keeping things cheap and reliable without unnecessary complexity.

I invest around 1-2h/month to maintain and (mainly) adjust my setup. Before I head multiple Proxmox instances and a backup server that cost me around $250/month, I was spending 1-2h/week just to keep everything running. The difference is night and day.

However, I've personally had bad experiences with consumer hardware like the Raspberry Pi and hardware failures. Most of the time, I didn't feel motivated to replace the hardware and set up all the services again (even if I had a backup). As an Unify alternative i can recommand GL-iNET; build modern hardware for OpenWRT with some additions and the hardware has enough power to run Wifi7, AdGuard and Tailscale or ZeroTier. (Before I run Protectli Vaults with a virtual PfSense, Tailscale and AdGuard on Proxmox and extra OpenWRT access points) I can recommand the Protectli Hardware over a Raspberry Pi, especially if you want to run a single server/hardware homelab.

Thanks for the inspiration; it's always refreshing to see others embracing simplicity!

Re: I don't maintain my homelab, it maintains itself

#52
This is surprising to me and the exact opposite of what I want for a few reasons:

1. I don't like surprise breakages. I am not prepared to fix a service my family uses midday on a Tuesday when I am working since it auto updated. I'd like to specifically make sure I have dedicated time and plan if something is going to go wrong.

2. My family HATES when things change. I try to run LTS versions of things, but annoyingly, some software like nextcloud doesn't have LTS version. One of the things my family likes the most, is that the stuff I host isn't constantly changing like commercial products. Having google photos change or netflix have a new interface randomly is very, very frustrating for them.

Since my homelab is completely internal, I avoid quickly doing updates (unless it is a critical security issue), and definitely avoid doing major version upgrades unless there is good value in it.

Re: I don't maintain my homelab, it maintains itself

#53
post #33

I thought this was going towards the "I have an agent do it". glad it didn't :) What this skips though is the complexity of services like NextCloud (stuck in maintenance mode again?), Immich (needs a compose file edit?), MineCraft worlds (Dad! my client is on another version again!), (dmn) AlbyHub (needs re-login and closed its channel). But to be fair this is really getting quite minimal these days indeed. I didn't…

Accidentally production.

Re: I don't maintain my homelab, it maintains itself

#54

> UniFi supports automatic and scheduled updates, Yeah, right until the moment it bricks after an update.

I've had automatic updates on for a decade without issue.

Good for you. That doesn't necessarily mean your experience is representative enough though.

Unifi stronk. Noone needs working ipv6 or 2+ gigabit pppoe throughput and many other things, like an ability to assign a name to an entry in embedded radius server.

Re: I don't maintain my homelab, it maintains itself

#55

I suspect my approach is even more controversial… I just open Claude code and type /routine-maintenance and it reads the skill file, logs into all my systems on my home network and runs updates, validate backups are still healthy, update any docker images, checks SMART stats, reviews some logs, and then fires off an email using brevo to tell me any future maintenance concerns I might have. Edit: zero minutes old alre…

Manual type? No cron job? Practically Luddite

Yeah lol I considered it but i hit my token max so often that I didn’t want to be deep in a development cycle and suddenly have to wait 4 more hours because my maintenance chat boiled over.

So keeping the trigger manual at least gives me the ability to skip it while I work.

Re: I don't maintain my homelab, it maintains itself

#57
post #6

> I've approximated it somewhere around 15 minutes of maintenance per month, barring an emergency. If that's normal to you, congrats - you've peaked in life. However, that's absolutely absurd to me. I used to spend days on end building, maintaining, and debugging various aspects of my servers, databases, apps, etc. It's been normal for me for the past 3 years thanks to using NixOS for all server infrastructure.

The same with NixOS and in production. Even less than 15 minutes month.

Re: I don't maintain my homelab, it maintains itself

#58
post #47
post #20

I wrote a small agent (single go binary) that does all the monitoring and maintenance for me. Possibly overkill but it is amusing to think there is a little ghost in the machine.

please elaborate

It is just a small agent using a remote harness, so the local dependencies are limited. It is specialised in the sense that the prompt, tools, and skills are all custom and baked into the binary, including the key to the harness API. I don't consider it sensitive in this context because the key is scoped.

The agent has a single mission to maintain the system it is dropped into. It has its own in-process heartbeat and is launched as a service.

As I said above, it is kind of amusing to think that it exists at all. It checks whether things are running fine and can possibly correct them if it finds something wrong.

I can see this becoming a more serious kind of system agent.

I try not to advertise in this forum but I can drop a link if interested.

Re: I don't maintain my homelab, it maintains itself

#59
post #33

I thought this was going towards the "I have an agent do it". glad it didn't :) What this skips though is the complexity of services like NextCloud (stuck in maintenance mode again?), Immich (needs a compose file edit?), MineCraft worlds (Dad! my client is on another version again!), (dmn) AlbyHub (needs re-login and closed its channel). But to be fair this is really getting quite minimal these days indeed. I didn't…

> NextCloud (stuck in maintenance mode again?)

The frustrating thing is that basically everyone who self hosts Nextcloud will know exactly what you're talking about, and the solution is generally quite simple, and yet the docs for this problem are terrible, scattered, out of date, and mostly just link to old GitHub issues that don't directly address the problem until you get to some comment half way down.

It's a terrible user experience - seriously, point an LLM at it for a few hours to write the first draft of a "So you're stuck in maintenance mode" page, get it reviewed by the community, publish. Immediate quality of life improvement for everyone struggling to get started self hosting it.

Re: I don't maintain my homelab, it maintains itself

#60

I've had "servers" or a "homelab" at home for de3cades. I stopped a while ago when I burned out. About 4 month ago, I bought a new motherboard and graphics card for my desktop and dropped the old ones into a $70 case I got from Best Buy and put Ubuntu on it. I think I spent 10x that on memory for my new desktop, but that's just a passing grumble. The new server now runs transcription and embeddings for me on the old…

I got sick of the work around it, so one weekend recently, while my partner was out of town, I used Claude to help re-architect my whole homelab to be effectively 100% IaC with OpenTofu and Salt. Went through the whole architecture, networking, etc. Now I would just need to pull down the repo to my Mac and can spin everything up via PXE to install Proxmox, Ubuntu, etc., and then I have a scheduled golden image build, this can spin up my network devices with a bunch of VLANs and east-west firewalls, Kea and Unbound, even more esoteric things like a VM that passes through a USB ADS-B receiver. It spins up Talos to drive a Kubernetes cluster and everything from there. The only thing it can't (or doesn't?) manage is my Synology servers.
Post reply on HN