Ask HN: Please recommend how to manage personal serverss

The points to consider:

- architecture. You have three boxes. One of them has lots of storage. One of them is cheap. One costs you monthly. I don't know that this is what you actually want. You probably need a main box that can do anything, a backup facility for that, and a proxy to expose services to the outside world.

- a common operating system on all nodes. I like Debian stable. Not everybody does. Being happy with it is more important than being the "best". But you should only have one.

- automatic backup of config and data. Snapshots are nice.

- if you can't have perfect snapshots, you can at least check your config into git. Use etckeeper.

- set up a common approach to running things. Make everything grab TLS certs from Lets Encrypt through nginx. Make a new user for each service. Make a new database user for each service that needs that, make a new PHP worker pool, whatever. Be consistent.

- document your policy and your exceptions. This can be a text file or a wiki or something weird.

- know where you are getting things, how to upgrade them, and how to get announcements of available updates.

My personal wiki has very short notes on how to rebuild each from scratch. (Pretty much: push Linode Web site buttons to make a new Debian Stable instance, get a shell and do this `apt install` command line, and edit config file like so). Data gets pushed/pulled via simple shell scripts run on laptop (usually using SSH and rsync).

Separate from those services, my GPU server is a separate box at home, and frequently changing at a low level, so blasting it entirely a few times has made pragmatic sense, so I'm glad it's not sharing config complexity any other resources. And setting up the large ML stacks down to proprietary drivers sometimes is initially very experimental, and I need to do it manually first anyway, not yet ready to make a Dockerfile or set up passthrough for containers, and after the experiment works, there's no reason to do that. Were I making a production setup, or something reproducible by others, I'd do more after the initial experiment setup.

Wrangling much more complex layers atop (e.g., K8s, Docker, Terraform, Ansible, NixOS, etc.) sometimes means more things that can go wrong, and sometimes more time spent learning someone else's bureaucracy. Most of tech work now is learning piles of other people's bureaucracy. That makes sense for businesses that actually need that complexity, and for people who just want to copy&paste cargo-cult command lines and hopefully it works, and for people who want to homelab it for experience (which is perfectly valid). But the way I run my important services and my experimental box seemed to be easier overall.

Of course, for curiosity/resume/masochism purposes, I do have a separate K8s cluster at home, which runs nothing important, and which I can obliterate and change and experiment with at will, without being encumbered by it running services I actually need.

note, there are two wikis now.