My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.
Rebuilding my homelab: Suffering as a service
11–20 of 65 posts
Re: Rebuilding my homelab: Suffering as a service
#12My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.
I really hope that is the case too, but for now Kubernetes sucked all the oxygen out of the room for everything else :(
Re: Rebuilding my homelab: Suffering as a service
#13No. Real suffering in the homelab is getting N 20 year old servers and swapping parts between them to get N-M servers that work. I feel like the project will be successful if I get all the drives wiped and I am within site of that although I discovered the wiping was going to be a process of triage: some drives did not spin up, one drive took 14 hours to wipe whereas a normal drive would take about 30 minutes. My col…
That is retrocomputing, not homelabbing. Look into the sub-$200 Intel N100 based systems with 16GB RAM.
Re: Rebuilding my homelab: Suffering as a service
#14After running NixOS for 6+ months on my homelab and also re-using part of the configuration on my work machine, I feel the same way as Xe each time I'm interacting with a non-declarative OS. There's just no simple way to share configuration between machines or to automagically clean things up after making changes.
Ansible feels like a thin layer of ice upon a deep ocean of the OS state, hiding in a multitude of non-tracked configuration files. It is simply not enough to build a layer of YAML around an OS which is imperative by nature.
Unfortunately, I can see the downsides of NixOS as well, being radically different from what we usually expect in a Linux distribution, adopting it in a already established environment will no doubt be hard. Steep learning curve, idiosyncracies of the Nix language (although after reading parts of the Nix thesis[1], I find it much more understandable and deeply thought out), just explaining Nix to people who don't have much experience with the functional way of doing things, let alone taking the functional approach all the way to defining an entire operating system - all of this sounds like a tough barrier to cross.
And yet, the desire to keep things reproducible and declarative (not to mention going back in time) persists once you've had the taste of NixOS.
Re: Rebuilding my homelab: Suffering as a service
#15This was a tremendous write-up. I appreciate the detail including your ingressd setup. I agree, though, that this is a pain. It's for this reason that we made Cloud Seeder [1] so you can have hands-free setup of your homelab and IPv6rs for painless ingress [2] [1] https://github.com/ipv6rslimited/cloudseeder [2] https://ipv6.rs
I'd set up IPv6 on there, but the problem is that I use Flannel (which is IPv4 only) and my 8 gigabit fiber ISP only gives me IPv4 connectivity. I'll look into more details, but I've slightly given up on IPv6 for now. Maybe I'll set up Calico or something, but IPv6 seems to have been made artificially difficult by everything in the stack. I hate it.
Re: Rebuilding my homelab: Suffering as a service
#16The CoreOS diversion was interesting to read. I've been daily driving CoreOS+i3 for the past year (I might be the only one in the world). I thought having a tiny immutable base OS would make the system easier to manage over time, but unfortunately that hasn't been the case. It's been an adventure but I'm ready to give up and switch to something more vanilla.
I've keep running Fedora coreos on my home server. My biggest issue with it, is that it is very cloud oriented and doesn't seem to allow to rerun the provisioning config on an already existing machine. This turns the thing again into a stateful pet instead of a "one cow cattle". Although I do very much like the rollback feature which has allowed me temporarily roll back an update a couple of times
Re: Rebuilding my homelab: Suffering as a service
#17This was a tremendous write-up. I appreciate the detail including your ingressd setup. I agree, though, that this is a pain. It's for this reason that we made Cloud Seeder [1] so you can have hands-free setup of your homelab and IPv6rs for painless ingress [2] [1] https://github.com/ipv6rslimited/cloudseeder [2] https://ipv6.rs
I'd set up IPv6 on there, but the problem is that I use Flannel (which is IPv4 only) and my 8 gigabit fiber ISP only gives me IPv4 connectivity. I'll look into more details, but I've slightly given up on IPv6 for now. Maybe I'll set up Calico or something, but IPv6 seems to have been made artificially difficult by everything in the stack. I hate it.
> 8 gigabit fiber ISP only gives me IPv4 connectivity.
Just like my ISP (but not 8 gbit!). Luckily, IPv6rs actually tunnels through IPv4 (or 6) and provides an IPv6 address. You don't need one to start!
I don't know the ins and outs for flannel, but maybe you could setup IPv4 internally and use IPv6 (and an IPv4 Reverse Proxy) for the public internet?
I agree, though, IPv6 on its own can be hard but thanks to WireGuard, tayga (NAT64), and nginx/caddy/etc. (reverse proxy), it's definitely quite usable!
Re: Rebuilding my homelab: Suffering as a service
#18Earlier quoted context omitted.
I'd set up IPv6 on there, but the problem is that I use Flannel (which is IPv4 only) and my 8 gigabit fiber ISP only gives me IPv4 connectivity. I'll look into more details, but I've slightly given up on IPv6 for now. Maybe I'll set up Calico or something, but IPv6 seems to have been made artificially difficult by everything in the stack. I hate it.
> 8 gigabit fiber ISP only gives me IPv4 connectivity. Just like my ISP (but not 8 gbit!). Luckily, IPv6rs actually tunnels through IPv4 (or 6) and provides an IPv6 address. You don't need one to start! I don't know the ins and outs for flannel, but maybe you could setup IPv4 internally and use IPv6 (and an IPv4 Reverse Proxy) for the public internet? I agree, though, IPv6 on its own can be hard but thanks to WireGua…
Can you email me at hackernews@xeserv.us? I'd like to hear more.
Re: Rebuilding my homelab: Suffering as a service
#19After running NixOS for 6+ months on my homelab and also re-using part of the configuration on my work machine, I feel the same way as Xe each time I'm interacting with a non-declarative OS. There's just no simple way to share configuration between machines or to automagically clean things up after making changes. Ansible feels like a thin layer of ice upon a deep ocean of the OS state, hiding in a multitude of non-t…
I’m picking this nit:
> When is a build reproducible?
> “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.”
> Neither Nix or NixOS gives you these guarantees.
This really makes me question whether all of the quirkiness of Nix is worth it if it can’t actually “pay off” with true reproducibility.
[1] “NixOS is not reproducible (2022) https://linderud.dev/blog/nixos-is-not-reproducible/
[2] “non reproducible issues in NixOS” https://github.com/orgs/NixOS/projects/30
Re: Rebuilding my homelab: Suffering as a service
#20The CoreOS diversion was interesting to read. I've been daily driving CoreOS+i3 for the past year (I might be the only one in the world). I thought having a tiny immutable base OS would make the system easier to manage over time, but unfortunately that hasn't been the case. It's been an adventure but I'm ready to give up and switch to something more vanilla.
I've wondered if that would be possible for a while, but I didn't imagine anyone would actually do that. What are the upsides and downsides of doing this? I'd love to read a writeup of how you did that and what you'll miss when you move away.