Live data from Hacker News

Rebuilding my homelab: Suffering as a service

xeiaso.net

41–50 of 65 posts

Re: Rebuilding my homelab: Suffering as a service

#41
post #31

> I ran a poll on Mastodon to see what people wanted me to do. The results were overwhelmingly in favor of Rocky Linux. As an online "content creator", who am I to not give the people what they want? Truman would be proud! > the NAS. It has all of our media and backups on it. It runs Plex and a few other services, mostly managed by docker compose. Does the NAS run NixOS?

Yes, it runs NixOS and I am too cowardly to bother changing that any time soon. It's got everything on a giant ZFS array and most distros have poor ZFS support.

> and most distros have poor ZFS support.

This is the complete opposite of my experience. For most "server" style distros (IE not arch/arch derivatives) you just install the zfs modules and forget about it. Ubuntu even has them pre baked on their kernel.

Arch gets complicated because it's a rolling release so the zfs module supported kernel versions gets out of sync with the latest kernel version which can prevent system updates due to irresolvable requirements which can go on for days/weeks as they play catch-up with eachother but you can just install the lts kernel and mostly avoid that issue.

For (mostly) every other distro they are a lot more coordinated in their releases so the zfs module will work with the newer kernel without any issues.

Other than that I can't think of what poor support even means, once it's installed it works. You can even have zfs on root for most distros.

Re: Rebuilding my homelab: Suffering as a service

#42
post #10

This was a tremendous write-up. I appreciate the detail including your ingressd setup. I agree, though, that this is a pain. It's for this reason that we made Cloud Seeder [1] so you can have hands-free setup of your homelab and IPv6rs for painless ingress [2] [1] https://github.com/ipv6rslimited/cloudseeder [2] https://ipv6.rs

I didn't know about IPv6rs, but thanks for supporting raw WireGuard configs[1]! Being able to use just WireGuard without having to install any additional daemons or wrappers is always appreciated.

If only I had known about this service before AWS launched its eu-south-2 region (Spain), I would have seriously considered it. But unfortunately I already have my stuff all setup and working, tunneling through an EC2 instance.

Still, bookmarked. I won't say I'll be using it in the future, but I'll definitely keep it in mind for the next time I need to change/update my homeserver stuff.

[1]: https://ipv6.rs/raw

Re: Rebuilding my homelab: Suffering as a service

#43
post #19

After running NixOS for 6+ months on my homelab and also re-using part of the configuration on my work machine, I feel the same way as Xe each time I'm interacting with a non-declarative OS. There's just no simple way to share configuration between machines or to automagically clean things up after making changes. Ansible feels like a thin layer of ice upon a deep ocean of the OS state, hiding in a multitude of non-t…

I’m picking this nit: > When is a build reproducible? > “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” > Neither Nix or NixOS gives you these guarantees. This really makes me question whether all of the quirkiness of Nix is worth it if it can’t actually “pay off” with true reproducibility…

Nonetheless, Nix/NixOS is more reproducible than the majority of other build systems and distros out of the box. But yes, if this is a hard requirement, you’ll be better off with a different choice.

Keep in mind that this is but one of the features NixOS provides. I would say the config-driven approach to OS management is extremely powerful.

As an example, I could bring up my homelab’s external reverse proxy on a generic VPS in a few minutes over SSH using a single command. This includes SSH keys, Telegraf, Nginx with LetsEncrypt certs, and automatic OS upgrades. No Ansible needed :)

See: https://github.com/nix-community/nixos-anywhere

Re: Rebuilding my homelab: Suffering as a service

#45

My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.

Isn’t the result of that research Kubernetes?

What do you expect about the new solution that will be better?

Genuine question, btw. While I see how kubernetes can feel overcomplicated, it has always felt as a consequence of how complicated it is to run such a large number of workloads in a scalable and robust manner.

Re: Rebuilding my homelab: Suffering as a service

#46
post #20

The CoreOS diversion was interesting to read. I've been daily driving CoreOS+i3 for the past year (I might be the only one in the world). I thought having a tiny immutable base OS would make the system easier to manage over time, but unfortunately that hasn't been the case. It's been an adventure but I'm ready to give up and switch to something more vanilla.

I've wondered if that would be possible for a while, but I didn't imagine anyone would actually do that. What are the upsides and downsides of doing this? I'd love to read a writeup of how you did that and what you'll miss when you move away.

I'm no blogger but here's a quick writeup.

# Setup

Setup was a process, no clicking through a nice UI for this one. I had to set up a web server on a second machine to serve the ignition yaml to the primary machine.

It was a very manual process despite CoreOS's promise of automation. There were many issues like https://github.com/coreos/fedora-coreos-tracker/issues/155 where the things I wanted to configure were just not configurable. I had some well-rehearsed post-setup steps to rename the default user from "core" to my name, set keyboard layout, move system dirs to a set of manually-created btrfs subvolumes, etc.

# Usage

The desktop and GUI worked flawlessly. All I had to do was install i3 and lightdm via rpm-ostree. Zero issues, including light 2D gaming like Terraria.

Audio was a pain. My speakers are fine. My mic worked out of the box in ALSA, but Pipewire didn't detect it for some reason, so I had to write some manual pipewire config to add it manually. Also, I had to learn what ALSA and Pipewire are...

I ran just about everything, including GUI apps, in distrobox/arch containers. This was very nice: Arch breaks itself during updates somewhat often and when that happens I can just blow the container away and install pkglist.txt and be back in 5 minutes. I get the benefits of Arch (super fast updates) without the downsides (upgrade brittleness). I plan on keeping distrobox even once I leave.

# Updates

I disabled Zincati (the unattended update service) and instead I ran `rpm-ostree upgrade` before my weekly reboots.

This is the reason I'm leaving. This was supposed to be the smoothest part of CoreOS, but those upgrades failed several times in the past year. To CoreOS's credit my system was never unbootable, but when the upgrades failed I had to do surgery using the unfamiliar rpm-ostree and its lower level ostree to get the system updating again. As of now it's broken yet again and I'm falling behind on updates. I could solve this, I've done it before! But I've had enough. I'm shuffling files to my NAS right now and preparing to hop distros. If anyone wants to try to sell me on NixOS, now's the time ;)

Re: Rebuilding my homelab: Suffering as a service

#47

Earlier quoted context omitted.

> reliability far exceeds the Raspberry Pis In which ways? Most commenters say the exact opposite, that cheap N100 mini PCs are less reliable than Raspberry Pis. I'm just now trying to decide which way to go. Raspberry Pi 5 is definitely much much more interesting from the nerdy point of view, but would cost about the same as some cheap N100, for half the power. Though half the electricity usage too.

If you get away with a 1/2 GB RAM version I'd say the Pi 5 is worth it, otherwise cheap mini PCs are better. I've found mini PCs to be more reliable than the Pi's I've owned as well (including the 5), though a lot of that is mitigated if you go for an m.2 hat on the Pi. In general though, if you're thinking of using a Pi more like a regular computer there isn't really anything special about it that makes it worthwhil…

My use case is Home Assistant, Pi-hole and such things. I would go for the 8 GB model and the NVMe hat, ending up somewhere around 170-180€. So practically the same price as N100.

With RPi I do like the tinkering aspect, an actual community, and lower power consumption -- I'd prefer passive cooling but I can't see a way to do that with the NVMe hat.

With N100 I would get double CPU power, RAM and storage. But I don't actually need that extra power, and the concept of buying a cheap pre-built PC is just a lot less appealing than something more nerdy. So I'll probably end up getting the RPi just for the, err, shall we call emotional reasons.

Re: Rebuilding my homelab: Suffering as a service

#48

My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.

This is incredibly popular a take, and this anti-k8s is rapidly upvoted almost every time.

The systemd hate has cooled a bit, but it too functions as a sizable attractor for disdain & accusation hurling. Let's look at one of my favorite excerpts from the article, on systemd:

> Fleet was glorious. It was what made me decide to actually learn how to use systemd in earnest. Before I had just been a "bloat bad so systemd bad" pleb, but once I really dug into the inner workings I ended up really liking it. Everything being composable units that let you build up to what you want instead of having to be an expert in all the ways shell script messes with you is just such a better place to operate from. Not to mention being able to restart multiple units with the same command, define ulimits, and easily create "oneshot" jobs. If you're a "systemd hater", please actually give it a chance before you decry it as "complicated bad lol". Shit's complicated because life is complicated.

Shits complicated because life is complicated. In both cases, having encompassing ways to compose connectivity has created a stable base (starting point to expert/advanced capable) that allowed huge communities to bloom. Rather than every person being out there by ourselves, the same tools work well for all users, the same tools are practiced with the same conventions.

Overarching is key to commonality being possible. You could walk up to my computer and run 'systemd cat' on any service on it, and quickly see how stuff was setup (especially on my computers which make heavy use of environment variables where possible); before every distro and to a sizable degree every single program was launched & configured differently, requires plucking through init scripts to see how or if the init script was modified. But everything has a well defined shape and form in systemd, a huge variety of capabilities for controlling launch characteristics, process isolation, ulimits, user/group privileges, special tmp directories is all provided out of the box in a way that means there's one man page to go to, and that's instantly visible with every option detailed, so we don't have to go spelunking.

The Cloud Native paradigm that Kubernetes practices is a similar work of revelation, offering similar batteries included capabilities. Is it confusing having pods, replicasets, and services? Yes perhaps at first. But it's unparalleled that one just POSTs resources one wants to an API-server and let's the system start & keep that running; this autonomic behavior is incredibly freeing, leaving control loops doing what humans have had to shepherd & maintain themselves for decades; a paradigm break turning human intent directly into consistent running managed systems.

The many abstractions/resource types are warranted, they are separate composable pieces that allow so much. Need to serve on a second port? Easy; new service since the service is separate from the deployment. Why are there so many different types? Because computers are complex, because this is a model of what really is. Maybe we can reshuffle to get different views, but most of that complexity will need to stay around, but perhaps in refactores shapes.

And like systemd, Kubernetes with it's Desired State Management and operators creates a highly visible highly explorable system; any practitioner can walk up to any cluster and start gleaning tons of information from it, can easily see it run.

It's a wrong hearted view to think that simpler is better. We should start with essential complexity & figure out simultaneously a) how to leverage and b) how to cut direct paths through our complex capable systems. We gain more by permitting and enabling than by pruning. We gain my by being capable of working at both big and small scales than we gain by winnowing down/down scoping our use cases. The proof is in the pudding. Today there's hundreds of guides one can go through in an hour to setup & get started running some services on k3s. Today there's a colossal communities of homelab operators sharing helm charts & resources (ex: https://github.com/onedr0p/home-ops), the likes of which has vastly outclassed where we have stood before. Being afraid of & shying away from complexity is a natural response, but i want people to show that they see so many of the underlying simplicities & conceptions that we have gotten from kube that do make things vastly simpler than the wild West untamed world we came from, where there weren't unified patterns of API servers & operators, handling different resources but all alike & consistent. To conquer complexity you must understand it, and I think very few of those with a true view of Kubernetes complexity have the sense that there are massive opportunities for better, for simpler. To me, the mission, the goal, the plan should be to better manage & better package Kubernetes to better onboard & help humans through it, to try to walk people into what these abstractions are for & shine lights on how they all mirror real things computers need to be doing.

(Technical note, Kubernetes typically runs 0 vm's, it runs containers. With notable exceptions being snap-in OCI runtimes like Firecracker and Kata which indeed host pods as vms. Kine relies on containers are far more optimizable; works like Puzzlefs and Composefs CSIs can snap-in to allow vastly more memory-and-storage-efficient filesystems to boot. So many wonderful pluggable/snappable layers; CNI for networking too.)

Re: Rebuilding my homelab: Suffering as a service

#49

So in summary a former NixOS user now use a preconfigured OS dedicated to run one single platform (k8s) and still suffer for having to tinker with everything. It would have been fun or wise if she had gone back to Nix in the end.

> a former NixOS user

Why did the author stop using NixOS? This is the first time I'm hearing about a veteran NixOS user giving up on it.

Re: Rebuilding my homelab: Suffering as a service

#50

Good place to tell people about CKD8S. It acts as an IaC translating TS to K3 Yalm greatly simplifying working in K8. Developers shouldn't be writing YAML.

cdk8s*

The project, afaik, is abandoned, but it does still work quite well. I use it for my homelab: https://github.com/shepherdjerred/servers/tree/main/cdk8s

Post reply on HN