Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

petekeen.net

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#51

Post by wannabe44 »

MisterKent wrote: post #7

Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup. Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem. If anything, it helped keep my stuff alive during all the hardware issues a lot longer. I think like 5-6 years a…

> All the enterprisey stuff makes it feel a lot more complex than it really is.

Kubernetes is not PaaS. It is a low level container orchestrator. But people start by thinking of it as a ready-made PaaS and then keep on trying to make it closer to that vision by continually slapping random plugins on it.

In corporate environments, incompetence is the norm. Every kubernetes deployment in such places ends up relying on rando docker images, helm charts and "operators" written by script kiddies and webshits which break every night and are hardly better than coding the same functionality yourself.

You need a strong instinct to say "No" and taste to keep the system complexity under reasonable limits. Sometimes this involves actually architecting your "platform" code around the barebones orchestration provided by kubernetes instead of slapping on latest webshit shown on CNCF dot org. That sort of judgement and taste is impossible to find in corporate environments in the age of resume driven development. That is how most people get Kubernetes scars.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#52

Post by drusklo »

The problem with using k8s on homelabs, is that a lot of the applications you would usually deploy, are not designed for it; having to manage a bunch of persistent volumes because most of your applications use sqlite is not very practical, and if the backend is sqlite, then you are probably running only one pod, so no real HA (if the pod goes down k8s will start a new one though), if you have to go through hoops to deploy an application that's not designed for it decreases its value.

Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#53

Post by eqvinox »

preisschild wrote: post #31
eqvinox wrote: post #6

> Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble. Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works…

I enjoy using Kubernetes on bare metal nodes at home and took it over Proxmox, quite happy with it (not so happy with costs of hardware to expand the cluster though)

If that works for you, great!

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#54

Post by zrail »

lloydatkinson wrote: post #21
zrail wrote: post #2

Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes". IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress. I'm curious…

Did you ever consider NixOS for your homelab?

I haven't, mostly for personal reasons rather than technical ones.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#55

Post by zrail »

eqvinox wrote: post #53
preisschild wrote: post #31

Earlier quoted context omitted.

I enjoy using Kubernetes on bare metal nodes at home and took it over Proxmox, quite happy with it (not so happy with costs of hardware to expand the cluster though)

If that works for you, great!

Agree! There are a ton of people out there that are happily running k8s on their homelab and there is absolutely nothing wrong with that. I tried it and found it was just too much for me.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#56

Post by zrail »

wsdn wrote: post #19

The deterministic IPv6 approach is fascinating. Have you run into any issues with ISP-provided IPv6 prefix delegation changing over time, or are you strictly using ULAs (Unique Local Addresses) internally to bypass that headache?

Hey thanks! That's my favorite recent addition.

The host machines resolve a GUA from an ISP-provided prefix like normal. On top of that, I l33t'd my way to a fun ULA prefix. Each host's prefix, the stack network prefixes, and container addresses are deterministically derived from hashes of each component's name.

As of today I haven't made any of the ULA stuff routable, although that was the original impetus for it. Clients see each host's GUA in DNS and generally route traffic through Caddy. Cross-host traffic is almost entirely through Tailscale. I run DNS refreshes at deploy time and hourly, but I haven't dealt with Comcast rotating my PD assignment yet. It's been stable for a couple years so I'm probably due.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#57

Post by zrail »

MisterKent wrote: post #7

Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup. Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem. If anything, it helped keep my stuff alive during all the hardware issues a lot longer. I think like 5-6 years a…

Sure! That's the setup I was running before all of this. My problems are likely entirely my own making, tbh, as described in the article.

IME K8s in general and longhorn especially behaved poorly in the face of heterogeneous hardware where the largest node is flaky and has more cores and memory than the rest of the nodes combined.

My point is that, for me, in my setup, with my own idiosyncratic constrains, trying to keep up with Talos + longhorn + fluxcd expended way too many spoons in any given month.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#58

Post by zrail »

maccard wrote: post #32
SMFloris wrote: post #5

IMO, kubernetes is overkill for a small non-homogeneous home cluster. What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign l…

I disagree - I think k8s is absolutely the right choice at that level. We use ECS instead of k8s for small projects because the control plane is more than the actual service, but when you want container management, some sort of basic deployment management, service discovery, storage management and secret management, k8s is super simple to run and works pretty much out of the box. For a homelab, I'd expect you to want…

> I'd expect you to want this

The thing is, I don't want any of that. At least, I don't want any of it to be dynamic. For my purposes the more I can determine at deploy time and make static the better.

K8s is a stack of dozens-to-hundreds of control loops that are built around the idiosyncrasies of a particular distributed key-value store that is only moderately fit for purpose. This is all great fun if you want to learn about it (I did!) and/or if you get to be a user rather than an operator.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#59

Post by chuckadams »

catdog wrote: post #28
pelasaco wrote: post #26

I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens…

On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.

K8s uses watchers on etcd, not polling, so if there's no changes, there should be no activity. But just collecting metrics like CPU usage is probably a watcher on constantly changing data, like every second or so. So your control nodes are likely to stay warm most of the time, but there's also things like MicroK8s, which is made so you can stop the control plane and the worker containers keep running.

Re: Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

#60

Post by MisterKent »

gbalduzzi wrote: post #24
MisterKent wrote: post #7

Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup. Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem. If anything, it helped keep my stuff alive during all the hardware issues a lot longer. I think like 5-6 years a…

Did you already know kubernetes, or did you have to learn it for this setup? I believe this is the key difference, because it doesn't make sense to learn such a complicated tool for a simple use case. If you already know it sure, it may make sense. But you are removing a huge initial effort required otherwise.

I learned it for this setup. I do zero devops at work, there's a separate team that handles that (and even then, they're not on kubernetes).

I learned it because it seemed like the right way to do things and fun. Cut myself a few times in the earliest days having no clue, but no regrets.

Post reply on HN