Live data from Hacker News

Domesticating Kubernetes

blog.quickbird.uk

61–70 of 100 posts

Re: Domesticating Kubernetes

#61
Why would you do this for simple home apps? k8s is complete overkill. At home, there is little reason to drag in all that complexity to host a blog or whatever.

Kubernetes drags in a tremendous amount of complexity, background knowledge, and strange constraints, even on small installations. I really do not see its benefit for small apps and projects.

Are you hosting your blog? You don't need k8s. Are you running a small app server at home? Nope, don't need it. Are you running an auto-scaling app with many hundreds or thousands of worker instances, and need to support non-disruptive roll-in upgrades across multiple clouds and have many services which need to scale independently? Maybe you need k8s, but only in some circumstances.

For the record, I run many k8s clusters across several clouds, using 1,000 - 50,000 cores at any one time, so I've dealt with quite a bit of k8s complexity, and I'm still on the fence whether it's the right answer for us, but it's allowed us to standardize our software to k8s and only worry about getting that running well in each deployment, which put the cross-cloud work on the infrastructure teams versus the software development teams. The price you pay for this is that you still need to do non-k8s stuff if you need some cloud specific resources more than simple compute and routing, so you have both k8s and cloud specific code.

Re: Domesticating Kubernetes

#62
I recently gave up on home network clustering/kubernetes. I've since moved to digital ocean. I was working on data heavy apps for testing. I had several SSD zfs pools which was the driving reason, and a lot of RAM to work with.

My setup did work initially. I had a dedicated server acting as the metal lb, about 20$ a month. That was then connected through a wireguard tunnel to my home dmz network. Which backed to a dual xeon v2 work station. The latency was very good under 20ms, and really good speeds. I'm lucky to have fios in my area.

The Xeon workstation died, I then backed up to several think pads. Those were not performant either. So I got several HP t610 nodes, raspberry pi speed with sata 3. Rook took all CPU, and to boot they were running at 90c constantly. Even after re paste and with fan mods. I didn't want five little space heaters next to my desk.

After all this I ditched the home setup. I had gotten parts for a local Epyc server after my xeon's died. But sold them due the current situation.

In the end I had wanted to start a series of blogs on Kubernetes and micro service development. To help me learn and flesh out my under standing of Kubernetes.

I don't feel I wasted several months setting up Kube. I now know about under the cover stuff. Having deployed OkD, Rancher, kubespray, and kube adm. The initial wire guard set up helped cement a lot of the internal networking model. I was mildly acquainted with having worked on openvswitch and open stack before.

If you're doing local. I would really recommend a Ryzen 1600af (6c/12t zen+ 65w), either asrock rack x470, or a basic b450. Both can take ECC, and should land a little node under 500ish or so. There are also SFF PCs Lenovo ThinkCentre M93, comes to mind. But at that price 90$ a node, I'd rather move up the stack a little bit.

If you're waiting to buy local. DigitalOcean has been very well priced. If you want to learn the internals, grab a dedicated server and set up KVM, to get familiar with the internals. On the upper end you can get a new ryzen 3rd gen dedicated for 90$ a month. I look at it as a 90$ class I take once.

I'm not saying Kube is always right. But I avoided it for a long time, backing up to docker swarm. Now, I feel at a base it's not much more complicated than docker swarm, now that I've done this deep dive. Kube adm is on ease of use with docker swarm to me. Add in Metal LB, and Gitlab to help manage the cluster, you have a personal little cloud. It's also good to know for future job searches.

Re: Domesticating Kubernetes

#63

I think what type of k8s environment you use very much depends on what you're looking to get out of it. If it's experience deploying applications into containerized environments, then micro-k8s and k3s seem like reasonable choices, you don't really care about the setup of the underlying components, just that they present the k8s API. If you're looking for experience of managing k8s clusters, then either the distribut…

I think you are right on the money - the effort of hosting it at home is only worthwhile from educational standpoint I do try to keep things as simple as possible so that I could re-deploy the cluster quickly. Also learning to debug issues in kubernetes is a skill in it's own right.

Regarding Kubeadm - some storage solutions don't even work on K3S, or at least i don't see how you can make them work. For example EdgeFS Rook integrated CSI driver requires you to deal with feature gates.

Re: Domesticating Kubernetes

#64
All of these trap your mind in the k8s spiral arm and watch how adroit I am at building something no one would want to troubleshoot articles are hipster reminders of why I only create occasional throwaways for this site.

Re: Domesticating Kubernetes

#65
post #32
post #2

I can attest to RPI i/o speed: its horrible. Combined with the fact that I have to build Docker containers on the RPIs themselves(because of arm), its more of a hassle than a cool add-on (to be clear I'm running Docker Swarm in a way that isn't too different from the setup of OP). Only reason I can see RPIs in kubernetes is if you're exclusively using ARM everywhere and/or are running some distributed cluster among d…

Is there an RPi alternative (ie. similar size, form etc.) that has better IO/Network speed? I used RPis for my robot ( https://sendc.at/dl/Kjjbt3dij6T733YgWsyv3p3Vv5GPGKO1q5IEFStV... ) with daughter boards for motor control) and they wre pretty convenient, but never really considered running anything server-like on them because they have reportedly woeful network performance.

The pi 4 has gigabit ethernet. I'm not sure why you'd think it has "woeful network performance". The 3B+ could only do a couple hundred megabit, which was somewhat mediocre, though.

Re: Domesticating Kubernetes

#66
post #4

What I miss from all of these tutorials is one very important piece - how to handle network routing and dns automation within your home network, that's in typical scenario is being handled by the ingress/cloud controller. Without having automated (or easy enough) way of reaching the apps you're deploying there, each of these clusters is pretty much useless for users except for maybe learning basics of k8s, what's eas…

Thanks for pointing it out - I tried to cover this area and drew a diagram, but on second through, I don't think I managed to do it justice.

I have a static LAN IP for an Ingress Controller, say 192.168.2.100. All HTTP/HTTPS requests are port-forwarded to it from my router. From there, the Ingress is in charge of directing the request based on domain name.

As for DNS, I use a single domain name, and have a record for a wildcard subdomain - so any subdomain will end up at my router, I don't have to configure anything at my DNS registrar when I add yet another application, as long as it's using a subdomain.

ExternalDNS is a superior solution, but most people will only have 1 or 2 domain names.

Re: Domesticating Kubernetes

#67

Why would you do this for simple home apps? k8s is complete overkill. At home, there is little reason to drag in all that complexity to host a blog or whatever. Kubernetes drags in a tremendous amount of complexity, background knowledge, and strange constraints, even on small installations. I really do not see its benefit for small apps and projects. Are you hosting your blog? You don't need k8s. Are you running a sm…

Learning?

Re: Domesticating Kubernetes

#68
post #46
post #40

I've given up running my own k8s at home a couple of times. It does seem to be getting easier, but then upgrading and maintaining breaks me again. + the other stuff like running your own container registry. Then with new centos podman breaks everything that worked OK in docker. I hate this stuff, so many problems its worth ignoring the whole stack.

what do you end up using instead of k8s?

I mostly use docker compose. Perfect fit for tinkering but not over complicated and huge ram eater.

Also:k3s might be an option.

Re: Domesticating Kubernetes

#69

Why would you do this for simple home apps? k8s is complete overkill. At home, there is little reason to drag in all that complexity to host a blog or whatever. Kubernetes drags in a tremendous amount of complexity, background knowledge, and strange constraints, even on small installations. I really do not see its benefit for small apps and projects. Are you hosting your blog? You don't need k8s. Are you running a sm…

I'll second you. I full time admin our K8s infrastructure and clusters. It's a great solution for our production workloads and I've bootstrapped RancherOS/k3s a handful of times to look at using it at home but K8s is purpose built for cloud. I can get all the same benefits with significantly reduced complexity with Docker + Systemd at home.

Re: Domesticating Kubernetes

#70
post #65
post #32

Earlier quoted context omitted.

Is there an RPi alternative (ie. similar size, form etc.) that has better IO/Network speed? I used RPis for my robot ( https://sendc.at/dl/Kjjbt3dij6T733YgWsyv3p3Vv5GPGKO1q5IEFStV... ) with daughter boards for motor control) and they wre pretty convenient, but never really considered running anything server-like on them because they have reportedly woeful network performance.

The pi 4 has gigabit ethernet. I'm not sure why you'd think it has "woeful network performance". The 3B+ could only do a couple hundred megabit, which was somewhat mediocre, though.

If they've fixed it with the 4 then that's great, ISTR in previous versions the Ethernet was backed by the shared USB controller (https://raspberrypi.stackexchange.com/questions/45130/why-do...), hence my question.
Post reply on HN