Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Domesticating Kubernetes

blog.quickbird.uk

Post reply on HN
Print view View original

21–30 of 100 posts

Re: Domesticating Kubernetes

#21

Post by erinaceousjones »

charlieegan3 wrote: post #12
rtempaccount1 wrote: post #7

I think what type of k8s environment you use very much depends on what you're looking to get out of it. If it's experience deploying applications into containerized environments, then micro-k8s and k3s seem like reasonable choices, you don't really care about the setup of the underlying components, just that they present the k8s API. If you're looking for experience of managing k8s clusters, then either the distribut…

> if you're looking for something to run your home services long term, I would recommend not using Kubernetes If we disregard one’s experience with Kubernetes as a factor, are there any other reasons you see to not use k8s at home?

I suppose it depends on what your intentions are:

* "I want to run my stuff from home indefinitely, in the background": Don't use kubernetes. You'll spend ages setting up your cluster, you'll be fighting to keep it up. Most things you'll want to run probably have an OS package and run happily as systemd services. It's unnecessary overhead, both on the hardware, and mentally. Internal certificates [used to, at least] expire after a year on kubeadm created clusters, so if you don't look at it occasionally your control plane goes down. It doesn't handle the case where all your machines are nearly at full capacity (i.e. >85% disk, or nearly all memory used) and the defaults are to kick things off nodes with "pressure" which is a huge PITA when you don't have a dynamically scalable cluster - nothing makes my blood boil than seeing 200 lines of "Evicted" in kubectl . It's designed for huge workloads in datacentres, after all. You really don't need it for hosting a single user blog and a NAS.

* "I want to set up a homelab and learn k8s": Definitely use kubernetes. You'll learn how painful and time-consuming it is to manage onprem installs, but you'll also learn a lot. A lot of packaged kubernetes solutions follow the same patterns (ingress controllers like nginx / ambassador / envoy, istio service mesh, flannel etc for VLANs, prometheus for monitoring, helm / argocd deployments, ...) so it's super useful if you need to use kubernetes at work. You'll come to realize just how much awful, awful stupid rolling-release "this was deprecated on Tuesday and all our config schemas have changed" bullshit you're protected from when using someone else's (i.e. BigCorp's cloud solution) managed kubernetes cloud thing, and screaming "NO" at anyone who starts to utter "on-prem" will become ingrained in you.

:-)

Re: Domesticating Kubernetes

#22

Post by detaro »

sandGorgon wrote: post #3

you can run a kubernetes stack using k3s on a rpi stack within minutes. and it runs far better than all of these. https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ here's a live walkthrough - https://www.youtube.com/watch?v=DjpVtNjiXSU

> and it runs far better than all of these.

in what way?

Re: Domesticating Kubernetes

#23

Post by politelemon »

charlieegan3 wrote: post #14
guiriduro wrote: post #10

Or... you could just install a Docker swarm in 10 minutes https://www.bretfisher.com/10-minutes-to-highly-available-do... No good reasons to use K8s outside multi-team enterprise stateful multi-tier architectures.

Kubernetes provides a standard way to deploy applications. If I’m familiar with deploying applications on Kubernetes at work, why not have the same for my side projects at home?

The same can be said of deployment mechanism that one is accustomed to.

GP's point is around time to setup and complexity involved.

Re: Domesticating Kubernetes

#24

Post by globular-toast »

sdan wrote: post #2

I can attest to RPI i/o speed: its horrible. Combined with the fact that I have to build Docker containers on the RPIs themselves(because of arm), its more of a hassle than a cool add-on (to be clear I'm running Docker Swarm in a way that isn't too different from the setup of OP). Only reason I can see RPIs in kubernetes is if you're exclusively using ARM everywhere and/or are running some distributed cluster among d…

Don't build stuff on the Pi! You just need to set up a cross-compiler toolchain and compile it on your much more capable AMD64 machine.

Re: Domesticating Kubernetes

#25

Post by sdan »

GordonS wrote: post #5
sdan wrote: post #2

I can attest to RPI i/o speed: its horrible. Combined with the fact that I have to build Docker containers on the RPIs themselves(because of arm), its more of a hassle than a cool add-on (to be clear I'm running Docker Swarm in a way that isn't too different from the setup of OP). Only reason I can see RPIs in kubernetes is if you're exclusively using ARM everywhere and/or are running some distributed cluster among d…

You can actually build ARM images on your x64 machine using newer versions of Docker with BuildKit built into it. I think it uses QEMU under the hood. I recently did this at work, using a single Dockerfile to built multi-arch images for ARM and x64 - it's pretty nice!

Previously I've done this, but found it takes a bunch of time. I'll give it another go, though and see how it goes!

Re: Domesticating Kubernetes

#26

Post by sdan »

guiriduro wrote: post #10

Or... you could just install a Docker swarm in 10 minutes https://www.bretfisher.com/10-minutes-to-highly-available-do... No good reasons to use K8s outside multi-team enterprise stateful multi-tier architectures.

Swarm is dead in the water.

No progress and there's a ton of basic features that aren't being supported. In my use I have to literally remove every single node and reset the cluster if 1 goes down (I run into some weird grpc error via swarm). Combined with weak GPU support.... its there for basic use, but not for anything "intermediate" or higher.

Re: Domesticating Kubernetes

#27

Post by vetinari »

davestephens wrote: post #17
tachion wrote: post #4

What I miss from all of these tutorials is one very important piece - how to handle network routing and dns automation within your home network, that's in typical scenario is being handled by the ingress/cloud controller. Without having automated (or easy enough) way of reaching the apps you're deploying there, each of these clusters is pretty much useless for users except for maybe learning basics of k8s, what's eas…

Few things to do: - specify a LAN IP for your ingress controller so it doesn't change. - Use ddwrt/dnsmasq to point *.k8s.myhomenetwork.local to said IP Once that's configured, you just configure the ingress hostname on services as you would "normally".

Also - do not use .local tld. It is a reserved one (RFC6762), for mDNS/Bonjour:

> This document specifies that the DNS top-level domain ".local." is a special domain with special semantics, namely that any fully qualified name ending in ".local." is link-local, and names within this domain are meaningful only on the link where they originate. This is analogous to IPv4 addresses in the 169.254/16 prefix or IPv6 addresses in the FE80::/10 prefix, which are link-local and meaningful only on the link where they originate.

Microsoft used to recommend .local TLD for AD deployment as a best practice, and nowadays there are companies stuck with this decision. Do not make the same mistake; unlike companies, you probably want your zeroconf stuff to work.

Re: Domesticating Kubernetes

#28

Post by pas »

detaro wrote: post #22
sandGorgon wrote: post #3

you can run a kubernetes stack using k3s on a rpi stack within minutes. and it runs far better than all of these. https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ here's a live walkthrough - https://www.youtube.com/watch?v=DjpVtNjiXSU

> and it runs far better than all of these. in what way?

k3s is a lightweight stripped down version of k8s, so it's smaller and faster, and tuned for more resource scarce environments, like an RPi.

Re: Domesticating Kubernetes

#29

Post by rkachowski »

sandGorgon wrote: post #3

you can run a kubernetes stack using k3s on a rpi stack within minutes. and it runs far better than all of these. https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/ here's a live walkthrough - https://www.youtube.com/watch?v=DjpVtNjiXSU

The author mentions the differences between k3s and micro k8s in the post and concludes that due to the quirks of k3s (subset of features running in a single binary) it's only really preferable if you want to run rancher.

Re: Domesticating Kubernetes

#30

Post by jimmcslim »

tachion wrote: post #19
jimmcslim wrote: post #15

Earlier quoted context omitted.

I think the OP might be more asking about how to get local DNS automatically provisioned with the IP address of a service/container that has been deployed to the LAN.

That's exactly what I meant. I know how to do it myself, but then again I'm not the target for these articles, but when I read them with the mindset of someone who is supposed to have a use of them, that's always the big missing bit for me.

Did davestephen's comment help though? Apart from suggesting a .local domain... that's a bad idea, try .lan instead.

Then you could type (for Plex, for example) plex.k8s.lan

Maybe we need the equivalent of traefik for DNS?

https://news.ycombinator.com/item?id=23040478

Post reply on HN