Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

jeffgeerling.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#21

Post by geerlingguy »

vyshane wrote: post #17

3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.

Very nice; I've considered doing something similar and running some production sites on it from my home, but the limitation has always been my terrible Internet bandwidth through Spectrum.

We almost got Verizon gigabit fiber a few years ago... then AT&T ran fiber to the front of my neighborhood last year, and then never ran it to the houses. As it is, I'm stuck with 10 mbps uplink, which is not enough to be able to do most of what I would want to do with a more powerful local cluster.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#22

Post by ricardbejarano »

Mountain_Skies wrote: post #10

Wonder if some old netbooks could be used for this purpose. Doubt I am the only one with a small pile of them laying around.

From experience, anything with 2GB or more RAM can be a master node. Workers can even have 1GB and work just fine.

Be warned, though, in my experience etcd requires a somewhat decent read/write latency, or else it's going to fail, and when etcd fails everything fails. Your changes don't apply, etc.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#23

Post by geerlingguy »

ricardbejarano wrote: post #22
Mountain_Skies wrote: post #10

Wonder if some old netbooks could be used for this purpose. Doubt I am the only one with a small pile of them laying around.

From experience, anything with 2GB or more RAM can be a master node. Workers can even have 1GB and work just fine. Be warned, though, in my experience etcd requires a somewhat decent read/write latency, or else it's going to fail, and when etcd fails everything fails. Your changes don't apply, etc.

_Technically_ 1GB is the limit, but yeah, at 1GB I was hitting OOM errors every day or so (this was on the Pi 3 B+). I was very happy when I found out the Pi 4 had 2 or 4GB models! No matter what, you should also make sure to not schedule Pods on the master node(s) when you have limited memory (or in most cases, really...).

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#26

Post by [deleted] »

geerlingguy wrote: post #21
vyshane wrote: post #17

3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.

Very nice; I've considered doing something similar and running some production sites on it from my home, but the limitation has always been my terrible Internet bandwidth through Spectrum. We almost got Verizon gigabit fiber a few years ago... then AT&T ran fiber to the front of my neighborhood last year, and then never ran it to the houses. As it is, I'm stuck with 10 mbps uplink, which is not enough to be able to d…

[deleted]

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#27

Post by fyfy18 »

salamander014 wrote: post #19
SirMonkey wrote: post #4

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ (Networking) https://rook.io/ (Persistent Storage)

I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router ( https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (…

I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device.

Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It hasn't lost data though, even though one of the SSDs died and was unrecoverable, so props for that. To be fair I haven't tried upgrading since I installed it so maybe my issues would be resolved by that.

Overall Rook feels somewhat overkill for a homelab user, but I haven't seen any recommended alternatives.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#28

Post by [deleted] »

raesene9 wrote: post #18
mkesper wrote: post #8

Earlier quoted context omitted.

How does this compare to k3s? https://rancher.com/docs/k3s/latest/en/

k3s is usually (AFAIK) run on VMs for production purposes, rather than being used for testing, which is where kind shines. The major advantage of kind is you download a single binary, run one command and quickly have a functional, disposable cluster, anywhere that Docker runs.

[deleted]

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#29

Post by bndw »

mkesper wrote: post #8
raesene9 wrote: post #6

A good method for learning how clusters work and playing with them without having to spend a lot of time re-building when you break things is kind ( https://kind.sigs.k8s.io/ ). Each node is a Docker container, but the version of Kubernetes running inside it is vanilla Kubeadm, so it's quite representative of what "real" clusters would look like. The great thing about it is you can spin up a cluster in < 2 mins local…

How does this compare to k3s? https://rancher.com/docs/k3s/latest/en/

Kind is explicitly not for production workloads[0]:

  Non-Goals: Being “production workload ready” - kind is meant to be used:

    for testing Kubernetes itself
    for testing against Kubernetes (EG in CI on Travis, Circle, etc.)
    for “local” clusters on developer machines
    NOT to host workloads serving user traffic etc.
K3s seems happy to support production workloads. It's a single binary and gets you a cluster in one command:

  curl -sfL https://get.k3s.io | sh -
  sudo kubectl get nodes
Personally, I use both Kind and k3s. Kind is great for running locally and k3s is awesome for single-node kubernetes on a vps.

[0] - https://kind.sigs.k8s.io/docs/contributing/1.0-roadmap/#non-...

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#30

Post by moondev »

mkesper wrote: post #8
raesene9 wrote: post #6

A good method for learning how clusters work and playing with them without having to spend a lot of time re-building when you break things is kind ( https://kind.sigs.k8s.io/ ). Each node is a Docker container, but the version of Kubernetes running inside it is vanilla Kubeadm, so it's quite representative of what "real" clusters would look like. The great thing about it is you can spin up a cluster in < 2 mins local…

How does this compare to k3s? https://rancher.com/docs/k3s/latest/en/

KIND uses kubeadm which is the upstream standard for bootstrapping. It also supports multi masters and even launches a haproxy load balancer for kube-apiserver! K3S uses it's own bootstrapping, as well as non upstream binaries. It also replaces etcd with it's own shim.

Non-kubeadm bootstrapped clusters are snowflakes in my opinion. You will find many docs that will not work with k3s due to missing systemd. You also lose some flexibility. For example there is no way to change the container runtime cgroup driver.

KIND is designed for local use on your workstation. Cluster API is the project to use for production deployments. You can drive Cluster API with KIND for testing though!

Post reply on HN