Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

jeffgeerling.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#31

Post by ricardbejarano »

geerlingguy wrote: post #23
ricardbejarano wrote: post #22

Earlier quoted context omitted.

From experience, anything with 2GB or more RAM can be a master node. Workers can even have 1GB and work just fine. Be warned, though, in my experience etcd requires a somewhat decent read/write latency, or else it's going to fail, and when etcd fails everything fails. Your changes don't apply, etc.

_Technically_ 1GB is the limit, but yeah, at 1GB I was hitting OOM errors every day or so (this was on the Pi 3 B+). I was very happy when I found out the Pi 4 had 2 or 4GB models! No matter what, you should also make sure to not schedule Pods on the master node(s) when you have limited memory (or in most cases, really...).

Or set tight resource limits.

Setting resource requests also enables the use of HorizontalPodAutoscalers.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#32

Post by jptoto »

geerlingguy wrote: post #12
jptoto wrote: post #7

I LOVE Jeff's work and I own his Ansible book. The setup is pretty awesome. FWIW I think you could do this with local VMs a little more easily and provision with Vagrant. Having said that, a cluster of Pis is super fun!

What do you know, that's exactly how I do local development work for the cluster ;) https://github.com/geerlingguy/raspberry-pi-dramble/tree/mas...

Aha! Makes sense.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#33

Post by yumraj »

vyshane wrote: post #17

3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.

This is very cool. Curious, roughly, how much did this setup cost?

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#34

Post by bogomipz »

SirMonkey wrote: post #4

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ (Networking) https://rook.io/ (Persistent Storage)

Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio?

I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is that correct?

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#35

Post by pstadler »

Great article! Never stop tinkering.

Here‘s how I got to know Kubernetes:

By end of 2016, the Kubernetes hype was just about to pick up real steam. As somebody who always liked the idea of running something like an own cluster in the cloud, I attended KubeCon Europe in early 2017. The event was sold out, and took place in a venue almost too small for the number of attendees. It was great. During the event I was just about to finish the Hobby Kube[1] project. Back then weren’t any guides that addressed all the problems encountered when running Kubernetes on a budget, using low cost VPS on random cloud providers. So I dived into the subject in the second half of 2016 and started writing a guide, including automated provisioning using Terraform. I discovered WireGuard in the process of looking for a solution to efficiently securing network traffic between hosts. This still makes me feel like I was an early adopter to something that’s becoming hugely popular.

If somebody would like to add a Terraform module for deploying to Raspberry Pi, please ping me or open a PR here[2].

[1] https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#36

Post by moondev »

bogomipz wrote: post #34
SirMonkey wrote: post #4

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ (Networking) https://rook.io/ (Persistent Storage)

Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is…

Not directly, rook provides a storage backend for in-cluster workload persistent volumes.

In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#37

Post by ownagefool »

bogomipz wrote: post #34
SirMonkey wrote: post #4

I learned k8s with some NUCs we had laying around at work. Might be easier than Pi, but not as cheap. Some things I used: https://metallb.universe.tf/ (LoadBalancer) https://cilium.io/ (Networking) https://rook.io/ (Persistent Storage)

Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is…

https://rook.io/docs/rook/v1.1/

Rook will give you:

- Ceph: Block Storage, Shared storage, Object storage

- EdgeFS: Object but apparently will function as Block and File

- Minio: Object

- NFS4: Shared

- Cockroach: Database

- Cassandra: Database

- YugabyteDB: distributed SQL database (new to me)

Only the first two are marked as stable.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#38

Post by lhuser123 »

bndw wrote: post #29
mkesper wrote: post #8

Earlier quoted context omitted.

How does this compare to k3s? https://rancher.com/docs/k3s/latest/en/

Kind is explicitly not for production workloads[0]: Non-Goals: Being “production workload ready” - kind is meant to be used: for testing Kubernetes itself for testing against Kubernetes (EG in CI on Travis, Circle, etc.) for “local” clusters on developer machines NOT to host workloads serving user traffic etc. K3s seems happy to support production workloads. It's a single binary and gets you a cluster in one command:…

What are the benefits you find in using Kubernetes for a single-node cluster on a vps?

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#39

Post by Methusalah »

This seems like a fun project and a great way to learn Kubernetes, but if I'm dropping this much money on it, I'd like it to have some productive purpose afterwards.

I'm a full-stack web dev primarily using node/react/postgres. I've also got some projects currently hosted on a Linode instance. Ideas on fun/productive uses for this cluster after I've built it and messed around with Kubernetes?

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#40

Post by ownagefool »

moondev wrote: post #36
bogomipz wrote: post #34

Earlier quoted context omitted.

Does Rook give you the equivalent of EBS root volumes for your nodes then? Is that the function you have it providing? Does it offer something beyond using local host storage and minio? I ask because I've generally been confused about the use case for Rook despite having read the "what is Rook?" paragraph many times on the project home page. My assumption is that it lets you build your own internal cloud provider. Is…

Not directly, rook provides a storage backend for in-cluster workload persistent volumes. In theory you could manage kvm machines with kubevirt and back the machines with PVCs from rook. I have not tried this and would be curious how the performance is.

I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume.

You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.

Post reply on HN