Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

jeffgeerling.com

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#61

Post by moondev »

darren0 wrote: post #54
moondev wrote: post #30

Earlier quoted context omitted.

KIND uses kubeadm which is the upstream standard for bootstrapping. It also supports multi masters and even launches a haproxy load balancer for kube-apiserver! K3S uses it's own bootstrapping, as well as non upstream binaries. It also replaces etcd with it's own shim. Non-kubeadm bootstrapped clusters are snowflakes in my opinion. You will find many docs that will not work with k3s due to missing systemd. You also l…

k3s has no direct relationship to systemd, I'm not sure what things not working you are referring too. k3s has full flexibility to change any k8s option or component, it merely defaults to settings to meet it's goals of being a simple, lightweight, and secure distro.

> k3s has no direct relationship to systemd

Exactly, this differs from every kubeadm bootstrapped cluster which uses systemd to run the kubelet, as well using common config locations and behavior.

> k3s has full flexibility to change any k8s option or component

Quick Example: How do you change the kubelet's cgroup-driver? This looks to be hard-coded into the k3s binaries.

https://github.com/rancher/k3s/blob/c79743d4c3d9f2d21b352ed7...

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#63

Post by MuffinFlavored »

segmondy wrote: post #62

You can also build a cluster on your computer if it's beefy enough by running multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few years ago and that's my k8s experiment land.

I don’t fully understand where the line goes from “all I need is a Docker container for nginx + Postgres + Redis + my services” to “I need Kubernetes”

When does one need to go from just Docker containers to container orchestration like k8s?

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#65

Post by Nullabillity »

fyfy18 wrote: post #27
salamander014 wrote: post #19

Earlier quoted context omitted.

I also use MetalLB and Rook. Can't say enough good things about them. I have also used Kube-Router ( https://kube-router.io - Digital Ocean's non Virtual networking plugin for bare metal environments; it puts your containers on your physical network, which is freaking neat) and loved that, but since I started deploying kubernetes with Rancher I've found for dev clusters I'm not caring about what networking is used. (…

I've had a lot of performance issues with Rook. I setup a k8s cluster six months ago on a few ThinkCenter Tiny's (more powerful than a Pi). They each have a single 120GB SSD, so I've allocated a directory on the filesystem (ext4) to Rook rather than a dedicated device. Originally I only had 2GB of RAM and Rook ended up using most of this. Even now the CPU usage by Rook is quite high even when the system is idle. It h…

You need to set a memory target for your OSDs, or they will gobble up as much as possible for their cache. In Rook you do that by specifying the resource limits on your CephCluster[0].

Sadly, Rook enforces some pretty high minimum values (2GiB per OSD, 1 GiB per mon), which is fine for production but can make homelab experimentation annoying.

[0]: https://rook.io/docs/rook/v1.1/ceph-cluster-crd.html#cluster...

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#66

Post by Nullabillity »

MuffinFlavored wrote: post #63
segmondy wrote: post #62

You can also build a cluster on your computer if it's beefy enough by running multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few years ago and that's my k8s experiment land.

I don’t fully understand where the line goes from “all I need is a Docker container for nginx + Postgres + Redis + my services” to “I need Kubernetes” When does one need to go from just Docker containers to container orchestration like k8s?

Do you care about availability? Do you care about rolling upgrades? Do you want to practice any of those two? Then it might be time to make the switch.

Yes, Docker has Swarm Mode. And yes, it is easier to get started with than K8s.

The problem is that it locks you into a lot of choices when you first spin up each (Docker) Service, that can't be changed without taking down the Service and then recreating it with the new option. Want to change how upgrades are rolled out? Too bad!

Kubernetes still has a few settings like that, but since (K8s) Services are just responsible for routing traffic to separately-managed Pods, you can always fall back to rolling the upgrade manually, or doing a blue-green deployment.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#67

Post by rwmj »

vyshane wrote: post #17

3 years ago, I also wanted a bare metal cluster for my homelab. I wanted x86-64, low power consumption, small footprint, and low cost. I ended up building this 5 node nano ITX tower: https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-to... I think that the exposed boards adds to its charm. Doesn't help with dust though.

Yours is a lot neater than the four node bare cluster I built a few years ago: https://rwmj.wordpress.com/2014/04/28/caseless-virtualizatio...

One issue with caseless machines is the amount of RF they emit. Makes it hard to listen to any broadcast radio near one and probably disturbs the neighbours similarly.

I'm now using a cluster of NUCs which is considerably easier to deal with although quite a lot more expensive: https://rwmj.wordpress.com/2018/02/06/nuc-cluster-bring-up/

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#68

Post by ownagefool »

moondev wrote: post #44
ownagefool wrote: post #40

Earlier quoted context omitted.

I would say they're functionality quite equivalent, you just wouldn't pull them in as a root volume. You should be able to point your default StorageClass at Ceph and have it create RBD block storage devices for you, which would auto create pvc, that you mount you actual data in in your kube manifests.

The parent comment was asking about root volumes for kubernetes nodes > Does Rook give you the equivalent of EBS root volumes for your nodes then? I think we are saying the same thing if i'm not mistaken?

I doubt the parent really cares about whether or not it's a root volume, but for the record you can mount it to root.

It'll just essentially be empty, which means you have to add a step of populating root with something useful. You'd have to either try and use an initContainer to copy in a filesystem at runtime, have ceph give you a pre-populated directory, probably via thin provisioning, or do something out of band (I've did this in k8s).

This is usually more effort than it's worth though, as container runtimes populate the root for you with whatever you want anyway. Plus, if you start treating containers as blobs like VMs, you'll end up in the situation where you don't know where your important variable data is, which leads to situations where people forget to back it up and test it.

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#69

Post by mpfundstein »

ricardbejarano wrote: post #3

Everything I know about Kubernetes I learned from a single-node (pseudo)clustered refurbished ThinkPad X201. It cost me 90€ at an IBM refurbished sell. It is downstairs by the router, and it has been hosting everything for me except my email and my blog (which I want to host but I'm not sure about the reliability of my ISP's service, this morning it went down for 5h12m, without prior notice or anything). It is amazin…

I only really learn by doing stuff. I read books usually to get started but mostly quickly have to turn to building something. Then revisiting the book on the go. If I only read the book/paper/whatever, I usually ‘think’ I get it but nearly always that isn’t the case :-)

Re: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis

#70

Post by imtringued »

MuffinFlavored wrote: post #63
segmondy wrote: post #62

You can also build a cluster on your computer if it's beefy enough by running multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few years ago and that's my k8s experiment land.

I don’t fully understand where the line goes from “all I need is a Docker container for nginx + Postgres + Redis + my services” to “I need Kubernetes” When does one need to go from just Docker containers to container orchestration like k8s?

As soon as you want load balancing for high availability but I personally am not a fan of Kubernetes.
Post reply on HN