Just finished reading part one and wow, what an excellently written and presented post. This is exactly the series I needed to get started with Kubernetes in earnest. It’s like it was written for me personally. Thanks for the submission MathiasPius!
Bare-Metal Kubernetes, Part I: Talos on Hetzner
61–70 of 79 posts
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#62I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.
> Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent. curious what do you mean? To me Postgresql doesn't have disadvantages over SQLite, everything is just better..
PostgreSQL is more complex to use and operate and requires more setup than SQLite. If you don’t need the capabilities of PostgreSQL then you can avoid paying the setup and maintenance costs by using the simpler SQLite.
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#63Earlier quoted context omitted.
Interesting read. I have just setup a very similar cluster this week: 3 node bare metal cluster in a 10G mesh network. Decided for Debian, RKE2, Calico and Longhorn. Encryption is done using LUKS FDE. For Load Balancing I am using the HCloud Load Balancer (in TCP mode). At first I had some problems with the mesh network as the CNI would only bind to a single interface. Finally solved it using a bridge, veth and isola…
Using containerd I assume? I've been trying to get RKE2 or k3s play nicely with CRI-O and it's been a long exercise in frustration.
which distro? it should just work out of the box.
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#64Earlier quoted context omitted.
> Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent. curious what do you mean? To me Postgresql doesn't have disadvantages over SQLite, everything is just better..
PostgreSQL is more complex to use and operate and requires more setup than SQLite. If you don’t need the capabilities of PostgreSQL then you can avoid paying the setup and maintenance costs by using the simpler SQLite.
In simplest case, you do sudo apt install ... in both cases, connect to database and do your work..
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#65Earlier quoted context omitted.
PostgreSQL is more complex to use and operate and requires more setup than SQLite. If you don’t need the capabilities of PostgreSQL then you can avoid paying the setup and maintenance costs by using the simpler SQLite.
In simplest case, you do sudo apt install ... in both cases, connect to database and do your work..
I have never had a Postgres install go that easily. There’s still initialization and setup of the server and users. And you’ll have to do something about upgrades as well. Postgres isn’t difficult to set up but SQLite is just a file. It’s much simpler.
And that’s only the installation. Interaction with SQLite as a database is also simpler.
They both have uses but it’s strange to me to assert that they’re equally complex.
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#66What performance numbers are you seeing on pods with Ceph PVs? e.g., what does `rados bench` give?
I rand rados benchmarks and it seems writes are about 74MB/s, whereas both random and sequential reads are running at about 130MB/s, which is about wire speed given the 1Gbit/s NICs. Complete results are here: https://gist.github.com/MathiasPius/cda8ae32ebab031deb054054...
Thanks!
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#67Earlier quoted context omitted.
In simplest case, you do sudo apt install ... in both cases, connect to database and do your work..
I have never had a Postgres install go that easily. There’s still initialization and setup of the server and users. And you’ll have to do something about upgrades as well. Postgres isn’t difficult to set up but SQLite is just a file. It’s much simpler. And that’s only the installation. Interaction with SQLite as a database is also simpler. They both have uses but it’s strange to me to assert that they’re equally comp…
> There’s still initialization and setup of the server and users.
that command will create postgres user in the system, you do su to that user, run psql and you all set.
> And you’ll have to do something about upgrades as well
apt will take care of it too
> but SQLite is just a file
there is some "file" in postgresql distribution, people just don't use it, because why they would?
> Interaction with SQLite as a database is also simpler.
any specifics?
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#68Earlier quoted context omitted.
I have never had a Postgres install go that easily. There’s still initialization and setup of the server and users. And you’ll have to do something about upgrades as well. Postgres isn’t difficult to set up but SQLite is just a file. It’s much simpler. And that’s only the installation. Interaction with SQLite as a database is also simpler. They both have uses but it’s strange to me to assert that they’re equally comp…
> There’s still initialization and setup of the server and users. that command will create postgres user in the system, you do su to that user, run psql and you all set. > And you’ll have to do something about upgrades as well apt will take care of it too > but SQLite is just a file there is some "file" in postgresql distribution, people just don't use it, because why they would? > Interaction with SQLite as a databa…
SQLite datatypes are advisory so it pretty much just accepts anything you give it.
PostgreSQL definitely says no to schema violations.
These are both features.
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#69I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…
Ah man just looking at that list makes me glad for EKS. But thanks for the effort, I will read to learn more.
If you ever want to have fun with setting up your own k8s, I recommend to start small. The author is already knowledgeable, so they probably knew from the start what they want, but a lot of this complexity is not essential.
When I deployed my first kubernetes "cluster", I just spinned a single-node "cluster" using kubeadm (today k3s is an option too) and started deploying services (with no distributed storage - everything stored using hostPath). You only need to know kubernetes basics to do this. Then you probably want to configure CNI (I recommend flannel when starting, later cilium), spin an ingress controller (I recommend nginx or traefik), deploy cert-manager (this was hard for me when I started) and you can go a long way. With time I scaled up, decided to use GitOps, and deployed many more services (including my own registry - I started with docker's own, then migrated to Gitea. Harbor is too heavy for me). And of course over time you add monitoring, alerting etc - the fun never ends (but it's all optional, you should to decide when is the right time).
Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner
#70Earlier quoted context omitted.
Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud…
I was a k8s contrib since 2015, version 1.1. I even worked at Rancher and Google Cloud. If you don't need absolutely granular control over a PAAS/SAAS (complex networking w/ circuit breaking yadda yadda, deep stack tracing, vms controlled by k8s (kubevirt etc), multi-tenancy in cpu or gpu) you don't need k8s and will absolutely flourish using a container solution like ECS. Use fargate and arm64 containers and you wil…
Can completely agree. We went k8s several years ago and we battle the complexity continually.
Want to change the pod spec to alter a variable under Vitess? That will be restarting several thousands of pods.
DNS lookups failing as IPVS tuple collision?
API server memory balooning to OOM due to watch/list cm storm?
It all basically boils down to several FTE servicing K8s instead of building our platform. And on call is a nightmare.