Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bare-Metal Kubernetes, Part I: Talos on Hetzner

datavirke.dk

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#71

Post by CoolCold »

> Ceph is designed to host truly massive amounts of data, and generally becomes safer and more performant the more nodes and disks you have to spread your data across.

I'm very pessimistic on CEPH usage in the scenario you have - may be I've missed it, but seen nothing about upgrading networking, as by default you gonna have 1Gbit on single interface used for public network/internal vSwitch.

Even by your benchmarks, write test is 19 iops (block size is huge though)

    Max bandwidth (MB/sec): 92
    Min bandwidth (MB/sec): 40
    Average IOPS:           19
    Stddev IOPS:            2.62722
    Max IOPS:               23
    Min IOPS:               10
while single HDD drive would give ~ 120 iops. single 3 years old NVMe datacenter edition, gives ~ 33000 iops with 4k block + fdatasync=1

CEPH would be very limiting factor in 1Gbit networking I believe - I'd put clear disclaimer on that for fellow sysadmins.

P.S. The amount of work you done is huge and appreciated.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#72

Post by ChoHag »

doctorpangloss wrote: post #26
wg0 wrote: post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

Any sufficiently complicated Docker Swarm, Heroku, Elastic Beanstalk, Nomad or other program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of vanilla Kubernetes.

[dead]

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#73

Post by sureglymop »

Here's what I don't really get.. So, let's say you have three hosts and create your cluster. But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down. So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also and use DNS to make sure one of the reverse proxies is used? Maybe I'm just misunderstanding how it works but multiple nodes in a cluster still need some sort of central entry point right? So what is the correct way to do this.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#74

Post by ralgozino »

sureglymop wrote: post #73

Here's what I don't really get.. So, let's say you have three hosts and create your cluster. But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down. So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also…

You almost answered your own question. One common solution is to have 2 nodes with haproxy (or similar) sharing a virtual IP with keepalived that load balance de traffic to the control plane nodes and to the nodes where your ingress controller runs.

There are other options, like running the haproxy in the control plane nodes.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#75

Post by MathiasPius »

sureglymop wrote: post #73

Here's what I don't really get.. So, let's say you have three hosts and create your cluster. But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down. So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also…

My solution for this setup is having ingress controllers on all three nodes, and then specifying all three IPs in all DNS records. That way the end user will "load balance" based on the DNS randomization.

Of course, if a node goes down, a third of the traffic will be lost, but with low TTLs and some planning, you can minimoze the impact of this.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#76

Post by sureglymop »

ralgozino wrote: post #74
sureglymop wrote: post #73

Here's what I don't really get.. So, let's say you have three hosts and create your cluster. But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down. So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also…

You almost answered your own question. One common solution is to have 2 nodes with haproxy (or similar) sharing a virtual IP with keepalived that load balance de traffic to the control plane nodes and to the nodes where your ingress controller runs. There are other options, like running the haproxy in the control plane nodes.

Thank you, this was very helpful! I read up on keepalived and the used protocols now!

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#77

Post by sureglymop »

MathiasPius wrote: post #75
sureglymop wrote: post #73

Here's what I don't really get.. So, let's say you have three hosts and create your cluster. But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down. So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also…

My solution for this setup is having ingress controllers on all three nodes, and then specifying all three IPs in all DNS records. That way the end user will "load balance" based on the DNS randomization. Of course, if a node goes down, a third of the traffic will be lost, but with low TTLs and some planning, you can minimoze the impact of this.

It's an interesting approach. I did it a bit differently. I set up three Proxmox nodes on three hetzner servers. Then I deployed virtual routers. I then set up HAProxy and k3s nodes as LXC containers. What's nice about the whole setup is that a proxmox node can go down and it all still works. I will now set up keepalived as mentioned in the other reply so the HAProxies will also be fully HA. Proxmox also works well with zfs and backups. I set up the proxmox nodes manually and did the rest with terraform + ansible. One `terraform destroy` cleans up everything nicely. I wonder how the performance difference is between bare metal and k8s node in LXC.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#78

Post by fireflash38 »

KyleSanderson wrote: post #63
fireflash38 wrote: post #55

Earlier quoted context omitted.

Using containerd I assume? I've been trying to get RKE2 or k3s play nicely with CRI-O and it's been a long exercise in frustration.

which distro? it should just work out of the box.

Initially Ubuntu 20.04, but I upgraded to 22.04. Finally got it working -- turns out a lot of things that reference `--cgroup-driver="systemd"` are doing it as if it were run in shell, which means that the quotes around "systemd" get removed by shell, and would lead to an error & ignored options.

Nothing was showing whatsoever when using 20.04, so I wonder if there were some missing dependencies somewhere there...

I'll probably write up everything I discovered at some point, there's a lot of pieces that you have to cobble together from pretty disparate sources (network plugins, config files (which!?), etc).

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#79

Post by dave-at-koor »

Great post. We (Koor) have been going through something similar to create a demo environment for Rook-Ceph. In our case, we want to show different types of data storage (block, object, file) in a production-like system, albeit at the smaller end of scale.

Our system is hosted at Hetzner on Ubuntu. KubeOne does the provisioning, backed by Terraform. We are using Calico for networking, and we have our own Rook operator.

What would have made the Rook-Ceph experience better for you?

Post reply on HN