Live data from Hacker News

Bare-Metal Kubernetes, Part I: Talos on Hetzner

datavirke.dk

21–30 of 79 posts

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#21

I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…

Thank you for the amazing write up!

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#22
post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

amount of nodes is a poor position to take... it should be the features and requirements you need for the cluster.

If Docker Swarm satisfies, then yes.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#23
post #11

What performance numbers are you seeing on pods with Ceph PVs? e.g., what does `rados bench` give?

I haven't had an excuse to test it yet, but since it's only 6 OSDs across 3 nodes and all of them are spinning rust, I'd be surprised if performance was amazing.

I'm definitely curious to find out though, so I'll run some tests and get back to you!

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#24

Speaking of k8s, anyone here know of ready-made solutions for getting XCode (i.e. xcodebuild) running in pods? As far as I'm aware, there are no good solutions for getting XCode running on Linux, so at the moment I'm just futzing about with a virtual-kubelet[0] implementation that spawns MacOS VMs. This works just fine, but the problem seems like such an obvious one that I expect there to be some existing solution(s)…

There are no good ready made solutions.

Someone has submitted patches to containerd and authored “rund” (d for darwin) to run HostProcess containers on macOS.

The underlying problem is poorly familiarity with Kubernetes on Windows among Kubernetes maintainers and users. Windows is where all similar problems have been solved, but the journey is long.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#25

Speaking of k8s, anyone here know of ready-made solutions for getting XCode (i.e. xcodebuild) running in pods? As far as I'm aware, there are no good solutions for getting XCode running on Linux, so at the moment I'm just futzing about with a virtual-kubelet[0] implementation that spawns MacOS VMs. This works just fine, but the problem seems like such an obvious one that I expect there to be some existing solution(s)…

https://blog.darlinghq.org/2023/08/21/progress-report-q2-202... talks about running darling in flatpak, so it's not too much of a stretch to imagine it in a pod someday, but I don't think it's there today.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#26
post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

Any sufficiently complicated Docker Swarm, Heroku, Elastic Beanstalk, Nomad or other program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of vanilla Kubernetes.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#27
post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

> I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Personally, I'd also consider throwing Portainer in there, which gives you both a nice way to interact with the cluster, as well as things like webhooks: https://www.portainer.io/

With something like Apache, Nginx, Caddy or something else acting as your "ingress" (taking care of TLS, reverse proxy, headers, rate limits, sometimes mTLS etc.) it's a surprisingly simple setup, at least for simple architectures.

If/when you need to look past that, K3s is probably worth a look, as some other comments pointed out. Maybe some other of Rancher's offerings as well, depending on how you like to interact with clusters (the K9s tool is nice too).

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#28

I recently rebuilt my Kubernetes cluster running across three dedicated servers hosted by Hetzner and decided to document the process. It turned into a (so far) 8-part series covering everything from bootstrapping and firewalls to setting up persistent storage with Ceph. Part I: Talos on Hetzner https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... Part II: Cilium CNI & Firewalls https://datavirke.dk/posts/…

Have you tried KubeOne? Also with the benefits of machine-deployments. Works like a charm, didn’t go through your blogs, but KubeOne on Hetzner [0] seems easier than your deployment. And yes, also Open Source and German support available.

[0] https://docs.kubermatic.com/kubeone/main/architecture/suppor...

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#29
post #17
post #13

Earlier quoted context omitted.

I agree in part - the features and simplicity of Docker Swarm are very appealing over k8s, but it also feels like so neglected that I'd be waiting every day for the EOL announcement.

It's built from another separate project called swarm-kit. So if it comes to that where it is abandoned, the forks would be out in the wild soon enough. I see more risk of docker engine as a whole pulling some terraform/elastic search licensing someday as investors get desperate to cash out.

Docker is largely irrelevant in modern container orchestration platforms. Kubernetes dropped docker support as of 1.24 in favor of CRI-O.

Docker is just one of many implantations of the Open Container Initiative (OCI) specifications. It’s not even fully open source at this point.

Under the hood Docker leverages containerd which in tern leverages runc which leverages libcontainer for spawning processes.

Linux containers at this point will exist perfectly fine if Docker as a corporate entity disappears. The most impact that would be felt would be Dockerhub being shutdown.

They also sort of already did pull something like Hashicorp with their Docker Desktop product for MacOS.

That’s a little different than if Docker disappeared completely, but one could easily switch to Podman (which has a superset of the docker syntax).

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#30
post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

Node count driven infrastructure decisions make little sense.

A better approach is to translate business requirements to systems capabilities and evaluate which tool best satisfies those requirements given the other constraints within your organization.

Managed Kubernetes solutions like GKE require pretty minimal operational overhead at this point.

Post reply on HN