Bare-Metal Kubernetes, Part I: Talos on Hetzner • Hacker News Board

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#41

Post by wg0 » Sat, Sep 09, 2023, 5:52 PM UTC

amazingman wrote: post #36

husarcik wrote: post #20

Earlier quoted context omitted.

The docker swarm ecosystem is very poor as far as tooling goes. You're better off using docker-compose (? maybe docker swarm) and then migrating to k3s if you need a cluster. My docker swarm config files are nearly the same craziness as my k3s config files so I figured I might as well benefit from the tooling in Kubernetes. Edit for more random thoughts: being able to use helm to deploy services helped me switch to k…

This is almost exactly my experience with Docker Compose, which is lionized by commenters in nearly every Kubernetes thread I read on HN. It's great and super simple and easy ... until you want to wire multiple applications together, you want to preserve state across workload lifecycles for stateful applications, and/or you need to stand up multiple configurations of the same application. The more you want to run app…

Then the Helm and layers of kustomize are not easy to reason with either.

That's system configuration and that'll become tedious for sure.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#42

Post by ori_b » Sat, Sep 09, 2023, 5:52 PM UTC

amazingman wrote: post #37

ori_b wrote: post #32

Earlier quoted context omitted.

Unfortunately, the above statement also applies to kubernetes.

A pithy response to be sure, but is it true? Every Kubernetes object type exists within a well-specified hierarchy, has a well-specified specification, an API version, and documentation. Most of the object families' evolution are managed by a formal SIG. Not sure how any of that qualifies as ad-hoc or informal.

"It's not a mess! it was designed by committee!"

I'm not sure what to say here. The kubernetes docs and code speak for themselves. If you actually think that it's clean, simple, well designed, and easy to operate, with smooth interop between the parts, I can't change your mind. But in practice, I have found it very unpleasant. It seems this is common, and the usual suggestion is to pay someone else to operate it.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#43

Post by wg0 » Sat, Sep 09, 2023, 5:55 PM UTC

doctorpangloss wrote: post #26

wg0 wrote: post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

Any sufficiently complicated Docker Swarm, Heroku, Elastic Beanstalk, Nomad or other program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of vanilla Kubernetes.

Most smaller teams do not need a full fledge kubernetes anyways.

There's no one size fits all approach. There are trade offs. The Kubernetes tractor needs lots of oiling and what not for all the bells and whistles.

Trade offs is the keyword here.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#44

Post by yjftsjthsd-h » Sat, Sep 09, 2023, 5:58 PM UTC

linuxdude314 wrote: post #29

wg0 wrote: post #17

Earlier quoted context omitted.

It's built from another separate project called swarm-kit. So if it comes to that where it is abandoned, the forks would be out in the wild soon enough. I see more risk of docker engine as a whole pulling some terraform/elastic search licensing someday as investors get desperate to cash out.

Docker is largely irrelevant in modern container orchestration platforms. Kubernetes dropped docker support as of 1.24 in favor of CRI-O. Docker is just one of many implantations of the Open Container Initiative (OCI) specifications. It’s not even fully open source at this point. Under the hood Docker leverages containerd which in tern leverages runc which leverages libcontainer for spawning processes. Linux containe…

> Docker is just one of many implantations of the Open Container Initiative (OCI) specifications. It’s not even fully open source at this point.

How so? I know Docker Desktop wraps its own stuff around docker, but AFAIK docker itself is FOSS.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#45

Post by MathiasPius » Sat, Sep 09, 2023, 6:14 PM UTC

dhess wrote: post #11

What performance numbers are you seeing on pods with Ceph PVs? e.g., what does `rados bench` give?

I rand rados benchmarks and it seems writes are about 74MB/s, whereas both random and sequential reads are running at about 130MB/s, which is about wire speed given the 1Gbit/s NICs.

Complete results are here: https://gist.github.com/MathiasPius/cda8ae32ebab031deb054054...

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#46

Post by mythz » Sat, Sep 09, 2023, 6:19 PM UTC

Thankfully we've never had the need for such complexity and are happy with our current GitHub Actions > Docker Compose > GCR > SSH solution [1] we're using to deploy 50+ Docker Containers.

Requires no infrastructure dependencies, stateless deployment scripts checked into the same Repo as Project and after GitHub Organization is setup (4 secrets) and deployment server has Docker compose + nginx-proxy installed, deploying an App only requires 1 GitHub Action Secret, as such it doesn't get any simpler for us and we'll look to continue to use this approach for as long as we can.

[1] https://servicestack.net/posts/kubernetes_not_required

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#47

Post by seabrookmx » Sat, Sep 09, 2023, 6:28 PM UTC

mythz wrote: post #46

Thankfully we've never had the need for such complexity and are happy with our current GitHub Actions > Docker Compose > GCR > SSH solution [1] we're using to deploy 50+ Docker Containers. Requires no infrastructure dependencies, stateless deployment scripts checked into the same Repo as Project and after GitHub Organization is setup (4 secrets) and deployment server has Docker compose + nginx-proxy installed, deploy…

I used to do something similar at a previous company and this works well if you don't have to worry about scaling. YAGNI principal and all that. When you run hundreds of containers for different workloads, k8s bin packing and autoscaling (both on the pod and node level) tips the balance in my experience.

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#48

Post by mythz » Sat, Sep 09, 2023, 6:39 PM UTC

seabrookmx wrote: post #47

mythz wrote: post #46

Thankfully we've never had the need for such complexity and are happy with our current GitHub Actions > Docker Compose > GCR > SSH solution [1] we're using to deploy 50+ Docker Containers. Requires no infrastructure dependencies, stateless deployment scripts checked into the same Repo as Project and after GitHub Organization is setup (4 secrets) and deployment server has Docker compose + nginx-proxy installed, deploy…

I used to do something similar at a previous company and this works well if you don't have to worry about scaling. YAGNI principal and all that. When you run hundreds of containers for different workloads, k8s bin packing and autoscaling (both on the pod and node level) tips the balance in my experience.

Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face.

Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud.

[1] https://stackexchange.com/performance

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#49

Post by Patrickmi » Sat, Sep 09, 2023, 7:05 PM UTC

wg0 wrote: post #12

I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for Docker swarm is to Kubernetes what SQLite is to PostgreSQL. To some extent.

I was using docker swarm cause of the simplicity and easy setup but the one feature that I really really need was to be able to specify which runtime to use, either I use runsc (and docker plugins don’t work with runsc) or runc as the default and it was too inefficient to have groups of node with certain runtime, I really do like swarm but it misses too much features that are important

Re: Bare-Metal Kubernetes, Part I: Talos on Hetzner

#50

Post by doctorpangloss » Sat, Sep 09, 2023, 7:07 PM UTC

wg0 wrote: post #41

amazingman wrote: post #36

Earlier quoted context omitted.

This is almost exactly my experience with Docker Compose, which is lionized by commenters in nearly every Kubernetes thread I read on HN. It's great and super simple and easy ... until you want to wire multiple applications together, you want to preserve state across workload lifecycles for stateful applications, and/or you need to stand up multiple configurations of the same application. The more you want to run app…

Then the Helm and layers of kustomize are not easy to reason with either. That's system configuration and that'll become tedious for sure.

Helm and Kustomize are low-budget custom resource definitions. They serve their purpose well and they have few limitations considering how much they can achieve before you write your own controllers.

In my opinion, the complexity is symptomatic of success: once you make a piece of some kind of seemingly narrowly focused software that people actually use, you wind up also creating a platform, if not a platform-of-platforms, in order to satisfy growth. Kubernetes can scale for that business case in ways Docker Swarm, ELB, etc. do not.

Is system configuration avoidable? In order to use AWS, you have to know how a VPC works. That is the worst kind of configuration. I suppose you can ignore that stuff for a very long time, you'll be paying ridiculous amounts of money for the privilege - almost the same in bandwidth costs, transiting NAT gateways and all your load balancers, whatever mistakes you made, as you do in compute usage. Once you learn that bullshit, you know, Kubernetes isn't so tedious after all.