Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K8s with 1M nodes

bchess.github.io

Post reply on HN
Print view View original

71–80 of 85 posts

Re: K8s with 1M nodes

#71

Post by jamesblonde »

nonameiguess wrote: post #64
jamesblonde wrote: post #51

Earlier quoted context omitted.

Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it? Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...

The setting is configurable, but by default, etcd's Raft implementation requires a voting node to write to disk before it makes a vote, as in actually flushing to disk, not just writing to the file cache. Since you need a majority vote before a client can get a response, this is why it's strongly recommended you use the fastest possible disks, keep the nodes geographically close to each other, and etcd's default stor…

RonDB uses a non-blocking 2PC algorithm - commits in memory, and then does a group commit of transactions to disk every 500ms. This means it can handle insane write throughput, as well as read throughput. However, if both your DB nodes fail, you could lose 500ms of data - which is not the end of the world for k8s. Normally, you would locate DB nodes in different AZes, reducing the probabilty of correlated failures.

Re: K8s with 1M nodes

#72

Post by cyberax »

dmlittle wrote: post #47
cyberax wrote: post #44

Earlier quoted context omitted.

How often do you have sudden host failures? Especially if you use a half-decent server with redundant components for the DB node? Once in maybe 10 years?

The node failure rate is much higher than that. On a 1M node cluster of cloud-managed instances (AWS, GCP, Azure, etc.) you'd likely see failures a few times a month, if not more.

Yep. And the chances that the DB node with the control plane fails are therefore less than one in ten thousand.

Re: K8s with 1M nodes

#73

Post by tbrockman »

breakingcups wrote: post #70
pluc wrote: post #61

> Early on in this project, I asked ChatGPT “I want to scale Kubernetes to 1 million nodes. What types of problems would I need to overcome?” click

It's a shame the author led with something that carries about the same authority as a horoscope, since the rest of the article is actually quite interesting.

If you have sufficient knowledge in the subject matter you're questioning ChatGPT about, you can fairly reliably discern complete bullshit from something plausibly true that warrants additional investigation (which I'd say is more useful than your typical horoscope). In isolation it seems worth the gamble to me, so long as you don't view it as much more than consulting the tea leaves.

Re: K8s with 1M nodes

#74

Post by ymelnyk »

jeffinhat wrote: post #21

This is an awesome experiment and write up. I really appreciate the reproducibility. I would like to see how moving to database that scales write throughput with replicas would behave, namely FoundationDB. I think this will require more than an intermediary like kine to be efficient, as the author illustrates the apisever does a fair bit of its own watching and keeping state. I also think there's benefit, at least fo…

Here you go, Kine FoundationDB backend https://github.com/melgenek/f8n

To be honest, I was building it with the purpose of matching the Etcd scale, but making foundationdb a multitenant data store.

But with the recent craze of scalability , I'll be investing time into understanding how far foundationdb can be pushed as a K8s data store. Stay tuned.

Re: K8s with 1M nodes

#75

Post by __turbobrew__ »

jamesblonde wrote: post #71
nonameiguess wrote: post #64

Earlier quoted context omitted.

The setting is configurable, but by default, etcd's Raft implementation requires a voting node to write to disk before it makes a vote, as in actually flushing to disk, not just writing to the file cache. Since you need a majority vote before a client can get a response, this is why it's strongly recommended you use the fastest possible disks, keep the nodes geographically close to each other, and etcd's default stor…

RonDB uses a non-blocking 2PC algorithm - commits in memory, and then does a group commit of transactions to disk every 500ms. This means it can handle insane write throughput, as well as read throughput. However, if both your DB nodes fail, you could lose 500ms of data - which is not the end of the world for k8s. Normally, you would locate DB nodes in different AZes, reducing the probabilty of correlated failures.

At that point it is apples to oranges. One of the main reasons why etcd writes are slow is because they are guaranteed to be durably persisted across the quorum.

If you just turned off file system syncs in etcd you could probably get an order of magnitude better performance as well.

Re: K8s with 1M nodes

#76

Post by adsharma »

I saw one fleeting reference to a CNI in the article.

Anyone familiar with the space will tell you this is the biggest blocker in production.

You will have to pay for an "enterprise" CNI to make it work.

Re: K8s with 1M nodes

#77

Post by jauntywundrkind »

dmlittle wrote: post #48
jauntywundrkind wrote: post #7

Earlier quoted context omitted.

The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct. https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute e…

It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).

Agreed. The subscriptions really is a huge huge part of the magic, and it's a weakpoint of Kine. Thanks for chiming in.

Ideally, i'd love to see a database specific offering. Use postgres async replication (ideally somehow sharded so there's not a single consumer node) to some fan out system that's doing all the watching.

But etcd mostly does the job, seems unlikely to be going anywhere. It's be cool though.

Re: K8s with 1M nodes

#78

Post by moondev »

trenchpilgrim wrote: post #25
itsnowandnever wrote: post #20

Earlier quoted context omitted.

not exactly a fair assessment since neither of those were out and/or available to the kubernetes team at the time. sure, some things at many times from now into eternity may be or become better suited for the kubernetes data plane but at the time if etcd wasn't used there would be no kubernetes today

The Kubernetes team chose etcd specifically because they were trying to replace Borg's master/slave database at Google. Nothing about Kubernetes requires etcd; the team was trying to solve a Google-internal problem with it (and in the end, didn't gain traction within Google.) k3s uses sqlite by default which was an option at the time, other clusters today use PostgreSQL. Have you looked at the etcd keys and values in…

Upstream kubernetes literally requires etcd. Anything that changes that is a fork

Re: K8s with 1M nodes

#79

Post by cyberax »

itsnowandnever wrote: post #63
cyberax wrote: post #45

Earlier quoted context omitted.

> etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. Sorry, this is just BS. etcd is a fifth wheel in most k8s installations. Even the largest clusters are better off with something like a large-ish instance running a regular DB for the control plane state storage. Yes, etcd theoretically protects again…

that's not my point - my point is it would not have gotten the adoption it has without etcd and the fact that it was resilient and scalable out of the box

It probably would have gotten even faster adoption if it used a saner embedded DB.

Re: K8s with 1M nodes

#80

Post by trenchpilgrim »

moondev wrote: post #78
trenchpilgrim wrote: post #25

Earlier quoted context omitted.

The Kubernetes team chose etcd specifically because they were trying to replace Borg's master/slave database at Google. Nothing about Kubernetes requires etcd; the team was trying to solve a Google-internal problem with it (and in the end, didn't gain traction within Google.) k3s uses sqlite by default which was an option at the time, other clusters today use PostgreSQL. Have you looked at the etcd keys and values in…

Upstream kubernetes literally requires etcd. Anything that changes that is a fork

There are many forms of Kubernetes that are not using etcd that are certified by the CNCF as conformant (https://www.cncf.io/training/certification/software-conforma...). The version in the kubernetes/kubernetes repository is a reference implementation, intended to be customized by the community.
Post reply on HN