> Early on in this project, I asked ChatGPT “I want to scale Kubernetes to 1 million nodes. What types of problems would I need to overcome?”
click
K8s with 1M nodes
61–70 of 85 posts
Re: K8s with 1M nodes
#62If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art
Agree this is a consideration if your only workload is an existing or greenfield ErlangVM-compatible project.
From what I know basically everyone approaching this scale with k8s has different problems to solve, namely multi-tenancy (shared hosting/internal plattform providers) and compatibility with legacy or standard software.
Re: K8s with 1M nodes
#63Earlier quoted context omitted.
etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.
> etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. Sorry, this is just BS. etcd is a fifth wheel in most k8s installations. Even the largest clusters are better off with something like a large-ish instance running a regular DB for the control plane state storage. Yes, etcd theoretically protects again…
that's not my point - my point is it would not have gotten the adoption it has without etcd and the fact that it was resilient and scalable out of the box
Re: K8s with 1M nodes
#64I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…
Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it? Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...
The setting is configurable, but by default, etcd's Raft implementation requires a voting node to write to disk before it makes a vote, as in actually flushing to disk, not just writing to the file cache. Since you need a majority vote before a client can get a response, this is why it's strongly recommended you use the fastest possible disks, keep the nodes geographically close to each other, and etcd's default storage is only 2GB per node.
All in all, it was a poor choice for Kubernetes to use this as its backend in the first place. Apparently, Google uses its own shim, but there is also kine, which was created a long time ago for k3s and allows you to use a RDBMS. k3s used sqlite as its default originally, but any API equivalent database would work.
We should keep in mind etcd was meant to literally be the distributed /etc directory for CoreOS, something you would read from often but perform very few writes to. It's a configuration store. Kubernetes deciding to also use it for /var was never a great idea.
Re: K8s with 1M nodes
#65Earlier quoted context omitted.
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
Interestingly the public of Azure’s etcd-compatible service was withdrawn before exiting preview. [1] https://learn.microsoft.com/en-us/answers/questions/154061/a...
It's interesting they ever exposed it at all really! I don't think you can use Google's Spanner-based etcd replacement for a self-managed Kubernetes cluster, for example.
Re: K8s with 1M nodes
#66I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
While I can’t speak for the others, AWS doesn’t replace all of etcd. Only the raft consensus layer which is replaced with Journal which is an internal AWS service.
Re: K8s with 1M nodes
#67Earlier quoted context omitted.
This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
There are similar libraies in Elixir. Is the ecosystem for ML as developed as for python? Nope, but not every ML project needs the most obscure libraries etc. (For the record I don't really see Erlang clusters as a replacement for k8s)
You aren’t going to teach a research scientist Erlang in spite of Python and R. You aren’t going to win that fight, ever.
Re: K8s with 1M nodes
#68If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art
This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?
I don’t think there are very many k8s clusters running 100s of pods per node. The default maximum is 110. You can, of course, scale beyond this, but you’ll run into etcd performance issues, IP space issues, max connection, IOPS and networking limitations for most use cases.
At 1M nodes I’d still expect an average of a dozen or so pods per node.
Re: K8s with 1M nodes
#69Earlier quoted context omitted.
This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?
This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
You can with Pyrlang or whatever other cursed implementation of Python on top of the Erlang VM you’d prefer.
Re: K8s with 1M nodes
#70> Early on in this project, I asked ChatGPT “I want to scale Kubernetes to 1 million nodes. What types of problems would I need to overcome?” click
It's a shame the author led with something that carries about the same authority as a horoscope, since the rest of the article is actually quite interesting.