People don’t realize how crucial Ben was in the forming of OpenAI as it is known today. This is an extremely underrated post.
K8s with 1M nodes
41–50 of 85 posts
Re: K8s with 1M nodes
#42I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently.
[1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazo...
[2]: https://cloud.google.com/blog/products/containers-kubernetes...
[3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
Re: K8s with 1M nodes
#43I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
Interestingly the public of Azure’s etcd-compatible service was withdrawn before exiting preview.
[1] https://learn.microsoft.com/en-us/answers/questions/154061/a...
Re: K8s with 1M nodes
#44I don't get the point of benchmarking k8s without the guarantees of etcd. At some point, you are just competing with clusterssh.
How often do you have sudden host failures? Especially if you use a half-decent server with redundant components for the DB node?
Once in maybe 10 years?
Re: K8s with 1M nodes
#45“Perhaps my spiciest take from this entire project: most clusters don’t actually need the level of reliability and durability that etcd provides.” This assumption is completely out of touch, and is especially funny when the goal is to build an extra large cluster.
etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.
> etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd.
Sorry, this is just BS. etcd is a fifth wheel in most k8s installations. Even the largest clusters are better off with something like a large-ish instance running a regular DB for the control plane state storage.
Yes, etcd theoretically protects against any kind of node failures and network partitions. But in practice, well, nobody really cares about the control plane being resilient against meteorite strikes and Cthulhu rising from the deeps.
Re: K8s with 1M nodes
#46I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…
It,s nice to know that the upper bound of the resiliency of a k8s cluster is the amount of redundancy etcd has - which is in essence a horizontally scaled monolith.
Re: K8s with 1M nodes
#47I don't get the point of benchmarking k8s without the guarantees of etcd. At some point, you are just competing with clusterssh.
How often do you have sudden host failures? Especially if you use a half-decent server with redundant components for the DB node? Once in maybe 10 years?
The node failure rate is much higher than that. On a 1M node cluster of cloud-managed instances (AWS, GCP, Azure, etc.) you'd likely see failures a few times a month, if not more.
Re: K8s with 1M nodes
#48Earlier quoted context omitted.
etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.
The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct. https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute e…
It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).
Re: K8s with 1M nodes
#49That's really impressive and an interesting experiment.
I was about to say that Nomad did something similar, but that was 2 million Docker containers across 6100 nodes, https://www.hashicorp.com/en/c2m
Re: K8s with 1M nodes
#50Earlier quoted context omitted.
This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?
This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
There are similar libraies in Elixir. Is the ecosystem for ML as developed as for python? Nope, but not every ML project needs the most obscure libraries etc.
(For the record I don't really see Erlang clusters as a replacement for k8s)