Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K8s with 1M nodes

bchess.github.io

Post reply on HN
Print view View original

21–30 of 85 posts

Re: K8s with 1M nodes

#21

Post by jeffinhat »

This is an awesome experiment and write up. I really appreciate the reproducibility.

I would like to see how moving to database that scales write throughput with replicas would behave, namely FoundationDB. I think this will require more than an intermediary like kine to be efficient, as the author illustrates the apisever does a fair bit of its own watching and keeping state. I also think there's benefit, at least for blast radius, to shard the server by api group or namespace.

I think years ago this would have been a non starter with the community, but given AWS has replaced etcd (or at least aspects) with their internal log service for their large cluster offering, I bet there's some appetite for making this interchangable and bringing and open source solution to market.

I share the authors viewpoint that for modern cloud based deployments, you're probably best avoiding it and relying on VMs being stable and recoverable. I think reliability does matter if you want to actually realize the "borg" value and run it on bare metal across a serious fleet. I haven't found the business justification to work on that though!

Re: K8s with 1M nodes

#22

Post by theptip »

wppick wrote: post #18

If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art

This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?

Re: K8s with 1M nodes

#23

Post by [deleted] »

itsnowandnever wrote: post #20
trenchpilgrim wrote: post #14

Earlier quoted context omitted.

The API and controller loops are the point of k8s. etcd is an implementation detail and lots of clusters swap it out for something else like sqlite. I'm pretty sure that GCP and Azure are using Spanner or Cosmos instead of etcd for their managed offerings.

not exactly a fair assessment since neither of those were out and/or available to the kubernetes team at the time. sure, some things at many times from now into eternity may be or become better suited for the kubernetes data plane but at the time if etcd wasn't used there would be no kubernetes today

[deleted]

Re: K8s with 1M nodes

#24

Post by alphabettsy »

trenchpilgrim wrote: post #14
itsnowandnever wrote: post #5

Earlier quoted context omitted.

etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.

The API and controller loops are the point of k8s. etcd is an implementation detail and lots of clusters swap it out for something else like sqlite. I'm pretty sure that GCP and Azure are using Spanner or Cosmos instead of etcd for their managed offerings.

Yep. K3s can use SQLite or Postgres.

Re: K8s with 1M nodes

#25

Post by trenchpilgrim »

itsnowandnever wrote: post #20
trenchpilgrim wrote: post #14

Earlier quoted context omitted.

The API and controller loops are the point of k8s. etcd is an implementation detail and lots of clusters swap it out for something else like sqlite. I'm pretty sure that GCP and Azure are using Spanner or Cosmos instead of etcd for their managed offerings.

not exactly a fair assessment since neither of those were out and/or available to the kubernetes team at the time. sure, some things at many times from now into eternity may be or become better suited for the kubernetes data plane but at the time if etcd wasn't used there would be no kubernetes today

The Kubernetes team chose etcd specifically because they were trying to replace Borg's master/slave database at Google. Nothing about Kubernetes requires etcd; the team was trying to solve a Google-internal problem with it (and in the end, didn't gain traction within Google.) k3s uses sqlite by default which was an option at the time, other clusters today use PostgreSQL.

Have you looked at the etcd keys and values in a Kubernetes cluster? It's a _remarkably_ simple schema you could do in pretty much any database with fast prefix or path scans.

Re: K8s with 1M nodes

#26

Post by trenchpilgrim »

itsnowandnever wrote: post #16
geoctl wrote: post #8

Earlier quoted context omitted.

Is it? I honestly kinda believe that etcd is probably the weakest point in vanilla k8s. It is simply unsuitable for heavy write environments and causes lots of consistency problems under heavy write loads, it's generally slow, it has value size constraints, it offers very primitive querying, etc... Why not replace etcd altogether with something like Postgres + Redis/NATS?

that touches on what I consider the dichotomy of k8s: it's a really scalable system that makes it easy to spin up a cluster locally on your laptop and interact with the full API locally just like in prod. so it's a super scalable system with a dense array of features. but paradoxically most shops won't need the vast majority of k8s features ever and by the time they scale to where they do need a ton of distributed in…

The other draw: Because k8s is open, you can easily hire employees, contractors, consultants and vendors and have them immediately solve problems within the k8s ecosystem. If you run a bespoke system, you have to train engineers on the system before they can make large contributions.

Re: K8s with 1M nodes

#27

Post by pas »

itsnowandnever wrote: post #16
geoctl wrote: post #8

Earlier quoted context omitted.

Is it? I honestly kinda believe that etcd is probably the weakest point in vanilla k8s. It is simply unsuitable for heavy write environments and causes lots of consistency problems under heavy write loads, it's generally slow, it has value size constraints, it offers very primitive querying, etc... Why not replace etcd altogether with something like Postgres + Redis/NATS?

that touches on what I consider the dichotomy of k8s: it's a really scalable system that makes it easy to spin up a cluster locally on your laptop and interact with the full API locally just like in prod. so it's a super scalable system with a dense array of features. but paradoxically most shops won't need the vast majority of k8s features ever and by the time they scale to where they do need a ton of distributed in…

but it's also standard, you can hire for it, outsource it, etc.

and it's pretty modular too, so it can even serve as the host for the bespoke whatever that's needed

though I remember reading the fly.io blog post about their custom scheduler/allocator which illustrates nicely how much of a difference a custom in-house solution makes if works well

Re: K8s with 1M nodes

#28

Post by halayli »

without publishing mem_etcd code, and without telling us what happens when one of the etcd/mem_etcd node dies to compare, this write up doesn't provide much information.

Re: K8s with 1M nodes

#29

Post by merb »

stackskipton wrote: post #3
ktpsns wrote: post #2

Typical large scale high performance computing clusters are at a size of 10k nodes (for instance Jupiter and SuperMUC in Germany) [1]. These centers are quite remarkably big buildings. I wonder how much 1M node single k8s clusters there are in the world right now. Most likely at the hyperscalers. [1] what is a node? Typically it is a synonym for "server". In some configurations HPC schedulers allow node sharing. Then…

I doubt any Hyperscalers are running 1M Node clusters either. They probably just have groups of clusters at each datacenter and some overall scheduler that determines which cluster is best suited for workload during deployment then connects to that cluster and schedules the workload.

Some hyperscalers even have services for that. Which even makes it possible to have cross cluster ingress. And other things. And it makes it possible to have multiple cluster ingress different regions that somewhat work together.

Re: K8s with 1M nodes

#30

Post by reactordev »

theptip wrote: post #22
wppick wrote: post #18

If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art

This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?

This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
Post reply on HN