Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K8s with 1M nodes

bchess.github.io

Post reply on HN
Print view View original

51–60 of 85 posts

Re: K8s with 1M nodes

#51

Post by jamesblonde »

__turbobrew__ wrote: post #31

I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…

Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it?

Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...

Re: K8s with 1M nodes

#52

Post by throwdbaaway »

This looks impressive. As someone who is not familiar with ML, I do have a question -- surely in 2025 there must be a way to schedule a large pytorch job across multiple k8s clusters? EKS and GKE already provide VPC native flat network by default .

Re: K8s with 1M nodes

#53

Post by czhu12 »

If anyone is looking for a gentler, Heroku like onramp to Kubernetes, its exactly why I built Canine [1].

In retrospect, at my previous company, what we really needed in the early days was something that was Heroku-like (don't make me think about infra (!)) but could be easily added to and scaled up over time, as our service grew. We eventually grew to about 10M users, using the site monthly, and had to do a huge effort to migrate to Kubernetes.

Canine's philosophy is: full Kubernetes, with a deployment layer on top. If you ever out grow it, just dump Canine entirely, and work directly with the Kubernetes system it's operating. It even gives you all the K8s YAML config needed to offboard.

It's also similar to how the dev infra works at Airbnb (where I worked before that) -- Kubernetes underneath, a user friendly interface on top.

Re: K8s with 1M nodes

#54

Post by rootlocus »

czhu12 wrote: post #53

If anyone is looking for a gentler, Heroku like onramp to Kubernetes, its exactly why I built Canine [1]. In retrospect, at my previous company, what we really needed in the early days was something that was Heroku-like (don't make me think about infra (!)) but could be easily added to and scaled up over time, as our service grew. We eventually grew to about 10M users, using the site monthly, and had to do a huge eff…

You're not the first one to think that k9 is a good name for a kubernetes related technology https://k9scli.io

Re: K8s with 1M nodes

#55

Post by davidgl »

jamesblonde wrote: post #51
__turbobrew__ wrote: post #31

I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a…

Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it? Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...

See https://github.com/k3s-io/kine, k3s uses this to shim etcd to MySQL, Postgres and sqlite

Re: K8s with 1M nodes

#58

Post by rvz »

> Many limitations are imposed by software.

Or the amount of funding a startup has.

The bottom line is, you are not OpenAI or Google.

Re: K8s with 1M nodes

#59

Post by dbattaglia »

brimstedt wrote: post #56

It's an interesting and fun experiment, but what are real usecases for such a cluster?

At my last employer Elastic we definitely ran into these limits on the cloud SaaS team moving Elastocsearch node containers from our proprietary orchestration to k8s. I’m not sure how they eventually solved it but I believe the plan was essentially sharding ES clusters to different regional k8s clusters.

Re: K8s with 1M nodes

#60

Post by sailingparrot »

throwdbaaway wrote: post #52

This looks impressive. As someone who is not familiar with ML, I do have a question -- surely in 2025 there must be a way to schedule a large pytorch job across multiple k8s clusters? EKS and GKE already provide VPC native flat network by default .

The issue isn’t so much scheduling as it is stability.

More clusters means one more layer of things that can crash your (very expensive) training.

You also then still need to write tooling to manage cross cluster trainings correctly just as starting/stopping roughly at the same time, resuming from checkpoints, node health monitoring etc.

Nothing dealbreaking, but if it could just work in a single cluster that would be nicer.

Post reply on HN