Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the largest known Kubernetes cluster

cloud.google.com

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Building the largest known Kubernetes cluster

#21

Post by jcims »

rvz wrote: post #2

> While we don’t yet officially support 130K nodes, we're very encouraged by these findings. If your workloads require this level of scale, reach out to us to discuss your specific needs Obviously this is a typical experiment at Google on running a K8s cluster at 130K nodes but if there is a company out their that "requires" this scale, I must question their architecture and their infrastructure costs. But of course…

I work for a mature public company that most people in the US have at least heard of. We're far from the largest in our industry and we run jobs with more than that almost every night. Not via k8s though.

Re: Building the largest known Kubernetes cluster

#25

Post by DougBTX »

ZeroCool2u wrote: post #20
Sayrus wrote: post #18

Earlier quoted context omitted.

GKE uses Spanner as an etcd replacement.

But, and I'm honestly asking, you as a GKE user don't have to manage that spanner instance, right? So, you should in theory be able to just throw higher loads at it and spanner should be autoscaling?

Yes, from the article:

> To support the cluster’s massive scale, we relied on a proprietary key-value store based on Google’s Spanner distributed database... We didn’t witness any bottlenecks with respect to the new storage system and it showed no signs of it not being able to support higher scales.

Re: Building the largest known Kubernetes cluster

#26

Post by chrisandchris »

belter wrote: post #7

130k nodes...cute...but can Google conquer the ultimate software engineering challenge they warn you about in CS school? A functional online signup flow?

The could team up with Microsoft, because their signup flow is fine but the login flow is badly broken.

Re: Building the largest known Kubernetes cluster

#27

Post by PunchyHamster »

hazz99 wrote: post #3

I’m sure this work is very impressive, but these QPS numbers don’t seem particularly high to me, at least compared to existing horizontally scalable service patterns. Why is it hard for the kube control plane to hit these numbers? For instance, postgres can hit this sort of QPS easily, afaik. It’s not distributed, but I’m sure Vitess could do something similar. The query patterns don’t seem particularly complex eithe…

it's not really bottlenecked by the store but by the calculations performed on each pod schedule/creation.

It's basically "take global state of node load and capacity, pick where to schedule it", and I'd imagine probably not running in parallel coz that would be far harder to manage.

Re: Building the largest known Kubernetes cluster

#28

Post by thundergolfer »

dijit wrote: post #16
xyse53 wrote: post #4

They mention GCS fuse. We've had nothing but performance and stability problems with this. We treat it as a best effort alternative when native GCS access isn't possible.

fuse based filesystems in general shouldn’t be treated as production ready in my experience. They’re wonderful for low volume, low performance and low reliability operations. (browsing, copying, integrating with legacy systems that do not permit native access), but beyond that they consume huge resources and do odd things when the backend is not in its most ideal state.

AWS Lambda uses FUSE and that’s one of the largest prod systems in the world.

Re: Building the largest known Kubernetes cluster

#29

Post by senorrib »

PunchyHamster wrote: post #27
hazz99 wrote: post #3

I’m sure this work is very impressive, but these QPS numbers don’t seem particularly high to me, at least compared to existing horizontally scalable service patterns. Why is it hard for the kube control plane to hit these numbers? For instance, postgres can hit this sort of QPS easily, afaik. It’s not distributed, but I’m sure Vitess could do something similar. The query patterns don’t seem particularly complex eithe…

it's not really bottlenecked by the store but by the calculations performed on each pod schedule/creation. It's basically "take global state of node load and capacity, pick where to schedule it", and I'd imagine probably not running in parallel coz that would be far harder to manage.

No a k8s dev, but I feel like this is the answer. K8s isn't usually just scheduling pods round robin or at random. There's a lot of state to evaluate, and the problem of scheduling pods becomes an NP-hard problem similar to bin packing problem. I doubt the implementation tries to be optimal here, but it feels a computationally heavy problem.

Re: Building the largest known Kubernetes cluster

#30

Post by dijit »

thundergolfer wrote: post #28
dijit wrote: post #16

Earlier quoted context omitted.

fuse based filesystems in general shouldn’t be treated as production ready in my experience. They’re wonderful for low volume, low performance and low reliability operations. (browsing, copying, integrating with legacy systems that do not permit native access), but beyond that they consume huge resources and do odd things when the backend is not in its most ideal state.

AWS Lambda uses FUSE and that’s one of the largest prod systems in the world.

An option exists, but they prefer you use the block storage API.
Post reply on HN