Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the largest known Kubernetes cluster

cloud.google.com

Post reply on HN
Print view View original

71–80 of 94 posts

Re: Building the largest known Kubernetes cluster

#71

Post by xyse53 »

dijit wrote: post #16
xyse53 wrote: post #4

They mention GCS fuse. We've had nothing but performance and stability problems with this. We treat it as a best effort alternative when native GCS access isn't possible.

fuse based filesystems in general shouldn’t be treated as production ready in my experience. They’re wonderful for low volume, low performance and low reliability operations. (browsing, copying, integrating with legacy systems that do not permit native access), but beyond that they consume huge resources and do odd things when the backend is not in its most ideal state.

I think it's possible to write a solid fuse filesystem. Not as performant as in-kernel but it could easily not be the bottleneck depending on the backend.

I commented though because GCP highlights it in a few places as component for AI workloads. I'm curious if anyone is using it in an important application and happy with it.

Re: Building the largest known Kubernetes cluster

#72

Post by thundergolfer »

dijit wrote: post #30
thundergolfer wrote: post #28

Earlier quoted context omitted.

AWS Lambda uses FUSE and that’s one of the largest prod systems in the world.

An option exists, but they prefer you use the block storage API.

No, as in Lambda itself uses FUSE as an implementation detail of their container filesystem.

Re: Building the largest known Kubernetes cluster

#73

Post by dilyevsky »

__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

It's totally possible to run tens of thousands of QPS on etcd if your disks are NVMEs (or if you disable fdatasync which is not recommended). If you use kine+cockroachdb or tidb you can go even higher which is what I'm guessing is equivalent to their spanner setup.

Re: Building the largest known Kubernetes cluster

#74

Post by dilyevsky »

iwontberude wrote: post #58
__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

For those not aware, if you create too many resources you can easily use up all of the 8GB hard coded maximum size in etcd which causes a cluster failure. With compaction and maintenance this risk is mitigated somewhat but it just takes one misbehaving operator or integration (e.g. hundreds of thousands of dex session resources created for pingdom/crawlers) to mess everything up. Backups of etcd are critical. That de…

It's not hardcoded and you can increase it via flag.

Re: Building the largest known Kubernetes cluster

#75

Post by dijit »

thundergolfer wrote: post #72
dijit wrote: post #30

Earlier quoted context omitted.

An option exists, but they prefer you use the block storage API.

No, as in Lambda itself uses FUSE as an implementation detail of their container filesystem.

It seems there were some major issues, but AWS has developed around them and optimised for its needs; (https://www.madebymikal.com/on-demand-container-loading-in-a...)

Fair, but far from a common advice I’m willing to tell people (other CTOs) to do.

Re: Building the largest known Kubernetes cluster

#76

Post by stevefan1999 »

OvervCW wrote: post #38
senorrib wrote: post #29

Earlier quoted context omitted.

No a k8s dev, but I feel like this is the answer. K8s isn't usually just scheduling pods round robin or at random. There's a lot of state to evaluate, and the problem of scheduling pods becomes an NP-hard problem similar to bin packing problem. I doubt the implementation tries to be optimal here, but it feels a computationally heavy problem.

In what way is it NP-hard? From what I can gather it just eliminates nodes where the pod wouldn't be allowed to run, calculates a score for each and then randomly selects one of the nodes that has the lowest score, so trivially parallelizable.

That's greedy

Re: Building the largest known Kubernetes cluster

#77

Post by preisschild »

__turbobrew__ wrote: post #53
preisschild wrote: post #39

Earlier quoted context omitted.

Meh, I've had had clusters with close to 1k nodes (w/ cilium as CNI) and didnt have major issues

When I was involved about a year ago, cilium falls apart at around a few thousand nodes. One of the main issues of cilium is that the bpf maps scale with the number of nodes/pods in the cluster, so you get exponential memory growth as you add more nodes with the cilium agent on them. https://docs.cilium.io/en/stable/operations/performance/scal...

Thats true and I definitely had to "tune" the bpf map limits, but it wasn't really that difficult to do.

Re: Building the largest known Kubernetes cluster

#79

Post by ozgrakkurt »

__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

There was a blogpost about creating an alternative to etcd for super high scale kubernetes cluster. All code was open too. It was from someone named Benjamin I think but not sure.

I’m not able to find the blogpost but maybe someone else can!

Post reply on HN