Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the largest known Kubernetes cluster

cloud.google.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Building the largest known Kubernetes cluster

#81

Post by Saser »

ozgrakkurt wrote: post #79
__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

There was a blogpost about creating an alternative to etcd for super high scale kubernetes cluster. All code was open too. It was from someone named Benjamin I think but not sure. I’m not able to find the blogpost but maybe someone else can!

This might be what you're thinking of: https://bchess.github.io/k8s-1m/

Re: Building the largest known Kubernetes cluster

#82

Post by leo_e »

Papers like this are fascinating engineering, but dangerous marketing.

They convince every Series A startup that they need a multi-region federated control plane for their 50 microservices. I spend half my time convincing my team not to emulate Google, because we don't have Google's scale problems—we have velocity problems.

Complexity is an asset for Google (it's a moat), but a liability for the rest of us. I just want a cluster that doesn't require a dedicated ops team to upgrade.

Re: Building the largest known Kubernetes cluster

#84

Post by kritr »

__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

I’ve seen some talk of replacing etcd with FoundationDB, which could yield similar improvements.

Re: Building the largest known Kubernetes cluster

#85

Post by solatic »

zkmon wrote: post #78

What business usecase requires a single cluster with thousands of pods? Wouldn't having multiple clusters, each hosting a few namespaces, be a better architecture?

This. I may not work with AI training workflows, but I struggle to understand why they supposedly require launching a thousand pods per second to use GPUs that need to fundamentally be installed across different baremetal machines. Once the GPUs are on different machines, if there are 1k+ such machines, just start putting them on different Kubernetes clusters. Build a scheduling layer above the Kubernetes control plane to decide which Kubernetes cluster to schedule the pod onto.

The whole thing stinks of, AI investors are throwing money at AI companies, so go to GCP and tell them to solve the problem at any price so that they can keep scaling without needing to build the scheduling layer above the Kubernetes control planes.

Re: Building the largest known Kubernetes cluster

#86

Post by game_the0ry »

BruSwain wrote: post #50
game_the0ry wrote: post #33

Earlier quoted context omitted.

> You are not Google. 100% agree. People at my co are horny to adopt k8s. Really, tech leads want to put it on their resume ("resume driven development") and use a tool that was made to solve a particular problem we never had. The downside is now we now need to be proficient it at, know how to troubleshoot it, etc. It was sold to leadership as something that would make our lives easier but the exact opposite has happ…

I think k8s has a learning curve, absolutely, and there are absolutely cases where it can be unnecessary overhead. But I actually think those cases are pretty small. If you're running multiple apps, k8s is valuable. There is initial investment in learning the system, but its v-extensible, flexible, & portable. (Yes, every hyperscaler's implementation of k8s has its own nuance in certain places, but the core concept o…

We must be terrible at implementation bc we have a had a prod outage and our DX is objectively worse. Its caused more problems and headaches for us.

Re: Building the largest known Kubernetes cluster

#87

Post by dilyevsky »

rvz wrote: post #2

> While we don’t yet officially support 130K nodes, we're very encouraged by these findings. If your workloads require this level of scale, reach out to us to discuss your specific needs Obviously this is a typical experiment at Google on running a K8s cluster at 130K nodes but if there is a company out their that "requires" this scale, I must question their architecture and their infrastructure costs. But of course…

> You are not Google.

You think they are just running it for fun? It's literally non-Google customers who wanted this as was explained in the article

Re: Building the largest known Kubernetes cluster

#88

Post by scottyah »

game_the0ry wrote: post #33
rvz wrote: post #2

> While we don’t yet officially support 130K nodes, we're very encouraged by these findings. If your workloads require this level of scale, reach out to us to discuss your specific needs Obviously this is a typical experiment at Google on running a K8s cluster at 130K nodes but if there is a company out their that "requires" this scale, I must question their architecture and their infrastructure costs. But of course…

> You are not Google. 100% agree. People at my co are horny to adopt k8s. Really, tech leads want to put it on their resume ("resume driven development") and use a tool that was made to solve a particular problem we never had. The downside is now we now need to be proficient it at, know how to troubleshoot it, etc. It was sold to leadership as something that would make our lives easier but the exact opposite has happ…

Use killercoda and get your CKA, I bet most of the confusion will be gone. I've basically started mandating it for newer folks on my team since it covers so many of the gaps that get created by people who try Just In Time learning on the systems. K9s is great for visual people who are used to vim.

Re: Building the largest known Kubernetes cluster

#89

Post by scoodah »

prescriptivist wrote: post #70
scoodah wrote: post #62

Earlier quoted context omitted.

This is why I’ve always thought Tekton was a strange project. It feels inevitable that if you buy into Tekton CI/CD you will hit issues with etcd scaling due to the sheer number of resources you can wind up with.

What boundaries does this 8GB etcd limit cut across? We've been using Tekton for years now but each pipeline exists in its own namespace and that namespace is deleted after each build. Presumably that kind of wholesale cleanup process keeps the DB size in check, because we've never had a problem with Etcd size... We have multiple hundreds of resources allocated for each build and do hundreds of builds a day. The curr…

Yeah I mean if you’re deleting namespaces after each run then sure, that may solve it. They have a pruner now that you can enable too to set up retention periods for pipeline runs.

There’s also some issues with large Results, though I think you have to manually enable that. From their site

> CAUTION: the larger you make the size, more likely will the CRD reach its max limit enforced by the etcd server leading to bad user experience.

And then if you use Chains you’re opening up a whole other can of worms.

I contracted with a large institution that was moving all of their cicd to Tekton and they hit scaling issues with etcd pretty early in the process and had to get Red Hat to address some of them. If they couldn’t get them addressed by RH they were going to scrap the whole project.

Re: Building the largest known Kubernetes cluster

#90

Post by zkmon »

solatic wrote: post #85
zkmon wrote: post #78

What business usecase requires a single cluster with thousands of pods? Wouldn't having multiple clusters, each hosting a few namespaces, be a better architecture?

This. I may not work with AI training workflows, but I struggle to understand why they supposedly require launching a thousand pods per second to use GPUs that need to fundamentally be installed across different baremetal machines. Once the GPUs are on different machines, if there are 1k+ such machines, just start putting them on different Kubernetes clusters. Build a scheduling layer above the Kubernetes control pla…

Yep, it's just saying "you should now launch 1000's pods in a single cluster, just because we said it makes sense, and please don't look at the costs, business sense and operational issues."
Post reply on HN