Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the largest known Kubernetes cluster

cloud.google.com

Post reply on HN
Print view View original

51–60 of 94 posts

Re: Building the largest known Kubernetes cluster

#51

Post by __turbobrew__ »

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb.

I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super clean right now, so the first step in getting off boltdb would be creating a clean interface that could be implemented by another db.

Re: Building the largest known Kubernetes cluster

#52

Post by __turbobrew__ »

PunchyHamster wrote: post #27
hazz99 wrote: post #3

I’m sure this work is very impressive, but these QPS numbers don’t seem particularly high to me, at least compared to existing horizontally scalable service patterns. Why is it hard for the kube control plane to hit these numbers? For instance, postgres can hit this sort of QPS easily, afaik. It’s not distributed, but I’m sure Vitess could do something similar. The query patterns don’t seem particularly complex eithe…

it's not really bottlenecked by the store but by the calculations performed on each pod schedule/creation. It's basically "take global state of node load and capacity, pick where to schedule it", and I'd imagine probably not running in parallel coz that would be far harder to manage.

The k8s scheduler lets you tweak how many nodes to look at when scheduling a pod (percentage of nodes to score) so you can change how big “global state” is according to the scheduler algorithm.

Re: Building the largest known Kubernetes cluster

#53

Post by __turbobrew__ »

preisschild wrote: post #39
jakupovic wrote: post #13

Doing this at anything > 1k nodes is a pain in the butt. We decided to run many <100 nodes clusters rather than a few big ones.

Meh, I've had had clusters with close to 1k nodes (w/ cilium as CNI) and didnt have major issues

When I was involved about a year ago, cilium falls apart at around a few thousand nodes.

One of the main issues of cilium is that the bpf maps scale with the number of nodes/pods in the cluster, so you get exponential memory growth as you add more nodes with the cilium agent on them. https://docs.cilium.io/en/stable/operations/performance/scal...

Re: Building the largest known Kubernetes cluster

#54

Post by tayo42 »

Nextgrid wrote: post #42

K8S clusters on VMs strike me as odd. I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality? And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.

In a large organization their more efficient to run on VMS. You can colocate services that fit together on one machine.

And in reality no one sizes their machines correctly. They always do some handwavey thing like we need 4 cores, but maybe well burst and maybe there will be an outage so lets double it. Now all that utilization can be watched and you can take advantage of over subscription.

Re: Building the largest known Kubernetes cluster

#55

Post by oasisaimlessly »

__turbobrew__ wrote: post #53
preisschild wrote: post #39

Earlier quoted context omitted.

Meh, I've had had clusters with close to 1k nodes (w/ cilium as CNI) and didnt have major issues

When I was involved about a year ago, cilium falls apart at around a few thousand nodes. One of the main issues of cilium is that the bpf maps scale with the number of nodes/pods in the cluster, so you get exponential memory growth as you add more nodes with the cilium agent on them. https://docs.cilium.io/en/stable/operations/performance/scal...

Wouldn't that be quadratic rather than exponential?

Re: Building the largest known Kubernetes cluster

#58

Post by iwontberude »

__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

For those not aware, if you create too many resources you can easily use up all of the 8GB hard coded maximum size in etcd which causes a cluster failure. With compaction and maintenance this risk is mitigated somewhat but it just takes one misbehaving operator or integration (e.g. hundreds of thousands of dex session resources created for pingdom/crawlers) to mess everything up. Backups of etcd are critical. That dex example is why I stopped it for my IDP.

Re: Building the largest known Kubernetes cluster

#59

Post by travem »

phrotoma wrote: post #9
hazz99 wrote: post #3

I’m sure this work is very impressive, but these QPS numbers don’t seem particularly high to me, at least compared to existing horizontally scalable service patterns. Why is it hard for the kube control plane to hit these numbers? For instance, postgres can hit this sort of QPS easily, afaik. It’s not distributed, but I’m sure Vitess could do something similar. The query patterns don’t seem particularly complex eithe…

I am extremely Not A Database Person but I understand that the rationale for Kubernetes adopting etcd as its preferred data store was more about its distributed consistency features and less about query throughput. etcd is slower cause it's doing RAFT things and flushing stuff to disk. Projects like kine allow K8s users to swap sqlite or postgres in place of etcd which (I assume, please correct me otherwise) would de…

There are also distributed databases that use RAFT but can still scale while delivering distributed consensus don’t is not a challenge that can’t be solved. For example, TiDB handles millions of QPS while delivering ACID transactions, e.g. https://vivekbansal.substack.com/p/system-design-study-how-f...

Re: Building the largest known Kubernetes cluster

#60

Post by locknitpicker »

__turbobrew__ wrote: post #51

It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from. Etcd is the main bottleneck in upstream k8s and it seems like there is no real steam to build an upstream replacement for etcd/boltdb. I did poke around a while ago to see what interfaces that etcd has calling into boltdb, but the interface doesn’t seem super cl…

> It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from.

I'm not so sure. I mean, everything has tradeoffs, and what you need to do to put together the largest cluster known to man is not necessarily what you want to have to put together a mundane cluster.

Post reply on HN