I worked in DHTs in grad school. I still double take that Google and other companies "computers dedicated to a task" numbers are missing 2 digits from what I expected. We have a lot of room left for expansion, we just have to relax centralized management expectations.
Building the largest known Kubernetes cluster
41–50 of 94 posts
Re: Building the largest known Kubernetes cluster
#42K8S clusters on VMs strike me as odd.
I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality?
And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.
Re: Building the largest known Kubernetes cluster
#43I’m sure this work is very impressive, but these QPS numbers don’t seem particularly high to me, at least compared to existing horizontally scalable service patterns. Why is it hard for the kube control plane to hit these numbers? For instance, postgres can hit this sort of QPS easily, afaik. It’s not distributed, but I’m sure Vitess could do something similar. The query patterns don’t seem particularly complex eithe…
It says in the blog that they require 13,000 queries per second to update lease objects, not that 13,000 is the total for all queries. I don't know why they cite that instead of total, but etcd's normal performance testing indicates it can handle at least 50,000 writes per second and 180,000 reads: https://etcd.io/docs/v3.6/op-guide/performance/. So, without them saying what the real number is, I'm going to guess their reads and writes outside of lease updates are at least much larger than those numbers.
Re: Building the largest known Kubernetes cluster
#44Imagine a Beowulf cluster of these
Re: Building the largest known Kubernetes cluster
#45K8S clusters on VMs strike me as odd. I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality? And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.
Because k8s gives you lots of other things out of the box like easy scaling of apps etc. Harder to do on VM:s where you would either have to dedicate one VM per app (might be a waste of resources) or you have to try and deploy and run multiple apps on multiple VM:s etc.
(For the record I’m not a k8s fanatic. Most of the time a regular VM is better. But a VM isn’t = a kubernetes cluster).
Re: Building the largest known Kubernetes cluster
#46K8S clusters on VMs strike me as odd. I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality? And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.
The reason to target k8s on cloud vms is that cloud VMs don't subdivide as easily or as cleanly. Managing them is a pain. K8s is an abstraction layer for that - Rather than building whole machine images for each product, you create lighter weight docker images (how light weight is a point of some contention), and you only have to install your logging, monitoring, and etc once.
Your advice about bigger machines is spot on - K8s biggest problem is how relatively heavyweight the kublet is, with memory requirements of roughly half a gig. On a modern 128g server node that's a reasonable overhead, for small companies running a few workloads on 16g nodes it's a cost of doing business, but if you're running 8 or 4g nodes, it looks pretty grim for your utilization.
Re: Building the largest known Kubernetes cluster
#47K8S clusters on VMs strike me as odd. I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality? And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.
> I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality?
I think you're not familiar with Kubernetes and what features it provides.
For example, kubernetes supports blue-green deployments and rollbacks, software-defined networks, DNS, node-specific purges and taints, etc. Those are not hypervisor features.
Also, VMs are the primitives of some cloud providers.
It sounds like you heard about how Borg/Kubernetes was used to simplify the task of putting together clusters with COTS hardware and you didn't bothered to learn more about Kubernetes.
Re: Building the largest known Kubernetes cluster
#48K8S clusters on VMs strike me as odd. I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality? And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.
because if you just do a few huge VMs you still have all the problems that k8s solves out of the box. Except now you have to solve them yourself, which will likely end up being a crappier less robust version of kubernetes.
Re: Building the largest known Kubernetes cluster
#49They mention GCS fuse. We've had nothing but performance and stability problems with this. We treat it as a best effort alternative when native GCS access isn't possible.
fuse based filesystems in general shouldn’t be treated as production ready in my experience. They’re wonderful for low volume, low performance and low reliability operations. (browsing, copying, integrating with legacy systems that do not permit native access), but beyond that they consume huge resources and do odd things when the backend is not in its most ideal state.
I started rewriting gcsfuse using https://github.com/hanwen/go-fuse instead of https://github.com/jacobsa/fuse and found it rock-solid. FUSE has come a long way in the last few years, including things like passthrough.
Honestly, I'd give FUSE a second chance, you'd be surprised at how useful it can be -- after all, it's literally running in userland so you don't need to do anything funky with privileges. However, if I starting afresh on a similar project I'd probably be looking at using 9p2000.L instead.
Re: Building the largest known Kubernetes cluster
#50> While we don’t yet officially support 130K nodes, we're very encouraged by these findings. If your workloads require this level of scale, reach out to us to discuss your specific needs Obviously this is a typical experiment at Google on running a K8s cluster at 130K nodes but if there is a company out their that "requires" this scale, I must question their architecture and their infrastructure costs. But of course…
> You are not Google. 100% agree. People at my co are horny to adopt k8s. Really, tech leads want to put it on their resume ("resume driven development") and use a tool that was made to solve a particular problem we never had. The downside is now we now need to be proficient it at, know how to troubleshoot it, etc. It was sold to leadership as something that would make our lives easier but the exact opposite has happ…
I think k8s has a learning curve, absolutely, and there are absolutely cases where it can be unnecessary overhead. But I actually think those cases are pretty small. If you're running multiple apps, k8s is valuable. There is initial investment in learning the system, but its v-extensible, flexible, & portable. (Yes, every hyperscaler's implementation of k8s has its own nuance in certain places, but the core concept of k8s translates very well)