Intelligent Kubernetes Load Balancing at Databricks
databricks.com
Intelligent Kubernetes Load Balancing at Databricks
1–10 of 30 posts
Re: Intelligent Kubernetes Load Balancing at Databricks
#2[deleted]
Re: Intelligent Kubernetes Load Balancing at Databricks
#3Thanks for writing - I found the Power of Two Choices algorithm particularly interesting (I haven't seen it before).
From the recent grpConf ( https://www.youtube.com/playlist?list=PLj6h78yzYM2On4kCcnWjl... ) it seems gRPC as a standard is also moving in this "proxyless" model - gRPC will read xDS itself.
Re: Intelligent Kubernetes Load Balancing at Databricks
#4Less featureful than this, but we’ve been doing GRPC client side load balancing with kuberesolver[1] since 2018. It allows GRPC to handle the balancer implementations. It’s been rock solid for more than half a decade now.
Re: Intelligent Kubernetes Load Balancing at Databricks
#5Less featureful than this, but we’ve been doing GRPC client side load balancing with kuberesolver[1] since 2018. It allows GRPC to handle the balancer implementations. It’s been rock solid for more than half a decade now. 1: https://github.com/sercand/kuberesolver
What is the difference between Kuberesolver and using a Headless Service?
In the README.md file, they compare it with a ClusterIP service, but not with a Headless on "ClusterIP: None".
The advantages of using Kuberesolver are that you do not need to change DNS refresh and cache settings. However, I think this is preferable to the application calling the Kubernetes API.
Re: Intelligent Kubernetes Load Balancing at Databricks
#6we have the same issue with HTTP as well, due to HTTP keepalive, which many clients have out-of-the box.
the "impact" can be reduced by configuring an overall connection-ttl, so it takes some time when new pods come up but it works out over time.
--
that said, i'm not surprised that even a company as large as databricks feels that adding a service mesh is going to add operational complexity.
looks like they've taken the best parts (endpoint watch, sync to clients with xDS) and moved it client-side. compared to the failure mode of a service mesh, this seems better.
Re: Intelligent Kubernetes Load Balancing at Databricks
#7> kube-proxy supports only basic algorithms like round-robin or random selection
this is "partially" true.
if you're using ipvs, you can configure the scheduler to just about anything ipvs supports (including wrr). they removed the validation for the scheduler name quite a while back.
kubernetes itself though doesn't "understand" (i.e., can NOT represent) the nuances (e.g., weights per endpoint with wrr), which is the problem.
Re: Intelligent Kubernetes Load Balancing at Databricks
#8Less featureful than this, but we’ve been doing GRPC client side load balancing with kuberesolver[1] since 2018. It allows GRPC to handle the balancer implementations. It’s been rock solid for more than half a decade now. 1: https://github.com/sercand/kuberesolver
kuberesolver is an interesting take as well. Directly watching the K8s API from each client could raise scaling concerns at very large scale, but it does open the door to using richer Kubernetes metadata for smarter load-balancing decisions. thanks for sharing!
Re: Intelligent Kubernetes Load Balancing at Databricks
#9we have the same issue with HTTP as well, due to HTTP keepalive, which many clients have out-of-the box. the "impact" can be reduced by configuring an overall connection-ttl, so it takes some time when new pods come up but it works out over time. -- that said, i'm not surprised that even a company as large as databricks feels that adding a service mesh is going to add operational complexity. looks like they've taken…
Yes, we’ve leaned toward minimizing operational overhead. Taking the useful parts of a mesh (xDS endpoint and routing updates) into the client has worked extremely well in practice and has been very reliable, without the extra moving parts of a full mesh.
Re: Intelligent Kubernetes Load Balancing at Databricks
#10we have the same issue with HTTP as well, due to HTTP keepalive, which many clients have out-of-the box. the "impact" can be reduced by configuring an overall connection-ttl, so it takes some time when new pods come up but it works out over time. -- that said, i'm not surprised that even a company as large as databricks feels that adding a service mesh is going to add operational complexity. looks like they've taken…
When we started we had a lot of pieces like Certificate Management in-house and adding a full blown Service Mesh was a big operational overhead. We started with building only the parts we needed and started integrating things like xDS natively in rest of our clients.