Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intelligent Kubernetes Load Balancing at Databricks

databricks.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Intelligent Kubernetes Load Balancing at Databricks

#21

Post by atombender »

hanikesn wrote: post #20
shizcakes wrote: post #4

Less featureful than this, but we’ve been doing GRPC client side load balancing with kuberesolver[1] since 2018. It allows GRPC to handle the balancer implementations. It’s been rock solid for more than half a decade now. 1: https://github.com/sercand/kuberesolver

I've been using a standardized xds resolver[1]. The benefit here is that you don't have to patch grpc clients. [1] https://github.com/wongnai/xds

Do you know how this compares to the Nginx ingress controller, which has a native gRPC mode?

Re: Intelligent Kubernetes Load Balancing at Databricks

#22

Post by walth »

bbkane wrote: post #3

Thanks for writing - I found the Power of Two Choices algorithm particularly interesting (I haven't seen it before). From the recent grpConf ( https://www.youtube.com/playlist?list=PLj6h78yzYM2On4kCcnWjl... ) it seems gRPC as a standard is also moving in this "proxyless" model - gRPC will read xDS itself.

You might be interested in nginx's implementation

https://nginx.org/en/docs/http/ngx_http_upstream_module.html...

Re: Intelligent Kubernetes Load Balancing at Databricks

#23

Post by pm90 »

thewisenerd wrote: post #6

we have the same issue with HTTP as well, due to HTTP keepalive, which many clients have out-of-the box. the "impact" can be reduced by configuring an overall connection-ttl, so it takes some time when new pods come up but it works out over time. -- that said, i'm not surprised that even a company as large as databricks feels that adding a service mesh is going to add operational complexity. looks like they've taken…

I haven’t been keeping up but is there still hype over full mesh like istio/linkerd? Ive seen it tried in a couple of places but didn’t work super well; the last place couldn’t because datadog apparently bills sidecar containers as additional hosts so using sidecar proxy would have doubled our datadog bill.

Re: Intelligent Kubernetes Load Balancing at Databricks

#24

Post by dastbe »

jedberg wrote: post #11

I wonder why they didn't use rendezvous hashing (aka HRW)[0]? It feels like it would solve all the requirement that they laid out, is fully client side, and doesn't require real time updates for the host list via discovery. [0] https://en.wikipedia.org/wiki/Rendezvous_hashing

the problem is that they want to apply a number of stateful/lookaside load balancing strategies, which become more difficult to do in a fully decentralized system. it’s generally easier to asynchronously aggregate information and either decide routing updates centrally or redistribute that aggregate to inform local decisions.

Re: Intelligent Kubernetes Load Balancing at Databricks

#26

Post by dastbe »

pm90 wrote: post #23
thewisenerd wrote: post #6

we have the same issue with HTTP as well, due to HTTP keepalive, which many clients have out-of-the box. the "impact" can be reduced by configuring an overall connection-ttl, so it takes some time when new pods come up but it works out over time. -- that said, i'm not surprised that even a company as large as databricks feels that adding a service mesh is going to add operational complexity. looks like they've taken…

I haven’t been keeping up but is there still hype over full mesh like istio/linkerd? Ive seen it tried in a couple of places but didn’t work super well; the last place couldn’t because datadog apparently bills sidecar containers as additional hosts so using sidecar proxy would have doubled our datadog bill.

> the last place couldn’t because datadog apparently bills sidecar containers as additional hosts so using sidecar proxy would have doubled our datadog bill.

that seems like the tail wagging the dog

Post reply on HN