Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

131–140 of 243 posts

Re: Load Balancing

#131

Post by interroboink »

A small note on "least connections" load balancing. The article says:

    Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has.
and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the load balancer. So, the load balancer doesn't know how busy the server is, unless that information is communicated separately, through some other channel. One variation is "least sessions", where the number of active sessions rather than connections is used, and as mentioned, that info is stored separately from the "main" connection, and available on the load balancer for its decision-making.

Example: https://www.youtube.com/watch?v=tAAmZ3bz8AA&t=681s

Re: Load Balancing

#132

Post by jedberg »

AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

Leastconn is basically a pull model. The server sends the TCP close to the load balancer, which is its way of saying "I'm done with that request and I'm ready for the next one". It's essentially pulling the next connection from the LB.

Re: Load Balancing

#133

Post by strunz »

interroboink wrote: post #131

A small note on "least connections" load balancing. The article says: Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has. and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the loa…

Yeah this was the most confusing part of the whole article to me, and seemed like the most important. I was like "how is it keeping track of how many connections are still being processed?"

Re: Load Balancing

#134

Post by strunz »

Nijikokun wrote: post #109

Interactive diagrams and visuals are worth so much with these topics. I just created documentation at ngrok (full disclosure, I work there!) for how to do load balancing with their edges product for this the other day! Now I have something to refer people to when describing these. https://ngrok.com/docs/guides/how-to-round-robin-load-balanc...

Boy have I got the perfect site for you - https://ciechanow.ski/archives/

Re: Load Balancing

#136

Post by kccqzy »

interroboink wrote: post #131

A small note on "least connections" load balancing. The article says: Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has. and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the loa…

I would in fact argue that in large systems it is infeasible to have a load-balancer sit in between. Especially if you adopt a microservice-based architecture: it doesn't make sense for every microservice to have its own load balancer to proxy all traffic. Instead the load balancer simply assigns traffic to specific services without being in the middle. It is the client's responsibility to obey the load balancer's decision.

Re: Load Balancing

#137

Post by edude03 »

I love this simulation it's the best visualization I've seen of load balancing concepts.

One thing though - in my experience the biggest challenge I typically see is that different requests take different amounts of time for example you're running multiple instances of your monolith and there is an endpoint that returns a static response and another one that generates a huge report.

This is actually something that can be handled with a load balancer that can introspect layer 7 but that's a whole other thing

Re: Load Balancing

#138

Post by AYBABTME »

jedberg wrote: post #132
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

Leastconn is basically a pull model. The server sends the TCP close to the load balancer, which is its way of saying "I'm done with that request and I'm ready for the next one". It's essentially pulling the next connection from the LB.

Yes but only if connections aren't reused.

Re: Load Balancing

#139

Post by bityard »

naikrovek wrote: post #72

Why is there not a queue system where a server can pull requests when it's ready to handle one? Why must the work to determine where to send a request go to the load balancer, when you could just pull from a queue and save yourself the trouble of finding the exact right load balancer strategy for your application?

The short answer is that it's more complicated to engineer and has its own set of trade-offs. With the push-based models shown in the article, everything works regular HTTP(S), no fancy routing is needed, and the LB needs to keep very little state to do its job.

Re: Load Balancing

#140

Post by bityard »

AYBABTME wrote: post #100
orthecreedence wrote: post #99

Earlier quoted context omitted.

The pull model works beautifully for background work queues. For load balancing it seems it would take a bit of re-plumbing on the expectations of request/response. How would you envision this working with something like HTTP? It seems it would reverse the direction, which would be interesting.

Very similar to background queues but the load balancer holds the request and dispatches it to a waiting worker, or waits until someone comes to serve it. While the worker handles the request, the load balancer proxies. The request serving is the same, the way the work is dispatched is just inverted.

Seems like this wouldn't work as well in situations where you care a lot about latency or resource utilization. If the workers are polling the LB every n ms, then that's an average of n / 2 ms added to _every_ request. Plus additional CPU cycles and network traffic on both ends due to the polling mechanism.
Post reply on HN