Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

141–150 of 243 posts

Re: Load Balancing

#141

Post by samwho »

interroboink wrote: post #131

A small note on "least connections" load balancing. The article says: Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has. and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the loa…

I didn't really want to go in to direct server return. I think most people will be more familiar with using something like nginx as a reverse proxy and load balancer at the same time. It is very cool tech, though. :)

Re: Load Balancing

#142

Post by samwho »

edude03 wrote: post #137

I love this simulation it's the best visualization I've seen of load balancing concepts. One thing though - in my experience the biggest challenge I typically see is that different requests take different amounts of time for example you're running multiple instances of your monolith and there is an endpoint that returns a static response and another one that generates a huge report. This is actually something that ca…

I do talk about variable request cost a fair bit, but I really don't think I did a good job visualising it. The rate at which each request shrinks while it's being processed by a server is meant to represent cost, but it's pretty subtle and easy to miss unless you're really watching closely.

If I were to do this again, making request cost more obvious would be something I'd like to do. I had initially had requests by variable in size on screen, and moved away from that because load balancers don't typically know the cost of a request up front.

Re: Load Balancing

#143

Post by hinkley »

jedberg wrote: post #132
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

Leastconn is basically a pull model. The server sends the TCP close to the load balancer, which is its way of saying "I'm done with that request and I'm ready for the next one". It's essentially pulling the next connection from the LB.

I'm still looking out for a better explanation of why 'choose 2' is so much more empirically effective than intuition says it should be.

Other than the fact that randomization avoids pathological cases, nothing I know about distributed computing or queuing theory really gets to the bottom of it. It feels like there should be more to it than that.

Re: Load Balancing

#144

Post by martin_ »

bityard wrote: post #140
AYBABTME wrote: post #100

Earlier quoted context omitted.

Very similar to background queues but the load balancer holds the request and dispatches it to a waiting worker, or waits until someone comes to serve it. While the worker handles the request, the load balancer proxies. The request serving is the same, the way the work is dispatched is just inverted.

Seems like this wouldn't work as well in situations where you care a lot about latency or resource utilization. If the workers are polling the LB every n ms, then that's an average of n / 2 ms added to _every_ request. Plus additional CPU cycles and network traffic on both ends due to the polling mechanism.

couldn’t the workers longpoll vs interval based?

Re: Load Balancing

#145

Post by hinkley »

AYBABTME wrote: post #138
jedberg wrote: post #132

Earlier quoted context omitted.

Leastconn is basically a pull model. The server sends the TCP close to the load balancer, which is its way of saying "I'm done with that request and I'm ready for the next one". It's essentially pulling the next connection from the LB.

Yes but only if connections aren't reused.

I keep having to turn off keepalive between the service and the reverse proxies and I bet this is part of why.

I think Node 12 introduced LIFO queuing for connection pools because they found the cost of silent disconnects from the server to be too high. They got much better histograms by using the most recently freed connection instead of round robin of available sockets.

Re: Load Balancing

#146

Post by hinkley »

p4l4g4 wrote: post #126
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

I like the idea. I guess you could do something with pubsub to simulate this model. A regular http stack (load balancer and reverse proxies in front of some http framework) would probably need major modifications to work. It would be great if such a mechanism was part of the http standard, so you could easily connect compliant tools.

You can get a lot of traction with long polling. I use that in a couple of places where we talk to Consul to avoid a lot of polling nonsense. If I could figure out one last spot (where it's buried behind a service call due to separation of concerns), I could remove a lot of the remaining jitter from a rendezvous algorithm (tell me when all servers have made a change)

Re: Load Balancing

#147

Post by akhayam »

notyourwork wrote: post #62
akhayam wrote: post #9

If a picture is worth a thousand words.... a well done animation is gotta be at least 10K words. Thanks for making an old topic fun to read about again!

100% agree, although I have experience and knew the material, I read this through because it was a pleasure to read and the visualizations were engaging. Well done!

[dead]

Re: Load Balancing

#149

Post by ed_balls »

zokier wrote: post #71

Least connections is intuitively very sensible algorithm, but it has one pitfall: if backend server starts returning errors for whatever reason (e.g. load shedding) quicker than typical responses, it can lead to situation where disproportionate number of requests are directed to that one server. Also typically you'd have more than one load balancer instance, and at least naive least connections requires shared state…

you can improve this be picking top 3 server with least connections and then pick one randomly from these 3 servers.

Re: Load Balancing

#150

Post by zokier »

throwaway894345 wrote: post #75

I’ve always been more curious about how load balancers are supposed to be highly available? Presumably you run many instances, but then it seems like you need something to balance the load across your load balancer instances? Also, how do many load balancer instances listen on the same IP address (or do they use DNS to map a domain name to multiple IP addresses?)?

Floating/Virtual IPs are one networking solution for HA. With Corosync/Pacemaker a cluster of hosts can decide how the ips are then assigned to physical hosts. CARP maybe/probably can do the same. I just googled and apparently there is also some IPVS/LVS project for Linux for HA loadbalancing?
Post reply on HN