Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

151–160 of 243 posts

Re: Load Balancing

#151

Post by lowflyingcondor »

hinkley wrote: post #143
jedberg wrote: post #132

Earlier quoted context omitted.

Leastconn is basically a pull model. The server sends the TCP close to the load balancer, which is its way of saying "I'm done with that request and I'm ready for the next one". It's essentially pulling the next connection from the LB.

I'm still looking out for a better explanation of why 'choose 2' is so much more empirically effective than intuition says it should be. Other than the fact that randomization avoids pathological cases, nothing I know about distributed computing or queuing theory really gets to the bottom of it. It feels like there should be more to it than that.

https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2...

and

> Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches. https://brooker.co.za/blog/2012/01/17/two-random.html

Re: Load Balancing

#152

Post by lowflyingcondor »

ed_balls wrote: post #149
zokier wrote: post #71

Least connections is intuitively very sensible algorithm, but it has one pitfall: if backend server starts returning errors for whatever reason (e.g. load shedding) quicker than typical responses, it can lead to situation where disproportionate number of requests are directed to that one server. Also typically you'd have more than one load balancer instance, and at least naive least connections requires shared state…

you can improve this be picking top 3 server with least connections and then pick one randomly from these 3 servers.

Inverting this is actually better because it avoids the stale data / herd behavior. i.e. randomly choose X candidate servers and then pick the one with least load. also, 2 may be better than 3

https://brooker.co.za/blog/2012/01/17/two-random.html

Re: Load Balancing

#154

Post by er4hn »

Beautiful animations! I love how they make understanding this more intuitive and can be adjusted to play with different settings.

One thing that I generally felt is missing in these discussions is how TLS messes up load balancing strategies. I did my own research on what to do when you want to have TLS and not terminate at the load balancer. I covered that in a blog post available here: https://er4hn.info/blog/2023.02.18-tls-load-balancer/

Re: Load Balancing

#155

Post by ricardobeat »

samwho wrote: post #127
Abimelex wrote: post #124

Earlier quoted context omitted.

indeed, a really great post! In case you decide to change the colors or need another reason I also want to add, that in general red/green is difficult for color-blind people. About 8% of male population is affected by red/green color blindness.

The colour palette I used on the graphs is the Wong palette I found at https://davidmathlogic.com/colorblind/ . Or are you talking about the animations? I reached out to some colourblind folks I knew before publishing and they didn’t flag those, just the graphs. Happy to change the animations as well if they’re problematic. :)

The PEWMA vs LC bar graph uses red and green, and ends up looking like single set of brown bars.

Re: Load Balancing

#156

Post by hamilyon2 »

I always thought that for hundreds of backends and above 10K RPS those algorithms scale poorly. First of all, scaling beyond single load balancer becomes a problem. The cost of updating counters becomes significant. If responses are quick, the cost additional bookkeeping adds doesn't pay off.

Of you use weighted random all these problems go away.

Is my knowledge outdated?

Re: Load Balancing

#157

Post by null0pointer »

toast0 wrote: post #85
throwaway894345 wrote: post #75

I’ve always been more curious about how load balancers are supposed to be highly available? Presumably you run many instances, but then it seems like you need something to balance the load across your load balancer instances? Also, how do many load balancer instances listen on the same IP address (or do they use DNS to map a domain name to multiple IP addresses?)?

A classical load balancer runs in an HA hot-warm pair, with IP takeover --- when the secondary senses the primary has failed, it takes over the IP and begins serving. Depending on the type of load balancing and the software involved, this could be nearly seamless, or it could end all sessions in progress. If you want to run hot-hot load balancing on a single IP, it's generally done with routing protocols. Equal cost…

Awesome response. You've answered a lot of questions I've been wondering about. Thanks!

Re: Load Balancing

#158

Post by xyzzy4747 »

If the requests are similar (i.e. don't have unique user data in them), it's best to put a high performance reverse proxy in front that has some caching. This way you don't have to execute lots of code for every request.

Re: Load Balancing

#159

Post by danielheath »

bityard wrote: post #140
AYBABTME wrote: post #100

Earlier quoted context omitted.

Very similar to background queues but the load balancer holds the request and dispatches it to a waiting worker, or waits until someone comes to serve it. While the worker handles the request, the load balancer proxies. The request serving is the same, the way the work is dispatched is just inverted.

Seems like this wouldn't work as well in situations where you care a lot about latency or resource utilization. If the workers are polling the LB every n ms, then that's an average of n / 2 ms added to _every_ request. Plus additional CPU cycles and network traffic on both ends due to the polling mechanism.

If you care about that, having requests queue on a worker that’s already busy (while another worker is idle) is currently the most widely used alternative.

Re: Load Balancing

#160

Post by francislavoie »

lowflyingcondor wrote: post #152
ed_balls wrote: post #149

Earlier quoted context omitted.

you can improve this be picking top 3 server with least connections and then pick one randomly from these 3 servers.

Inverting this is actually better because it avoids the stale data / herd behavior. i.e. randomly choose X candidate servers and then pick the one with least load. also, 2 may be better than 3 https://brooker.co.za/blog/2012/01/17/two-random.html

Caddy has this with `lb_policy random_choose 2`. See https://caddyserver.com/docs/caddyfile/directives/reverse_pr....
Post reply on HN