Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

121–130 of 243 posts

Re: Load Balancing

#121

Post by bombela »

AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

The pull model, is similar to having some metrics to choose which backend to push to.

I believe push/pull are the two faces of the same coin. You might physically initiate a connection one way or the other. You might in abstract push or pull information.

Ultimately you are trying build an oracle that predicts the future.

Re: Load Balancing

#122

Post by throwway120385 »

samwho wrote: post #115
0x457 wrote: post #97

Earlier quoted context omitted.

You still need health checks, though? Otherwise, how do you tell the difference between: "no traffic + server alive" vs "some traffic + server is dead". Yeah, you can monitor throughput on a load balancer, but if I ever again wake up from an alert about not traffic being served - I will throw hands.

Yeah, they’d have to exist but it would be for quite a different purpose. I wonder if that would mean we could implement them in different ways, e.g have a health check service that everything pings and if a ping isn’t received for N minutes, assume it’s dead and trigger some replacement routine or alert.

This begins to look a lot more like a software watchdog at that point, and you can even have each service provide a count of outstanding/processed requests per tick and if a server never gets outstanding or processed requests you could have the watchdog kill it off.

Re: Load Balancing

#123

Post by throwway120385 »

zokier wrote: post #88
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

That sort pull architecture is interesting thought. I do see it working best when a worker nodes ability to do work is very clear cut, i.e. the response time does not significantly vary with the number of concurrent requests it is handling. Here is an example where it would not work well: lets imagine workers are doing a CPU bound task and each worker has one cpu (for simplicity), but we have also identified that we…

With a pull architecture you wouldn't identify a request queue depth up front. Rather, each incoming connection gets delayed until a worker attaches to EG the TCP stream. If you're serving a webpage the client wouldn't see anything other than a little load lag, especially if your load balancer took care of the TLS connection before pausing to wait for a worker.

So if you can handle every incoming request with one worker that worker gets them all. Otherwise each worker pops off the stack as it becomes free. And if you can service more concurrent requests you just add more workers.

Re: Load Balancing

#124

Post by Abimelex »

NKosmatos wrote: post #49

Excellent post, very informative and with nice animations (check the bonus interactive animation at the end of the page), my only comment is to use different colours/colors (or dashed lines) for the graphs, it’s a bit confusing having the same color for different percentiles. Also take the time to check out other posts/pages by Sam! Well done mate ;-)

indeed, a really great post! In case you decide to change the colors or need another reason I also want to add, that in general red/green is difficult for color-blind people. About 8% of male population is affected by red/green color blindness.

Re: Load Balancing

#125

Post by c7DJTLrn »

samwho wrote: post #83
throwaway894345 wrote: post #75

I’ve always been more curious about how load balancers are supposed to be highly available? Presumably you run many instances, but then it seems like you need something to balance the load across your load balancer instances? Also, how do many load balancer instances listen on the same IP address (or do they use DNS to map a domain name to multiple IP addresses?)?

You've got it, DNS is what's often used to solve this problem. Do a few resolutions of reddit.com, you can see them returning 4 different IP addresses in a randomised order. :)

CDNs such as Cloudflare also do anycast routing which means you can hit the same IP in different places in the world and get a response from the nearest point-of-presence.

Re: Load Balancing

#126

Post by p4l4g4 »

AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

I like the idea. I guess you could do something with pubsub to simulate this model. A regular http stack (load balancer and reverse proxies in front of some http framework) would probably need major modifications to work.

It would be great if such a mechanism was part of the http standard, so you could easily connect compliant tools.

Re: Load Balancing

#127

Post by samwho »

Abimelex wrote: post #124
NKosmatos wrote: post #49

Excellent post, very informative and with nice animations (check the bonus interactive animation at the end of the page), my only comment is to use different colours/colors (or dashed lines) for the graphs, it’s a bit confusing having the same color for different percentiles. Also take the time to check out other posts/pages by Sam! Well done mate ;-)

indeed, a really great post! In case you decide to change the colors or need another reason I also want to add, that in general red/green is difficult for color-blind people. About 8% of male population is affected by red/green color blindness.

The colour palette I used on the graphs is the Wong palette I found at https://davidmathlogic.com/colorblind/.

Or are you talking about the animations? I reached out to some colourblind folks I knew before publishing and they didn’t flag those, just the graphs. Happy to change the animations as well if they’re problematic. :)

Re: Load Balancing

#128

Post by capableweb »

p4l4g4 wrote: post #126
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

I like the idea. I guess you could do something with pubsub to simulate this model. A regular http stack (load balancer and reverse proxies in front of some http framework) would probably need major modifications to work. It would be great if such a mechanism was part of the http standard, so you could easily connect compliant tools.

> It would be great if such a mechanism was part of the http standard, so you could easily connect compliant tools.

Not sure HTTP is the best approach to the suggested method. Since you control both ends, there are surely better protocols to use, like QUIC or something similar (or just straight up UDP).

Re: Load Balancing

#129

Post by zokier »

throwway120385 wrote: post #123
zokier wrote: post #88

Earlier quoted context omitted.

That sort pull architecture is interesting thought. I do see it working best when a worker nodes ability to do work is very clear cut, i.e. the response time does not significantly vary with the number of concurrent requests it is handling. Here is an example where it would not work well: lets imagine workers are doing a CPU bound task and each worker has one cpu (for simplicity), but we have also identified that we…

With a pull architecture you wouldn't identify a request queue depth up front. Rather, each incoming connection gets delayed until a worker attaches to EG the TCP stream. If you're serving a webpage the client wouldn't see anything other than a little load lag, especially if your load balancer took care of the TLS connection before pausing to wait for a worker. So if you can handle every incoming request with one wor…

> So if you can handle every incoming request with one worker that worker gets them all. Otherwise each worker pops off the stack as it becomes free

As I alluded, that works great if "can handle" and "being free" are clear-cut binary properties. But for complex applications when you are driving for high utilization while keeping latency down those questions become complicated; a worker might have some free capacity to handle requests but it doesn't mean that it would produce response as quickly as some other (more idle) worker.

In other words, the problem is not just assigning requests to workers that can handle them but assigning requests to workers that can handle them with lowest latency.

Re: Load Balancing

#130

Post by donalhunt »

Would love to see the following scenarios explored too:

- failure of a backend node

- adding additional capacity

From experience both scenarios can trigger non-obvious behaviours (especially when you have backends with different power levels).

Post reply on HN