Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

21–30 of 243 posts

Re: Load Balancing

#21

Post by throwaway71271 »

gbuk2013 wrote: post #19
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

Moving to async model adds new set of operational challenges as well as some interesting failure scenarios. (Edit) Also, in practice you would need at least one more system to enqueue request into the broker, as the latter would typically not be exposed to the outside world. Request/response on the other hand is much simpler to configure and operate.

this is not async, its sync queue

the lb puts a request where it has some reply_to (ip:port) where it waits (blockingly) for response from whoever picked up the request, it just does now know who that is until a reply comes

Re: Load Balancing

#22

Post by jqpabc123 »

yabones wrote: post #14
jqpabc123 wrote: post #10

I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…

That's very common with stateful applications. Lots of people use HAProxy or other application-aware LB's to keep the sessions "sticky" to a single app server. https://www.haproxy.com/blog/enable-sticky-sessions-in-hapro... https://www.haproxy.com/blog/load-balancing-affinity-persist...

Keeping a session "sticky" once it is assigned to a server is a somewhat different issue than controlling the assignment.

"Persistence" is more what I want but digging this out of HAProxy configuration and making changes on the fly looks troublesome.

Re: Load Balancing

#23

Post by paulddraper »

throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

One of the nicest things about pull load balancing is that it completely replaces the need for a separate registration + health check system.

Once you consider that, IDK if someone can really claim "more complicated."

Re: Load Balancing

#24

Post by paulddraper »

deepfriedrice wrote: post #20
mnutt wrote: post #7

The simulations are really cool. I'm trying to think of a real-world situation where you'd have differently-sized backends though. I guess it might help if your server runs hourly cron jobs that take away system resources from requests.

Had the same thought. With AWS et al. it's just too easy to use the same instances and avoid that variable.

On the contrary, with AWS et al, it's just too easy to have noisy neighbors :)

Re: Load Balancing

#25

Post by throwaway71271 »

Mavvie wrote: post #17
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

So what do you think the disadvantage is of a pull approach? Presumably it's not just better or else tools would use it?

i never ran this in prod, so not sure

i think the retries are a bit strange, and debugging was weirder as well, but its probably just me not used to it

even though i had multiple chances to use it in prod, i always go for http somehow, it just feels so familiar

also because of the way the REPLY_TO address:port worked in my experiments, sometimes having half open tcp connections really messes up things

Re: Load Balancing

#26

Post by jgtrosh »

Mavvie wrote: post #17
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

So what do you think the disadvantage is of a pull approach? Presumably it's not just better or else tools would use it?

Not an expert in load balancing, but in similar problems (work sharing / work stealing, MPI get/put) it makes sense to pull only if you can pull fast enough to avoid incurring prohibitive latency at every request/message.

Multithreading-based work stealing à la Cilk relies on extremely cheap thread mechanisms and implementation to minimize communication.

In another similar situation, HPC switches are credit based so that until you hit congestion, you can “instantaneously” know if a remote is ready to receive.

This isn't a formal explanation, of course.

Edit: after some thought, that's not really the distinction that is made for load balancing. There's already knowledge of the remote state required for pushing to the last loaded queue. So the difference between pull and push is about having one queue vs several. In that sense it is like supermarkets that implement the more efficient one queue to every cashier Vs the more traditional one queue per cashier. In supermarkets there is a choice to make because there's other constraints, but just optimising for load balancing it's strictly better to have a single queue, if you only have one input.

Re: Load Balancing

#27

Post by jqpabc123 »

mkl95 wrote: post #13
jqpabc123 wrote: post #10

I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…

Sounds like sticky sessions

"Sticky" applies after a session has been assigned to a server. I'm interested in control over the assignment algorithm.

Re: Load Balancing

#29

Post by photochemsyn »

I'm going to guess the style of this excellent effort was inspired in part by this popular HN poster? Creating a post in this manner with animations does take a lot of work but it's very effective and instructional:

https://ciechanow.ski/gps/

Incidentally load balancing and dynamic routing in satellite networks looks like a complicated problem and is a subject of active research:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9231381/

Re: Load Balancing

#30

Post by samwho »

bbatchelder wrote: post #16

The playground simulation is pretty cool. One thing I found interesting, is it you go with PEWMA and create a scenario where the cluster is stressed, and then add 1 server, it pummels the shit out of the new server and you have a brief surge in failed requests. Not sure if that is a real world issue, or just with the simulation...

This is very likely a bug in the simulation. My simplified implementation of PEWMA prioritises servers that have had no traffic, in order to send at least 1 request to all servers. There will be a window, until this new server serves its first request, where it is considered the highest priority server.

I doubt very much that this would be part of any real world implementation

Post reply on HN