Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

41–50 of 243 posts

Re: Load Balancing

#41

Post by gbuk2013 »

throwaway71271 wrote: post #35
gbuk2013 wrote: post #32

Earlier quoted context omitted.

If you have a queue then you have async model. As an example of a failure scenario, how does your system distinguish between a request timeout, a response that didn’t get sent back because of network failure and the consumer crashing and losing the message?

pretty much the same way a push load balancer does https://github.com/jackdoe/back-to-back/blob/master/broker/b... for { select { case reply := not much different than what you do with normal http timeouts, you send a request, sometimes a response comes sometimes it doesnt, up to the load balancer to decide if it wants to retry or error out also, queue does not mean async model, it means a queue, there are many queue…

> queue does not mean async model

“Message queues implement an asynchronous communication pattern between two or more processes/threads whereby the sending and receiving party do not need to interact with the message queue at the same time.”

https://en.m.wikipedia.org/wiki/Message_queue

Re: Load Balancing

#42

Post by cett »

jqpabc123 wrote: post #27
mkl95 wrote: post #13

Earlier quoted context omitted.

Sounds like sticky sessions

"Sticky" applies after a session has been assigned to a server. I'm interested in control over the assignment algorithm.

Definitely doable with HAProxy + lua. I've used it extensively for load balancing stateful apps.

Re: Load Balancing

#43

Post by samwho »

throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

Thank _you_ for striking up a spirited discussion! :)

Re: Load Balancing

#44

Post by alex_sf »

jqpabc123 wrote: post #10

I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…

As others have mentioned, this is a kind of sharding strategy. It doesn't get rid of the need for load balancing, and it's really going the opposite direction of what we know to be reliable.

> Manual/statistical load balancing --- assign users to a specific server based on their login credentials.

What happens when that specific server goes down? Needs an upgrade/deployment? You'll have to failover to a different server, which brings you back to an automatic load balancing strategy.

> Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete.

Modern load balancers add a meaningless amount of latency per request. If you're truly forwarding it in the networking sense, then a load balancer/reverse proxy is still involved.

If you mean something like redirecting them to an endpoint that points directly at an individual server, you get back to the first problem. What happens when that server goes down?

> A statistical model of server utilization can be maintained and users assigned or re-assigned as needed.

This is one of those things that sounds _very simple_, but in practice is incredibly complicated.

Re: Load Balancing

#45

Post by bbss »

Great post! I would have loved to see P2C (Power of 2 Choices) in there as well, which is typically a better alternative to Round Robin and Least Connections.

Re: Load Balancing

#46

Post by jqpabc123 »

cett wrote: post #42
jqpabc123 wrote: post #27

Earlier quoted context omitted.

"Sticky" applies after a session has been assigned to a server. I'm interested in control over the assignment algorithm.

Definitely doable with HAProxy + lua. I've used it extensively for load balancing stateful apps.

I'll definitely take a look at this.

What I have in mind isn't really a "proxy" but more of a login/redirection server.

A "proxy" is middleware which directs all communication through a single server which adds to latency.

What I have in mind will run logins through a single server. But once the login is complete, any further communication is redirected to the proper work server to continue without any proxy middleware involved.

This won't entirely eliminate downtime issues but it does limit the effects to a reasonable level while offering increased efficiency and decreased latency.

Re: Load Balancing

#47

Post by samwho »

bbss wrote: post #45

Great post! I would have loved to see P2C (Power of 2 Choices) in there as well, which is typically a better alternative to Round Robin and Least Connections.

Thank you! P2C is really cool, but it would have meant having to talk about load balancers with incomplete information. This felt like slightly too much to add to an already-quite-long post. It also would have added an extra layer of complexity to my already-quite-complex simulation code :sweat_smile:

Re: Load Balancing

#48

Post by artworx »

throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

The AWS Lambda Runtime API is pull based:

- GET events from ${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next

- process it (run your handler)

- POST the response to /runtime/invocation/AwsRequestId/response

https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.ht...

Re: Load Balancing

#49

Post by NKosmatos »

Excellent post, very informative and with nice animations (check the bonus interactive animation at the end of the page), my only comment is to use different colours/colors (or dashed lines) for the graphs, it’s a bit confusing having the same color for different percentiles.

Also take the time to check out other posts/pages by Sam! Well done mate ;-)

Re: Load Balancing

#50

Post by rektide »

Mavvie wrote: post #17
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

So what do you think the disadvantage is of a pull approach? Presumably it's not just better or else tools would use it?

Making a high reliability queue that can also soak load spikes is non-trivial.

Now that we have hundreds of Gbps ethernet and TB of memory the idea has more merit, can scale pretty absurdly high with mundane systems. Or maybe you have sharding, which means now you have a load balancing problem again, of picking which work queue to take work from.

The HA bit is still hard. You have to to figure out if there's a netsplit (some folks can't connect to one server) or if one server really is gone. Probably just multicast to each queue all the incoming work & all the incoming pulls. Ideally each queue could also hear all the outgoing traffic. If ethernet capacity were unidirectional this would be great, box #2 could autonomously detect faults & take over. But ethernet is bidirectional, and now it needs all box #1's incoming traffic and it's outgoing traffic too. So instead maybe have the clients fail over. We can iterate on resign but HA is non-trivial.

Post reply on HN