Good post. Although, this just scratches the surface. There's also caching mechanisms involved in handling requests that might make similar requests faster when served by the same host. Now you're tempted to add sticky sessions to the mix and deal with the perks and problems that comes with that.
Load Balancing
51–60 of 243 posts
Re: Load Balancing
#52The playground simulation is pretty cool. One thing I found interesting, is it you go with PEWMA and create a scenario where the cluster is stressed, and then add 1 server, it pummels the shit out of the new server and you have a brief surge in failed requests. Not sure if that is a real world issue, or just with the simulation...
It should happen in the real world as well, at least that's what I've been told when I started my first job as a system admin. The reason people cited to me back then was that the balancer usually isn't particularly smart when balancing, so they only see a free node, thus every free request is routed to it. The errors (mostly timeout) will happen once the request start to actually get processed. Normally, the node ge…
I don't know anything about this subject, but my first thought (which may be wrong) would be to just set the weight of the new server to be the same as one of the other servers that are receiving messages (perhaps one of the lower ranks). In that way, it would not be overloaded so easily and adjust its ranking after a while
Re: Load Balancing
#53Fantastic visualizations!! This makes a complex & dry topic simple, understandable, and engaging. Very well done!
Thank you for your early reviews! Genuinely couldn't do it without the people around me.
Re: Load Balancing
#54Good post. Although, this just scratches the surface. There's also caching mechanisms involved in handling requests that might make similar requests faster when served by the same host. Now you're tempted to add sticky sessions to the mix and deal with the perks and problems that comes with that.
Absolutely! I had to pick a stopping point in order to restore balance to my personal life :D
I'd love to see people taking this as inspiration and covering the more advanced topics. I've already started on my next post and it's about a completely different area to load balancing. Very excited.
Re: Load Balancing
#55Earlier quoted context omitted.
So what do you think the disadvantage is of a pull approach? Presumably it's not just better or else tools would use it?
Not an expert in load balancing, but in similar problems (work sharing / work stealing, MPI get/put) it makes sense to pull only if you can pull fast enough to avoid incurring prohibitive latency at every request/message. Multithreading-based work stealing à la Cilk relies on extremely cheap thread mechanisms and implementation to minimize communication. In another similar situation, HPC switches are credit based so…
Having enough workers to do the work shouldn't be any more constraining than it was before.
The article does nicely mention that simple round Robin actually has lower latency, because some traffic gets lucky & goes to under-utilized machines. Unfairness helps some traffic go faster. The queue is probably going to eliminate this, but the unfairness advantage comes at the cost of a lot of other traffic getting put into long queues on workers, so it wasn't really a good thing anyways. The p90+ is usually awful.
Re: Load Balancing
#56Earlier quoted context omitted.
pretty much the same way a push load balancer does https://github.com/jackdoe/back-to-back/blob/master/broker/b... for { select { case reply := not much different than what you do with normal http timeouts, you send a request, sometimes a response comes sometimes it doesnt, up to the load balancer to decide if it wants to retry or error out also, queue does not mean async model, it means a queue, there are many queue…
And what happens if your request modifies state but the response is not received?
exactly the same as push load balancing when it doesnt receive the instance's response due to network failure, but the state is modified
Re: Load Balancing
#57I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…
As others have mentioned, this is a kind of sharding strategy. It doesn't get rid of the need for load balancing, and it's really going the opposite direction of what we know to be reliable. > Manual/statistical load balancing --- assign users to a specific server based on their login credentials. What happens when that specific server goes down? Needs an upgrade/deployment? You'll have to failover to a different ser…
What happens when that specific server goes down?
Good point.
You'll have to failover to a different server, which brings you back to an automatic load balancing strategy.
Or to a manual load balancing strategy. What I have in mind is being able to easily re-direct users from one server to another using a simple CLI utility. This won't entirely eliminate downtime issues but it will (hopefully) mitigate effects to a manageable level.
In the era of cloud computing, downtime has become less of an issue.
This is one of those things that sounds _very simple_, but in practice is incredibly complicated.
I like attempting to simplify supposedly complicated issues. What I have in mind is simply counting the requests each server handles and using this as a simple measure to compare utilization. It's true that all requests are not equal but statistically, over time, with all servers being similar, the differences will tend to balance out.
Re: Load Balancing
#58we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…
Honestly, I disagree. It's not like queue systems are rare. They're fairly common and they're often seen as an over complication! Once and only once queuing is hard. Async programming is hard.
Streaming requests or responses is out the window! Now you're storing the full request and response blobs in your infrastructure instead of chunks at a time in the network layer. Is a whole Netflix movie queued? If we're talking about queued chunks then its just UDP you're describing.
Connection oriented designs tend to more transparency end to end. The synchronous nature means a failed call can be bubbled back through the remote call chain. Failed async calls can be dropped, which leaves ambiguity. The callers need to resort to timeouts instead of closed connection signals.
Not to mention the issue that this doesn't work for client calls. The response handling from the client is more complex. The async callback would need to be demuxed such that a response can be associated with a call. There's no open connection so you'd need to punch the firewall somehow...honestly its very messy.
Re: Load Balancing
#59Earlier quoted context omitted.
pretty much the same way a push load balancer does https://github.com/jackdoe/back-to-back/blob/master/broker/b... for { select { case reply := not much different than what you do with normal http timeouts, you send a request, sometimes a response comes sometimes it doesnt, up to the load balancer to decide if it wants to retry or error out also, queue does not mean async model, it means a queue, there are many queue…
> queue does not mean async model “Message queues implement an asynchronous communication pattern between two or more processes/threads whereby the sending and receiving party do not need to interact with the message queue at the same time.” https://en.m.wikipedia.org/wiki/Message_queue
if you read what i wrote, you will see that it is not asynchronous communication.
and it is not a message queue in the sense you mean, it is a request response queue, it just happens to be using messages
when people think of message queue they are thinking of async event driven communication, and this is not it
Re: Load Balancing
#60Earlier quoted context omitted.
Definitely doable with HAProxy + lua. I've used it extensively for load balancing stateful apps.
I'll definitely take a look at this. What I have in mind isn't really a "proxy" but more of a login/redirection server. A "proxy" is middleware which directs all communication through a single server which adds to latency. What I have in mind will run logins through a single server. But once the login is complete, any further communication is redirected to the proper work server to continue without any proxy middlewa…
HAProxy can also do redirects