181–190 of 243 posts
If the requests are similar (i.e. don't have unique user data in them), it's best to put a high performance reverse proxy in front that has some caching. This way you don't have to execute lots of code for every request.
Least connections is intuitively very sensible algorithm, but it has one pitfall: if backend server starts returning errors for whatever reason (e.g. load shedding) quicker than typical responses, it can lead to situation where disproportionate number of requests are directed to that one server. Also typically you'd have more than one load balancer instance, and at least naive least connections requires shared state…
Why is there not a queue system where a server can pull requests when it's ready to handle one? Why must the work to determine where to send a request go to the load balancer, when you could just pull from a queue and save yourself the trouble of finding the exact right load balancer strategy for your application?
The short answer is that it's more complicated to engineer and has its own set of trade-offs. With the push-based models shown in the article, everything works regular HTTP(S), no fancy routing is needed, and the LB needs to keep very little state to do its job.
seems like the queue could accept the request, keep the https connection open, wait for a response from whatever server accepts the request, and pass the response to the client upon receipt just fine.
I dunno. web stuff is poorly done imo. poorly designed.
Also the "overrun" of dropped connections shouldn't really happen if
* incoming connections are So that part of the description is subtly wrong; you WANT to queue on loadbalancer, not on app servers.
The app server should have queue just long enough to feed all the threads but not enough to start dropping anything
I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.
They KNOW how busy they are. They are the ones tracking and forwarding connections to them. That's why leastconn works in the first palce
But, for example HAProxy have option to directly back-feed weights via healthchecks from app, so there is an option for app to signal back-pressure in RR balancing
Well written and animated!! Anyone knows what technology was used for the animation?
I used https://pixijs.com/ . This was my first project with it and it was really nice to use. I fell in to a few traps here and there, but generally it wasn't too bad to get the results I wanted with it. :)
I'd guess AWS default to round robin because it's the least complex, and doesn't have issues like if a worker responds extremely quickly due to an error (eg. returns a 500 immediately) then using LOR it seems it would consume all the requests until it's taken out of service by any health checks. But maybe there are other potential downsides?
[1] - https://aws.amazon.com/about-aws/whats-new/2019/11/applicati...
I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.
This is a fascinating blind spot I didn't know I had! Difficult to do much with as so many request/response load balancing software is inherently push based, but it's a great eye opener.
That's why leastconn works in the first place. And leastconn is almost always one you want. It's almost magical. GC stall on one server ? That means it isn't processing, which means every new request will go to other servers.
One server processing 2x as fast as the other ? Well, it keeps its connection count low, so it gets more of them
I think it might be worth adding to this some discussion of how the load balancer itself handles the load, and the related fact that _within_ some backends, the responsibility for load-balancing falls on rich clients that maintain knowledge of the server sets to which they can route requests.
Then in case of leastconn there will be no dropped connections as long as total server capacity < incoming traffic.