Load Balancing

This is so Correct!

Load Balancing is essential for large websites with huge user base.

Really excellent simulation and explanation. Thank you!

I came here to say the same thing! This is also a great tool for explaining it to non-technical customers.

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized

you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties

but if you just let instances pull work, things work out for themselves

sadly, the whole industry is stuck on http push

i was playing with it few years ago making a simple queue and wrote my ideas here: https://punkjazz.org/jack/we-got-it-all-wrong.txt and some basic benchmarks https://punkjazz.org/jack/we-got-it-all-wrong-2.txt

in the same time, there is so much tooling for http, and its so natural to use it, that it is actually hard to switch to another transport layer, so now the best we have is some naive bayesian classifiers in the LB and some exponential backoff

here is the difference i had in timings using synchronous io queue vs http for 2 endpoints, one fast and one slow (endpoints had the same code, just transport was different)

    2019/02/19 22:26:27 synchronous QUEUE ... 20000 messages, took: 29.95s, speed: 667.74 per second
    -   0+10=9711  48.55% ******************************
    -  10+ 5= 276  49.94% 
    -  15+ 8=  20  50.03% 
    -  23+12=  16  50.11% 
    -  35+18=3398  67.11% **********
    -  53+27=6549  99.85% ********************
    -  80+40=  30 100.00% 
    - 120+60=   0 100.00%
    - 180+90=   0 100.00%
    - 270+ 0=   0 100.00%


    2019/02/19 22:26:56 http ... 20000 messages, took: 29.55s, speed: 676.91 per second
    -   0+10=3274  16.37% ********************
    -  10+ 5=1523  23.98% *********
    -  15+ 8=4439  46.18% ****************************
    -  23+12=4756  69.96% ******************************
    -  35+18=3318  86.55% ********************
    -  53+27=2037  96.73% ************
    -  80+40= 581  99.64% ***
    - 120+60=  70  99.99% 
    - 180+90=   2 100.00% 
    - 270+ 0=   0 100.00%

you can see how the fast endpoint is always fast with the queue transport, but with classic push load balancing it spills latency a lot, because the instance is sometime busy servicing the slow request

PS: this post is absolutely amazing! and the animations are brilliant! thanks a lot for making it

I have a simple alternative method in mind for SAAS (software as a service) apps.

Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete.

The obvious downside is a custom load balancer implementation is required.

Does anyone have any experience using NodeJS as a load balancer for something like this?