Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

111–120 of 243 posts

Re: Load Balancing

#111

Post by samwho »

chasd00 wrote: post #110

iirc there was a famous case where an outage occurred at a company using some least connection or least resource algorithm in the load balancers. The load balancer queue'd up requests and when a new server was brought online it immediately dos'd it out of existence. Then another server was brought online and the same thing occurred and another and another. They eventually switched to round-robin to give the new serve…

Yes! This is called a “thundering herd” problem: https://en.m.wikipedia.org/wiki/Thundering_herd_problem.

The topic is definitely much more complex than what I was able to cover in a single post :)

Re: Load Balancing

#112

Post by samwho »

Nijikokun wrote: post #109

Interactive diagrams and visuals are worth so much with these topics. I just created documentation at ngrok (full disclosure, I work there!) for how to do load balancing with their edges product for this the other day! Now I have something to refer people to when describing these. https://ngrok.com/docs/guides/how-to-round-robin-load-balanc...

Big ngrok fan, keep up the awesome work!

Re: Load Balancing

#113

Post by chasd00 »

bbatchelder wrote: post #15
jqpabc123 wrote: post #10

I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…

Did this 20 years ago by having the name of the server as part of the user's profile. User's 1 through 50 (light users) log in and their profile says they go to app-1.myapp.com. User's 51 through 60 (heavy users) log in and their profile says they go to app-2.myapp.com. A specific user may pay extra to have a non-shared environment, and this supports that as well.

i did something similar way back when only the server identifier was part of the session cookie iirc. Whichever server behind the LB started the session got all the requests for that session. It was more like a user load balancer vs a request load balancer.

Re: Load Balancing

#114

Post by throwawayc3ff »

411111111111111 wrote: post #64
throwawayc3ff wrote: post #52

Earlier quoted context omitted.

I don't know anything about this subject, but my first thought (which may be wrong) would be to just set the weight of the new server to be the same as one of the other servers that are receiving messages (perhaps one of the lower ranks). In that way, it would not be overloaded so easily and adjust its ranking after a while

I guess my explanation was lacking then, as that wouldn't help. reducing the weight below the old nodes might work, but it would also extend the duration you're overloaded, which would also cause requests to fail.

That makes sense. I guess there's no simple fix for it.

Re: Load Balancing

#115

Post by samwho »

0x457 wrote: post #97
samwho wrote: post #40

Earlier quoted context omitted.

I honestly had never thought hard about reversing the relationship and having workers pull. This point about no longer requiring health checks is a real "woah" moment. Thanks for expanding my mind!

You still need health checks, though? Otherwise, how do you tell the difference between: "no traffic + server alive" vs "some traffic + server is dead". Yeah, you can monitor throughput on a load balancer, but if I ever again wake up from an alert about not traffic being served - I will throw hands.

Yeah, they’d have to exist but it would be for quite a different purpose. I wonder if that would mean we could implement them in different ways, e.g have a health check service that everything pings and if a ping isn’t received for N minutes, assume it’s dead and trigger some replacement routine or alert.

Re: Load Balancing

#116

Post by eximius »

AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

This is a fascinating blind spot I didn't know I had!

Difficult to do much with as so many request/response load balancing software is inherently push based, but it's a great eye opener.

Re: Load Balancing

#117

Post by blipvert »

wmf wrote: post #98
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

For most protocols like HTTP there's no software to do this and it's not worth developing something nonstandard.

Should be something that could quite easily be added to a uWSGI subscription router - subscribing node periodically says “I have N slots available” and the router keeps track of how many requests are in flight.

Re: Load Balancing

#118

Post by vlunkr »

jqpabc123 wrote: post #10

I have a simple alternative method in mind for SAAS (software as a service) apps. Manual/statistical load balancing --- assign users to a specific server based on their login credentials. A statistical model of server utilization can be maintained and users assigned or re-assigned as needed. Latency can be reduced to zero by simply forwarding the connection to the proper server once the login is complete. The obvious…

There are lots of good replies already, but I'm curious why you would even want to do this. Web applications have been moving away from stateful servers for ages. Ideally your server instances are completely disposable. If you need to persist and share state between them, there are great mechanisms: the database, a queue, cookies, etc.

The 12-factor app describes this well: https://12factor.net/processes

Re: Load Balancing

#119

Post by kriz9 »

It is nice to see someone writing that a single server is enough for a lot of websites/apps - if not most. Computers are insanely fast nowadays.

Re: Load Balancing

#120

Post by paulddraper »

0x457 wrote: post #97
samwho wrote: post #40

Earlier quoted context omitted.

I honestly had never thought hard about reversing the relationship and having workers pull. This point about no longer requiring health checks is a real "woah" moment. Thanks for expanding my mind!

You still need health checks, though? Otherwise, how do you tell the difference between: "no traffic + server alive" vs "some traffic + server is dead". Yeah, you can monitor throughput on a load balancer, but if I ever again wake up from an alert about not traffic being served - I will throw hands.

Your server scaling system will need healthchecks. But:

1. It can be more tolerant of failures, since an unresponsive server doesn't mean failed requests.

2. Hooking up your server scaling system to HAProxy, etc. isn't the easiest thing in the world anyway.

Post reply on HN