Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

191–200 of 243 posts

Re: Load Balancing

#191

Post by AdamN »

dazbradbury wrote: post #188

AWS Application Load Balancers seem to default to round-robin, would be interested to hear how many people change this to their "least connections" equivalent called "LOR" [1]? And why they don't support any other options? I'd guess AWS default to round robin because it's the least complex, and doesn't have issues like if a worker responds extremely quickly due to an error (eg. returns a 500 immediately) then using L…

You point out a good gotcha that most people don't notice - many host-specific errors are often faster than the standard response and more traffic will route to those hosts with a performance-based heuristic. A well configured LB would have hosts with responses that take them out of rotation for known fault conditions - but at scale that's hard for companies to validate.

Re: Load Balancing

#192

Post by adql »

interroboink wrote: post #131

A small note on "least connections" load balancing. The article says: Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has. and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the loa…

> But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the load balancer. So, the load balancer doesn't know how busy the server is, unless that information is communicated separately, through some other channel.

Loadbalancer can still see it's an incoming connection as long as user is sending something (even if returning traffic goes directly) so at the very least it can keep number of connections same.

Of course one might stream 240p while other user streams 4k but for example in HAProxy there is an option to use external agent for health checks, that also have option to modify traffic weights and cut traffic to ones that are being swamped

Re: Load Balancing

#193

Post by adql »

paulddraper wrote: post #23
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

One of the nicest things about pull load balancing is that it completely replaces the need for a separate registration + health check system. Once you consider that, IDK if someone can really claim "more complicated."

"Wait, why this is dead for 4 weeks and not processing anything

> One of the nicest things about pull load balancing is that it completely replaces the need for a separate registration + health check system.

"Jeff, stop taking random advice off HN as gospel, that shit is stuck in GC for last week ,we told you we still need monitoring"

Re: Load Balancing

#194

Post by adql »

throwaway894345 wrote: post #78
samwho wrote: post #40

Earlier quoted context omitted.

I honestly had never thought hard about reversing the relationship and having workers pull. This point about no longer requiring health checks is a real "woah" moment. Thanks for expanding my mind!

Agreed on the “whoa moment”. Not needing health checks seems pretty compelling. I’m curious if there are any off-the-shelf pull-based load balancer that don’t require the full HTTP request copied onto the queue before handing off to a worker?

You need health checks. If not for LB then for your own monitoring.

Re: Load Balancing

#195

Post by zamnos »

AdamN wrote: post #191
dazbradbury wrote: post #188

AWS Application Load Balancers seem to default to round-robin, would be interested to hear how many people change this to their "least connections" equivalent called "LOR" [1]? And why they don't support any other options? I'd guess AWS default to round robin because it's the least complex, and doesn't have issues like if a worker responds extremely quickly due to an error (eg. returns a 500 immediately) then using L…

You point out a good gotcha that most people don't notice - many host-specific errors are often faster than the standard response and more traffic will route to those hosts with a performance-based heuristic. A well configured LB would have hosts with responses that take them out of rotation for known fault conditions - but at scale that's hard for companies to validate.

At scale, with also having scaled systems engineers, it's not, like, impossibly hard. A sidecar like Envoy can be configured to emit health stats which can then be read by the load balancer to consider a given server unhealthy. Again, at scale, but each team is already responsible for a dashboard with health metrics for their service, so the load balancer team doesn't have to try and determine everybodies health metrics, only their own.

Re: Load Balancing

#196

Post by adql »

throwaway71271 wrote: post #21
gbuk2013 wrote: post #19

Earlier quoted context omitted.

Moving to async model adds new set of operational challenges as well as some interesting failure scenarios. (Edit) Also, in practice you would need at least one more system to enqueue request into the broker, as the latter would typically not be exposed to the outside world. Request/response on the other hand is much simpler to configure and operate.

this is not async, its sync queue the lb puts a request where it has some reply_to (ip:port) where it waits (blockingly) for response from whoever picked up the request, it just does now know who that is until a reply comes

That's push model when per-server maxconn is full tho.

The biggest benefit from pull model is not having to update backend server list every time you add/remove one but outside of that it isn't really all that beneficial.

You also get added latency, unless each backend server is actively listening and connected but if it is, you're just wasting extra RTT to say "hey, there is a request in queue, do you want it?"

Re: Load Balancing

#197

Post by adql »

jayd16 wrote: post #76
throwaway71271 wrote: post #21

Earlier quoted context omitted.

this is not async, its sync queue the lb puts a request where it has some reply_to (ip:port) where it waits (blockingly) for response from whoever picked up the request, it just does now know who that is until a reply comes

If we're not talking about an async model then the suggestion is much less drastic than it sounded at first. In that case the crux of your desire is simply allowing the hosts to signal readiness more directly. You would almost never actually wait for host machines to dial in. You would have a list of hosts that are ready or not ready as they would almost always be ready for more. You want to assume readiness (as this…

...that's just push model. "Signalling" via "well, the loadbalancer have 10 sessions max per server" is enough.

Pull model just adds unnecesary RTT.

> Off the top of my head I'm trying to think how a host could actively signal to an LB that's its ready for more requests... I suppose its trivial and common to use a health check. Is it even a change to say that these need to be updated to achieve your goal of host to LB pulling?

Like this.

There is rarely a case where you decide to not serve the next request after serving previous one so push is most optimal for short ones. And if it doesn't want to it can just signal that via healthcheck.

Pull makes more sense for latency-insensitive jobs like "take a task from queue, do it, and put the results back", as if you say make video encoding service that dynamically scales itself in the background and "just do one encode and exit" is commonplace.

Re: Load Balancing

#198

Post by adql »

dano wrote: post #90
toast0 wrote: post #70

Earlier quoted context omitted.

A pull approach seems difficult to manage when you have many layers in your load balancing. In small setups, you may just have one layer with a single load balancer (well, hopefully at least a hot-warm pair), but larger setups often have multiple levels. There may be a network level traffic split to multiple frontend load balancers via something like ECMP; those frontends may connect directly to the origin hosts, or…

Nice to see someone mention direct server return or as BigIP called it nPath routing. This was an effective scaling method for handling small request that returned large payloads (audio and video files). I don't know how well known this configure is or whether it is still viable in an all TLS world.

It kinda fell out of favour when even midrange server can do tens of gigabits of TLS traffic. So you need to be very big traffic wise to make it worth.

Re: Load Balancing

#199

Post by mst »

hinkley wrote: post #145
AYBABTME wrote: post #138

Earlier quoted context omitted.

Yes but only if connections aren't reused.

I keep having to turn off keepalive between the service and the reverse proxies and I bet this is part of why. I think Node 12 introduced LIFO queuing for connection pools because they found the cost of silent disconnects from the server to be too high. They got much better histograms by using the most recently freed connection instead of round robin of available sockets.

I don't remember the URL (sorry) but I saw something recently talking about Facebook's queueing approach for requests being FIFO under normal load and LIFO when overloaded, so that under an overload state you serve the number of requests you -can- serve reasonably quickly rather than -everything- timing out.

For node's situation that sounds probably workable but I think a lot of the time I'd prefer to send periodic keepalives to the idle pool connections. However, you can't exactly bake -that- into the runtime so I can see their argument and perhaps the best thing would be "both, plus periodically do a clean shutdown on excess pool members until you need them again."

Re: Load Balancing

#200

Post by adql »

jqpabc123 wrote: post #46
cett wrote: post #42

Earlier quoted context omitted.

Definitely doable with HAProxy + lua. I've used it extensively for load balancing stateful apps.

I'll definitely take a look at this. What I have in mind isn't really a "proxy" but more of a login/redirection server. A "proxy" is middleware which directs all communication through a single server which adds to latency. What I have in mind will run logins through a single server. But once the login is complete, any further communication is redirected to the proper work server to continue without any proxy middlewa…

Start here:

https://github.com/haproxy/spoa-example

https://www.haproxy.com/blog/extending-haproxy-with-the-stre...

We used that to make an SSO login site that works independently on what is on the backend. Logic was basically:

* if there is no/invalid SSO cookie, SPOA set a flag which made haproxy redirect to the SSO app * if there is valid cookie, decode it and send the data (usually just logged user name) to the app in header

Once cookie is correct it doesn't need SSO server so it is pretty fast for users that already logged in.

It can be also used for blocking requests based on external engine, it's pretty flexible overall

https://docs.fastly.com/signalsciences/install-guides/other-...

Post reply on HN