Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

201–210 of 243 posts

Re: Load Balancing

#201

Post by base »

dazbradbury wrote: post #188

AWS Application Load Balancers seem to default to round-robin, would be interested to hear how many people change this to their "least connections" equivalent called "LOR" [1]? And why they don't support any other options? I'd guess AWS default to round robin because it's the least complex, and doesn't have issues like if a worker responds extremely quickly due to an error (eg. returns a 500 immediately) then using L…

Depending on the app might not have better performance. The reason is that a host with more connections might not be using more resources than one with less connections. The busy connections might be waiting for a reply from an external service.

Re: Load Balancing

#202

Post by CrimsonRain »

adql wrote: post #189
eximius wrote: post #116

Earlier quoted context omitted.

This is a fascinating blind spot I didn't know I had! Difficult to do much with as so many request/response load balancing software is inherently push based, but it's a great eye opener.

It's not difficult. loadbalancer knows how many requests are in-flight to server . The OP is plainly wrong. That's why leastconn works in the first place. And leastconn is almost always one you want. It's almost magical. GC stall on one server ? That means it isn't processing, which means every new request will go to other servers. One server processing 2x as fast as the other ? Well, it keeps its connection count lo…

What if a server has most of its request processing io blocked? It can process more requests because it has empty CPU RAM but load balancer will think the server is fully booked.

Re: Load Balancing

#203

Post by mst »

bityard wrote: post #140
AYBABTME wrote: post #100

Earlier quoted context omitted.

Very similar to background queues but the load balancer holds the request and dispatches it to a waiting worker, or waits until someone comes to serve it. While the worker handles the request, the load balancer proxies. The request serving is the same, the way the work is dispatched is just inverted.

Seems like this wouldn't work as well in situations where you care a lot about latency or resource utilization. If the workers are polling the LB every n ms, then that's an average of n / 2 ms added to _every_ request. Plus additional CPU cycles and network traffic on both ends due to the polling mechanism.

High volume low latency message queueing is already a problem people have had and implemented solutions for that (for their use case) cost little enough in terms of latency that the advantages were well worth it.

Also there's no reason to poll the LB/queue/etc., you just tell it "I'm ready" and it sends you something to handle when it's got something.

Pull/push here is about who decides when a server is ready to receive another request.

So ... I'm not an expert on actually implementing it, but I've seen systems in practice that -were- in such situations and it worked out extremely well.

Re: Load Balancing

#204

Post by adql »

hamilyon2 wrote: post #156

I always thought that for hundreds of backends and above 10K RPS those algorithms scale poorly. First of all, scaling beyond single load balancer becomes a problem. The cost of updating counters becomes significant. If responses are quick, the cost additional bookkeeping adds doesn't pay off. Of you use weighted random all these problems go away. Is my knowledge outdated?

Well, in short, yes.

> I always thought that for hundreds of backends and above 10K RPS those algorithms scale poorly

https://www.haproxy.com/blog/haproxy-forwards-over-2-million...

> First of all, scaling beyond single load balancer becomes a problem.

If you are in DC or have L3 access to the underlying network ECMP is VERY easy way to scale to 4-16 loadbalancers. Over that just... use multiple IPs. We use ECMPed setup of 4 nodes since forever and it works very well.

There are theoretical improvements that could be made by doing L4 loadabalancing into L7 loadbalancing, but you'd need a hell lot of traffic to ever need that

> The cost of updating counters becomes significant. If responses are quick, the cost additional bookkeeping adds doesn't pay off.

Essentially, yes but if you have some architecture that talks very little (say just few packets per connection, let's assume some IoT garbage), you probably just want different architecture altogether; like "first request gets you assigned server, then you talk to it directly without LB". L4 balancing is also an option.

In essence, if somehow performance of leastconn algorithm is your bottleneck, you're both big enough and specialized enough to go with another approach than "just dump all traffic at few loadbalancers"

Re: Load Balancing

#205

Post by mst »

samwho wrote: post #80
zokier wrote: post #71

Least connections is intuitively very sensible algorithm, but it has one pitfall: if backend server starts returning errors for whatever reason (e.g. load shedding) quicker than typical responses, it can lead to situation where disproportionate number of requests are directed to that one server. Also typically you'd have more than one load balancer instance, and at least naive least connections requires shared state…

Least connections doesn't need shared state between the instances, but it would need shared state between the load balancers if you had more than one for sure. You're totally right, I glossed right over errors. With PEWMA and weighted round robin you can make instances incur a "penalty" for serving errors, which can help you isolate bad servers. It would have been fun to visualise this, definitely.

It also strikes me that it might be interesting to model least connections with two or three LBs with and without such shared state.

My instinct is that you'd still get most of the advantages even with each LB keeping its own independent connection counts, but given the number of times I've fired up a profiler and gone "wait, seriously, -that-s the slow part?!" I'd suggest nobody believes me and measures it instead.

Re: Load Balancing

#206

Post by adql »

ahmadnoid wrote: post #101

What I don't understand is since it's the same service can't we just run a performance test, determine the ballpark of the needed resources and just have them capped like in k8s? why would some servers be more powerful than others in the first place?

One perfectly homogenous server gets 20MB POST with image that then needs to be reencoded/cropped

Other gets tiny GET for CSS.

Boom, now your equal servers are inequally loaded. The beauty of leastconn is that if 1st server is loaded it will just naturally get less connections than the lucky ones that only got easy jobs

Re: Load Balancing

#207

Post by zamnos »

dazbradbury wrote: post #188

AWS Application Load Balancers seem to default to round-robin, would be interested to hear how many people change this to their "least connections" equivalent called "LOR" [1]? And why they don't support any other options? I'd guess AWS default to round robin because it's the least complex, and doesn't have issues like if a worker responds extremely quickly due to an error (eg. returns a 500 immediately) then using L…

Conspiratorially speaking, the way to handle AWS' load balancer algorithm shortcomings is to run more EC2 instances, which, naturally AWS profits from.

Re: Load Balancing

#208

Post by philsnow »

adql wrote: post #186
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

> All push load balancing algorithms try to somehow predict how busy downstreams are. They KNOW how busy they are. They are the ones tracking and forwarding connections to them. That's why leastconn works in the first palce But, for example HAProxy have option to directly back-feed weights via healthchecks from app, so there is an option for app to signal back-pressure in RR balancing

> They KNOW how busy they are. They are the ones tracking and forwarding connections to them.

Well, they do when they're the only ones sending work to the workers.

The article uses a literal black box for the load balancer, but there are workloads that are too heavy for a single machine, so in those cases (and others, like HA) you have to have a pool of load balancers. You can try to make those load balancers know everything about what's happening in the whole system but it can be hard and expensive.

Or, you can have them operate on less-than-perfect knowledge. This is what the round-robin strategy does, and just like round-robin, has its tradeoffs (much simpler, worse 95%ile latency).

All of this is assuming the load balancers and workers servicing connections are the only things running on those machines. In real world usage there can often be other loads on the same hardware, belonging to tenants your team doesn't even have a relationship with, which can complicate things quite a bit.

Re: Load Balancing

#209

Post by SteveNuts »

adql wrote: post #186
AYBABTME wrote: post #95

I feel like something missing from people's perspective when thinking of load balancing is to consider pull models. All push load balancing algorithms try to somehow predict how busy downstreams are. Some even go as far as having downstreams send back some utilization metrics. But if you use a pull-based approach, this is all sort of moot. Downstreams will pull work when they're ready for it.

> All push load balancing algorithms try to somehow predict how busy downstreams are. They KNOW how busy they are. They are the ones tracking and forwarding connections to them. That's why leastconn works in the first palce But, for example HAProxy have option to directly back-feed weights via healthchecks from app, so there is an option for app to signal back-pressure in RR balancing

That’s interesting I haven’t heard of this in HAproxy. I’ve seen that it has a static weight for how many requests to send to a backend in the config file, what’s this functionality called for dynamic weights? And how do you expose that?

Re: Load Balancing

#210

Post by philsnow »

zokier wrote: post #150
throwaway894345 wrote: post #75

I’ve always been more curious about how load balancers are supposed to be highly available? Presumably you run many instances, but then it seems like you need something to balance the load across your load balancer instances? Also, how do many load balancer instances listen on the same IP address (or do they use DNS to map a domain name to multiple IP addresses?)?

Floating/Virtual IPs are one networking solution for HA. With Corosync/Pacemaker a cluster of hosts can decide how the ips are then assigned to physical hosts. CARP maybe/probably can do the same. I just googled and apparently there is also some IPVS/LVS project for Linux for HA loadbalancing?

This brought to mind the OpenBSD 3.5 release song https://www.openbsd.org/lyrics.html#35

  VRRP, philosophically,
  must ipso facto standard be
  But standard it
  needs to be free
  vis-à-vis
  the IETF
  you see?
  
  But can VRRP
  be said to be
  or not to be
  a standard, see,
  when VRRP can not be free,
  due to some Cisco patentry..
Post reply on HN