Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Load Balancing

samwho.dev

Post reply on HN
Print view View original

31–40 of 243 posts

Re: Load Balancing

#31

Post by samwho »

photochemsyn wrote: post #29

I'm going to guess the style of this excellent effort was inspired in part by this popular HN poster? Creating a post in this manner with animations does take a lot of work but it's very effective and instructional: https://ciechanow.ski/gps/ Incidentally load balancing and dynamic routing in satellite networks looks like a complicated problem and is a subject of active research: https://www.ncbi.nlm.nih.gov/pmc/arti…

Yes! Bartosz Ciechanowski is a huge inspiration to me. This post was my first attempt to imbue some of his style into my writing. I'm not up to his standard by a long way, but I'm very happy with my results. :)

Re: Load Balancing

#32

Post by gbuk2013 »

throwaway71271 wrote: post #21
gbuk2013 wrote: post #19

Earlier quoted context omitted.

Moving to async model adds new set of operational challenges as well as some interesting failure scenarios. (Edit) Also, in practice you would need at least one more system to enqueue request into the broker, as the latter would typically not be exposed to the outside world. Request/response on the other hand is much simpler to configure and operate.

this is not async, its sync queue the lb puts a request where it has some reply_to (ip:port) where it waits (blockingly) for response from whoever picked up the request, it just does now know who that is until a reply comes

If you have a queue then you have async model.

As an example of a failure scenario, how does your system distinguish between a request timeout, a response that didn’t get sent back because of network failure and the consumer crashing and losing the message?

Re: Load Balancing

#33

Post by samwho »

paulddraper wrote: post #24
deepfriedrice wrote: post #20

Earlier quoted context omitted.

Had the same thought. With AWS et al. it's just too easy to use the same instances and avoid that variable.

On the contrary, with AWS et al, it's just too easy to have noisy neighbors :)

I don't think many people are intentionally deploying their apps with replicas of different sizes, you're right.

But this is it: noisy neighbours, inherent physical differences in even identical hardware, using different node sizes in your clusters. I think incidental differences in servers are very common, even within the same AWS instance category.

Re: Load Balancing

#34

Post by samwho »

akhayam wrote: post #9

If a picture is worth a thousand words.... a well done animation is gotta be at least 10K words. Thanks for making an old topic fun to read about again!

Thanks for your lovely words! <3

Re: Load Balancing

#35

Post by throwaway71271 »

gbuk2013 wrote: post #32
throwaway71271 wrote: post #21

Earlier quoted context omitted.

this is not async, its sync queue the lb puts a request where it has some reply_to (ip:port) where it waits (blockingly) for response from whoever picked up the request, it just does now know who that is until a reply comes

If you have a queue then you have async model. As an example of a failure scenario, how does your system distinguish between a request timeout, a response that didn’t get sent back because of network failure and the consumer crashing and losing the message?

pretty much the same way a push load balancer does https://github.com/jackdoe/back-to-back/blob/master/broker/b...

    for {
      select {
      case reply :=
not much different than what you do with normal http timeouts, you send a request, sometimes a response comes sometimes it doesnt, up to the load balancer to decide if it wants to retry or error out

also, queue does not mean async model, it means a queue, there are many queues in http requests responses (e.g. the listen(2) backlog queue itself) and it does not make it async :)

Re: Load Balancing

#36

Post by 411111111111111 »

bbatchelder wrote: post #16

The playground simulation is pretty cool. One thing I found interesting, is it you go with PEWMA and create a scenario where the cluster is stressed, and then add 1 server, it pummels the shit out of the new server and you have a brief surge in failed requests. Not sure if that is a real world issue, or just with the simulation...

It should happen in the real world as well, at least that's what I've been told when I started my first job as a system admin.

The reason people cited to me back then was that the balancer usually isn't particularly smart when balancing, so they only see a free node, thus every free request is routed to it. The errors (mostly timeout) will happen once the request start to actually get processed.

Normally, the node gets a steady amount of requests over time, thus the load is constant (generally speaking, a request will require the most resources at the same relative time of their lifecycle). As all requests are fresh, they'll all hit the same load bottleneck at the same time, causing all the timeouts.

The answer is to both aggressively scale horizontally and then quickly decommission until you're back to baseline.

Or just accept the failed requests

Its been over 10 years though, it mightve been improved since.

Re: Load Balancing

#37

Post by samwho »

dswalter wrote: post #3

The simulations in this are great. It must have taken a ton of work to get this to such an apparently simple and almost self-explanatory state. Really effective as teaching material; kudos to the author!

I really appreciate you saying this, you have no idea how much it means to me.

I spent many evenings and weekends tweaking this asking myself "is this intuitive to someone whose only experience with this topic is everything prior in this post?"

It's important to me that every section is grounded only in all of the previous sections. One of my fundamental beliefs is that anyone can learn anything, provided they're presented the material in the right order.

Re: Load Balancing

#38

Post by gbuk2013 »

throwaway71271 wrote: post #35
gbuk2013 wrote: post #32

Earlier quoted context omitted.

If you have a queue then you have async model. As an example of a failure scenario, how does your system distinguish between a request timeout, a response that didn’t get sent back because of network failure and the consumer crashing and losing the message?

pretty much the same way a push load balancer does https://github.com/jackdoe/back-to-back/blob/master/broker/b... for { select { case reply := not much different than what you do with normal http timeouts, you send a request, sometimes a response comes sometimes it doesnt, up to the load balancer to decide if it wants to retry or error out also, queue does not mean async model, it means a queue, there are many queue…

And what happens if your request modifies state but the response is not received?

Re: Load Balancing

#39

Post by throwaway71271 »

paulddraper wrote: post #23
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

One of the nicest things about pull load balancing is that it completely replaces the need for a separate registration + health check system. Once you consider that, IDK if someone can really claim "more complicated."

oh yea, the amount of outages we had in my last company because of weighted roundrobbin registration/healthcheck/watermark/latch issues were way more than anyone would guess

Re: Load Balancing

#40

Post by samwho »

paulddraper wrote: post #23
throwaway71271 wrote: post #8

we are so stuck with this push request load balancing its crazy, if we just switch to pull instead of push things get much smoother, and resources get better utilized you cant reliably guess if the instance where you will push your request actually has capacity to handle it, even using ML to guess it will still have thrashing properties but if you just let instances pull work, things work out for themselves sadly, th…

One of the nicest things about pull load balancing is that it completely replaces the need for a separate registration + health check system. Once you consider that, IDK if someone can really claim "more complicated."

I honestly had never thought hard about reversing the relationship and having workers pull. This point about no longer requiring health checks is a real "woah" moment. Thanks for expanding my mind!
Post reply on HN