Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interns with toasters: how I taught people about load balancers

rachelbythebay.com

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Interns with toasters: how I taught people about load balancers

#101

Post by bigiain »

pitaj wrote: post #94
abakker wrote: post #71

Earlier quoted context omitted.

There are a variety of mechanical systems used to combat this problem, not just thick grease. Limited Slip differentials can be employed to ensure that one wheel can only slip so far before the differential locks and then both wheels are forced to rotate together. LSDs can be employed in 3 positions front, rear, and center. The front and rear will lock left and right wheels together, the center one will lock the rota…

There's also the Torsen differential [1] [1]: https://youtu.be/JEiSTzK-A2A

And for those on a budget - what we in Australia call "The CIG Locker"...

http://www.billzilla.org/diffs.htm

(They _kinda_ work on rally cars and hillclimbers...)

Re: Interns with toasters: how I taught people about load balancers

#102

Post by xchaotic »

jlgaddis wrote: post #16
scopecreep wrote: post #7

I feel like if you have to teach them about load balancers as college interns, FB needs to find a better school to pull interns from.

Where else might they learn about load balancers? I doubt there are many high schools teaching these concepts.

Vendors like Cisco have excellent training, but it's biased towards their own gear.

Re: Interns with toasters: how I taught people about load balancers

#103

Post by jakobegger »

rollcat wrote: post #91
jakobegger wrote: post #73

Earlier quoted context omitted.

I'm not sure how that would solve the problem. Not every server is a front end web server. Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others…

Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurr…

Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.

Re: Interns with toasters: how I taught people about load balancers

#104

Post by darshitpp »

masklinn wrote: post #42
darshitpp wrote: post #38

> This is what happened when one bad web server decided it was going to fail all of its requests, and would do so while incurring the absolute minimum amount of load on itself. Good ELI5 explanation, but it doesn't really explain why the webserver failed the requests as it did. Or maybe I'm missing something?

> it doesn't really explain why the webserver failed the requests as it did. Or maybe I'm missing something? That the article is about load-balancers and how a single "rogue actor" can have outsized effect on the entire thing. The failing webserver is besides the point.

Oh, ok

Re: Interns with toasters: how I taught people about load balancers

#105

Post by darshitpp »

Twirrim wrote: post #98
darshitpp wrote: post #38

> This is what happened when one bad web server decided it was going to fail all of its requests, and would do so while incurring the absolute minimum amount of load on itself. Good ELI5 explanation, but it doesn't really explain why the webserver failed the requests as it did. Or maybe I'm missing something?

Take your pick. There's a remarkable number of things that can cause a server to start returning 500s while the rest of the fleet is fine. Doesn't even have to be that specific server's fault (e.g. database behind it could have reached a connection limit handling the connection pools from all the other webservers, leaving this one in the dust. Fun part from this is it can result in the misbehaving server _moving_ as…

Understood! Thanks for explaining!

Re: Interns with toasters: how I taught people about load balancers

#106

Post by grosjona »

kd5bjo wrote: post #85
grosjona wrote: post #84

Earlier quoted context omitted.

If each server can handle thousands of operations at a time, the duration of each operation shouldn't matter because long operations should also be evenly spread across servers... Also due to probability. When you're dealing with big numbers across relatively few servers, randomness averages out everything.

It depends on how long your long operations are. Back when I was working on Justin.tv, we had to write a custom load balancer because we had to maintain a sizable number of TCP connections that lasted for hours or days without disconnecting.

I've worked on a cryptocurrency trading platform that had WebSocket connections which also sometimes lasted for days (including trading bots). Each user-facing server could handle 10k concurrent sockets and the long connections were spread out evenly.

When you have a sample of 10k users, they tend to behave consistently with any other 10k sample... Assuming that all machines are the same size and the code is the same on all machines.

Re: Interns with toasters: how I taught people about load balancers

#107

Post by ithkuil »

cecilpl2 wrote: post #80

I see that on my continuous integration system. We use Teamcity with ~50 agents for build tasks that take 20-30 minutes. Each agent can only be running a single task. During the day, all agents are busy and the queue fills up with 30 or so pending tasks. If one agent gets into a bad state where, say, it fails to checkout from source control and fails within the first 20 seconds of a build, then it will very quickly c…

> this particular failure mode actually becomes more common the more agents you add

I'm surprised. I would expect the length of the pending job queue to also have an impact on this failure _mode_ (as opposed to the failure _cause_), and the queue length is inversely proportional to the number of agents, flooring to 0 when you have to many agents than necessary to deal with peak demand.

If the bad agents (I'm assuming that the fault to check out from SCM lies in the agents themselves, not in the overloaded SCM server due to too many agents polling, otherwise this is not an example relevant to the thread) are e.g. 1 every 50 agents and the number of agents is much larger than the peak load (e.g. a million agents), the probability of a job of failing approaches 1/50, because the good agents blocked on doing work are just a small fraction and thus they don't significantly skew the probability of a new job to get scheduled to a bad agent (which are more likely to be ready).

If on the other hand you have less agents than peak load, the queue will contain some jobs, and bad agents will chew through the whole queue when they get the chance, as you described.

And they always get the chance, since they are almost always ready (if they fail very quickly), so that the pool of ready agents will almost always contain only bad agents.

If you add more agents you have to deal with these bad agents "floating on the top of the pool", and the new ones you add (at the rate of e.g. 1 every 50). So, if you have 5 bad agents and you add another 5 you still have a pretty high chance that new jobs will land on the bad ones.

But you can add more and the probability of success just gets better.

Re: Interns with toasters: how I taught people about load balancers

#108

Post by nathan_f77 »

mabbo wrote: post #72
vasco wrote: post #4

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check AP…

There's a great gem for Rails that you can use for comprehensive health checks: https://github.com/ianheggie/health_check

It can check the database connection, redis, cache, email, up-to-date migrations, and S3 credentials.

Re: Interns with toasters: how I taught people about load balancers

#109

Post by rollcat »

jakobegger wrote: post #103
rollcat wrote: post #91

Earlier quoted context omitted.

Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurr…

Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.

> A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks.

Except now you can use a round-robin/"dumb" LB on the frontend, fixing that exact issue (and a whole class of other problems).

Yes, a broken worker will quickly drain the queue, except once you find out and remove it, you can (hopefully) resubmit the failed jobs. It gives you a new primitive to work with - HTTP requests are 1:1 to HTTP responses, but a job's result can be updated, and the new result propagated up the chain. (Think: media transcoding / post-processing, rendering pipelines, data analysis, etc.)

> that would require totally changing the application architecture

True, usual cost/benefit assessment when dealing with technical debt, etc.

Re: Interns with toasters: how I taught people about load balancers

#110

Post by deaps »

vasco wrote: post #4

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

I was going to say something similar. I get that this was a very basic explanation of what load balancing can be but very rarely have I seen it implemented in such a fashion, as described - that is, 'serving toast, when finished with the toast, that toaster gets more toast, etc.'

There are so many types of load balancing, and so many different ways to guard, simply with best practices, against what this story explains.

I also found the 'quadruple hump' in a graph reference difficult to follow without more of a backstory of what the graph was representing.

I also don't understand why it would be so difficult to grep through the logs of that load balancer (or load balancers) and find that common fault on the backend.

Post reply on HN