Interns with toasters: how I taught people about load balancers • Hacker News Board

Re: Interns with toasters: how I taught people about load balancers

#71

Post by abakker » Sun, Apr 22, 2018, 3:57 PM UTC

snovv_crash wrote: post #39

This reminds me strongly of another system of distribution, which suffers from the same effect: differential gearboxes in cars. Since the torque to all the wheels is equal, if one wheel slips it very quickly takes all of the power of the engine, since power = rotational velocity * torque. Only if the speeds are similar is the allocation of power to the wheels similar. In cars, the solution is to make sure that the po…

There are a variety of mechanical systems used to combat this problem, not just thick grease.

Limited Slip differentials can be employed to ensure that one wheel can only slip so far before the differential locks and then both wheels are forced to rotate together. LSDs can be employed in 3 positions front, rear, and center. The front and rear will lock left and right wheels together, the center one will lock the rotation speed of the front and back axles.

You can also add manually selectable locking differentials in any of those positions instead. Frequently, you see LSDs in front and back and manual lockers in the center.

In off road applications it is more common to see manually selectable lockers in every position. Because of the lack of differentials, though, manual lockers put a lot of load on the drivetrain if the surface isn’t slippery.

The alternative, traction control can use variable differentials to send different power amounts to different wheels, typically by employing the brakes on the slipping wheel to force power to distribute evenly.

Obviously, there is a lot to this, but highlights some of the options.

Re: Interns with toasters: how I taught people about load balancers

#72

Post by mabbo » Sun, Apr 22, 2018, 4:00 PM UTC

vasco wrote: post #4

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true".

Now you've got a host that can't access a database or can't do much of anything- but it can return true!

Health check APIs should always do some level of checks on dependencies and complex internal logic (maybe a few specific unit/integration tests?) to ensure things are truly healthy.

Re: Interns with toasters: how I taught people about load balancers

#73

Post by jakobegger » Sun, Apr 22, 2018, 4:12 PM UTC

rollcat wrote: post #48

jakobegger wrote: post #37

Earlier quoted context omitted.

That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to distribute over a number of servers, then your load balancer needs to track which servers are busy, or you'll spend a lot of time waiting for the randomly assigned server to finish, while others are idle.

Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use…

I'm not sure how that would solve the problem. Not every server is a front end web server.

Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others are idle.

Re: Interns with toasters: how I taught people about load balancers

#74

Post by s_kilk » Sun, Apr 22, 2018, 4:25 PM UTC

mabbo wrote: post #72

vasco wrote: post #4

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check AP…

At work we have two endpoints on every service, `/status` and `/health-check`.

The former is basically a "return true" endpoint, which can tell you if the service is alive and reachable. The latter will usually do something like "select 1;" from any attached databases and only succeed if everything is OK.

Re: Interns with toasters: how I taught people about load balancers

#75

Post by indigodaddy » Sun, Apr 22, 2018, 4:37 PM UTC

Not sure how relevant this is to Rachel's incident, but looks like they use ECMP/BGP->shiv(L4)->proxygen(L7). So hard to believe that health checking wasn't in the mix. If the nodes were passing the health check, but not properly serving requests still, then I'd assume that a post-mortum items would have involved improving health checks.

Found this pretty cool presentation/PDF about FB's load-balancing architecture. Stays fairly high level: http://www.esnog.net/gore16/gore16-files/Mikel_Billion_user....

Re: Interns with toasters: how I taught people about load balancers

#76

Post by tzs » Sun, Apr 22, 2018, 4:46 PM UTC

jakobegger wrote: post #34

amingilani wrote: post #2

Who is Rachel and how do her short and simple stories always hit the front-page? They're interesting but I always think they're a little _too_ simple. I mean this entire thing can be summed up with: 500s (and other errors) are returned faster than processed requests. Load-balancers will find a misbehaving server's queue empty more often and give it all the requests

Everything seems trivial once you understand it. The author seems to be good at making people understand, so that it seems trivial afterwards. Knowing nothing about load balancers, had I read your summary before Rachels article, I would probably just glossed over it, but I would not have understood its profound importance for designing load balancers. Rachel's story kept me engaged and explained the problem in a way…

> Everything seems trivial once you understand it

That has an interesting consequence with patents. In a patent infringement trial plaintiff is going to have to explain to the jury what the patent covers. But if plaintiff succeeds in getting the jury to understand the patent well enough to be able to realize that defendant infringed, there is a good chance they will understand the patent well enough to think that it was not non-obvious. The defendant also explains to the jury what the patent does, in order to explain their theory as to why they are not infringing.

Re: Interns with toasters: how I taught people about load balancers

#77

Post by Johnny555 » Sun, Apr 22, 2018, 4:56 PM UTC

jakobegger wrote: post #37

grosjona wrote: post #22

That's why I like random load balancing. If each machine is powerful enough and can handle a few thousand users then the distribution averages out. Smart load balancers are only really necessary if you have inefficient servers that can't handle more than 100 connections per second and they're difficult to get right. If you toss a coin 10 times, you're much more likely to get >=80% heads than if you were to toss that…

That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to distribute over a number of servers, then your load balancer needs to track which servers are busy, or you'll spend a lot of time waiting for the randomly assigned server to finish, while others are idle.

Also if you have moderately long lived requests, if one server starts to get overloaded and it starts to slow down, new requests will still be sent to that server, which will make it more overloaded and slower, and jobs start to build up on that server, making it slower and slower.

Re: Interns with toasters: how I taught people about load balancers

#78

Post by randerson » Sun, Apr 22, 2018, 4:58 PM UTC

s_kilk wrote: post #74

mabbo wrote: post #72

Earlier quoted context omitted.

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check AP…

At work we have two endpoints on every service, `/status` and `/health-check`. The former is basically a "return true" endpoint, which can tell you if the service is alive and reachable. The latter will usually do something like "select 1;" from any attached databases and only succeed if everything is OK.

And the former is the one you want your Load Balancers to be checking. With a deep health check, even a brief database outage will cause every web server to be taken out of rotation, and then you're completely down for at least as many health check intervals as it takes for the LB to consider a host healthy again. Same goes for any other shared resource that is likely to affect all web servers if it becomes unavailable.

Re: Interns with toasters: how I taught people about load balancers

#79

Post by randerson » Sun, Apr 22, 2018, 5:00 PM UTC

matt4077 wrote: post #35

grosjona wrote: post #22

That's why I like random load balancing. If each machine is powerful enough and can handle a few thousand users then the distribution averages out. Smart load balancers are only really necessary if you have inefficient servers that can't handle more than 100 connections per second and they're difficult to get right. If you toss a coin 10 times, you're much more likely to get >=80% heads than if you were to toss that…

Random balancing only works with homogenous servers.

Isn't it almost always advisable to have homogenous servers in a web farm? What are some example cases there that doesn't make sense?

Re: Interns with toasters: how I taught people about load balancers

#80

Post by cecilpl2 » Sun, Apr 22, 2018, 5:18 PM UTC

I see that on my continuous integration system. We use Teamcity with ~50 agents for build tasks that take 20-30 minutes. Each agent can only be running a single task.

During the day, all agents are busy and the queue fills up with 30 or so pending tasks.

If one agent gets into a bad state where, say, it fails to checkout from source control and fails within the first 20 seconds of a build, then it will very quickly chew through the entire queue of pending tasks, failing them all.

You'd think the more agents you have the better insulated you are from the failure of a single one, but this particular failure mode actually becomes more common the more agents you add!!!