Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interns with toasters: how I taught people about load balancers

rachelbythebay.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: Interns with toasters: how I taught people about load balancers

#111

Post by lathiat »

pitaj wrote: post #94
abakker wrote: post #71

Earlier quoted context omitted.

There are a variety of mechanical systems used to combat this problem, not just thick grease. Limited Slip differentials can be employed to ensure that one wheel can only slip so far before the differential locks and then both wheels are forced to rotate together. LSDs can be employed in 3 positions front, rear, and center. The front and rear will lock left and right wheels together, the center one will lock the rota…

There's also the Torsen differential [1] [1]: https://youtu.be/JEiSTzK-A2A

It takes a while to get your head around how these work but they are fantastic. Fortunately there are a few videos on it now to help.

Re: Interns with toasters: how I taught people about load balancers

#112

Post by mtreis86 »

abakker wrote: post #71
snovv_crash wrote: post #39

This reminds me strongly of another system of distribution, which suffers from the same effect: differential gearboxes in cars. Since the torque to all the wheels is equal, if one wheel slips it very quickly takes all of the power of the engine, since power = rotational velocity * torque. Only if the speeds are similar is the allocation of power to the wheels similar. In cars, the solution is to make sure that the po…

There are a variety of mechanical systems used to combat this problem, not just thick grease. Limited Slip differentials can be employed to ensure that one wheel can only slip so far before the differential locks and then both wheels are forced to rotate together. LSDs can be employed in 3 positions front, rear, and center. The front and rear will lock left and right wheels together, the center one will lock the rota…

The Viscous Coupling is really neat.

The design consists of a series of plates with holes cut in them and relies on a dilitant fluid (silicon paste) filling the holes to have a significant resistance to shearing. This allows the outputs to run at different speeds than each other, up to a point where the paste gets hot from shearing and the pressure increase locks the plates to each other. When the output speeds are similar, the diff doesn't lock up but will allow one to overspeed the other.

One main benefit to this design over others, is that by changing the number of plates you can change the amount of torque sent to the outputs. When installed with twice as many discs to the rear than the front you end up with twice as much torque to the back wheels, as in the transfer case of a E30 BMW 325iX. Effectively there is no mechanical connection between the engine and the wheels until the transfer case input starts moving which starts to lock up and transfer torque to the rear. The rear diff outputs are not mechanically connected until the shearing in it causes an increases in pressure. But the increase in pressure happens so quickly, lockup being within milliseconds, that you wouldn't know it while driving.

Re: Interns with toasters: how I taught people about load balancers

#113

Post by KaiserPro »

> "some of the commenters dismissed either it ("impossible") or me ("doesn't know anything about load balancing"). "

This to me is odd. Was this posted when AWS's ELBs were new and shiny?

Most of the big failure cases I've dealt with are along these lines.

One server does something stupid and gobbles up the world.

That being said, this is a very neat way of describing the problem, I shall be referencing this in the future, I might start putting that in an interview question....

Re: Interns with toasters: how I taught people about load balancers

#114

Post by aaronmu »

mabbo wrote: post #72
vasco wrote: post #4

And that's why you always set health checks on servers behind a load balancer that should take bad ones out as soon as possible. You then get another interesting problem, which is if the server is only spitting errors when serving requests, it'll then be marked healthy again and forever toggle between healthy and unhealthy. But that you can also solve.

The mistake I've seen here is having the "health check" do little or nothing. In many web services, I've seen a /ping or /health api that the load balancer calls to ask "are you healthy?". But people get rushed or lazy or requirements change, and they wind up with that api just doing "return true". Now you've got a host that can't access a database or can't do much of anything- but it can return true! Health check AP…

The best way I can think of is to aggregate errors over time, categorize them and build a health check around those metrics.

Re: Interns with toasters: how I taught people about load balancers

#115

Post by sammorrowdrums »

randerson wrote: post #78
s_kilk wrote: post #74

Earlier quoted context omitted.

At work we have two endpoints on every service, `/status` and `/health-check`. The former is basically a "return true" endpoint, which can tell you if the service is alive and reachable. The latter will usually do something like "select 1;" from any attached databases and only succeed if everything is OK.

And the former is the one you want your Load Balancers to be checking. With a deep health check, even a brief database outage will cause every web server to be taken out of rotation, and then you're completely down for at least as many health check intervals as it takes for the LB to consider a host healthy again. Same goes for any other shared resource that is likely to affect all web servers if it becomes unavailab…

That's not necessarily true. In kubernetes, for example, you have Liveness and Readiness probes, which can both have a period, timeout, initial delay and importantly a number of failures before you kill the service and spawn a new one.

This allows you to have less frequent checks that are more in-depth, and basic ones that are just the `return true` type.

I guess you're correct that you can have a widespread db outage that makes many of them fail, but then there should be new ones coming very fast, as you can set the deployment minimum for services taking requests. I think you can get very close to stability even in this circumstance.

https://kubernetes.io/docs/tasks/configure-pod-container/con...

Re: Interns with toasters: how I taught people about load balancers

#116

Post by AdmiralAsshat »

KaiserPro wrote: post #113

> "some of the commenters dismissed either it ("impossible") or me ("doesn't know anything about load balancing"). " This to me is odd. Was this posted when AWS's ELBs were new and shiny? Most of the big failure cases I've dealt with are along these lines. One server does something stupid and gobbles up the world. That being said, this is a very neat way of describing the problem, I shall be referencing this in the f…

Given the author's presumed gender, I understood it to be thinly-veiled sexism. But I could be wrong.

Re: Interns with toasters: how I taught people about load balancers

#117

Post by bestes »

gaius wrote: post #87
pulkitsh1234 wrote: post #23

Maybe some people dismissed her problem as 'impossible' because she didn't inform what kind of load balancing technique was the load balancer using? I suspect what happened is that they didn't understand the problem, and so resorted to ineffective means to steer the attention away from their own inadequacies. ^ This statement is kinda harsh.

Every post on that blog is about how the author is incredibly clever and surrounded by dolts.

Many of her posts are about teaching or communicating issues like this to others. In other words, there is no reason to give a class like this if she doesn’t actually know more than the interns.

Re: Interns with toasters: how I taught people about load balancers

#119

Post by hluska »

rollcat wrote: post #90
hluska wrote: post #57

Earlier quoted context omitted.

I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?

So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Cele…

Thanks for the response - I understand it much better now.

Re: Interns with toasters: how I taught people about load balancers

#120

Post by KaiserPro »

AdmiralAsshat wrote: post #116
KaiserPro wrote: post #113

> "some of the commenters dismissed either it ("impossible") or me ("doesn't know anything about load balancing"). " This to me is odd. Was this posted when AWS's ELBs were new and shiny? Most of the big failure cases I've dealt with are along these lines. One server does something stupid and gobbles up the world. That being said, this is a very neat way of describing the problem, I shall be referencing this in the f…

Given the author's presumed gender, I understood it to be thinly-veiled sexism. But I could be wrong.

Perhaps, or just typical dickishness. (mind you its the same thing really.)

As a devop I get told or sorts of "facts" about various building blocks.

We are currently looking at low latency API (read not using HTTP) half of the suggestions I got back were about using haskell or Go. No one bothered with the low hanging fruit, like re-using TCP sockets, using predefined schemas instead of json, etc etc.

Not one person thought about LB latency...

Post reply on HN