Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

At Scale, Rare Events Aren’t Rare

perspectives.mvdirona.com

Post reply on HN
Print view View original

21–30 of 59 posts

Re: At Scale, Rare Events Aren’t Rare

#21

Post by InclinedPlane »

sametmax wrote: post #9
graphememes wrote: post #7

Unfortunately most people don't realize this until they are at scale.

It's ok, since most people never get "at scale". When you start getting these problems, it means you reached the stars you were shooting for. Having those issues are a good thing.

Not always. It's possible to bury yourself under so much technical debt that you can't get out from under it. To extend the metaphor, you can end up spending all your revenue merely servicing the debt and you cannot invest in paying it down. Your system can be so flawed that it hampers innovation, and requires ongoing costly maintenance and support which eats up all of your dev resources. The end result is that innovation is blocked off for you and if you run into a bump that reduces your review significantly for a while you can easily end up bankrupt.

Re: At Scale, Rare Events Aren’t Rare

#23

Post by sametmax »

InclinedPlane wrote: post #21
sametmax wrote: post #9

Earlier quoted context omitted.

It's ok, since most people never get "at scale". When you start getting these problems, it means you reached the stars you were shooting for. Having those issues are a good thing.

Not always. It's possible to bury yourself under so much technical debt that you can't get out from under it. To extend the metaphor, you can end up spending all your revenue merely servicing the debt and you cannot invest in paying it down. Your system can be so flawed that it hampers innovation, and requires ongoing costly maintenance and support which eats up all of your dev resources. The end result is that innov…

Same answer as to the other comment.

Re: At Scale, Rare Events Aren’t Rare

#24

Post by SamReidHughes »

Rainymood wrote: post #13
kyleschiller wrote: post #3

As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...

Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?

I'd say, hey, how do you calculate (1-h)^k? First take the natural log: ln((1-h)^k) = k ln(1-h) = -kh. And then exponentiate back up: e^(-kh). (For small values of h, ln(1-h) = -h by linear approximation.) (Edit: Wiped out looong comment.)

Re: At Scale, Rare Events Aren’t Rare

#25

Post by Minikloon »

InclinedPlane wrote: post #20
jasonallen wrote: post #5

It reminds me of an old story about Microsoft Windows. Back in the early 2000's, compiling and building Windows from source code took many hours on very specialized build hardware. Meanwhile there were thousands of developers who contributed to the full Windows stack. If any developer checked in a build failure, it would cause the build to be delayed. Well, at that scale (of thousands of developers), you can't compil…

There was essentially instability and chaos in the big dev heavy divisions at MS when they all worked in one branch, but that led very rapidly to more sophisticated models with more points of validation between the average dev and the common code/builds that qa and everyone else shared and used.

If the bar is set at breaking the build once a year, sounds like we're all average devs.

Re: At Scale, Rare Events Aren’t Rare

#26

Post by StavrosK »

SamReidHughes wrote: post #24
Rainymood wrote: post #13

Earlier quoted context omitted.

Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?

I'd say, hey, how do you calculate (1-h)^k? First take the natural log: ln((1-h)^k) = k ln(1-h) = -kh. And then exponentiate back up: e^(-kh). (For small values of h, ln(1-h) = -h by linear approximation.) (Edit: Wiped out looong comment.)

I think by "intuition" the GP meant "for the non-mathematicians" :P

Re: At Scale, Rare Events Aren’t Rare

#27

Post by stavros »

Minikloon wrote: post #25
InclinedPlane wrote: post #20

Earlier quoted context omitted.

There was essentially instability and chaos in the big dev heavy divisions at MS when they all worked in one branch, but that led very rapidly to more sophisticated models with more points of validation between the average dev and the common code/builds that qa and everyone else shared and used.

If the bar is set at breaking the build once a year, sounds like we're all average devs.

Speak for yourself, I'm an abysmal dev by that metric.

Re: At Scale, Rare Events Aren’t Rare

#28

Post by jacquesm »

kev009 wrote: post #17

Storage devices regularly go berserk in really novel and interesting ways when you have a large enough pool. Most projects I've worked on, I've known enough to fix the bugs, had a working theory of what the issues were and could fix if I really needed to after higher priorities, or could somehow work around. With storage devices, I'm frequently bewildered and stuck maybe to the last category at best. There are times…

> There are times when I sit back and just think, wow, how amazing is it that computers work at all knowing the things that do go wrong.

There is a special category of bugs named for that kind of feeling, they're called schrödinbug. The idea is that once you've noticed that something couldn't work it promptly stops working.

Re: At Scale, Rare Events Aren’t Rare

#29

Post by lolive »

InclinedPlane wrote: post #20
jasonallen wrote: post #5

It reminds me of an old story about Microsoft Windows. Back in the early 2000's, compiling and building Windows from source code took many hours on very specialized build hardware. Meanwhile there were thousands of developers who contributed to the full Windows stack. If any developer checked in a build failure, it would cause the build to be delayed. Well, at that scale (of thousands of developers), you can't compil…

There was essentially instability and chaos in the big dev heavy divisions at MS when they all worked in one branch, but that led very rapidly to more sophisticated models with more points of validation between the average dev and the common code/builds that qa and everyone else shared and used.

The common pattern nowadays is to code review each commit, build each commit on Jenkins & pass tests, git bisect, etc... What was the procedure back then ?

Re: At Scale, Rare Events Aren’t Rare

#30

Post by obstinate »

kyleschiller wrote: post #3

As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...

Or as my first boss and mentor had a habit of saying, when you run a billion trials, one in a million events will happen about a thousand times.
Post reply on HN