[deleted]
At Scale, Rare Events Aren’t Rare
11–20 of 59 posts
Re: At Scale, Rare Events Aren’t Rare
#12Earlier quoted context omitted.
It's ok, since most people never get "at scale". When you start getting these problems, it means you reached the stars you were shooting for. Having those issues are a good thing.
Assuming you have the resources to cover these events, sometimes you hit scale, but not profitability..
Yes but in that case the scaling technical issues are not your main problem. Your main problem is your business model. Priorities, priorities...
Re: At Scale, Rare Events Aren’t Rare
#13As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...
Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?
Re: At Scale, Rare Events Aren’t Rare
#14As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...
Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?
Honestly I don't remember, it's 1-(1/e) if that helps.
EDIT: http://math.stackexchange.com/questions/82034/prove-that-1-f...
Re: At Scale, Rare Events Aren’t Rare
#15As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...
Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?
If a system has probability 1/n to fail, then it has probability 1 - 1/n to not fail. The probability it will not fail after n trials is (1 - 1/n) ^ n. The limit of this quantity when n->+inf is 1/e.
If you want to know the probability it will fail, just take 1 - probability_success = 1 - 1/e.
Re: At Scale, Rare Events Aren’t Rare
#16As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...
And this is the generalization: https://en.m.wikipedia.org/wiki/Poisson_limit_theorem
Re: At Scale, Rare Events Aren’t Rare
#17Storage devices regularly go berserk in really novel and interesting ways when you have a large enough pool. Most projects I've worked on, I've known enough to fix the bugs, had a working theory of what the issues were and could fix if I really needed to after higher priorities, or could somehow work around. With storage devices, I'm frequently bewildered and stuck maybe to the last category at best. There are times when I sit back and just think, wow, how amazing is it that computers work at all knowing the things that do go wrong.
Re: At Scale, Rare Events Aren’t Rare
#18One quote I've heard, that i find helps non-experts:
You don't have to personally prepare for winning the lottery, but the lottery has to prepare for somebody winning.
Re: At Scale, Rare Events Aren’t Rare
#19Rare events are self-similar.
Re: At Scale, Rare Events Aren’t Rare
#20It reminds me of an old story about Microsoft Windows. Back in the early 2000's, compiling and building Windows from source code took many hours on very specialized build hardware. Meanwhile there were thousands of developers who contributed to the full Windows stack. If any developer checked in a build failure, it would cause the build to be delayed. Well, at that scale (of thousands of developers), you can't compil…
There was essentially instability and chaos in the big dev heavy divisions at MS when they all worked in one branch, but that led very rapidly to more sophisticated models with more points of validation between the average dev and the common code/builds that qa and everyone else shared and used.