And yet we still haven't observed proton decay.
Partly because observing it requires very specific conditions. That makes the scale really tiny actually.
At Scale, Rare Events Aren’t Rare
41–50 of 59 posts
Re: At Scale, Rare Events Aren’t Rare
#42As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10. Graph: http://www.meta-calculator.com/online/?panel-102-graph&data-...
Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?
10 machines with a 10% chance of failure roughly equal to 100 machines with a 1% of failure.
I think confusingly worded, as n increases the reliability of each node has to increase correspondingly to get the convergence. I'm not sure what real system this reflects, but I suppose it's an indication of at what point the problems of scale will bite (if you know your rough failure rate).
Re: At Scale, Rare Events Aren’t Rare
#43Love this article but think the headline actually makes the wrong point - this is a product management issue, not a "it may never happen" issue. That it takes someone like James to know two wildly different domains - both the business-level details at risk (a $1M generator is worth potentially damaging if the alternative is a guaranteed $100M revenue loss) and the details of power engineering (overriding the switch only risks the generator, not a datacenter fire or loss of life) - is a shame.
Could the power engineering team have made this tradeoff more clear to the project managers doing the initial install? And yet, exposing a million little configuration options to the end-user isn't the right approach either.
Re: At Scale, Rare Events Aren’t Rare
#44There was an article a while back that used the term "Walmart scale". If some weird customer interaction happens one time in a million transactions, it happens ten times a day at Walmart.
Re: At Scale, Rare Events Aren’t Rare
#45Earlier quoted context omitted.
There was essentially instability and chaos in the big dev heavy divisions at MS when they all worked in one branch, but that led very rapidly to more sophisticated models with more points of validation between the average dev and the common code/builds that qa and everyone else shared and used.
The common pattern nowadays is to code review each commit, build each commit on Jenkins & pass tests, git bisect, etc... What was the procedure back then ?
"don't break the build" was the procedure.
Re: At Scale, Rare Events Aren’t Rare
#46Earlier quoted context omitted.
Very nice rule of thumb, honestly I did not expect it to (sort of) converge to ~63%. Does anyone have some intuition for this?
10 machines with a 10% chance of failure roughly equal to 100 machines with a 1% of failure. I think confusingly worded, as n increases the reliability of each node has to increase correspondingly to get the convergence. I'm not sure what real system this reflects, but I suppose it's an indication of at what point the problems of scale will bite (if you know your rough failure rate).
[deleted]
Re: At Scale, Rare Events Aren’t Rare
#47Re: At Scale, Rare Events Aren’t Rare
#48Re: At Scale, Rare Events Aren’t Rare
#49Earlier quoted context omitted.
It's always amusing when someone asks for a layman/non-math/intuitive reason why something works out and HN responds with a 3-paragraph long proof that seems to always require university-level math. And it seems those comments almost invariably start with "Oh, you just..."
Reminds me of my favorite "HN isn't the normal world" exchange: https://news.ycombinator.com/item?id=35079
In a sibling thread on that page:
cperciva 3548 days ago [-]
That is my startup idea. I don't want to take this
thread even more off-topic (if that's even possible),
but please feel free to contact me at the address in
that first post to explain why you think it is a bad
idea.
dhouston 3548 days ago [-]
we're in a similar space -- http://www.getdropbox.com
(and part of the yc summer 07 program) basically,
sync and backup done right (but for windows and os
x). i had the same frustrations as you with existing
solutions. let me know if it's something you're
interested in, or if you want to chat about it
sometime.
drew (at getdropbox.com)Re: At Scale, Rare Events Aren’t Rare
#50Unfortunately most people don't realize this until they are at scale.
Amazing how most developers don't know about probability and think software is not expected to survive hardware failures.