Reminds me of this: http://spectrum.ieee.org/tech-talk/computing/it/nsa-data-cen...
At Scale, Rare Events Aren’t Rare
31–40 of 59 posts
Re: At Scale, Rare Events Aren’t Rare
#32Well, Murphy's law. Any non-zero probability event will occur at least once given infinite time.
Re: At Scale, Rare Events Aren’t Rare
#33It's a one in a million shot. But it might just work.
Re: At Scale, Rare Events Aren’t Rare
#34Earlier quoted context omitted.
I'd say, hey, how do you calculate (1-h)^k? First take the natural log: ln((1-h)^k) = k ln(1-h) = -kh. And then exponentiate back up: e^(-kh). (For small values of h, ln(1-h) = -h by linear approximation.) (Edit: Wiped out looong comment.)
I think by "intuition" the GP meant "for the non-mathematicians" :P
It's always amusing when someone asks for a layman/non-math/intuitive reason why something works out and HN responds with a 3-paragraph long proof that seems to always require university-level math. And it seems those comments almost invariably start with "Oh, you just..."
Re: At Scale, Rare Events Aren’t Rare
#35It may be silly but if something is unlikely I stop and remind myself: if something is a 'one in a million' chance, then it happened to 7000 people today.
Re: At Scale, Rare Events Aren’t Rare
#36Though it's not all that rare, there's the question of dealing with death in large online social or identity networks.
With Google having some 3 billion Android / Chrome / Gmail profiles, and Facebook roughly as many users, standard actuarial statistics suggest that, even if allowing for multiple profiles per human, the number of newly dead accounts per day probably runs to the tens of thousands.
(Globally, deaths run about 120,000/day, so the figure's within reason.)
Which means you probably want to consider your processes for such matters, as well as various related issues, such as mistakenly presuming a user has died, or being falsely informed, how to handle data assets after death, etc., etc.
Scale matters.
Re: At Scale, Rare Events Aren’t Rare
#37It may be silly but if something is unlikely I stop and remind myself: if something is a 'one in a million' chance, then it happened to 7000 people today.
"one in a million" happens several times per second, in todays CPUs :)
Fun small experiment. Pick a random number between 1 and 15000000 (odds for winning the lottery in my country). Loop until you pick it again (i know it's pseudorandom and cylic; but the period seems big enough for the experiment).
Watch how freaking fast it happens (sub-second), and how many iterations it took (dozens of millions).
Re: At Scale, Rare Events Aren’t Rare
#38Earlier quoted context omitted.
I think by "intuition" the GP meant "for the non-mathematicians" :P
It's always amusing when someone asks for a layman/non-math/intuitive reason why something works out and HN responds with a 3-paragraph long proof that seems to always require university-level math. And it seems those comments almost invariably start with "Oh, you just..."
Reminds me of my favorite "HN isn't the normal world" exchange:
Re: At Scale, Rare Events Aren’t Rare
#39Well, Murphy's law. Any non-zero probability event will occur at least once given infinite time.
More like infinity times
x * ∞ = ∞ where X > 0
Re: At Scale, Rare Events Aren’t Rare
#40It may be silly but if something is unlikely I stop and remind myself: if something is a 'one in a million' chance, then it happened to 7000 people today.
"one in a million" happens several times per second, in todays CPUs :) Fun small experiment. Pick a random number between 1 and 15000000 (odds for winning the lottery in my country). Loop until you pick it again (i know it's pseudorandom and cylic; but the period seems big enough for the experiment). Watch how freaking fast it happens (sub-second), and how many iterations it took (dozens of millions).
Also why ECC memory is used... and chances to corrupt a bit of memory are much lower than that.