Estimating the chances of something that hasn’t happened yet • Hacker News Board

Re: Estimating the chances of something that hasn’t happened yet

#151

Post by GuB-42 » Sat, Nov 17, 2018, 3:14 PM UTC

cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

In the specific case of code, it is worth noting that runs are typically not independent, because of caching.

It is very common for the first run to be slower. It means that by "random chance" slow-slow-slow-fast-fast-fast is more likely than fast-fast-fast-slow-slow-slow. I personally tend to discard first runs as outliers when profiling.

Re: Estimating the chances of something that hasn’t happened yet

#152

Post by Mauricio_ » Sat, Nov 17, 2018, 3:37 PM UTC

YeGoblynQueenne wrote: post #84

So, according to this rule, if I wait for 10 minutes for a bus to come and none does, and then I wait for another 10 minutes for an alien invasion and none happens, the two have the same upper bound on their probability? Or are we going to start talking about priors, on buses and alien invasions, in which case the rule of three is not really useful? If I want to know how likely a specific book is to have typos, can't…

>the two have the same upper bound on their probability? Using only that information, yes. Replace the names of the two events with "event A" and "event B" and it doesn't sound so weird.

Re: Estimating the chances of something that hasn’t happened yet

#153

Post by Mauricio_ » Sat, Nov 17, 2018, 3:41 PM UTC

This is very similar to the sunrise problem. Laplace said the probability of the sun rising tomorrow is (k+1)/(k+2), where k is the number of days we know the sun has risen consecutively, if we always saw it rise, and if we don't have any other information.

https://en.m.wikipedia.org/wiki/Sunrise_problem

Re: Estimating the chances of something that hasn’t happened yet

#154

Post by neolefty » Sun, Nov 18, 2018, 6:01 AM UTC

hawkice wrote: post #72

neolefty wrote: post #57

Earlier quoted context omitted.

If you're truly estimating, though, wouldn't you use a 50% confidence interval — which gives you the rule of 1 (chances of the thing being true are 1/n — if you've seen 20 pages without a typo, chances are 1/20 that there's a typo on a page, with 50% confidence)?

So, for a 50% confidence interval, you could look at the first word -- if it is a typo, boom, done. Otherwise, flip a coin. This stinks. Estimating is about approximating an answer with low information -- it's about efficiency of using data, not _only_ doing better than guessing.

Sorry, I meant once you've examined n trials, what is your 50% confidence interval about the odds for a single trial? I think it would be 1/n.

Re: Estimating the chances of something that hasn’t happened yet

#155

Post by agonmon » Sun, Nov 18, 2018, 9:45 PM UTC

hinkley wrote: post #149

hinkley wrote: post #89

Earlier quoted context omitted.

Statistics is a tricky beast. If 3 runs gives you a 95% chance that your change is an improvement, and you use this process on ten different improvements, there's a 40% chance that at least one of them is bogus.

Would love to see a refutation from a downvoter.

I believe you have it right. Odds that all changes are improvements in 10 changes is: (.95)^10 Odds that at least one change is not improvement is: 1-(.95)^10 = 0.40

One could probably make an argument that not all trials are independent, but not sure what else down-voters saw.