Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Estimating the chances of something that hasn’t happened yet

johndcook.com

Post reply on HN
Print view View original

31–40 of 155 posts

Re: Estimating the chances of something that hasn’t happened yet

#31

Post by comboy »

cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

I agree that based on combination (n! / k!(n-k)!) it seems to be 1 in 20, but when you think of it as running A and B and checking if X or Y is faster 3 times, then you get 1 in 8. Quite a big difference. Where does it come from? Am I doing something wrong? I mean combination approach counts same time results as a win but given enough time precision we could skip this scenario.

edit: Ah, got it, in my case 3rd result for X can be higher than the first result for Y. You said all times must be smaller.

Re: Estimating the chances of something that hasn’t happened yet

#32

Post by jdmichal »

olooney wrote: post #19
blowski wrote: post #2

I’m not very good at maths, so I didn’t understand the whole post. However, does the size of the whole population affect the “3/n” thing? For example, if I’ve read 200 pages of a 201 page book and not discovered a typo the chances are 3/200 if I’ve understand the post correctly. If the book has 20000 pages, is the probability still 3/200?

The rule of three gives an upper bound estimate that each page has a 3/200 or 1.5% chance of having a typo. For the 20,000 page book, the probability of typo per page is still 1.5%, but you have to roll the dice 19,800 more times, each time you turn a new page. The formula for the upper bound on probability of getting through all the remaining pages without a typo is 1 - (1-3/200)^19800 Which of course is basically 1…

Minor correction to your otherwise excellent final statement:

"... or equivalently the total number of [pages with] typos in the 20,000 page book is between 0 and 300."

We're not measuring the number of typos on a page, but only whether the page contains a typo or not. So we can't speak about the number of typos.

Re: Estimating the chances of something that hasn’t happened yet

#33

Post by gboudrias »

rspeer wrote: post #21

If the author is reading this: When someone provides you a pro-bono translation, by all means credit them, but do not let them host it on their own site. Frequently, they are siphoning your PageRank. They will eventually replace the translation with monetized content of their choice. A good translation takes work! If you got a translation for free, why should you believe it's good, or that it has no ulterior motive?…

Yeah this is weird. Also, if OP doesn't speak Italian themselves, how can they attest to the quality of the translation?

Re: Estimating the chances of something that hasn’t happened yet

#34

Post by jedberg »

ajkjk wrote: post #15

So basically the '3' comes entirely from the choice of a 95% confidence interval. If you want a 99% confidence interval it's instead the 'rule of 4.6', which doesn't roll off the tongue as well.

You could do the 'rule of 5' though and have a confidence of 99.3%, which is pretty close to 99.

Re: Estimating the chances of something that hasn’t happened yet

#35

Post by JoshuaDavid »

comboy wrote: post #31
cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

I agree that based on combination (n! / k!(n-k)!) it seems to be 1 in 20, but when you think of it as running A and B and checking if X or Y is faster 3 times, then you get 1 in 8. Quite a big difference. Where does it come from? Am I doing something wrong? I mean combination approach counts same time results as a win but given enough time precision we could skip this scenario. edit: Ah, got it, in my case 3rd result…

Let's say there are 6 trials -- name them x1, x2, x3, y1, y2, y3.

In the case of checking if x is faster than y 3 times, you're doing x1 In the case of checking if all three measurements of x are smaller than all three measurements of y, you're checking x1 In other words, the latter case is checking whether the slowest x of three trials is faster than the fastest y of three trials.

Re: Estimating the chances of something that hasn’t happened yet

#37

Post by rivp931 »

cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

Going down this rabbit hole eventually leads you to nonparametric statistical tests, e.g. Mann-Whitney-U and so on.

Re: Estimating the chances of something that hasn’t happened yet

#38

Post by HAL9000Ti »

rspeer wrote: post #21

If the author is reading this: When someone provides you a pro-bono translation, by all means credit them, but do not let them host it on their own site. Frequently, they are siphoning your PageRank. They will eventually replace the translation with monetized content of their choice. A good translation takes work! If you got a translation for free, why should you believe it's good, or that it has no ulterior motive?…

You are exceedingly suspicious for no reasons, people just like to translate articles that they enjoyed reading, the vast majority of times that's the only motive.

Re: Estimating the chances of something that hasn’t happened yet

#39

Post by cperciva »

rivp931 wrote: post #37
cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

Going down this rabbit hole eventually leads you to nonparametric statistical tests, e.g. Mann-Whitney-U and so on.

Right. And those are also more powerful (and don't need any knowledge of the distribution of measurement errors).

I mentioned the "three old and three new" test because it's simple, not because it's powerful.

Re: Estimating the chances of something that hasn’t happened yet

#40

Post by comboy »

JoshuaDavid wrote: post #35
comboy wrote: post #31

Earlier quoted context omitted.

I agree that based on combination (n! / k!(n-k)!) it seems to be 1 in 20, but when you think of it as running A and B and checking if X or Y is faster 3 times, then you get 1 in 8. Quite a big difference. Where does it come from? Am I doing something wrong? I mean combination approach counts same time results as a win but given enough time precision we could skip this scenario. edit: Ah, got it, in my case 3rd result…

Let's say there are 6 trials -- name them x1, x2, x3, y1, y2, y3. In the case of checking if x is faster than y 3 times, you're doing x1 In the case of checking if all three measurements of x are smaller than all three measurements of y, you're checking x1 In other words, the latter case is checking whether the slowest x of three trials is faster than the fastest y of three trials.

You're right. I actually edited before you replied. Should have deleted it or preferably think more before typing. But now it's forever.
Post reply on HN