Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Estimating the chances of something that hasn’t happened yet

johndcook.com

Post reply on HN
Print view View original

61–70 of 155 posts

Re: Estimating the chances of something that hasn’t happened yet

#61

Post by dahart »

madrox wrote: post #14

What the author glosses over somewhat is the method of sampling. If you read the first 20 pages, find no typos, and use this rule to arrive at 15%, that could be way off. He's assuming the risk of typos are evenly distributed when there's a lot of reasons it may not be. For example, the first half of the book could've been more heavily proof-read than the latter half. It's not out of the question that editors get laz…

> It's understandable to not mention this

I feel like the very first word, "Estimating" made what you're talking about pretty clear. It's extremely common for estimates to make simplifying assumptions that don't necessarily hold if you care about accuracy or specific situations. He also called the rule of three "quick and dirty".

> If you were to randomly read 20 pages in a book and find no typos, 15% probability makes more sense.

OTOH, if typos are actually uniformly distributed, then reading the first 20 is better than a random 20 unless you take care to pick random pages without repeats.

Re: Estimating the chances of something that hasn’t happened yet

#62

Post by zawerf »

cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughly the same (or so I think - I don't really know and want a better workflow here).

But if you do this test repeatedly, even if the code had identical performance, you'll get a false positive 5% of the time. And depending on the spread of the timings you might not get a clear XXXYYY win even when it is indeed a minor improvement.

Re: Estimating the chances of something that hasn’t happened yet

#63

Post by madrox »

nitwit005 wrote: post #56
madrox wrote: post #14

What the author glosses over somewhat is the method of sampling. If you read the first 20 pages, find no typos, and use this rule to arrive at 15%, that could be way off. He's assuming the risk of typos are evenly distributed when there's a lot of reasons it may not be. For example, the first half of the book could've been more heavily proof-read than the latter half. It's not out of the question that editors get laz…

Doesn't the book example need to factor in the book length? If you read 20 pages, and it's 20 pages long, you should be 100% confident, but if you read 20 pages and it's 5000 pages long, your confidence should be near 0.

Not in this case, because the probably being estimated is "the chance a single page could contain a typo" and not "the chance the rest of the book could contain a typo."

To do the latter, you'd need to know the length of the book, yes.

Re: Estimating the chances of something that hasn’t happened yet

#64

Post by olooney »

whack wrote: post #59

In the example given, the author says that the odds of a given page having a typo is less than 3/20 . Sure, but if we don't want a range, but an exact number ? That sounds like a more interesting challenge to me. Formal statement: - You have observed N events, with 0 occurrences of X - Someone wants to make a bet with you about the likelihood of X happening - Once you've quoted a number, your counter-party then has t…

https://en.m.wikipedia.org/wiki/Rule_of_succession

https://en.m.wikipedia.org/wiki/Sunrise_problem

Re: Estimating the chances of something that hasn’t happened yet

#65

Post by sonnyblarney »

mirimir wrote: post #4

There's a trivial corollary, which I remember from my first physics class. Never base anything on less than three measurements. Or maybe it wasn't even physics, but rather carpentry. The old "Measure twice, cut once." rule is iffy.

" Or maybe it wasn't even physics, but rather carpentry. "

That's the quote of the day.

My father was a carpenter, and every day I think software has more in common with carpentry than computer science.

Re: Estimating the chances of something that hasn’t happened yet

#66

Post by c3534l »

nitwit005 wrote: post #56
madrox wrote: post #14

What the author glosses over somewhat is the method of sampling. If you read the first 20 pages, find no typos, and use this rule to arrive at 15%, that could be way off. He's assuming the risk of typos are evenly distributed when there's a lot of reasons it may not be. For example, the first half of the book could've been more heavily proof-read than the latter half. It's not out of the question that editors get laz…

Doesn't the book example need to factor in the book length? If you read 20 pages, and it's 20 pages long, you should be 100% confident, but if you read 20 pages and it's 5000 pages long, your confidence should be near 0.

Logically not, since your sampling method is imperfect and will not detect all typos and, in fact, some typos might make perfect sense given the context and be inherently undetectable forever.

Re: Estimating the chances of something that hasn’t happened yet

#67

Post by madrox »

robluxus wrote: post #24
madrox wrote: post #14

What the author glosses over somewhat is the method of sampling. If you read the first 20 pages, find no typos, and use this rule to arrive at 15%, that could be way off. He's assuming the risk of typos are evenly distributed when there's a lot of reasons it may not be. For example, the first half of the book could've been more heavily proof-read than the latter half. It's not out of the question that editors get laz…

Could we apply the rule of three for the larger context here? Something along the lines of: "if I haven't seen a book with unequal typos distribution yet, what are the odds this current one will be an instance where editors got lazier in the second half?"

You could, though that wouldn't be a very powerful test. You could probably construct a more powerful test if you're going through the trouble of reading that many books.

Re: Estimating the chances of something that hasn’t happened yet

#68

Post by blt »

cperciva wrote: post #39
rivp931 wrote: post #37

Earlier quoted context omitted.

Going down this rabbit hole eventually leads you to nonparametric statistical tests, e.g. Mann-Whitney-U and so on.

Right. And those are also more powerful (and don't need any knowledge of the distribution of measurement errors). I mentioned the "three old and three new" test because it's simple , not because it's powerful.

more powerful in the statistical sense? I though non-parametric tests were usually less powerful than those where a certain distribution is assumed.

Re: Estimating the chances of something that hasn’t happened yet

#70

Post by ma2rten »

zawerf wrote: post #62
cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughly the same (or so I think - I don't really know and want a better workflow here). But if you do this test repeatedly, even if the code had identical performance, you'll get a false positive 5% of the time. And depending on the spread of the timings you might n…

It is not a good method to use in that case.
Post reply on HN