Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Estimating the chances of something that hasn’t happened yet

johndcook.com

Post reply on HN
Print view View original

71–80 of 155 posts

Re: Estimating the chances of something that hasn’t happened yet

#71

Post by oldgradstudent »

The rule of three requires quite a lot of assumptions about the nature of the phenomena.

Or as Taleb says it:

> Consider a turkey that is fed every day. Every single feeding will firm up the bird's belief that it is the general rule of life to be fed every day by friendly members of the human race "looking out for its best interests," as a politician would say.

> On the afternoon of the Wednesday before Thanksgiving, something unexpected will happen to the turkey. It will incur a revision of belief.

Re: Estimating the chances of something that hasn’t happened yet

#72

Post by hawkice »

neolefty wrote: post #57
jedberg wrote: post #34

Earlier quoted context omitted.

You could do the 'rule of 5' though and have a confidence of 99.3%, which is pretty close to 99.

If you're truly estimating, though, wouldn't you use a 50% confidence interval — which gives you the rule of 1 (chances of the thing being true are 1/n — if you've seen 20 pages without a typo, chances are 1/20 that there's a typo on a page, with 50% confidence)?

So, for a 50% confidence interval, you could look at the first word -- if it is a typo, boom, done. Otherwise, flip a coin.

This stinks. Estimating is about approximating an answer with low information -- it's about efficiency of using data, not _only_ doing better than guessing.

Re: Estimating the chances of something that hasn’t happened yet

#73

Post by adamc »

blt wrote: post #68
cperciva wrote: post #39

Earlier quoted context omitted.

Right. And those are also more powerful (and don't need any knowledge of the distribution of measurement errors). I mentioned the "three old and three new" test because it's simple , not because it's powerful.

more powerful in the statistical sense? I though non-parametric tests were usually less powerful than those where a certain distribution is assumed.

That depends on whether the assumed error distribution is accurate. Obviously, if the distribution is known, you will do better by incorporating it. But assuming Gaussian errors when the data doesn't match can lead to bad analyses.

Re: Estimating the chances of something that hasn’t happened yet

#74

Post by thedirt0115 »

gboudrias wrote: post #33
rspeer wrote: post #21

If the author is reading this: When someone provides you a pro-bono translation, by all means credit them, but do not let them host it on their own site. Frequently, they are siphoning your PageRank. They will eventually replace the translation with monetized content of their choice. A good translation takes work! If you got a translation for free, why should you believe it's good, or that it has no ulterior motive?…

Yeah this is weird. Also, if OP doesn't speak Italian themselves, how can they attest to the quality of the translation?

I have a couple L2’s that I mostly speak and read, virtually no writing - my grammar is poor and my vocabulary is limited. Because of this I would feel uncomfortable/embarrassed to translate any tech stuff I wrote to it. However, I could read someone else’s translation and know if it’s way off the mark. Also, I have plenty of friends that speak the language natively, so I could ask them to review. If they say it’s good, I’d vouch for the translation because I trust them (even if I couldn’t read it at all).

Re: Estimating the chances of something that hasn’t happened yet

#75

Post by gweinberg »

whack wrote: post #59

In the example given, the author says that the odds of a given page having a typo is less than 3/20 . Sure, but if we don't want a range, but an exact number ? That sounds like a more interesting challenge to me. Formal statement: - You have observed N events, with 0 occurrences of X - Someone wants to make a bet with you about the likelihood of X happening - Once you've quoted a number, your counter-party then has t…

I don't think you can do this. You have a reasonably good upper bound of the probability, but you don't have any justification for putting any lower bound other than zero on the probability.

In particular, if you're considering the probability of a catastrophic event, it's probably better to find some other rationale for estimating the probability than just saying 'it has never happened before'.

Re: Estimating the chances of something that hasn’t happened yet

#76

Post by luckyt »

cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

This is incorrect: there's no reason to expect that X and Y will each appear 3 times in 6 trials if their probabilities are equal. If all 3 measurements of X are smaller than all 3 measurements of Y, then you have X < Y with confidence 1 - 1/8 or 87.5% confidence. You'd need at least 5 measurements to be 95% confident.

Re: Estimating the chances of something that hasn’t happened yet

#77

Post by eximius »

I feel like I'm having a math stroke.

The posterior probability of p being less than 3/N for Beta(1, N+1) should be integral(Beta(1, N+1), 0, 3/N), right?

That trends toward zero, so I must be wrong, but I can't for the life of me remember why.

EDIT: Ah! I was accidentally using Beta instead of the PDF for Beta.

Re: Estimating the chances of something that hasn’t happened yet

#78

Post by mark-r »

dahart wrote: post #61
madrox wrote: post #14

What the author glosses over somewhat is the method of sampling. If you read the first 20 pages, find no typos, and use this rule to arrive at 15%, that could be way off. He's assuming the risk of typos are evenly distributed when there's a lot of reasons it may not be. For example, the first half of the book could've been more heavily proof-read than the latter half. It's not out of the question that editors get laz…

> It's understandable to not mention this I feel like the very first word, "Estimating" made what you're talking about pretty clear. It's extremely common for estimates to make simplifying assumptions that don't necessarily hold if you care about accuracy or specific situations. He also called the rule of three "quick and dirty". > If you were to randomly read 20 pages in a book and find no typos, 15% probability mak…

Interesting that you would worry about repeats. Most people would interpret "randomly read 20 pages" as "choose 20 from n" not "take 20 rand(n)".

Re: Estimating the chances of something that hasn’t happened yet

#80

Post by saagarjha »

luckyt wrote: post #76
cperciva wrote: post #28

This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…

This is incorrect: there's no reason to expect that X and Y will each appear 3 times in 6 trials if their probabilities are equal. If all 3 measurements of X are smaller than all 3 measurements of Y, then you have X < Y with confidence 1 - 1/8 or 87.5% confidence. You'd need at least 5 measurements to be 95% confident.

Well, you’re the one measuring each three times. So this should always be the case?
Post reply on HN