I was just on a flight, hanging with a guy who's PhD was this topic. Josh is that you?? What are the odds.
Estimating the chances of something that hasn’t happened yet
101–110 of 155 posts
Re: Estimating the chances of something that hasn’t happened yet
#102Earlier quoted context omitted.
You could do the 'rule of 5' though and have a confidence of 99.3%, which is pretty close to 99.
If you're truly estimating, though, wouldn't you use a 50% confidence interval — which gives you the rule of 1 (chances of the thing being true are 1/n — if you've seen 20 pages without a typo, chances are 1/20 that there's a typo on a page, with 50% confidence)?
Yes. That's a very reasonable choice.
Re: Estimating the chances of something that hasn’t happened yet
#103This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…
> If the all three measurements of X are smaller than all three measurements of Y, you have X On a multitasking system, you should use the fastest benchmark run (assuming you're running the same code on the same data in all cases, and if you're not then you're not really benchmarking). This will be the one that is least influenced by any other processing going on.
Re: Estimating the chances of something that hasn’t happened yet
#104Earlier quoted context omitted.
Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughly the same (or so I think - I don't really know and want a better workflow here). But if you do this test repeatedly, even if the code had identical performance, you'll get a false positive 5% of the time. And depending on the spread of the timings you might n…
sqlite famously squeezed out a ~40% performance improvement (I think from v3 to v4?) by just combining tons of micro-optimizations of this kind where it wasn't obvious if each change even made an improvement. They measured the performance with cachegrind in order to identify very small improvements that get lost in the normal measurement noise.
That’s a really awesome idea. Also, maybe a few of the performance boosts in the final combination are mutualistic, but if measured independently would not make a difference.
Re: Estimating the chances of something that hasn’t happened yet
#105Although interesting, the article doesn’t relate to predicting things that haven’t happened yet, just things that aren’t known yet.
When predicting things that haven’t happened yet, a publicized “certain” prediction will inevitably influence the actual probability in an unpredictable way.
Re: Estimating the chances of something that hasn’t happened yet
#106Earlier quoted context omitted.
Interesting that you would worry about repeats. Most people would interpret "randomly read 20 pages" as "choose 20 from n" not "take 20 rand(n)".
You're right, and mathematicians and computer scientists aren't most people. We are being pedantic / precise here. If someone wants to caution that people need to be really careful about estimates because there are some assumptions involved, then it's fair to hold them to their own standard, isn't it? If typos are uniform, then the suggestion to read random pages doesn't help. And whether or not people interpret it o…
As a programmer it still never would have occurred to me to interpret it that way, but maybe I'm not typical. If your random library doesn't have a choose function, it's easy enough to just shuffle a list and take the first 20.
Re: Estimating the chances of something that hasn’t happened yet
#107This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…
Man, I wish I understood frequentist statistics to know if your reasoning makes sense. Bayesianly, if O = "the XXXYYY ordering", and F = "algo B is faster than algo" A, then
P(F|O) = P(O|F) * P(F) / P(O)
Re: Estimating the chances of something that hasn’t happened yet
#108If the author is reading this: When someone provides you a pro-bono translation, by all means credit them, but do not let them host it on their own site. Frequently, they are siphoning your PageRank. They will eventually replace the translation with monetized content of their choice. A good translation takes work! If you got a translation for free, why should you believe it's good, or that it has no ulterior motive?…
The obsession with SEO is what leads to things like this in the first place.
Maybe the sooner people stop paying attention to it, the sooner search engines will learn to find better metrics.
Re: Estimating the chances of something that hasn’t happened yet
#109I'd be interested to know how to estimate things that are very rare or don't have a normal distribution.
For instance, let's say I have a bold plan to protect us from meteor strikes that will cost $100B. How would a person make a decision about whether that's a good trade-off or not? And how would a mathematician help them make that decision?
How would it change for more complex cases, like a shield to prevent nuclear ICBM warfare which has never happened, but we all are worried about?
Re: Estimating the chances of something that hasn’t happened yet
#110This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…
I wish the people constantly complaining how n=20,000 is "far too small a sample size to call this science" (for every empirical study) would take not. Effect size matters!