121–130 of 155 posts
https://en.m.wikipedia.org/wiki/Sunrise_problem
A practical use of a Stilton to this is included in reddits "best" sorting of comments, leaving from room for doubt with low sample sizes of votes.
https://redditblog.com/2009/10/15/reddits-new-comment-sortin...
I've seen a very similar problem to this referred to as "black swan events"[1]. The whole point is you can't actually compute it. You see the period, it's there. What he's doing here isn't science, it's a guess. As others have pointed it, it's rather rare that events happen with perfectly distributed probability (probably the opposite). For instance, the chance of being in an accident is much higher if an accident ju…
Another approach would be this: Future is impossible to research, as you can only research entities that exist or phenomena that is happening. Future entities and future phenomena do not exist yet ( by definition ) so you cannot research them. -> no science of future.
What we are talking here is scientific prediction. "Scientific" is just another word for good and only means something when compared to another method of prediction that can be shown to be worse.
So we very much agree. I just wanted to write this out because the Future studies crowd has bothered me for some time now.
So, according to this rule, if I wait for 10 minutes for a bus to come and none does, and then I wait for another 10 minutes for an alien invasion and none happens, the two have the same upper bound on their probability? Or are we going to start talking about priors, on buses and alien invasions, in which case the rule of three is not really useful? If I want to know how likely a specific book is to have typos, can't…
This is the scenario where you know nothing else other than you waited 10 minutes for this event and it didn't happen. Sure, if you have more data, then you can get much tighter bounds for your estimate.
And if something has really never happened before, like my alien invasion example, what have we learned by applying the rule of three?
Honestly- perform the experiment yourself. Wait for X time, then calculate 3/X. Do you now have an upper bound on the probability that an alien invasion will happen?
This is exactly this approach is worthless for rare events which by definition have very skewed distributions. In that case, probably is overestimated a lot.
On the other hand, I'd the is a rare but systematic error, the error probably will likely be grossly underestimated.
Thought experiment: suppose you're writing a long string of digits that consists of 1 followed by a large number of zeroes (say 99 for simplicity) followed by (say 10000) uniformly distributed digits.
Your writing system has an issue that changes half of 5 digits into 6. What probability of error will be estimated by this dumb method after 100th digit?
Correct bayesian approach updates the prior based on input variability keeping the error estimates high when input has low variability etc. (This can be with variance or another method.) The even better method tries to estimate the shape of input distribution.
In other words, your result would be a difference of likelihood ratio of both input and output prior distributions (estimated to date - since no errors the ratio would be 1) minus likelihood ratio of posterior distributions.
Earlier quoted context omitted.
Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughly the same (or so I think - I don't really know and want a better workflow here). But if you do this test repeatedly, even if the code had identical performance, you'll get a false positive 5% of the time. And depending on the spread of the timings you might n…
sqlite famously squeezed out a ~40% performance improvement (I think from v3 to v4?) by just combining tons of micro-optimizations of this kind where it wasn't obvious if each change even made an improvement. They measured the performance with cachegrind in order to identify very small improvements that get lost in the normal measurement noise.
Earlier quoted context omitted.
Going down this rabbit hole eventually leads you to nonparametric statistical tests, e.g. Mann-Whitney-U and so on.
My personal favourite quick-and-dirty trick: A quick way to estimate any distribution's median is to draw 5 random samples. There's a >90% chance that the median is between the biggest and the smallest value.
In the example given, the author says that the odds of a given page having a typo is less than 3/20 . Sure, but if we don't want a range, but an exact number ? That sounds like a more interesting challenge to me. Formal statement: - You have observed N events, with 0 occurrences of X - Someone wants to make a bet with you about the likelihood of X happening - Once you've quoted a number, your counter-party then has t…
https://en.m.wikipedia.org/wiki/Rule_of_succession https://en.m.wikipedia.org/wiki/Sunrise_problem
Instead, you get to estimate the dependency between each observation as in advanced variants of Bayes chain rule. Ultimately, some place of the estimator will contain an assumption giving only bounded optimality.
Earlier quoted context omitted.
Why would this be true? What if we, by luck, had 14 days of good weather at the end of monsoon season? Your observation seems to be a function of the probabilities changing as the calendar date advances and not a historical run of good weather in the past.
Today's weather affect's tomorrows. They are not independent variables.
The rule of three requires quite a lot of assumptions about the nature of the phenomena. Or as Taleb says it: > Consider a turkey that is fed every day. Every single feeding will firm up the bird's belief that it is the general rule of life to be fed every day by friendly members of the human race "looking out for its best interests," as a politician would say. > On the afternoon of the Wednesday before Thanksgiving,…
It certainly does not require unwarranted assumptions, the proposed approach is consistent with the well-known turkey scenario. From the experience of feeding a turkey can infer that being slaughtered is a rare event, and that the likelihood it happening exactly tomorrow (without having access to a calendar) is not necessarily 0, but is below a certain rate - and it's definitely not likely to happen three times in th…
This is why a good experimental design will observe a measurement with some expected variability.
An even better trick is the sleeping beauty problem. To solve it you need external information.
This reminds me of a different "rule of 3": If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach --…
I wish the people constantly complaining how n=20,000 is "far too small a sample size to call this science" (for every empirical study) would take not. Effect size matters!