It just means A is slightly worse than B. Or equal to B. Or much worse than B, but that is quite unlikely (way less than 5%).
There’s a 5% chance that these results are total bullshit
11–20 of 27 posts
Re: There’s a 5% chance that these results are total bullshit
#12> Stop saying: “We’ve reached 95% statistical significance.” > And start saying: “There’s a 5% chance that these results are total bullshit.” Argh, no, no, no and no! 95% significance is NOT 95% probability! When you select a confidence level of a 95%, the probability that your results are nonsense is ZERO or ONE. There is no probability statement associated to it. Just because something is unknown does not mean that…
I'm not really sure what you're trying to say.
> Rather, 95% statistical significance means, we got this number from a procedure that 95% of the time produces the right thing, but we have no idea whether this particular number we got is correct or not.
I.e. We got this number from a procedure and there's a 5% chance it didn't produce the right thing.
Re: There’s a 5% chance that these results are total bullshit
#13No. In Frequentist thinking; p=0.05 means that if there was in reality no difference in your A and B and you repeated the experiment many times, 5% of the observed differences would be equal to or greater than the difference you just measured. No probabilistic statement about the results being correct or incorrect can be made from a Null-Hypothesis significance test.
+1 Well said.
This is correct and the original post is wrong.
Re: There’s a 5% chance that these results are total bullshit
#14> Stop saying: “We’ve reached 95% statistical significance.” > And start saying: “There’s a 5% chance that these results are total bullshit.” Argh, no, no, no and no! 95% significance is NOT 95% probability! When you select a confidence level of a 95%, the probability that your results are nonsense is ZERO or ONE. There is no probability statement associated to it. Just because something is unknown does not mean that…
> 95% statistical significance means, we got this number from a procedure that 95% of the time produces the right thing
The original post is incorrect about the probabilistic interpretation of the 95% confidence interface, but this interpretation is also wrong.
In classical statistics, p<0.05 means that, if there is no difference in our sample populations (i.e. the null hypothesis), then the probability of observing a difference at least this extreme is less than 0.05.
Re: There’s a 5% chance that these results are total bullshit
#15> Stop saying: “We’ve reached 95% statistical significance.” > And start saying: “There’s a 5% chance that these results are total bullshit.” Argh, no, no, no and no! 95% significance is NOT 95% probability! When you select a confidence level of a 95%, the probability that your results are nonsense is ZERO or ONE. There is no probability statement associated to it. Just because something is unknown does not mean that…
I'm not really sure what you're trying to say. > Rather, 95% statistical significance means, we got this number from a procedure that 95% of the time produces the right thing, but we have no idea whether this particular number we got is correct or not. I.e. We got this number from a procedure and there's a 5% chance it didn't produce the right thing.
Nope. It's "If we did this infinitely more times, there's a 5% of those samples wouldn't have significant results". It's a subtle but important distinction.
Though I'm surprised that his advice wasn't "Report confidence intervals at least". There's much more meaningful information in a point estimate and confidence interval than "p < 0.05"
Re: There’s a 5% chance that these results are total bullshit
#16I'm not a statistician, but lately I've been wondering: When we're A/B testing code, the code is already written. If there's a 5%, or even 15% chance of it being bullshit, who cares? The effort is usually exactly the same if I switch or not. It's my understanding that 95%, 99%, etc, were established for things that require extra change. We don't want to spend extra time developing and marketing a new drug if it isn't…
You would be better served by a Bayesian approach to A/B testing, and measure directly the probability of A converting more than B. http://www.evanmiller.org/bayesian-ab-testing.html You can then apply some sort of decision rule such as the difference being above a certain threshold.
Re: There’s a 5% chance that these results are total bullshit
#17Is it just me or this sentence makes no mathematical sense at all?
"If you’re running squeaky clean A/B tests at 95% statistical significance and you run 20 tests this year, odds are one of the results you report (and act on) is going to be straight up wrong."
Re: There’s a 5% chance that these results are total bullshit
#18"We’re taking techniques that were designed for static sample sizes and applying them to continuous datasets" - Wait, seriously? Do A/B testers not use the very well developed techniques that exist for time-series data?
Re: There’s a 5% chance that these results are total bullshit
#19No. In Frequentist thinking; p=0.05 means that if there was in reality no difference in your A and B and you repeated the experiment many times, 5% of the observed differences would be equal to or greater than the difference you just measured. No probabilistic statement about the results being correct or incorrect can be made from a Null-Hypothesis significance test.
What is the difference between what you wrote and saying "95% of the observed differences would be equal to or less than..."
The p-value (in my understanding) makes a prediction about what would occur if the experiment was repeated infinity times.
Re: There’s a 5% chance that these results are total bullshit
#20You have to wonder: what else from their junior year in college did mr. Avshalomov get completely wrong?
How many of the recent YC graduates fail at basic numeracy? Does node.js mean you don't have to understand data structures and algorithms to successfully "preneur" too?
I mean, in finance this doesn't do. Or in consulting. So there's adverse selection to worry too.