Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

41–50 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#41

Post by BenoitEssiambre »

tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

That interpretation makes the test basically useless because we know a priori that any two variable for things within each others light cone affect each other at least a little. More practically, since the test tells you nothing about size of effect, it will pick up on the tiniest of bias in your experimental procedure and always reject the null if you have enough data.

From the author of the article: "The general point reminds me of my dictum that statistical hypothesis testing works the opposite way that people think it does. The usual thinking is that if a hyp test rejects, you’ve learned something, but if the test does not reject, you can’t say anything. I’d say it’s the opposite: if the test rejects, you haven’t learned anything—after all, we know ahead of time that just about all null hypotheses of interest are false—but if the test doesn’t reject, you’ve learned the useful fact that you don’t have enough data in your analysis to distinguish from pure noise."

(https://statmodeling.stat.columbia.edu/2019/08/18/i-feel-lik...)

Re: It’s not just p=0.048 vs. p=0.052

#42

Post by lonelappde »

mehrdadn wrote: post #11
pacbard wrote: post #8

Earlier quoted context omitted.

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as…

I still don't get it. What p-value observations would not be "reasonable" here? It seems to me he's saying that anything between 0 and 1 is completely reasonable, which is a completely pointless statement as I see it.

His point is that the whole point of the field of statistics is that cherry picking results or getting lucky once is not proof of anything, unlike nonstatistical mathematics where one example is enough to justify a claim.

Re: It’s not just p=0.048 vs. p=0.052

#43

Post by lonelappde »

mehrdadn wrote: post #17
krona wrote: post #16

Earlier quoted context omitted.

Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.

So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"? At which point, what is even the point of this statement?

The point is that you need to replicate experiments and not cherry pick the one with the highest p-value.

Re: It’s not just p=0.048 vs. p=0.052

#44

Post by twelfthnight »

I think the problem with p-values is that it trains us to think about uncertainty without nuance. It hides the inherent trade off between the cost of taking on risk and the cost of reducing uncertainty, since it sets the threshold at p=0.05. Taken to the extreme, with a large enough sample we can nearly always find significant differences between populations, the difference will just be very small and n size will be enormous.

Recently I worked with a client to interpret results from an A/B test where A performed better than B with 85% confidence (based on credible intervals, accounting for multiple comparisons). We therefore recommended A. In a group phone call, the client told her colleagues that our company doesn't know what we're talking about because 85% confidence of a difference isn't statistically significant (i.e. isn't 95% confident). We lost their business.

This was a shame because gathering the data for the experiment was expensive and the downside of making the wrong choice was low. It is often the case that taking on more risk makes more sense than hitting diminishing returns on shrinking p-values with extra sample.

Re: It’s not just p=0.048 vs. p=0.052

#45

Post by lonelappde »

mruts wrote: post #31
JustFinishedBSG wrote: post #10

Earlier quoted context omitted.

.

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

mruts' (and most people who studied a little statistics) misunderstanding of the basic definition is exactly what author and others have been campaigning against for years.

Re: It’s not just p=0.048 vs. p=0.052

#46

Post by sweeneyrod »

mruts wrote: post #31
JustFinishedBSG wrote: post #10

Earlier quoted context omitted.

.

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

You don't understand what a p-value is.

> a P In other words, P(H0 | X) where H0 is the null hypothesis being true and X is the data observed. But that is not what a p-value is, they actually represent P(X | H0).

Re: It’s not just p=0.048 vs. p=0.052

#47

Post by BenoitEssiambre »

BenoitEssiambre wrote: post #41
tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

That interpretation makes the test basically useless because we know a priori that any two variable for things within each others light cone affect each other at least a little. More practically, since the test tells you nothing about size of effect, it will pick up on the tiniest of bias in your experimental procedure and always reject the null if you have enough data. From the author of the article: "The general po…

To add to this, the problem with t-tests in not the threshold. It is that the hypothesis you are rejecting (effect is exactly 0.0000000...) is infinitesimaly small. You've rejected basically nothing of your hypothesis space.

Your null should have a width. You should always be rejecting "effect is greater than some margin" which you should have to argue is greater than any bias you might expect in your experiment. There are always at least tiny biases.

Re: It’s not just p=0.048 vs. p=0.052

#48

Post by swsieber »

rom1v wrote: post #33

Related, about p-values: > Here's the problem in a nutshell: If you run 1000 experiments over the course of your career, and you get a significant effect (p > […] However, this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, a…

For about a year or so now Ive been wanting to make a game about science. It'd basically be a research and discovery simulator, and there would be a free-play mode. Some of the knobs would be # of required replication, required p-value and how good people are at generating hypothesis.

I think it'd be eye opening.

Re: It’s not just p=0.048 vs. p=0.052

#49

Post by Symmetry »

Put a Number on It! did a piece a while ago going through some psychology pieces that were part of a replication effort. They found that half failed to replicate but that people in a betting market could often tell which ones were going to replicate or not. The author also did a blind test himself and was also able to guess which ones would replicate. He laid out several rules of thumb, most significantly to the article Jacob’s Rule of Anti-Significance: A result with a p-value just above 0.05 could well be true. A result with a p-value just below 0.05 is almost certainly false.

More importantly, p=0.06 means that the researchers are honest. They could have easily p-hacked the results below 0.05 but chose not to. The opposite is true when p=0.049.

https://putanumonit.com/2018/09/07/the-scent-of-bad-psycholo...

Re: It’s not just p=0.048 vs. p=0.052

#50

Post by penagwin »

mehrdadn wrote: post #17
krona wrote: post #16

Earlier quoted context omitted.

Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.

So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"? At which point, what is even the point of this statement?

My understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)
Post reply on HN