Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

71–80 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#71

Post by swsieber »

marcosdumay wrote: post #57
swsieber wrote: post #48

Earlier quoted context omitted.

For about a year or so now Ive been wanting to make a game about science. It'd basically be a research and discovery simulator, and there would be a free-play mode. Some of the knobs would be # of required replication, required p-value and how good people are at generating hypothesis. I think it'd be eye opening.

There is a "people will mostly replicate/extend articles about X, and ignore articles about Y" (groupthink) effect that I imagine is also very relevant.

Oooh, that's a great one to add to the list!

Re: It’s not just p=0.048 vs. p=0.052

#72

Post by madhadron »

mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

Say you run an experiment. You get some data. From that data you calculate a p-value. The data has randomness, so the p-value is a random variable and has a distribution. By chance you can get results that look significant. By chance you can get results that look insignificant.

Re: It’s not just p=0.048 vs. p=0.052

#73

Post by rom1v »

kazinator wrote: post #52
rom1v wrote: post #33

Related, about p-values: > Here's the problem in a nutshell: If you run 1000 experiments over the course of your career, and you get a significant effect (p > […] However, this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, a…

That's not what the p-value means. It means that if you run 1000 of the experiments in a universe in which the hypothesis is false, around 50 of them will confirm the hypothesis anyway. If the hypothesis is true, then there are no false positives; all positives confirm the hypothesis. In a universe in which the hypothesis is true, there can only be false negatives. "False positive" means that the effect or condition…

> The false positive rate (Type I error rate) as defined by NHST is the probability that you will falsely reject the null hypothesis when the null hypothesis is true. In other words, if you reject the null hypothesis when p This is just a language issue: a false positive of the rejection of the null hypothesis.

Re: It’s not just p=0.048 vs. p=0.052

#74

Post by kgwgk »

penagwin wrote: post #50
mehrdadn wrote: post #17

Earlier quoted context omitted.

So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"? At which point, what is even the point of this statement?

My understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)

If the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy).

The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true).

I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases where it is statistically significant.”

Re: It’s not just p=0.048 vs. p=0.052

#75

Post by kgwgk »

pacbard wrote: post #8
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as…

> He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution.

That is the frequentist point of view!

Re: It’s not just p=0.048 vs. p=0.052

#76

Post by penagwin »

kgwgk wrote: post #74
penagwin wrote: post #50

Earlier quoted context omitted.

My understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)

If the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy). The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true). I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases w…

Sorry I was definitely kinda vague - I meant when the alternative hypothesis is true the distribution should be skewed towards 0.

Re: It’s not just p=0.048 vs. p=0.052

#77

Post by kgwgk »

mehrdadn wrote: post #20
pfortuny wrote: post #9

Earlier quoted context omitted.

What he says is (I gather) worse: those events are only separated be 1.1std deviations, which is little.

I think that's what he's saying too, but what is that supposed to show? Is he arguing against some claim that every interval of 1 standard deviation is equally significant? Did anybody make this claim? So far as I know, nobody considers (say) a 6-sigma effect to be 6 times stronger than a 1-sigma effect...

I don’t get it either. It’s a trivial consequence of having a threshold: if we say two cities are “far” when they are at least 1000 miles away then Washington D.C. is not far from Jacksonville while Boston is far from Jacksonville, even though Boston is not far from Washington.

Re: It’s not just p=0.048 vs. p=0.052

#80

Post by hadsed »

UncleMeat wrote: post #70
Symmetry wrote: post #49

Put a Number on It! did a piece a while ago going through some psychology pieces that were part of a replication effort. They found that half failed to replicate but that people in a betting market could often tell which ones were going to replicate or not. The author also did a blind test himself and was also able to guess which ones would replicate. He laid out several rules of thumb, most significantly to the arti…

This is true in more than just psych. A strategy for systems papers is to select the system that is second on the graphs. The author's results are not trustworthy, but the second place system was usable and ran reasonably well in the hands of somebody other than the original author.

For ML practitioners out there, this is a great method for a field also in a replication crisis.
Post reply on HN