Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

P values are not as reliable as many scientists assume (2014)

nature.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: P values are not as reliable as many scientists assume (2014)

#41

Post by tel »

haddr wrote: post #36

It is not that P-values are now bad by definition. It's only that they are many times wrongly intepreted. Putting too much confidence in P-values only might result in some wrong conclusions. And this is what some meta analyses discover. Many scientists try hard only to reach the "golden" <0.05 in order to claim discovery and publish it. This is why there is so many papers that misteriously cluster around 0.05...

There's also the systemic effect of prioritizing particular p values in that negative results are omitted leading to replication bias across the community.

Re: P values are not as reliable as many scientists assume (2014)

#42

Post by kgwgk »

jsprogrammer wrote: post #27
BorisVSchmid wrote: post #3

From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…

If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible. That only 36% were points to deep, fundamental errors.

Let me join the club of people claiming that you don't understand p-values.

It's not clear if you are talking about the rate of reproduction for a subset of the possible experimental outcomes (those rejecting the null at the alpha=5% level) or for the whole set.

When the null hypothesis is true (remember that there are fields where this is the norm, v.g. ESP), you would only reproduce (reject again) 5% of the rejections.

Of course you would reproduce (non-reject for the second time) 95% of the non-rejections. The global reproduction rate would be 0.95 x 0.95+0.05 x 0.5=0.905 (90.5% doesn't look ~95% either).

When the null hypothesis is not true, the probability of reproducing (in either sense) the result of a test depends on the effect size.

If the effect is huge, the test will be rejected with probability ~100% and the result will be reproduced with probability ~100%.

Or maybe you mean by reproducing "getting a lower p-value" in the the second trial? If the null is true, the probability of getting p-value2<p-value1 is precisely p-value1. If the null is not true, it will depend on the effect size. If you assume the effect size is the observed one, you expect p-value2 to be smaller than p-value1 with probability 50%.

Re: P values are not as reliable as many scientists assume (2014)

#43

Post by Dylan16807 »

jsprogrammer wrote: post #40
Dylan16807 wrote: post #38

Earlier quoted context omitted.

On a certain philosophical level you can never be absolutely sure of anything, and p-values are meaningless. On a practical level, p-values are the chance that a correlation is reported where 'reality' does not have a correlation. This is not the same number as the chance that the result agrees with 'reality'. You can reject the concept of objectivity, but you cannot reject that logic. So I have explained the alterna…

On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the mathematics, only that your reality has diverged from what you originally intended/believed. What is the meaning of the number that people call 'p-value' when it is not calculated on a well-formed experiment? I'm not sure if there is a general formula, but you ma…

You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say.

p does not tell you how likely a result is to be true.

Re: P values are not as reliable as many scientists assume (2014)

#44

Post by jsprogrammer »

kgwgk wrote: post #42
jsprogrammer wrote: post #27

Earlier quoted context omitted.

If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible. That only 36% were points to deep, fundamental errors.

Let me join the club of people claiming that you don't understand p-values. It's not clear if you are talking about the rate of reproduction for a subset of the possible experimental outcomes (those rejecting the null at the alpha=5% level) or for the whole set. When the null hypothesis is true (remember that there are fields where this is the norm, v.g. ESP), you would only reproduce (reject again) 5% of the rejecti…

>If the null is true, the probability of getting p-value2Precisely. P-value is only defined when the null hypothesis is true.

Apparently everyone else is overlooking this fact.

Re: P values are not as reliable as many scientists assume (2014)

#45

Post by kgwgk »

jsprogrammer wrote: post #44
kgwgk wrote: post #42

Earlier quoted context omitted.

Let me join the club of people claiming that you don't understand p-values. It's not clear if you are talking about the rate of reproduction for a subset of the possible experimental outcomes (those rejecting the null at the alpha=5% level) or for the whole set. When the null hypothesis is true (remember that there are fields where this is the norm, v.g. ESP), you would only reproduce (reject again) 5% of the rejecti…

>If the null is true, the probability of getting p-value2 Precisely. P-value is only defined when the null hypothesis is true. Apparently everyone else is overlooking this fact.

I don't see how does it contradict anything that I (and everyone else) wrote.

You seem to agree that when the null is true and the original result was p-value=0.05, the probability of reproducing the result (getting p-valueThis seems incompatible with your original claim: "If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible."

Could you explain exactly what do the following mean:

p-values were accurate (that the null is true?)

p-values averaged around 0.05 (that you're taking the subset of outcomes with p-value 0.05?)

~95% of results should be reproducible (that if you take the previous subset you will get p-value<0.05 always in 95% of them? or exactly 95% of the time in all of them?)

Re: P values are not as reliable as many scientists assume (2014)

#46

Post by jsprogrammer »

kgwgk wrote: post #45
jsprogrammer wrote: post #44

Earlier quoted context omitted.

>If the null is true, the probability of getting p-value2 Precisely. P-value is only defined when the null hypothesis is true. Apparently everyone else is overlooking this fact.

I don't see how does it contradict anything that I (and everyone else) wrote. You seem to agree that when the null is true and the original result was p-value=0.05, the probability of reproducing the result (getting p-value This seems incompatible with your original claim: "If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible." Could you explain exactly what do the following…

95% of the repeated observations that you make (in the same manner as the observations used to calculate a valid p-value of 0.05) will be consistent with the relevant null hypothesis.

What other meaning could there be? The result of an experiment is not a p-value, but a series of observations. Those are what need to be compared.

Re: P values are not as reliable as many scientists assume (2014)

#47

Post by jsprogrammer »

Dylan16807 wrote: post #43
jsprogrammer wrote: post #40

Earlier quoted context omitted.

On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the mathematics, only that your reality has diverged from what you originally intended/believed. What is the meaning of the number that people call 'p-value' when it is not calculated on a well-formed experiment? I'm not sure if there is a general formula, but you ma…

You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.

A well formed experiment tests only a null hypothesis.

p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate).

Wikipedia has a decent treatment that might help: https://en.wikipedia.org/wiki/P-value#Definition_and_interpr...

Re: P values are not as reliable as many scientists assume (2014)

#48

Post by kgwgk »

jsprogrammer wrote: post #46
kgwgk wrote: post #45

Earlier quoted context omitted.

I don't see how does it contradict anything that I (and everyone else) wrote. You seem to agree that when the null is true and the original result was p-value=0.05, the probability of reproducing the result (getting p-value This seems incompatible with your original claim: "If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible." Could you explain exactly what do the following…

95% of the repeated observations that you make (in the same manner as the observations used to calculate a valid p-value of 0.05) will be consistent with the relevant null hypothesis. What other meaning could there be? The result of an experiment is not a p-value, but a series of observations. Those are what need to be compared.

I guess the bit "results should be reproducible" made us think that you were talking about reproducing the previous results (i.e. if the null hypothesis was rejected in the first trial, obtaining again a rejection if the trial was repeated).

If I understand your point, you're saying: "If the null hypothesis is true then with 95% probability it won't be rejected. And, independently of the result of the first trial, if we do a second trial and the null hypothesis is true then with probability 95% it won't be rejected".

Which seems correct, but you might be overlooking the fact that it's not very interesting and unrelated to the discussion.

Re: P values are not as reliable as many scientists assume (2014)

#49

Post by Dylan16807 »

jsprogrammer wrote: post #47
Dylan16807 wrote: post #43

Earlier quoted context omitted.

You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.

A well formed experiment tests only a null hypothesis. p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate). Wikipedia has a decent treatment that might help: https://en.wikipedia.org…

But the importance of a p-value is showing when it's not the null hypothesis.

The only time you get 95% reproduction is a result that says the null hypothesis is true.

You're entirely right about that specific case.

But this only happens when nothing correlates. (And almost no science has been done, because most things in fact don't correlate.)

A result that disagrees with the null hypothesis at .05 does not imply any particular chance of another result that also disagrees with the null hypothesis at .05

If there is no correlation, then replication will happen 5% of the time. If there is correlation, it will be somewhere over 5%, but no particular value.

When people talk about reproduction, they talk about that chance. It will only be 95% by coincidence.

Re: P values are not as reliable as many scientists assume (2014)

#50

Post by Retra »

unabst wrote: post #22
geofft wrote: post #13

Earlier quoted context omitted.

How about the assumption that the fundamental constants of the universe are not slowly changing day-to-day?

A better word for that would be "premise" or "axiom". Premises and axioms are objective exceptions with objective merits. Assumptions are too personal because they infer belief which is purely subjective. Nature doesn't care about what anyone believes and science should never be a democracy. Assumptions also imply some independent existential entity as valid and are self-validating, whereas premises and axioms are hi…

You're making a distinction that I've never heard anyone make before, and I don't think you're making a convincing argument for it now.
Post reply on HN