Live data from Hacker News

P values are not as reliable as many scientists assume (2014)

nature.com

21–30 of 90 posts

Re: P values are not as reliable as many scientists assume (2014)

#21
post #17
post #6

Earlier quoted context omitted.

[deleted]

I agree that science strives to remove bias from its body of knowledge, but it's absolutely unavoidable for humans to paint their subjective experiences onto it. Humans have to bring their own preconceptions to any scientific experiment, even the medium by which we convey the knowledge is bathed in assumptions about what those words mean. Every facet of human knowledge is premised on how humans experience the univers…

> Every facet of human knowledge is premised on how humans experience the universe

Absolutely. But that is why this is where we start. Before science, we had no way to invalidate illusions and validate what was real because assumption on their own are neither. They are naked intuitions pending validation. For the longest time we were unable to validate them and we ended up with the mess we had before science. Basically, no one would ever have made it to Mars.

But with scientific validation knowledge becomes more than just an assumption or an intuition based on an experience, or a theory we came up with that we find ingenious because, well, we came up with it. By overcoming our assumptions we achieve objectivity, universality, and factuality. We discover knowledge that has rigid practical persistence. In this process something transcends from our subjective personal ideas to becoming objective impersonal facts. There is no self in science. And it is from this arduous feat that technology is born. There is nothing in this monitor or the components of this phone that are based on assumptions. These devices are selfless.

"Assumption" is as evil a word as "metaphysics" and "subjective" in science. Yet, there are still people who use the word as a synonym for axiom. This is simply bad word-choice. The correct term here would be "premise" and you used it yourself. Theories can have premises, but not assumptions. Are the premises assumed? No. They are granted.

Since this is HN, here is an analogy to software. A program that assumes certain behavior code or of external APIs will be rigged with bugs. Every aspect of it's execution must be tested, and the assumptions of the programmer must be eliminated by production. Of course, being human, we start with assumptions - such as "this would be the perfect library for this project". The "assumptions" that we being with however, eventually manifest themselves into "premises". And in software, these are the dependencies of a program. It is only natural for software to be dependent on other software. What is unnatural and anti-software would be to make assumptions about other software especially within its own execution.

The path from the assumptions of subjective raw experience to the subjective consumption of reliable technology is paved with the work of competent scientists (and analogously, by competent programmers).

> Given your definition, I don't think anything we know would be considered a fact.

If fact is to mean truth, then sure. But there is an abundance of statements that have been backed by evidence. And all these statements are truer than most. Measurably truthier, rather, and that is what counts because that leave room for progress. This is a better definition of "fact".

Re: P values are not as reliable as many scientists assume (2014)

#22
post #13
post #6

Earlier quoted context omitted.

[deleted]

How about the assumption that the fundamental constants of the universe are not slowly changing day-to-day?

A better word for that would be "premise" or "axiom". Premises and axioms are objective exceptions with objective merits. Assumptions are too personal because they infer belief which is purely subjective. Nature doesn't care about what anyone believes and science should never be a democracy.

Assumptions also imply some independent existential entity as valid and are self-validating, whereas premises and axioms are highly self-deprecating. Hence, assumptions are dogmatic self-fulfilling prophesies that are an end unto itself, whereas premises are unfortunate unavoidable constraints as a means to an end. They are what couldn't be eliminated despite towering doubt and cynicism. Premises lead to science, axioms to logic, and assumptions to religion (and the like -- not saying it is good or bad).

Re: P values are not as reliable as many scientists assume (2014)

#23
post #6

Earlier quoted context omitted.

[deleted]

First off, there's no need for all-caps. This isn't 4chan. > The fact that assumptions are considered as some unavoidable, forgivable, intricate part of science is part of what fuels anti-science and politics. No one here, as far as I can tell, is saying, 'oh well, science is full of assumptions therefore science is invalid.' The problem is not with science in general being valid or invalid, but rather with the sorts…

> 'oh well, science is full of assumptions therefore science is invalid.'

No, they are saying "science is based on assumption, but in this case 'many scientists' were making the wrong assumptions." The title should have said "scientists find errors in p-values as premises." Assumptions are avoided, not depended upon. Bad assumptions would certainly lead to irreplicable experiments also.

The distinction you make between science and its experiments is not a common distinction. If science = experiments, then you totally agree that this kind of science is bad science. Which was precisely my point.

Re: P values are not as reliable as many scientists assume (2014)

#24

From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…

A large amount of published results that are wrong is definitely something science can live with: we have to trade off Type I against Type II. But we should value accuracy: if we report something as being very, very unlikely if chance was at play, and it turns out that in fact (1) it'd be very likely even if the null hypothesis holds and (2) in fact even if P(D|H0) is low, P(H0|D) might be high... then what's the point in writing up all those fancy statistical analyses anyway? At that point significance testing becomes more of religious ritual and should either be discarded entirely or be amended.

Re: P values are not as reliable as many scientists assume (2014)

#25
Isn’t a main problem with p-values that you don’t know whether significance (low p-value) is a result of big effect and small sample or big sample and small effect. This is why you also need a measure for the effect, for example the distance of the two measurements in terms of standard derivations.

Re: P values are not as reliable as many scientists assume (2014)

#26
post #25

Isn’t a main problem with p-values that you don’t know whether significance (low p-value) is a result of big effect and small sample or big sample and small effect. This is why you also need a measure for the effect, for example the distance of the two measurements in terms of standard derivations.

That is a separate issue.

The main problem with p-values is that, without further information, one cannot infer from them how likely it is that a result is genuine.

Re: P values are not as reliable as many scientists assume (2014)

#27

From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…

If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible.

That only 36% were points to deep, fundamental errors.

Re: P values are not as reliable as many scientists assume (2014)

#28

From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…

If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible. That only 36% were points to deep, fundamental errors.

No. P-values don't work that way and don't mean what you think they mean. Read OP or heck, any of the classics like "Why most published research findings are false" http://dx.plos.org/10.1371/journal.pmed.0020124

(36% may or may not be bad, but you can't know without additional stuff like power or prior probability of hypotheses being true; p-values have no intuitive meaning and aren't an answer to any question that people are asking, which is a major reason why Bayesian approaches can be useful. And from a Bayesian perspective, I find 36% totally unsurprising - if anything, substantially better than I had expected given the gross underpowering of most psych studies, the statistical-significance publication filter, and the dubiousness of most hypotheses.)

Re: P values are not as reliable as many scientists assume (2014)

#29
post #28

Earlier quoted context omitted.

If the p-values were accurate and averaged around 0.05, ~95% of results should be reproducible. That only 36% were points to deep, fundamental errors.

No. P-values don't work that way and don't mean what you think they mean. Read OP or heck, any of the classics like "Why most published research findings are false" http://dx.plos.org/10.1371/journal.pmed.0020124 (36% may or may not be bad, but you can't know without additional stuff like power or prior probability of hypotheses being true; p-values have no intuitive meaning and aren't an answer to any question that…

A proper rebuttal would show what a p-value actually is and how it differs from what I claimed. Now, since a p-value is exactly what I previously claimed, you obviously can't do that. I'm not even sure what you are arguing against me here.

Re: P values are not as reliable as many scientists assume (2014)

#30
post #25

Isn’t a main problem with p-values that you don’t know whether significance (low p-value) is a result of big effect and small sample or big sample and small effect. This is why you also need a measure for the effect, for example the distance of the two measurements in terms of standard derivations.

I agree with TFA that p-hacking is a bigger problem.

Low p-value null hypothesis is unlikely.

Choose a shitty null hypothesis ("aliens did it!", "everything is Gaussian", etc) and you trivially get low p. Peer review checks this to some extent (you won't get away with "aliens did it") but there's a large gray area of null hypotheses shitty enough to give low p but not shitty enough to be rejected by peer review. Choosing the hypothesis after-the-fact is the most common strategy because it's undetectable except by repeating the experiment, which is hard.

Post reply on HN