P values are not as reliable as many scientists assume (2014)
1–10 of 90 posts
Re: P values are not as reliable as many scientists assume (2014)
#2[deleted]
Re: P values are not as reliable as many scientists assume (2014)
#3From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate.
I think the scientific system can work with that. It means that if you build follow-up experiments based on a single paper there is a good chance that the experiment fails. In some way, the scientific system of publishing is self-correcting in this regard, because you can then cast doubt on the previous paper, which is easier to publish than if you only have a fresh negative result (p-value > threshold).
Re: P values are not as reliable as many scientists assume (2014)
#4From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…
There is no way to know how many people tried to build a follow-up experiment which failed and was not published because the failure to replicate will usually be assumed to be due to some mistake, and even carefully finding p > threshold is not very publishable.
Re: P values are not as reliable as many scientists assume (2014)
#5[deleted]
All of us?
Scientific experiments always involve some degree of assumption. I'm most skeptical of those who don't state their assumptions.
Re: P values are not as reliable as many scientists assume (2014)
#6Re: P values are not as reliable as many scientists assume (2014)
#7Scientists have to do their work in a system that incentivizes bad science. How many people actually get to do their work in an environment that isn't hostile to them?
Re: P values are not as reliable as many scientists assume (2014)
#8Previous post with discussion, 563 days ago: https://news.ycombinator.com/item?id=7225739
PDF via same nature.com: https://news.ycombinator.com/item?id=8404620
Related, dupes of each other:
https://news.ycombinator.com/item?id=9463806
https://news.ycombinator.com/item?id=9486059
Related:
Re: P values are not as reliable as many scientists assume (2014)
#9Scientists have to do their work in a system that incentivizes bad science. How many people actually get to do their work in an environment that isn't hostile to them?
a) Are you serious about that second question and b) if so, can we discount thermodynamics in our answer? Otherwise it's kind of boring.
Re: P values are not as reliable as many scientists assume (2014)
#10Relevant, from Deborah Mayo.
http://errorstatistics.com/2015/03/16/stephen-senn-the-pathe...