Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is it time to up the statistical standard for scientific results?

arstechnica.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Is it time to up the statistical standard for scientific results?

#21

Post by chalst »

batbomb wrote: post #2

> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-va…

I agree with the diagnosis. The big problem is that it is not possible to be sure that some study accounts for all possible confounding factors, unless you have a correct theory that allows you to determine what factors might be confounding factors. Statistical methodology alone cannot tell you this.

Reproducibility by researchers who know no more about the experimental setup than what was published is a better way of guarding against this than upping the significance hurdle. Experienced researchers may have a long history of ignoring confounding factors - I am not sure maturity matters much. I think the best time to have your complacency about constructing statistical investogations torn apart is while you are doing your PhD.

Re: Is it time to up the statistical standard for scientific results?

#22

Post by eli_gottlieb »

apsec112 wrote: post #9
flatline wrote: post #5

Earlier quoted context omitted.

Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of…

This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis…

Right, you want this superhero cape with Bayes' Theorem printed on it?

It's got priiiiooooooors.

Re: Is it time to up the statistical standard for scientific results?

#23

Post by jmpeax »

apsec112 wrote: post #9
flatline wrote: post #5

Earlier quoted context omitted.

Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of…

This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis…

> What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data.

The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data.

I can see why elementary mistakes are so pervasive.

Re: Is it time to up the statistical standard for scientific results?

#25

Post by ProblemFactory »

RA_Fisher wrote: post #13

The real problem is that effect size is rarely discussed. P-values only relate to variability, sometimes high variability is acceptable, sometimes it's not. Taken by itself, a p-value is worthless, no matter how small it is. For example, suppose you develop a fertilizer that you're 99.99999% certain will produce one additional ear of corn in 10,000 bushels. Who cares!? You see a lot of that in published papers. A lot…

Effect size is important, because all events in the real world except for true random numbers have some correlation.

Eating an apple might help a broken leg heal faster or slower. There is certainly an extremely small correlation, and with a sufficiently large number of controlled experiments, a result rejecting the null hypothesis with p will be found. The required number of experiments might be astronomically large, but if any correlation exists, it can be found with enough samples.

But without looking at the effect size, the result is useless. Even if eating an apple helps your broken leg heal 2 seconds faster on average, it is pointless to suggest this as medical advice.

Re: Is it time to up the statistical standard for scientific results?

#27

Post by qwerta »

My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific.

It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough.

It is like heaving open-source, but without any source code.

Re: Is it time to up the statistical standard for scientific results?

#28

Post by dllthomas »

qwerta wrote: post #27

My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific. It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough. It is like heaving open-source, but without any source code.

While I think availability of data and code is important, note that it does not accomplish many of the functions of replication of experiments. If two people implement the described algorithm and get the same results, we can be a lot more confident than if two people run the same implementation, because the same implementation is more likely to have the same bugs.

Re: Is it time to up the statistical standard for scientific results?

#30

Post by qwerta »

dllthomas wrote: post #28
qwerta wrote: post #27

My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific. It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough. It is like heaving open-source, but without any source code.

While I think availability of data and code is important, note that it does not accomplish many of the functions of replication of experiments. If two people implement the described algorithm and get the same results, we can be a lot more confident than if two people run the same implementation, because the same implementation is more likely to have the same bugs.

Code has to be reviewed and _VERIFIED_. How can an article pass peer review, if nobody even checked code for basic mistakes?
Post reply on HN