Is it time to up the statistical standard for scientific results?
1–10 of 44 posts
Re: Is it time to up the statistical standard for scientific results?
#2> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that.
This screams systematic error and error propagation to me. It's possible that we don't need to up the p-value, we just need to make sure researchers aren't stupid and can properly account for all sources of errors. The problem is that's often an acquired skill over time, not something younger researchers typically think about, especially those who aren't multidisciplinary.
Re: Is it time to up the statistical standard for scientific results?
#3>In most fields, if there's less than a five percent chance that you'd get the two numbers by random chance, then you can reject chance—the results are considered significant. In statistical terms, this is called having a p value of less than 0.05.
No it isn't [1].
Re: Is it time to up the statistical standard for scientific results?
#4In physics, we use 2-sigma (95%) limits all the time. 5-sigma (99.9999%) is generally required only for a claim of detection.
If you're just feeling your way around in the dark, 2-sigma is a useful way to work, so we use that to guide exploration.
Why 2-sigma? Well, it's twice as big as 1-sigma.
Experiment didn't go well, but you need a more-impressive result? Use a 90% confidence interval instead of 95%.
Re: Is it time to up the statistical standard for scientific results?
#5> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-va…
Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of errors, up to and including blatantly cherry-picking or fabricating results, where the focus should be. There's been a series of recent posts encouraging scientists (and, moreover, journals) to publish negative results, which is probably a good starting point to promote greater transparency and honesty in the paper publishing game.
Re: Is it time to up the statistical standard for scientific results?
#6In physics, we use 2-sigma (95%) limits all the time. 5-sigma (99.9999%) is generally required only for a claim of detection. If you're just feeling your way around in the dark, 2-sigma is a useful way to work, so we use that to guide exploration. Why 2-sigma? Well, it's twice as big as 1-sigma. Experiment didn't go well, but you need a more-impressive result? Use a 90% confidence interval instead of 95%.
1-sigma is 68% confidence. 90% confidence intervals assume 1.64 sigma.
Re: Is it time to up the statistical standard for scientific results?
#7> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-va…
exactly wrong, it's the multidisciplinary scientists who don't get enough depth or rigor to come close to understanding sources of error and error propagation.
The thing I fear is underexperienced multidisciplinary researchers coming up with convoluted procedures that put their hands and eyes on their work in so many places that small sources of error have compounded themselves and entrenched themselves into nooks and crannies.
Re: Is it time to up the statistical standard for scientific results?
#8In physics, we use 2-sigma (95%) limits all the time. 5-sigma (99.9999%) is generally required only for a claim of detection. If you're just feeling your way around in the dark, 2-sigma is a useful way to work, so we use that to guide exploration. Why 2-sigma? Well, it's twice as big as 1-sigma. Experiment didn't go well, but you need a more-impressive result? Use a 90% confidence interval instead of 95%.
1-sigma is 68% confidence. 90% confidence intervals assume 1.64 sigma.
Agreed, I'll add a line break to denote that they're separate thoughts. 1-sigma is indeed 68%.
Re: Is it time to up the statistical standard for scientific results?
#9> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-va…
Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of…
This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis.
For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data".
It's also entirely possible to get a result at p less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5.
Now, this is a significant result at p if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.)
This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
Re: Is it time to up the statistical standard for scientific results?
#10>In most fields, if there's less than a five percent chance that you'd get the two numbers by random chance, then you can reject chance—the results are considered significant. In statistical terms, this is called having a p value of less than 0.05. No it isn't [1]. [1]: https://en.wikipedia.org/wiki/P-value
The Ars definition of a p value is not precisely worded but, I think, reasonably accurate. It doesn't include the bit about also including the probability of obtaining results more extreme than what you obtained.
The best definition I know is
> The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.
S. N. Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130:995–1004, 1999.
edit: oh dear, and then the Ars article says "Individual experiments may be wrong five percent of the time," but that's exactly what p values do not measure. Statistics is hard.