Is it ever NOT the time to up the statistical standard? Should science not be constantly seeking to improve itself?
Is it time to up the statistical standard for scientific results?
31–40 of 44 posts
Re: Is it time to up the statistical standard for scientific results?
#32Earlier quoted context omitted.
I haven't seen any reason to believe that being multidisciplinary makes this more likely. The whole problem is caused by the fact that most of the single-disciplinary researchers are using bad statistics. Very few degree programs give scientists enough background in statistics and experimental rigor to prevent these problems.
no doubt, the single-disciplinary researchers use bad statistics, too. My broader point is that multidisciplinary researchers are less likely to use statistics correctly, if for no other reason than that the fact that they are multidisciplinary means that they are less likely to be able to handle depth from a personality/self-habits point of view. Perhaps that's a little bit of projection, but I don't use any but the…
It would be interesting to look into this. My experience has been that multi-disciplinary researchers are more likely to be familiar with a broader range of research and familiar with statistical techniques across their different disciplines. Someone with a narrower perspective is more likely to get into bad habits related to their field.
I think your comment about them being less likely to be able to handle depth from a personality/self-habits point of view is really projecting and probably based on experiences with a few bad apples. My experience doesn't reflect that at all.
I feel like a lot of the problems with statistics come from overly insular journals and conferences where everyone does it wrong and nobody realizes that because they're all doing the same thing. I've seen this in some computer science fields.
Re: Is it time to up the statistical standard for scientific results?
#33My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific. It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough. It is like heaving open-source, but without any source code.
By this criteria, there is very little science going on anywhere. We don't teach philosophy of science enough anymore. In most of the fields that I'm familiar with, reproducibility is very hard and rarely bothered about by either the original researcher or others in the field.
Re: Is it time to up the statistical standard for scientific results?
#34Is it ever NOT the time to up the statistical standard? Should science not be constantly seeking to improve itself?
Many researchers in less mathematically rigorous fields lack basic statistical knowledge, such as how a p-value is calculated or what a chi-squared test is. And that isn't to be snarky towards them - they have a lot to absorb in their own field. It just means that they are slower to adopt or even know about various mathematical techniques.
This reminds me of the med researcher who rediscovered integration in 1993 and was cited many, many times:
Re: Is it time to up the statistical standard for scientific results?
#35My impression is there are a couple issues:
- Methodology - If you're mining for answers, and then presenting only the statistically significant, you will still find false positives on larger datasets, it will just take more work.
- Reproduction - If false positives were more fiercely chased down, then this would force researchers to be more careful. There is improvement along these lines.
Changing the p value is arbitrary, and may miss important results. I believe that encouraging reproduction of results, and reducing blind data mining is a better solution.
Re: Is it time to up the statistical standard for scientific results?
#36> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-va…
Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of…
It's not that arbitrary really. 95% equals 2 sigma.
Re: Is it time to up the statistical standard for scientific results?
#37Earlier quoted context omitted.
While I think availability of data and code is important, note that it does not accomplish many of the functions of replication of experiments. If two people implement the described algorithm and get the same results, we can be a lot more confident than if two people run the same implementation, because the same implementation is more likely to have the same bugs.
Code has to be reviewed and _VERIFIED_. How can an article pass peer review, if nobody even checked code for basic mistakes?
Yes, precisely. Publishing code and data is necessary for peer review, not for replication. Both are necessary parts of science.
Re: Is it time to up the statistical standard for scientific results?
#38Isn't the solution just to put funding into reproducing results? It's fine to have a wide-filter for exploratory research, but a big problem is that it can be decades before anybody goes back to reproduce.
Let's assume that 90% of results are false positives. Using a higher power experiment with even the same standard of p=0.05 would result in rejecting most of those, while hopefully keeping the majority of the true positives (due to the higher power of the new test). This would result in going from 90% false positives to less than half false positives, a considerable improvement.
It just seems that nobody ever advanced their career by trying to reproduce even a landmark result in their field.
Re: Is it time to up the statistical standard for scientific results?
#39No, it's not, unless you want even more data dredging and p-value hacking. This won't solve anything, and it'll make matters worse.
Re: Is it time to up the statistical standard for scientific results?
#40My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific. It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough. It is like heaving open-source, but without any source code.
By this criteria, there is very little science going on anywhere. We don't teach philosophy of science enough anymore. In most of the fields that I'm familiar with, reproducibility is very hard and rarely bothered about by either the original researcher or others in the field.
That's true, and it's a problem.