Live data from Hacker News

Is it time to up the statistical standard for scientific results?

arstechnica.com

41–44 of 44 posts

Re: Is it time to up the statistical standard for scientific results?

#41

Earlier quoted context omitted.

no doubt, the single-disciplinary researchers use bad statistics, too. My broader point is that multidisciplinary researchers are less likely to use statistics correctly, if for no other reason than that the fact that they are multidisciplinary means that they are less likely to be able to handle depth from a personality/self-habits point of view. Perhaps that's a little bit of projection, but I don't use any but the…

It would be interesting to look into this. My experience has been that multi-disciplinary researchers are more likely to be familiar with a broader range of research and familiar with statistical techniques across their different disciplines. Someone with a narrower perspective is more likely to get into bad habits related to their field. I think your comment about them being less likely to be able to handle depth fr…

Also, I'm in chemistry and biology (don't get me started about biophysics, which is populated by dilettantes who couldn't hack it in physics thinking that biophysics is "physics" - it's not. It's chemistry)... So what it means to me to be multidisciplinary may be different from what it means to you.

People seriously just take standard deviations and then plug it into the formula to spit out p values. Without thinking about things like: "should the values be normally distributed? or log-normal?" "How does error propagate through this formula I'm using?" "Is the major source of error in the replicates that I'm using (versus something I might be normalizing to, like a mass measurement)?" "Am I in the linear range of my calibration standards?" "Is using these data quantitatively (versus qualitatively) an honest thing to do?"

Re: Is it time to up the statistical standard for scientific results?

#42

Earlier quoted context omitted.

By this criteria, there is very little science going on anywhere. We don't teach philosophy of science enough anymore. In most of the fields that I'm familiar with, reproducibility is very hard and rarely bothered about by either the original researcher or others in the field.

That's true , and it's a problem .

Agreed, but I'm not sure the solution is upping the statistical standards so much as educating researchers about what the statistical standards actually mean, how to use them, and how to construct a valid, repeatable experiment.

Re: Is it time to up the statistical standard for scientific results?

#43
post #38

Isn't the solution just to put funding into reproducing results? It's fine to have a wide-filter for exploratory research, but a big problem is that it can be decades before anybody goes back to reproduce. Let's assume that 90% of results are false positives. Using a higher power experiment with even the same standard of p=0.05 would result in rejecting most of those, while hopefully keeping the majority of the true…

The issue in day to day science is that people will work their numbers to get p=0.05. They'll have a hypothesis, that, say, lactating adenomas confer a protective effect against DCIS. And they'll pull, say, all lactating adenoma cases from 2001 to 2013. If that doesn't work, they might actually try dropping the 2001 data and just using the 2002 to 2013 data if it gets them to p=0.05. The result is more brittle (we often test findings by asking "How would the p value change if we added one negative case?") but the alternative (pun intended) wouldn't otherwise be published.

This search for p=0.05 also leads to a lot of hair-splitting studies: take two diagnoses, say, usual and atypical ductal hyperplasia. Now, if you can find some constellation of parameters that define a middle category, say, "borderline ductal hyperplasia", you have a wide-open field to all sorts of p=0.05's, even if there's no change in treatment or outcome. You can say "cases previously characterized as UDH with a greater than are 67% more likely to have (p=0.002)" because you lumped together a bunch of stuff that people already mostly agreed on anyway.

Re: Is it time to up the statistical standard for scientific results?

#44

Earlier quoted context omitted.

That's true , and it's a problem .

Agreed, but I'm not sure the solution is upping the statistical standards so much as educating researchers about what the statistical standards actually mean, how to use them, and how to construct a valid, repeatable experiment.

And, importantly, providing more incentive to actually attempt replication.
Post reply on HN