Combating bad science

I am happy to have seen Ioannidis' talk on this very matter a while ago - very inspiring indeed.

However, I think, often such articles as the one in The Economist lead "outsiders" to the opinion that much of the current biomedical science is flawed. They say that "only" 10 out of 13 in the article have been shown to be reproducible, and while even three bad teeth are not good, it is not even clear what was wrong in the remaining three. It might be very trivial issues, or it might be fraud, who knows. Furthermore, it does mean that at least 3/4 of those papers were fine. Last, so far, the reproducibility results of the 50 cancer papers is still outstanding.

So while I do agree there is an issue at hand, it is a bit like with Apple vs. Windows issues: consumers of the former brand are used to high quality, so a single bug or issue in an Apple soft- or hardware quickly fills news outlets in the world, while nobody gives a damn about even the most critical issues in Windows, because their customers are used to it. While it is important to correct the (ab-)use of biomedical statistics, I think the way this is being presented to outsiders tends a bit towards sensationalism.

I do agree with Ioannidis' as far as that our papers should get more in-depth reviews (not only) of the statistics in them, but the problem is that we need to publish 2-3 papers per year that in turn need 3-5 reviewers per paper, so everybody has to do about 10-15 reviews per year. And by "doing a review" I do not mean just doing an "intensive" reading of the paper being reviewed in half a day, but actually looking at the data and methods and spending some days on it. I am not sure we have the time for that and I believe the reason is this madness of having to publish several papers per year just to survive. I.e., the real problem is our "publish or perish" system, nothing else. All this leads to the fact that it can be better to have dozens of junk papers than one good one, at least if you do not manage to get it to the very top (Nature, Science, Cell, etc.).

>After all, even with such waste, such bias, and such low rates of reproduction, science still gives us amazing insights into the world, enriches our understanding of the universe and our place in it

Not most science, it looks like. Most science gives us false insights into the world, reenforcing our prejudices and the sales of dubious products.

>and of course births incredible technology.

Or at least it once did. We're at the end of riding Moore's law, which confused us into thinking that science was progressing, when it was just transistor density that was progressing.

It's an incentive problem. People tend to game any system based on how there measured. And PHD's are generally measured by how much they publish as well as how quickly they publish. Take two academics that publish the same finding 3 months apart and it's the guy who published first who becomes famous. But, there are a lot more PHD's out there than famous ones so for the most part it's #of papers that's most important.

Ideally you want a lot of incremental research(easy), using other peoples data (cheap), and showing something novel(fame lottery). While most of this is practically useless it still gets published because there are plenty of Journals and they all need to publish something every month.

That's not to say people are not still doing high quality research but rather the signal to noise ratio often low for a reason.

Interesting initiative. For those interested in finding out more information about reproducibility in science, check out David Donoho and Roger D. Peng as well (and of course, Ioannidis has lots of other interesting publications in this area).

One of the outcomes from these efforts that I'm hoping for is that more scientists realize that reproducibility is not just a sideshow that's nice to achieve for some abstract reason. Rather, reproducibility is the core of science and reproducibility is the value that a scientist provides.

Based entirely on my personal experience, it's depressing how little other scientists even understand the problem (much less care about it). It's just not on anybody's radar. I'm often told that it's a waste of time and not worth bothering about because people have more important things to do. (This is at the same time that we [in our specific field] routinely throw out analysis results that took weeks or months to prepare, because they couldn't be reproduced).

Agree it's not a better solution, it's just that there is little talk of it in general. Paywalled science plays a big role here, it basically makes it impossible to gather even paper titles for further analysis.