The Irreproducibility Crisis of Modern Science

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific method. However, such use is invalid as the scientific self-criticism

1. assumes the validity of the scientific method

2. relies on the scientific method as its critical lens

Whereas those who critique science as a whole:

1. assume that the scientific method does not work and does not arrive at "truth"

2. then use scientists being self critical to prove #1.

Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200

In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."

Haven't downvoted, but I'd guess it probably starts with "Why are we bothering to educate anyone except the very, very best students in physics, maths and engineering?"

Sure, we will undoubtedly get better society-wide results if we identify a subset of elite students (notably defined primarily in terms of STEM fields) and only "bother" educating them, leaving everyone else to Just Trust Them™ and underutilizing the long tail of talent.

And then there's the premise that society somehow isn't looking to utilize the existing population of elite students, which of course it is, no shortage of organizations looking for outliers.

About the only good point there is that our system for allocating the best/brightest to certain problems could use improvement.

The rest of the comment explicitly disagrees with that thought. I have edited it to make it clearer.

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats:

1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list.

2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.

3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.

On the other hand...

4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.

That would never work. We have no accurate way to identify the "best" students so early in the educational pipeline. High school grades and standardized test scores are poor predictors of who will make major contributions to science.

More public funding for basic research (including reproduction studies) would probably deliver major long term benefits.