The Irreproducibility Crisis of Modern Science

University PR is far more guilty. Buzzfeed isn't scanning the current issue of "Physical Review Letters" for the latest scoop.

edit: When I read about ensemble theory, you receive support for the increased computational cost.

Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well.

But I found this interesting:

Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9]

..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.

More importantly, the example of the Nazis is useful in an ethical argument, because they're an almost universally acknowledged example of evil. Much like we aim to test our programs with extreme inputs, we can use the "extreme input" of Nazism to test our ethical arguments. If I posit some universal ethical standard, and then find that it doesn't seem to work if we take the Nazis, we've proven by counterexample that my standard is not universal after all.

The term 'science' is used in this way. Perhaps it should not be. Perhaps semantics drive the issue.

In the field in which I am familiar, we have 'expert witnesses' that will use reproducible scientific principles to analyze, for instance, blood scatter to determine the angle in which a shotgun was held when it discharged and blew off a particular person's head. The principles -- based on physics, chemistry, etc -- are sound and scientific.

The problem is, we will invariably have TWO very scientific expert witnesses who BOTH use reproducible and accepted methods. Yet, using their scientific tools, they will come to two utterly different conclusions.

It's left to a jury to decide which interpretation is correct.

While the tools used by the experts are scientific, their opinions are not. While their scientific tools are reproducible (for instance, the speed at which coagulating blood and brain tissue will slide down a stucco wall given all the variables such as temperature, humidity, friction coefficients, etc.) their ultimate reconstruction of what happened is not reproducible, and it is left to lay people to draw their conclusions.

There should be more of a distinction between Science and History. That is, science -- strictly defined -- is all about reproducibility. History -- by definition -- is not. Using scientific tools to reconstruct a past event does not result in a 'scientific' theory. Rather, it results in a historical theory arrived at with scientific tools.

Whether it is blood scatter from a murder (or suicide, depending on whom you ask), the formation of the Grand Canyon, or the development of finch beaks on Galapagos, the ultimate theories are inherently non-scientific as they can not be tested. The methods and tools used to derive the theories can be... but not the conclusion itself.

In short, perhaps the term 'science' is being used for things outside its domain. (Or, alternatively, if we wish to include such things inside the domain, we should broaden the strict definition of science. Like I said, there are some issues of semantics that may be driving some of the issues in the article... other issues, of course, pertain to sloppiness, errors, and the like.)