No One Peer-Reviews Scientific Software

I think this article presents a brief, well written summary of an important issue from someone who clearly knows what he's talking about and I would recommend reading it.

This is an excellent find. Thanks

How did we let this problem develop? I think it was simply a matter of creeping normalcy.

Probably so. I know from working with various industries, such as military radio guys, that gradually their jobs became all about software. Twenty years ago you'd have one software guy and 30 engineers and now you have 30 software guys and 1 engineer. It was very late in the day when folks sat down and said "Hey! We're really a software organization now, and this stuff is really important, so we'd better start tightening things up" I imagine some types of academic research are in that same place now.

It just creeps in over many years.

Open Source Science is looking better and better.

Two myths my non-science friends understand to be true.

a) A peer reviewed paper is very close to fact.

b) Scientists are essentially free from backbiting, tribalism and empire-building. They dispassionately rely on provable data.

Modify slightly to ignore any scientist funded by the wrong people.

I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact.

First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probably buggy) model, and (in my experience) reviewers are therefore extremely skeptical of results from computer models -- to the point that they're actually biased against publication. And since reviewers know that they can't verify correctness, they tend to look for theoretical papers that very closely reproduce experimental results.

Second, this type of critique is pushing hard on a straw man. The implicit assumption is that each paper is a stand-alone morsel of truth, and that the peer-review process somehow guarantees a paper's claims. This is not the case, and no scientist believes it to be true. It doesn't matter that peer review fails sometimes, because the system doesn't depend on it being flawless.

Science is about process, not people. Peer review is an important part of that process, but it's openly acknowledged that review has flaws. Thus, ask a scientist about any particular paper, and while s/he may be enthusiastic about the result or the theory, you'll almost never hear a scientist take the paper for granted as fact. More often than not, discussion of an interesting paper will center around designing other tests that can independently reproduce the paper's result. Ultimately, you have a system where a lot of genuinely smart, skeptical people are attacking ideas from all sides until either a consensus emerges, or the idea is destroyed.

The "skeptics" have it wrong, not because they're incorrect about human nature, but because they're attacking a fictional vision of science that exists only in their minds. They imagine a world revolving around a theory that is supported by one piece of data, extrapolated by buggy computer models, and presented by corrupt individuals who (for some unknown reason) have an incentive to doctor the results in the same way. In reality, science is massively redundant and competitive, computer models are treated as ancillary data (at best), and individual corruption is exposed by the redundancy of the system. If the "skeptics" put as much effort into understanding the science they critique as they do trying to find "smoking guns" for individual researchers, they would understand this dynamic.

Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large. It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Maybe it is understood by scientists that peer review has flaws, but when they appear on morning talk shows the term "peer review" is wielded as a holy sword.

Regardless of the state of the current process, how could review become anything but better from increased openness?

>"And since reviewers know that they can't verify correctness, they tend to look for theoretical papers that very closely reproduce experimental results."

This seems to be part of the problem. "We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out".

But then again, I'm no climate scientist. Although, one would think that no harm would be done by releasing the data, even if there were legitimate statistical reasons for truncating it.

>"Ultimately, you have a system where a lot of genuinely smart, skeptical people are attacking ideas from all sides until either a consensus emerges, or the idea is destroyed."

If you're a smart, skeptical person attacking a paper that shows global warming to be severe, Michael Mann will get you fired. Or ignore your ideas. Mann just published a paper that appears to use an inverted data series after McIntyre already correctly pointed out that it was inverted, out of pride I suppose. Fine science, that.