No One Peer-Reviews Scientific Software

We have a software group devoted to maintaining the core code. Our code has a nightly autobuild for development code and libraries are released every couple months.

The code is based on the ROOT package (about one million lines of code) maintained by folks at CERN and is well established within the community and elsewhere. Our libraries probably come to about a million more lines of code, which is maintained scrupulously by our collaboration.

Now neither of these sets of code actually do any analysis, they just make it so that the data is usable. In my analysis, there is probably ~10k lines of code to do the analysis and make the figures. In total, there's probably 500k or more lines of analysis code that is not officially maintained.

That doesn't include all of the code loaded onto FPGAs and custom chips reading out the detectors or the code in the trigger system. Nor does it include code written into various simulators used to determine "expected" response of the detectors.

So for peer review to look at code at the level suggested, people would have to look over literally millions of lines of code. To even be able to make it run, a user would have to set up an environment that would take between a few days and a week.

Instead, when I go to publish something, I lay out both my method and how I verified that it works. Reviewers then check that my method is sound and that my verification looks right. They have to trust that I implemented the method as I said I did.

In my collaboration, there is actually an internal review process that verifies code runs and for which a much longer private note must be written to explain all of the details of the analysis. However, there is still a high level of trust that everything was implemented as stated. This review does not qualify as "peer" review because it's conducted by people who will be listed as authors.

While I agree that more review is better than less, I hope this comment illustrates why software peer review is not a reasonable expectation.

To those guys, simply take a look at Google, or Amazon, or the credit-card processors. Lots of places deal with huge datasets in a peer-reviewed, transparent manner. Of course,use common sense here.

There's clearly a disconnect between how science is performed, and how it is perceived. It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong.

"It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists."

I have unpleasant news for you, then: the "skeptical" side is composed almost exclusively of politicians and economists.

"This seems to be part of the problem. 'We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out'."

That's not actually what they did, and the fact that you don't understand the nuances of the subject goes back to what I was saying about spending more time looking for "smoking guns" than actually understanding the science. The tree-ring data wasn't omitted to make models fit better (that paper wasn't even about a model) -- it was omitted because it was unreliable data.

If you ask me, the tree-ring data should never have been included in that paper at all. And if it weren't, as far as I can tell, it wouldn't have affected the results -- except that there would be an obvious (but largely irrelevant) gap in modern temperature data -- and the "skeptics" would no doubt be attacking some other absurd corner of the climate change research.

1. A company in the banking industry subjects every step of software development process to code review, plus security code review, and Q/A testing every time they release code for production.

So.. Shannon Rose wasn't talking about the banking industry or business code review. The author was talking about scientific systems. Ok.

2. Let's take a bioinformatic knowledge-base company. The code used to search and extract information from the knowledge base, though proprietary, is subject to code review and Q/A testing. In addition the knowledge base itself vigorously Q/A tested, peer reviewed and signed off before release.

But Shannon Rose wasn't talking about bioinformatics? Really? Ok....

3. Let's take a civil engineering company particularly, one who has been in the nuclear engineering industry (such as it has been). Structural engineering code used in nuclear power plants is constantly verified, running known engineering cases against the code base and rigorously checking the result, footnoting and explaining every detailed difference down to the nth decimal place.

But Shannon Rose wasn't talking about actual critical engineering code ... But wait. There's more.

The entire engineering software code base was open-source as are many scientific software projects. Further more, it was looked at in detail by physicists, engineers, and software engineers from within the company who weren't even working on the project. The code base was thus open to examination by engineers and scientists who use it as well as outsiders.

But, Shannon Rose wasn't talking about the nuclear engineering industry. No. This article concerns a particular set of useful but not life-threatening scientific finding that the author disagrees with - and where there are people who oppose learning and understanding what the truth is and what it means. Maybe that idea violate some persons understanding of some biblical text somewhere because somehow humans happen to be responsible mucking up a whole planet.

So, when that biblical text was written, who did peer review on that text? The interpretation?

The fact is, I can think of countless examples where people who work on scientific software, who after all believe that they are responsible for what they do, because maybe they're just that way, take extra precaution to verify their results and subject their work to review. (Because, after all, we don't want another Three Mile Island, or Y2K).

How many examples do I have to cite where there are responsible people before we can reject the idea that "no one" does peer review of any scientific software?

A footnote: the author also wrote that they think the Obama administration is "bringing martial law to the U.S." And why is this? Because the Bush administration was too incompetent to put heinous criminals on trial, we won't get to convict them.

And somehow this mistake is Obama's fault. If McCain had won it would presumably be his fault instead. Because one president couldn't be responsible enough to begin with. Khalid Sheikh Mohammed was actually put under military tribunal and allegedly "confessed." Who did peer review on the trial of Khalid Sheikh Mohammed?

Answer: No one. There was a document written up. No one signed it.

No. This article is not about science, or software. It claims instead there is this general widespread failure by people who work and review in scientific software. That claim is irresponsible on its own. It's follow-up conclusion that this "completely irresponsible" software review can be regulated by so-called "responsible" politicians (such as those who brought us Abu Ghraib, torture, and the possible untriability of Khalid Sheikh Mohammed) can "clean up" peer review is beyond hogwash.

Peer review certainly is an issue and always requires addressing. But I think this author should take their overactive bile somewhere else.