No One Peer-Reviews Scientific Software • Hacker News Board

Re: No One Peer-Reviews Scientific Software

#41

Post by jeremyw » Tue, Dec 01, 2009, 7:55 AM UTC

gjm11 wrote: post #11

jeremyw wrote: post #9

Earlier quoted context omitted.

Members of all fields of science are calling for open data and source code, so I read a political message in your opposition (and focus on climatology). As for correctness , the article doesn't use that CS term and I haven't seen anybody ask for that impossible standard (in the strict CS sense.) Find flaws earlier. That's the goal.

so I read a political message in your opposition (and focus on climatology) : Well, here's the very first sentence of the article he's commenting on: "Recent revelations that the peer review system in climatology might have been compromised by the biases of corrupt reviewers miss a much bigger problem." If focusing on climatology is evidence of a "political message", then I think you've fingered the wrong suspect.

Again, the calls for open access come from scientific fields broadly (where it already isn't happening voluntarily). If you're going to refute that on HN, where the audience isn't pitchfork-wielding, science-averse troglodytes, then you're probably making a narrow argument targeting a specific opposition, i.e. political.

Re: No One Peer-Reviews Scientific Software

#42

Post by billswift » Tue, Dec 01, 2009, 10:00 AM UTC

timr wrote: post #21

DanielBMarkham wrote: post #13

Earlier quoted context omitted.

It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Absolutely. I think we're going to have to have some kind of new certification system for research that affects public policy. Scientists simply can not be activists -- the conflict of interest is too great. If you find a meteor is going to strike the earth next year, you'd better b…

"Public-funded science should run from a public wiki or source control system where the scientists are the only authors but the public at large are readers." The IPCC report is free and public, for exactly this reason. It's quite plain from their arguments that few "skeptics" ever actually look at it. "Yes, I know that will drive some scientists mad compared to the current secretive system but it's time to let some d…

The IPCC report is a political document. It uses results that support its position and ignores everything else. The entire book "The Deniers" is a collection of complaints by scientists that the IPCC twisted their results to support its agenda. Ironically, nearly all of the scientists still "believe" in global warming, they just think THEIR work was misused.

Re: No One Peer-Reviews Scientific Software

#43

Post by billswift » Tue, Dec 01, 2009, 10:08 AM UTC

nvoorhies wrote: post #39

joe_the_user wrote: post #38

Earlier quoted context omitted.

Uh, the software industry may have a long way to go compared to where one might want it to be. But it has gone a ways compared to one person writing batch scripts or (the example from the article) one person writing spreadsheets.

It's also worth noting that in places where reliability is worth the massive decrease in productivity (e.g. robots on mars, avionics, etc.) tools like formal verification and multiple implementations for verification of behavior are used.

Those are situations where scientists hire professionals to do the programming, so that supports joe_the_users's point.

Re: No One Peer-Reviews Scientific Software

#44

Post by gjm11 » Tue, Dec 01, 2009, 1:46 PM UTC

jeremyw wrote: post #41

gjm11 wrote: post #11

Earlier quoted context omitted.

so I read a political message in your opposition (and focus on climatology) : Well, here's the very first sentence of the article he's commenting on: "Recent revelations that the peer review system in climatology might have been compromised by the biases of corrupt reviewers miss a much bigger problem." If focusing on climatology is evidence of a "political message", then I think you've fingered the wrong suspect.

Again, the calls for open access come from scientific fields broadly (where it already isn't happening voluntarily). If you're going to refute that on HN, where the audience isn't pitchfork-wielding, science-averse troglodytes, then you're probably making a narrow argument targeting a specific opposition, i.e. political.

I don't understand. The person you replied to didn't object to open access. He objected to this particular article, which (as it happens) says other things besides "open access would be a good thing".

The article suggests that part of the peer review process should be an examination of the software used, which is about as practical as saying that an experimental paper's reviewers should visit the lab and examine all the apparatus. (Actually, considerably less practical, as anyone who's ever done any sort of review on any nontrivial piece of software knows.) This is not the same as saying that there should be "open access".

The article opens by citing an alleged integrity problem in climatology and saying that lack of access to the relevant software is an even bigger problem. This is not the same as saying that there should be "open access".

The article has a very clear political agenda, as you can note from its beginning (climate-specific stuff, which you yourself have suggested taking as diagnostic for political agenda) and its end (approving links to Eric "famous computer scientist" Raymond and TCS Daily). timr is (1) pointing out the political agenda, for which you accuse him of being political, and (2) describing some specific problems he has with the article, which you characterize as "refuting calls for open access" even though his complaints do not at all take that form.

(I think it would be wonderful if every scientific publication were accompanied by every byte of code and data used in the work it describes, or at least the nearest thing to that that's legally and commercially feasible. But it wouldn't cure the problems that the article professes to find in the peer review process, and the only difference it would make to arguments over climate science is that the no-anthropogenic-global-warming people would find something else to complain about.)

Re: No One Peer-Reviews Scientific Software

#45

Post by michaelkeenan » Tue, Dec 01, 2009, 2:34 PM UTC

timr wrote: post #8

I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact. First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probabl…

timr:

Second, this type of critique is pushing hard on a straw man. The implicit assumption is that each paper is a stand-alone morsel of truth, and that the peer-review process somehow guarantees a paper's claims. This is not the case, and no scientist believes it to be true...The "skeptics" have it wrong, not because they're incorrect about human nature, but because they're attacking a fictional vision of science that exists only in their minds.

The article:

The concept of “peer review” is increasingly being treated in the popular discourse as synonymous with “the findings were reproduced and proven beyond a shadow of a doubt.”

This is never what peer review was intended to accomplish. Peer review functions largely to catch trivial mistakes and to filter out the loons. It does not confirm or refute a paper’s findings. Indeed, many scientific frauds have passed easily through peer review because the scammers knew what information the reviewers needed to see.