No One Peer-Reviews Scientific Software • Hacker News Board

Re: No One Peer-Reviews Scientific Software

#31

Post by timr » Tue, Dec 01, 2009, 12:04 AM UTC

jacoblyles wrote: post #23

timr wrote: post #16

Earlier quoted context omitted.

"Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large." There's clearly a disconnect between how science is performed, and how it is perceived. It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an e…

>"It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong." It sounds like you're excusing bad behavior by saying that other people sometimes behave badly. There is no "certain political group" that is over-representing the value of "peer review"…

Nope. I'm not excusing bad behavior. I think those scientists were being immature and irresponsible. But their bad behavior doesn't change my belief that the field of climate change research is valid.

Truth be told, the evidence I've seen doesn't even change my confidence level in their paper -- because unlike "skeptics", I never felt that any one source of historical temperature data was terribly likely to be precise in the first place. Fortunately for climate science, there are many other independent sources of historical temperature data, and all show the same trend. I believe it's extremely unlikely that they're all wrong. Certainly not in the same way.

(In case you were wondering, that's a core difference between skepticism, and "skepticism". A skeptic can be convinced that he's wrong. A "skeptic" wants only to convince others.)

Re: No One Peer-Reviews Scientific Software

#32

Post by joe_the_user » Tue, Dec 01, 2009, 3:05 AM UTC

mattheww wrote: post #12

I work in a fairly large scientific collaboration. Our data sets are on the scale of a few hundred terabytes per year. So, there's a lot of software to convert from the bits that stream out of the detectors to something that end users (grad students) can analyze. We have a software group devoted to maintaining the core code. Our code has a nightly autobuild for development code and libraries are released every couple…

The parent hopefully reflects the workings of large, established scientific organizations.

I did only a bit of work for a very small shop and where I worked coding standards were completely non-existent.

I think that using Matlab is a good way destroy the ability of anyone else to review one's work because of Matlab's nature as a closed, ad-hoc environment.

Some scientists could learn a lot from software best practices.

Re: No One Peer-Reviews Scientific Software

#33

Post by joe_the_user » Tue, Dec 01, 2009, 3:23 AM UTC

timr wrote: post #8

I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact. First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probabl…

First, the notion that anyone could "peer-review" software for correctness is obviously absurd. See the post above: http://news.ycombinator.com/item?id=968727

It's hard to believe that the parent has high rating. Seriously, as science uses more and more software and computers, science is going to have to move closer to standards of reliability of the software industry as well as scientific standards of reliability. Is that not obvious?

Uh... also... the parent also distorts the linked article, which readily admits peer-review isn't a panacea ("Worse, the concept of 'peer review' is increasingly being treated in the popular discourse as synonymous with 'the findings were reproduced and proven beyond a shadow of a doubt.'"). Did parent even read the original article???

Re: No One Peer-Reviews Scientific Software

#34

Post by lutorm » Tue, Dec 01, 2009, 3:53 AM UTC

spamizbad wrote: post #27

lutorm wrote: post #26

Earlier quoted context omitted.

The code I wrote for my simulations is not that big, but it's certainly big enough to be complicated and buggy. And it's GPL and available on my website. And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles shou…

What's your view on the mountains of scientific code that depend on proprietary math kernels and in some cases proprietary compiler platforms?

As you may guess, I'm unhappy about it. In astronomy, almost everyone uses IDL, which is a proprietary, closed-source language. (It also sucks, but that's beside the point.) I think this is bad, because it locks practically the entire research field into a closed platform. (Though GDL is working to be a replacement. I don't know how conformant it is, though.)

Still, I think this is less of a problem than the high-level scientific code. Most of the low-level functionality is pretty simple and (hopefully) much better tested than most scientific codes.

Re: No One Peer-Reviews Scientific Software

#35

Post by tel » Tue, Dec 01, 2009, 4:19 AM UTC

jacoblyles wrote: post #10

timr wrote: post #8

I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact. First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probabl…

Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large. It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Maybe it is understood by scientists that peer review has flaws, but when they appear on morning talk shows the term "peer review" is wielded as a hol…

Regardless of the state of the current process, how could review become anything but better from increased openness?

Increased openness is definitely a direction scientific computing needs to move toward. The trick is conflating that with "peer review" which often, for all its merit, is strongly tied to red tape and frustration. It's impossible to peer review software not because it's impossible to verify techniques but because without some level of trust somewhere it becomes impossible to publish, to communicate.

There's definitely a startup in there. There are tons of opportunities to kill off scientific journals as they are.

The challenge is that you can't kill peer review at the same time. Peer review is still, at the end of the day, critical. It's not a holy sword of truth but instead more like rinsing your plates in the sink before you throw them in the washing machine.

Re: No One Peer-Reviews Scientific Software

#36

Post by camccann » Tue, Dec 01, 2009, 4:29 AM UTC

joe_the_user wrote: post #33

timr wrote: post #8

I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact. First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probabl…

First, the notion that anyone could "peer-review" software for correctness is obviously absurd. See the post above: http://news.ycombinator.com/item?id=968727 It's hard to believe that the parent has high rating. Seriously, as science uses more and more software and computers, science is going to have to move closer to standards of reliability of the software industry as well as scientific standards of reliability. I…

standards of reliability of the software industry

Standards of reliability? Are you talking about the same software industry I'm familiar with?

Re: No One Peer-Reviews Scientific Software

#37

Post by aristoxenus » Tue, Dec 01, 2009, 5:47 AM UTC

lutorm wrote: post #26

mattheww wrote: post #12

I work in a fairly large scientific collaboration. Our data sets are on the scale of a few hundred terabytes per year. So, there's a lot of software to convert from the bits that stream out of the detectors to something that end users (grad students) can analyze. We have a software group devoted to maintaining the core code. Our code has a nightly autobuild for development code and libraries are released every couple…

The code I wrote for my simulations is not that big, but it's certainly big enough to be complicated and buggy. And it's GPL and available on my website. And while I certainly don't expect any of my referees to go over the code line-by-line, one of the rationales I had for GPLing the code was to make it possible for someone to replicate my study. One of the basic tenets about publishing research is that articles shou…

I'd just throw in that if someone uses your code to try to reproduce what you've done, they may actually just be reproducing a bug -- or in the sinister case, reproducing a faked result. So a more general point in this whole discussion would be that making the code available isn't always as important as making the mathematics or other theory behind the code available, and "reproduction" becomes "reimplementation", or some other corroborating element of the process as described by timr a few comments down (http://news.ycombinator.com/item?id=968456).

(Of course for large software endeavors that isn't practical either. So mattheww's point about trust unfortunately resonates.)

As for the proprietary issue, I think sometimes it might actually help reproducibility: if you can just drop a few lines of script into Matlab and examine them, it's potentially a much smaller task with a lot more battle-worn, common-ground code factored out than examining or re-implementing a full library of open code.

Re: No One Peer-Reviews Scientific Software

#38

Post by joe_the_user » Tue, Dec 01, 2009, 7:22 AM UTC

camccann wrote: post #36

joe_the_user wrote: post #33

Earlier quoted context omitted.

First, the notion that anyone could "peer-review" software for correctness is obviously absurd. See the post above: http://news.ycombinator.com/item?id=968727 It's hard to believe that the parent has high rating. Seriously, as science uses more and more software and computers, science is going to have to move closer to standards of reliability of the software industry as well as scientific standards of reliability. I…

standards of reliability of the software industry Standards of reliability? Are you talking about the same software industry I'm familiar with?

Uh, the software industry may have a long way to go compared to where one might want it to be. But it has gone a ways compared to one person writing batch scripts or (the example from the article) one person writing spreadsheets.

Re: No One Peer-Reviews Scientific Software

#39

Post by nvoorhies » Tue, Dec 01, 2009, 7:38 AM UTC

joe_the_user wrote: post #38

camccann wrote: post #36

Earlier quoted context omitted.

standards of reliability of the software industry Standards of reliability? Are you talking about the same software industry I'm familiar with?

Uh, the software industry may have a long way to go compared to where one might want it to be. But it has gone a ways compared to one person writing batch scripts or (the example from the article) one person writing spreadsheets.

It's also worth noting that in places where reliability is worth the massive decrease in productivity (e.g. robots on mars, avionics, etc.) tools like formal verification and multiple implementations for verification of behavior are used.

Re: No One Peer-Reviews Scientific Software

#40

Post by tomjen2 » Tue, Dec 01, 2009, 7:44 AM UTC

tybris wrote: post #28

*sigh, the general public will be heartbroken when they find out that scientists are in fact regular human beings. Science has become a surrogate for their faith.

I was. It wasn't that I assumed science was always right, but I did assume that scientists who works on something as important as climate change wouldn't cheat.

Just another example of why people shouldn't be trusted.