Why scientific programming does not compute • Hacker News Board

Re: Why scientific programming does not compute

#151

Post by xtracto » Thu, Jul 07, 2011, 8:02 AM UTC

rweba wrote: post #147

ajdecon wrote: post #114

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the p…

My PhD was in computer science and my experience was quite similar. I wrote probably around 3000 lines of code on 4 separate projects (mostly MATLAB, C and Java). This code was never shared with anyone, my advisors were not interested in the code, all they cared about were the results. To be honest it wasn't very good code, I would have a hard time understanding it now (although I could probably figure it out eventua…

I have a PhD in Comp. Sci. too, and continue working in academia.

Regarding your code, you could have just uploaded it to SourceForge or any other OpenSource repository. I know a guy (Steve Phelps) who did exactly that ( http://sourceforge.net/projects/jasa/ ) with his PhD code.

On a related note, the institute where I am working now has this "great" simulation program (housemade in C++) for which a lot of publicatios have been written. However, the code is closed source and thus cannot be third-party verified.

This is wrong, and actually, a colleague of mine who just started doing her PhD found an error in the simulation program, bad enough that it makes me question the previous research.

In my opinion it must be a requirement that all software related to a publication must be made open-source before (or at the same time) the paper is published.

In the traditional research method, computer programs are part of the methods of the reserach. It is amazing that nowadays researchers can publish research without clearly showing the process they used to arrive to those.

Re: Why scientific programming does not compute

#152

Post by xtracto » Thu, Jul 07, 2011, 8:06 AM UTC

gallamine wrote: post #44

I'm a PhD student in Electrical Engineering. I'm currently working on a Monte Carlo-type simulation for looking at the underwater light field for underwater optical communication (no sharks!). I'm doing the development in MATLAB and I recently put all my code up on Github ( https://github.com/gallamine/Photonator ) to help avoid some of these problems (lack of transparency). Even if nobody ever looks/uses the code, I…

"Essentially, all models are wrong, but some are useful."

[ http://en.wikiquote.org/wiki/George_E._P._Box ]

Re: Why scientific programming does not compute

#153

Post by xtracto » Thu, Jul 07, 2011, 8:18 AM UTC

jzila wrote: post #33

My girlfriend is a PhD student in a pharmacology lab. I'm a software engineer working for an industry leader. Once, she and the lab tech were having issues with their analysis program for a set of data. It was producing errors randomly for certain inputs, and the data "looked wrong" when it didn't throw an error. I came with her to the lab on a Saturday and looked through the spaghetti code for about 20 minutes. Once…

The problem I see with your girlfriend's program is more of a "verification" issue.

In the simulation sub-field I am there is this "research development process" which includes "verification" and "validation" after the model is performed.

Part of the verification is done by "third party code reviews" in which a party unrleated to the program/project reviews the model description (word document) and does a line-by-line analysis of the code to see that the program matches the code.

I did that during my PhD (a Professor at INSEAD paid me to do a code review of a model).

In the case of your girlfriend's lab, they catched the error via "face validation" (the results looked wrong).

Re: Why scientific programming does not compute

#154

Post by cool-RR » Thu, Jul 07, 2011, 8:23 AM UTC

My approach is to have the scientist write as little code as possible. That's why I'm working on GarlicSim:

http://garlicsim.org

GarlicSim's goal is to do all the technical, tedious work involved in writing a simulation while letting the scientist write only the code that's relevant to his field of study.

Re: Why scientific programming does not compute

#155

Post by wisty » Thu, Jul 07, 2011, 11:19 AM UTC

ajdecon wrote: post #114

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the p…

Another reason - a bug (and there's always bugs) would probably invalidate the paper, possibly causing a recall. Recalled papers are not seen in a positive light by the science establishment.

Careers could be destroyed, if people where held to account.

People in software see this as ludicrous. Of course there's bugs, just update the conclusions, and move on! But that's not how a lot of scientists think.

Re: Why scientific programming does not compute

#156

Post by draven » Thu, Jul 07, 2011, 11:35 AM UTC

JonnieCache wrote: post #104

Maybe we can do an outreach program? Hackers adopting scientists?

I wish this kind of mentoring program would be implemented here (I work in a big research center). Often us programmers end up having to integrate code written by scientists in our apps, and it's a pain. Even a quick glance over the code is often enough to see some problems.

Re: Why scientific programming does not compute

#157

Post by CurtHagenlocher » Thu, Jul 07, 2011, 11:46 AM UTC

w1ntermute wrote: post #145

tedjdziuba wrote: post #53

I know of a company, made up of scientists from academia, that develops software by writing the code (or "codes" as they call it) in Microsoft Word documents and e-mailing them to eachother. Some how, they are still in business. True story.

> writing the code (or "codes" as they call it) It's funny how this exact same mistake is made in the linked paper. For some reason, people outside of IT can't get it into their minds that "code" is an uncountable noun in this context.

This seems to be the standard in technical computing; people say "codes" instead of "code".

Re: Why scientific programming does not compute

#158

Post by bluekeybox » Thu, Jul 07, 2011, 12:40 PM UTC

JohnnyBrown wrote: post #131

ajdecon wrote: post #114

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the p…

I'm working in biology now, and a good example of researchers who produce quality, documented code that other people find useful is the Knight group at UC Boulder. They write python with good docs and support, publish the algorithms they come up with in bioinformatics journals, and people cite them all the time. Might be worth thinking about why there are incentives there and not elsewhere.

Another place where good software work is done is Broad Institute. The reasons are as follows: (a) Broad can hire the best people in bioinformatics, (b) they are a relatively large organization where focusing on the process pays off. Software ultimately is process (i.e. how you do stuff) and small labs often cannot afford to focus on the process and instead try to reach the goal (i.e. publishable results) more or less directly, regardless of the inefficiencies they may encounter.

In the past, the model of having many small labs in universities was a great idea. Today things are looking a bit different in some fields because larger labs can afford to do more automation (by hiring programmers instead graduate students).

Re: Why scientific programming does not compute

#159

Post by DennisP » Thu, Jul 07, 2011, 2:10 PM UTC

ajdecon wrote: post #114

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the p…

I'm still kind of amazed that when scientific results are based on original software, the sourcecode isn't required by the journals for peer review. How are people supposed to check the results?

CS papers without code are even worse.

Re: Why scientific programming does not compute

#160

Post by gte910h » Thu, Jul 07, 2011, 4:09 PM UTC

_delirium wrote: post #86

gte910h wrote: post #17

Earlier quoted context omitted.

What an excellent idea: Generate new jargon that's incompatible with the jargon used from the field you suck at. Perhaps concerned scientists and editors should reject the bifurcation here and take on the lingo of the field that creates the tool they have to use and need to learn better as a first step in learning to program in a more responsible manner?

I think you have the chronology backwards. The use of "code" as a mass noun dates to the 1960s at the earliest (actually, I can't find a good example before the 1970s in brief searching), while the use of "code" as a singular noun to mean "implemented algorithm", and "codes" as the plural, dates back at least to the 1950s. As one of many examples, here's a 1955 survey article compiling a list of "available digital co…

My Chronology may still be backwards, but they still should swap over to the language of the mature field of software development's language to better allow themselves to integrate in good practices.

The bifurcation is still harmful to them even if the software usage originated later than the science term.