Why scientific programming does not compute • Hacker News Board

Re: Why scientific programming does not compute

#121

Post by FrojoS » Thu, Jul 07, 2011, 1:16 AM UTC

gallamine wrote: post #44

I'm a PhD student in Electrical Engineering. I'm currently working on a Monte Carlo-type simulation for looking at the underwater light field for underwater optical communication (no sharks!). I'm doing the development in MATLAB and I recently put all my code up on Github ( https://github.com/gallamine/Photonator ) to help avoid some of these problems (lack of transparency). Even if nobody ever looks/uses the code, I…

The people at Willow Garage think there is a market for this [1].

http://www.willowgarage.com/blog/2010/04/27/reinventing-whee...

Re: Why scientific programming does not compute

#122

Post by shriphani » Thu, Jul 07, 2011, 1:17 AM UTC

ajdecon wrote: post #114

(Disclaimer: my background is in materials physics, and it may be different in other fields. But I doubt it.) Unfortunately there is very little direct incentive for research scientists to write or publish clean, readable code: - There are no direct rewards, in the tenure process or otherwise, for publishing code and having it used by other scientists. Occasionally code which is widely used will add a little to the p…

Matt Might has something on this topic. The CRAPL is what he calls it : http://matt.might.net/articles/crapl/

Re: Why scientific programming does not compute

#123

Post by bugsbunnyak » Thu, Jul 07, 2011, 1:29 AM UTC

droz wrote: post #80

mclin wrote: post #41

Rather than building these data analysis/visualization programs from scratch each time, my thought is that scientists should instead be writing them as modules for a data workflow application like RapidMiner. If you haven't heard of RapidMiner, you basically edit a flowchart where each step takes inputs and outputs, eg take some data and make a histogram, or perform a clustering analysis. Video of someone demoing it:…

Do you know of any other good data analysis applications similar to RapidMiner?

Knime http://www.knime.org/.

To a lesser (but free) extent, Orange http://orange.biolab.si/

Re: Why scientific programming does not compute

#124

Post by gallamine » Thu, Jul 07, 2011, 1:38 AM UTC

JabavuAdams wrote: post #120

gallamine wrote: post #44

I'm a PhD student in Electrical Engineering. I'm currently working on a Monte Carlo-type simulation for looking at the underwater light field for underwater optical communication (no sharks!). I'm doing the development in MATLAB and I recently put all my code up on Github ( https://github.com/gallamine/Photonator ) to help avoid some of these problems (lack of transparency). Even if nobody ever looks/uses the code, I…

Huh. Cool. Are you familiar with Metropolis light-transport? http://en.wikipedia.org/wiki/Metropolis_light_transport I've read about it in the context of speeding up global-illumination path-tracing for computer graphics. I think it's based on work that was originally done for neutron scattering.

I believe I came across it at some point or another, but I haven't looked closely at it. For better or worse, it doesn't seem to be employed by others doing light field simulations underwater ... not sure why. I'd have to poke into it further. I was, however, seriously thinking about writing a plugin for Blender that would utilize my specific scattering phase functions in their volumetric ray tracing renderer ... no time though.

Re: Why scientific programming does not compute

#125

Post by john_b » Thu, Jul 07, 2011, 1:47 AM UTC

hyperrail wrote: post #117

john_b wrote: post #72

Earlier quoted context omitted.

I'm a M.S. student in mechanical engineering facing a similar situation, except I haven't put any code on Github (my advisor wants to keep it proprietary, but I probably would not bother putting it up even if he were ok with it). I've written around 15000 lines of MATLAB for my research and only a handful of people will ever need to see it. Some is well-structured and nicely commented, but other parts are incomprehen…

> graduating a semester late (at the cost of several thousand dollars to myself) Really? Your funding isn't guaranteed? When I did my Master's I was funded as an R.A. without my advisor/lab having to tap her particular grants. Grants and fellowships were usually seen as something "extra" for master's and Ph.D pre-quals students, not their main source of funding. I find it surprising that your school or department see…

PhD student funding is guaranteed in my department, but not funding for M.S. students. Initially I did have a RA, supplied by my advisor's start up funding (he was fairly new at the time). But when his grant applications were rejected and his start up funding ran out, I had to do a TA instead. TA-ships aren't guaranteed though, and at my school if you stay too long on a TA they take you off of it to make sure that other students have a chance at funding too. My advisor did ultimately get some grant money in, but by that point there were other students who needed it more than me.

It might have something to do with recent state budget cuts (it's a state-funded university). My department has also grown dramatically over the past few years, both in terms of faculty and students, so the graduate student funding will probably lag behind for a few years more.

Re: Why scientific programming does not compute

#126

Post by john_b » Thu, Jul 07, 2011, 1:54 AM UTC

eru wrote: post #106

cdavoren wrote: post #100

Earlier quoted context omitted.

I think the problem is fundamentally one of economics. Research is good, but you have to decide how much money to allocate to it. In order to decide, you need a metric for performance. Really, only scientists are qualified to judge whether the results of other scientists are worth anything, so currently the only metric we really have is publishing in peer-reviewed journals. Ultimately, therefore, that's where the inc…

How about trying to fix the current system by making somebody else using your software count as a "super citation"? (It could even arguably count as much as co-authorship.)

I think this is an excellent idea. If published software could be tagged via a unique identifier (like the DOI of a paper), then it could be cited by that tag just like a paper. Well written software might even get cited more than the paper it was published in.

Re: Why scientific programming does not compute

#127

Post by kenjackson » Thu, Jul 07, 2011, 2:03 AM UTC

pygy_ wrote: post #11

bh42222 wrote: post #5

As a general rule, researchers do not test or document their programs rigorously, and they rarely release their codes, making it almost impossible to reproduce and verify published results generated by scientific software, say computer scientists. Just stop doing that! Seriously, testing is not wasted effort and for any project that's large enough it's not slowing you down. For a very small and simple project testing…

You won't see any clean code written by scientists until (major) journals make it mandatory to submit code for peer review and publication. When it happens, I hope that they'll manage to agree on a sensible license (even though I won't set my hopes too high).

That will never happen, because the universities are bigger than the journals and will push back. The universities want to own the code if there's money to be made. Stanford made a small fortune from Google, for example. If journals required code review, other journals would pop up that wouldn't require it.

Re: Why scientific programming does not compute

#128

Post by gwern » Thu, Jul 07, 2011, 2:03 AM UTC

There are a lot of suggestions that the code and data be required to publish.

Sorry guys, but that hasn't worked so far: the economics journal _Journal of Money, Credit and Banking _, which required researchers provide the data & software which could replicate their statistical analyses, discovered that Oops.

Re: Why scientific programming does not compute

#129

Post by gwern » Thu, Jul 07, 2011, 2:04 AM UTC

An interesting citation http://portal.acm.org/citation.cfm?id=188228 :

> This paper describes some results of what, to the authors' knowledge, is the largest N-version programming experiment ever performed. The object of this ongoing four-year study is to attempt to determine just how consistent the results of scientific computation really are, and, from this, to estimate accuracy. The experiment is being carried out in a branch of the earth sciences known as seismic data processing, where 15 or so independently developed large commercial packages that implement mathematical algorithms from the same or similar published specifications in the same programming language (Fortran) have been developed over the last 20 years. The results of processing the same input dataset, using the same user-specified parameters, for nine of these packages is reported in this paper. Finally, feedback of obvious flaws was attempted to reduce the overall disagreement. The results are deeply disturbing. Whereas scientists like to think that their code is accurate to the precision of the arithmetic used, in this study, numerical disagreement grows at around the rate of 1% in average absolute difference per 4000 fines of implemented code, and, even worse, the nature of the disagreement is nonrandom. Furthermore, the seismic data processing industry has better than average quality standards for its software development with both identifiable quality assurance functions and substantial test datasets.

Re: Why scientific programming does not compute

#130

Post by dasil003 » Thu, Jul 07, 2011, 2:16 AM UTC

schleyfox wrote: post #96

dasil003 wrote: post #71

For all the talk of "best practices" and "training" the depressing truth is that guaranteeing correct software is incredibly difficult and expensive. Professional software engineering practices aren't nearly sufficient to guarantee correctness with heavy math. The closest thing we have is NASA where the entire development process is designed and constantly refined in response to individual issues to create the checks…

I've spent a bit of time on the inside at NASA, specifically working on earth observing systems. There is a huge difference between the code quality of things that go into control systems for spacecraft (even then, meters vs. feet, really?) and the sort of analysis/theoretical code the article talks about. Spacecraft code gets real programmers and disciplined practices, while scientific code is generally spaghetti ID…

Definitely. Obviously the software industry has a lot of know-how that would be invaluable to the science community. The critical point I was trying to make is that scientists need to understand the fundamental difficulty of software correctness before they can be expected to apply best practices effectively.