Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

21–30 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#21

Post by jgeada »

goalieca wrote: post #11
cube00 wrote: post #6

The day when code used to produce a paper must also be published can not come soon enough.

Arguably, data is just as important. Academics hoard their data and try to milk out every paper they can from it. The reward system is based on publishing as many papers as possible rather than just making a meaningful contribution.

The reward system also prevents dead ends from being identified, publication of approaches that did not lead to the expected results or got nul results, publishing confirmations of prior papers, etc.

Basically, the reward system is designed to be easy to measure and administer, but is not actually useful in any way to the advancement of science.

Re: Challenge to scientists: does your ten-year-old code still run?

#22

Post by zimbatm »

matsemann wrote: post #5

Would an abandoned project I wrote 10 years ago still run? The code is probably fine, but getting it to actually run by linking up whatever libraries, sdks and environment correctly could be troublesome. Even a small pipeline a wrote a few weeks ago I had trouble re-running, because I forgot there was a manual step I had to do on the input file. Expecting more rigid software practices of scientists than software engi…

If the same project had been packaged with Nix, it would probably still compile. People regularly checkout older versions of nixpkgs to get access to older package releases.

One of the key property is that the build system enforces all the build inputs to be declared. And the other one is to keep a cache of all the build inputs like sources because upstream repositories tend to disappear over time.

Re: Challenge to scientists: does your ten-year-old code still run?

#23

Post by non-entity »

detaro wrote: post #19
throwanem wrote: post #12

Earlier quoted context omitted.

The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I…

And that scientists also are rarely supported by programmers, or if they are it's an unstable and unappreciated position.

Ive seen job listings for "scientific programmers" where what they're asking for is a scientist who happens to know a little programming.

Re: Challenge to scientists: does your ten-year-old code still run?

#24

Post by Sebb767 »

As someone who worked with bits of scientific code: Does the code you write right now work on another machine might be the more appropriate challenge. If seen a lot of hardcoded paths, unmentioned dependencies and monkey-patched libraries downloaded from somewhere; just getting the new code to work is hard enough. And let's not even begin to talk about versioning or magic numbers.

Similar to other comments I don't mean to fault scientists for that - their job is not coding and some of the dependencies come from earlier papers or proprietary cluster setups and are therefore hard to avoid - but the situation is not good.

Re: Challenge to scientists: does your ten-year-old code still run?

#25

Post by justinmeiners »

lordnacho wrote: post #4

You often run into code of the "just get it to work" variety, which has the problem that when it was written, maintainability was bottom of the list of priorities. Often the author has a goal that isn't described in terms of software engineering terms: calculate my option model, work out the hedge amounts, etc. And the people who write this kind of code tend not to think about version control, documentation, dependen…

Unit testing, readability version control, documentation, etc are all engineering practices for the purpose of making ongoing development organized (especially for teams).

Why would a researcher need to do this, when in most cases all that they use is the output, and in CS/math it's only a minimal prototype demonstrating operation of their principle?

All of the other stuff would certainly be nice, but they don't need to adopt our whole profession to write code

Re: Challenge to scientists: does your ten-year-old code still run?

#26

Post by qppo »

WanderPanda wrote: post #14
cube00 wrote: post #6

The day when code used to produce a paper must also be published can not come soon enough.

Making this mandatory might have bad downstream effects like prohibiting publication of some research at all (GPT-X I am looking at you)

Closed source research isn't publication, it's advertisement.

Re: Challenge to scientists: does your ten-year-old code still run?

#27

Post by belval »

goalieca wrote: post #11
cube00 wrote: post #6

The day when code used to produce a paper must also be published can not come soon enough.

Arguably, data is just as important. Academics hoard their data and try to milk out every paper they can from it. The reward system is based on publishing as many papers as possible rather than just making a meaningful contribution.

Data is much trickier because your data source for medical, education or even just regular businesses don't want the added legal weight of making data freely available.

This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blocked it because the information is too sensitive.

Re: Challenge to scientists: does your ten-year-old code still run?

#29

Post by arcanus »

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

> Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs.

You are blaming scientists but speaking from my personal experience as a computational scientist, this exists because there are few structures in place that incentivize strong programming practices.

* Funding agencies do not provide support for verification and validation of scientific software (typically)

* Few journals require assess code reproducibility and few require public code (few require even public data)

* There are few funded studies to reproduce major existing studies

Until these structural challenges are addressed, scientists will not have sufficient incentive to change their behavior.

> Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals.

I completely agree.

Re: Challenge to scientists: does your ten-year-old code still run?

#30

Post by throwanem »

detaro wrote: post #19
throwanem wrote: post #12

Earlier quoted context omitted.

The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I…

And that scientists also are rarely supported by programmers, or if they are it's an unstable and unappreciated position.

Having had that exact experience - yeah, that can be a big problem too.

Researchers and engineers can work really well together, because the strengths of each role complement the weaknesses of the other, and I think it would be very nice to see that actually happen some day.

Post reply on HN