Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

41–50 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#41

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

Can you describe a bit more about what is going on in the project? The file you linked is over 2.5k lines of c++ code, and that is just the “setup” file. As you say, this is supposed to be a statistical model, I expected this to be R, Python or one of the standard statistical packages.

Why is there so much c++ code?

Re: Challenge to scientists: does your ten-year-old code still run?

#42
In addition - does your ten year old protocol still work? Do your 10 year old results replicate? This isn't isolates to just programming - making robust and reproducible tools, code, equiptment, protocols, and results is undervalued across all areas of research, leading to situations where protocols published weren't robust so a change in reagent supplier leads to failure, or to protocols so dependent on weird local or unreported environmental conditions or random extra steps that attempting to replicate them leaves you nowhere. Robustness needs to be improved in general.

Re: Challenge to scientists: does your ten-year-old code still run?

#43

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am in 100% agreement and would like to point out that many papers based on code don't even come with code bases, and if they do those code bases are not going to contain or be accompanied by any documentation whatsoever. This is frequently by design as many labs consider code to be IP and they don't want to share it because it gives them a leg up on producing more papers and the shared code won't yield an authorship.

Re: Challenge to scientists: does your ten-year-old code still run?

#44
post #20

The two main problems in academia are that a) few researchers have formal training in best practices of software engineering, and that b) time pressure leads to "whatever worked two minutes before submission deadline" becoming what is kept for posteriority. When I started working as a full-time researcher, I had come from working two years in a software shop, only to find people at the research lab having never used…

Don't you think docker, dependencies, unit test frameworks, etc actually increase the need for ongoing maintenance as opposed to spitting out some C files or python scripts which last "forever"?

I don't think so. The source code is the same but there's now metadata that helps in setting up the same environment again, even years later. You still have the original code in case, e.g. Docker is no longer available.

For instance, if you just have a Python script importing a statistical library, what version are you going to use? Scipy had a pretty nasty change in one of its statistical functions, changing the outcome of significance tests in our project. Depending on which version you happened to have installed it'd give you a positive or negative result.

Re: Challenge to scientists: does your ten-year-old code still run?

#45
> Today, researchers can use Docker containers (see also ref. 7) and Conda virtual environments (see also ref. 8) to package computational environments for reuse.

Docker is also flawed. You can perfectly reproduce it today but what about in 10 years. I can barely go back to our previous release for some dockerfiles.

Re: Challenge to scientists: does your ten-year-old code still run?

#46
post #6

The day when code used to produce a paper must also be published can not come soon enough.

It won't happen until researchers are forced to do it. Please sign petition at https://publiccode.eu and have a look at my other comment here.

Re: Challenge to scientists: does your ten-year-old code still run?

#47
I wrote a C++ implementation of the AMBER force field in 2003. Still have the source code with its original modification times. Let's see:

  /usr/bin/g++   -I/home/dek/sw/rh9/gsl-1.3/include    -c -o NBEnergy.o NBEnergy.cpp
  NBEnergy.cpp: In member function ‘virtual double NBEnergy::Calculate(Coordinates&, std::vector)’:
  NBEnergy.cpp:20:68: error: no matching function for call to ‘find(std::vector::const_iterator, std::vector::const_iterator, const atom*&)’
   20 |       if (std::find(at1->Excluded.begin(), at1->Excluded.end(), at2) != at1->Excluded.end()) 
  {
        |                                                                    ^
  In file included from /usr/include/c++/9/bits/locale_facets.h:48,
                   from /usr/include/c++/9/bits/basic_ios.h:37,
                   from /usr/include/c++/9/ios:44,
                   from /usr/include/c++/9/ostream:38,
                   from GeneralParameters.h:6,
                   from NBEnergy.h:6,
                   from NBEnergy.cpp:1:
  /usr/include/c++/9/bits/streambuf_iterator.h:373:5: note: candidate: ‘template typename __gnu_cxx::__enable_if::__value, std::istreambuf_iterator >::__type std::find(std::istreambuf_iterator, std::istreambuf_iterator, const _CharT2&)’
    373 |     find(istreambuf_iterator __first,
      |     ^~~~
  /usr/include/c++/9/bits/streambuf_iterator.h:373:5: note:   template argument deduction/substitution failed:
  NBEnergy.cpp:20:68: note:   ‘__gnu_cxx::__normal_iterator >’ is not derived from ‘std::istreambuf_iterator’
   20 |       if (std::find(at1->Excluded.begin(), at1->Excluded.end(), at2) != at1->Excluded.end()) 
  {
        |                                                                    ^
  make: *** [: NBEnergy.o] Error 1

I still have a hardcoded reference to RedHat 9 apparently. But the only error has to do with an iterator, so clearly, something in C++ changed. Looks like a 1-2 line change.

Re: Challenge to scientists: does your ten-year-old code still run?

#48

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

Can you describe a bit more about what is going on in the project? The file you linked is over 2.5k lines of c++ code, and that is just the “setup” file. As you say, this is supposed to be a statistical model, I expected this to be R, Python or one of the standard statistical packages. Why is there so much c++ code?

It's a Monte-Carlo simulation, not a statistical model. These are usually written in C++ for performance reasons.

Re: Challenge to scientists: does your ten-year-old code still run?

#49

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am in 100% agreement and would like to point out that many papers based on code don't even come with code bases, and if they do those code bases are not going to contain or be accompanied by any documentation whatsoever. This is frequently by design as many labs consider code to be IP and they don't want to share it because it gives them a leg up on producing more papers and the shared code won't yield an authorshi…

If published research is based on a code base, then surely the documentation and working code is equally important than the carefully written paper.

Re: Challenge to scientists: does your ten-year-old code still run?

#50
post #27

Earlier quoted context omitted.

Arguably, data is just as important. Academics hoard their data and try to milk out every paper they can from it. The reward system is based on publishing as many papers as possible rather than just making a meaningful contribution.

Data is much trickier because your data source for medical, education or even just regular businesses don't want the added legal weight of making data freely available. This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blo…

It seems like there can be a balance between "the results are unverifiable because no one else can touch the data" and "effectively open-source the dataset"?

Something like: "To make it easier to verify the code behind this paper, we've used to generate a synthetic dataset with the same fields as the original and included it with the source code. The isn't comfortable with publishing the full dataset, but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to ..."

Post reply on HN