Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

71–80 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#71

Post by eythian »

dhosek wrote: post #39
tyingq wrote: post #8

Earlier quoted context omitted.

It's interesting that it's often easier to get something 25+ years old running because I need fewer things. Not so hard to find, say "DosBox" and and old version of Turbo Pascal.

When I was in my 20s I managed to get a contract updating some control software for a contact lens company on the basis of my happening to own an old copy of Borland C++ 1.0.

Had a similar experience getting a contract updating a mass spectrometer control system because I had extensive high school experience in Turbo Pascal.

Re: Challenge to scientists: does your ten-year-old code still run?

#72

Post by Sulfolobus »

goalieca wrote: post #45

> Today, researchers can use Docker containers (see also ref. 7) and Conda virtual environments (see also ref. 8) to package computational environments for reuse. Docker is also flawed. You can perfectly reproduce it today but what about in 10 years. I can barely go back to our previous release for some dockerfiles.

Similarly, conda envs can break in weeks due to package changes.

Even if you remove build versioning and all transitive dependencies from your env (making it less reproducible...) they will break pretty damn quick.

Re: Challenge to scientists: does your ten-year-old code still run?

#74

Post by xen0 »

jhrmnn wrote: post #64
cube00 wrote: post #6

The day when code used to produce a paper must also be published can not come soon enough.

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas h…

My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result.

And if that means scientists need to learn about version control, well... they should if they're writing code.

Re: Challenge to scientists: does your ten-year-old code still run?

#75

Post by wdwvt1 »

An excellent article full of good suggestions. I appreciated that it's less certain of the Best Practices TM than many comments on this subject. I am curious how the goals/techniques for reproducibility change with the percentage of software/computational work that a scientific project contains. It feels like as the percentage of a paper's ultimate conclusions that are computationally derived increases, the importance of strict "the tests pass and the numerical results are identical" reproducibility also increases. Most of my projects are mixed wet-lab/dry-lab - a fair amount of custom code is required, but it's usually less than 50% of the work. When I'm relying on other papers that have a similar mix of things, I'm often not interested if the continuous integration tests of their code pass. I am more interested in understanding well the specific steps they take computationally and in a sensitivity analysis of their computational portion (if you slightly alter your binning threshold do you still get that fantastic clustering?). I believe this is because in my field (microbiology), computational tools can guide, but physical reality and demonstrated biology are the only robust evidence of a phenomenon/mechanism/etc. For most research I do not demand tests of all the analytical pieces they are relying on (was their incubator actually set to 37C? was the pH of the media +- 0.2? etc) - I trust they've done good science. Why would I demand their code meet a higher standard?

Re: Challenge to scientists: does your ten-year-old code still run?

#76

Post by freeone3000 »

acutesoftware wrote: post #49
OminousWeapons wrote: post #43

Earlier quoted context omitted.

I am in 100% agreement and would like to point out that many papers based on code don't even come with code bases, and if they do those code bases are not going to contain or be accompanied by any documentation whatsoever. This is frequently by design as many labs consider code to be IP and they don't want to share it because it gives them a leg up on producing more papers and the shared code won't yield an authorshi…

If published research is based on a code base, then surely the documentation and working code is equally important than the carefully written paper.

No, the paper is what matters. The code is a means to generate the paper.

Re: Challenge to scientists: does your ten-year-old code still run?

#77

Post by stillsut »

It's not academia but Kaggle that's really been on the forefront of building portable and reproducible computational pipelines.

The real key is incentives and there are two that standout to me:

- Incentive to get others to "star" and fork your code makes the coder compete to not only have an accurate result, but also prioritize producing code/notebooks that are digestible and instructive. That includes liberal commenting/markup, idiomatic syntax and patterns, diagnostic figures, and the use of modern and standard libraries.

- There is an incentive to move with the community on best practices for the libraries while still allowing experimental libraries. Traditionally, there is the incentive of inertia: e.g. "I always do my modelling in Lisp, and I won't change because then I'd be less productive". But with kaggle, to learn from the insights and advances of others, you need to have an ability to work with the developing common toolset.

In academia, if these incentives were given weight on par with publication and citation then we'd see the tools and practices fall into place.

Re: Challenge to scientists: does your ten-year-old code still run?

#78

Post by jnxx »

Very related to this, see also Hinsens blog post: http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabilit...

I think that GNU Guix is extremely well-suited to improve this situation.

Also, one could think this is an academic problem, in the sense of am otherwise unimportant niche problem. It really isn't, it is just like in many other topics that academics get confronted first with this issue. I am sure that in many medium or large companies there are some Visual Basic or Excel code bases which are important but could turn out extremely hard to reproduce. This issue will only get more burning with today's fast-moving ecosystems where backward-compatibility is more a moral ideal than an enforced requirement.

It is well known that ransomware can wipe-out businesses if critical business data is lost. But more and more businesses and organizations also have critical, and non-standard, software.

Re: Challenge to scientists: does your ten-year-old code still run?

#79

Post by josefx »

dekhn wrote: post #47

I wrote a C++ implementation of the AMBER force field in 2003. Still have the source code with its original modification times. Let's see: /usr/bin/g++ -I/home/dek/sw/rh9/gsl-1.3/include -c -o NBEnergy.o NBEnergy.cpp NBEnergy.cpp: In member function ‘virtual double NBEnergy::Calculate(Coordinates&, std::vector )’: NBEnergy.cpp:20:68: error: no matching function for call to ‘find(std::vector ::const_iterator, std::vec…

You probably didn't include the algorithm header that defines find directly and it stopped compiling once the standard library maintainers cleaned up their own includes. The iostreams headers you include define their own stream iterator specific overload of find and that doesn't match.

Re: Challenge to scientists: does your ten-year-old code still run?

#80

Post by jnxx »

goalieca wrote: post #45

> Today, researchers can use Docker containers (see also ref. 7) and Conda virtual environments (see also ref. 8) to package computational environments for reuse. Docker is also flawed. You can perfectly reproduce it today but what about in 10 years. I can barely go back to our previous release for some dockerfiles.

Guix is arguably better.
Post reply on HN