Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

51–60 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#51

Post by neutronicus »

throwanem wrote: post #12
lordnacho wrote: post #4

You often run into code of the "just get it to work" variety, which has the problem that when it was written, maintainability was bottom of the list of priorities. Often the author has a goal that isn't described in terms of software engineering terms: calculate my option model, work out the hedge amounts, etc. And the people who write this kind of code tend not to think about version control, documentation, dependen…

The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I…

Nah.

The fundamental problem is that scientific code is produced by entry-level developers:

1. Paid below-market wages

2. With no way to move up in the organization

3. With lots of non-software responsibilities

4. With an expectation of leaving the organization in six years

As long as the grunt work of science is done by overworked junior scientists whose careers get thrown to the wolves no matter what they do, you're not going to get maintainable code out of it.

Re: Challenge to scientists: does your ten-year-old code still run?

#52

Post by justinmeiners »

slhck wrote: post #44
justinmeiners wrote: post #38

Earlier quoted context omitted.

Don't you think docker, dependencies, unit test frameworks, etc actually increase the need for ongoing maintenance as opposed to spitting out some C files or python scripts which last "forever"?

I don't think so. The source code is the same but there's now metadata that helps in setting up the same environment again, even years later. You still have the original code in case, e.g. Docker is no longer available. For instance, if you just have a Python script importing a statistical library, what version are you going to use? Scipy had a pretty nasty change in one of its statistical functions, changing the out…

It makes sense that having more information is better than less.

I would argue that they should use no dependencies to avoid this problem entirely, or download them and include them as source in the project, or at least include a note of which version of a major library they used in a README or comment. I think this is what is often done in practice currently.

Perhaps as you are saying, docker is just a stable way to document this stuff formally. But it is a large moving part that assumes a lot of stuff is still on the internet. What if the docker hub image is removed or dramatically changed? What if that OS package manager no longer exists? It just doesn't seem like our software is getting more longevity, but less. I don't know why we would bring that extra complexity to academic research if the goal is longevity.

Re: Challenge to scientists: does your ten-year-old code still run?

#53

Post by disgruntledphd2 »

paperwork wrote: post #41
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

Can you describe a bit more about what is going on in the project? The file you linked is over 2.5k lines of c++ code, and that is just the “setup” file. As you say, this is supposed to be a statistical model, I expected this to be R, Python or one of the standard statistical packages. Why is there so much c++ code?

Because much of this code was written in the 80's, I suspect. In general, there's a bunch of really old scientific codebases in particular disciplines because people have been working on these problems for a looooonnngg time.

Re: Challenge to scientists: does your ten-year-old code still run?

#54

Post by nanddalal »

GitHub offers a free tier for GitHub actions with 2,000 Actions minutes/month [1]. This could be useful:

1. write some unit tests which don't use too much compute resources (so you can stick to the free tier)

2. package your code into a docker where the tests can be run

3. wire up the docker with tests to GitHub Actions

This way now you have continuous testing and can make sure your codes keep running.

References:

[1] https://github.com/pricing

Re: Challenge to scientists: does your ten-year-old code still run?

#55

Post by recursivecaveat »

paperwork wrote: post #41
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

Can you describe a bit more about what is going on in the project? The file you linked is over 2.5k lines of c++ code, and that is just the “setup” file. As you say, this is supposed to be a statistical model, I expected this to be R, Python or one of the standard statistical packages. Why is there so much c++ code?

It is essentially a detailed simulation of viral spread, not just a programmed distribution or anything. It's all in C++ because it's pretty performance-critical.

Re: Challenge to scientists: does your ten-year-old code still run?

#56

Post by OminousWeapons »

acutesoftware wrote: post #49
OminousWeapons wrote: post #43

Earlier quoted context omitted.

I am in 100% agreement and would like to point out that many papers based on code don't even come with code bases, and if they do those code bases are not going to contain or be accompanied by any documentation whatsoever. This is frequently by design as many labs consider code to be IP and they don't want to share it because it gives them a leg up on producing more papers and the shared code won't yield an authorshi…

If published research is based on a code base, then surely the documentation and working code is equally important than the carefully written paper.

I completely agree, the problem is the journal editors and reviewers largely don't.

Re: Challenge to scientists: does your ten-year-old code still run?

#57

Post by detaro »

throwanem wrote: post #30
detaro wrote: post #19

Earlier quoted context omitted.

And that scientists also are rarely supported by programmers, or if they are it's an unstable and unappreciated position.

Having had that exact experience - yeah, that can be a big problem too. Researchers and engineers can work really well together, because the strengths of each role complement the weaknesses of the other, and I think it would be very nice to see that actually happen some day.

It doesn't help with the issue of hard-to-reproduce work, but apparently working for a company making products aimed at scientists can be a place to see this happen (if the company is good about talking to customers).

Re: Challenge to scientists: does your ten-year-old code still run?

#59

Post by MattGaiser »

nanddalal wrote: post #54

GitHub offers a free tier for GitHub actions with 2,000 Actions minutes/month [1]. This could be useful: 1. write some unit tests which don't use too much compute resources (so you can stick to the free tier) 2. package your code into a docker where the tests can be run 3. wire up the docker with tests to GitHub Actions This way now you have continuous testing and can make sure your codes keep running. References: [1…

Even if it broke, who would go back and fix it?

I do not see that happening, especially with complex library bugs.

Re: Challenge to scientists: does your ten-year-old code still run?

#60

Post by bumby »

dandelion_lover wrote: post #40
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

As a theoretical physicist doing computer simulations, I am trying to publish all my code whenever possible. However all my coauthors are against that. They say things like "Someone will take this code and use it without citing us", "Someone will break the code, obtain wrong results and blame us", "Someone will demand support and we do not have time for that", "No one is giving away their tools which make their compe…

I’m curious, are dedicated software assurance teams a thing in your research area? Or is quality left up to the primary researchers?
Post reply on HN