Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

111–120 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#111

Post by BeetleB »

djaque wrote: post #105
BeetleB wrote: post #87

Earlier quoted context omitted.

> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

> The point is that as a scientist your code is a tool to get the job done and not the product.

Everything you say is as true for experimental equipment and mathematical tools. Physicists are fantastic at mathematics, yet are one of the most anti-math people I know - in the sense of "Mathematics is just a tool to get results that explain nature! Doing mathematics for its own sake is a waste of time!"

The equation is not the product - the explanation of physical phenomena is. If the attitude of "I don't need to show how I got this equation" is unacceptable, the same should go for code.

Re: Challenge to scientists: does your ten-year-old code still run?

#112

Post by klyrs »

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I do research for a private company, and open-source as much of my work as I can. It's always a fight. So I'll take their side for the moment.

Many years ago, a paper on the PageRank algorithm was written, and the code behind that paper was monetized to unprecedented levels. Should computer science journals also require working proof of concept code, even if that discourages companies from sharing their results; even if it prevents students from monetizing the fruits of their research?

Re: Challenge to scientists: does your ten-year-old code still run?

#113

Post by BeetleB »

Sebb767 wrote: post #99
BeetleB wrote: post #87

Earlier quoted context omitted.

> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).

On one hand, yes. But on the other hand, reuseable code, dependency management, linting, portability etc are not that easy problems and something junior developers tend to struggle with (and its not like that problem never pops up for seniors, either). I really can't fault non-compsci scientist for not handling that problem well. Of course, part of it (like publishing the relevant code) is far easier and should be do…

> But on the other hand, reuseable code, dependency management, linting, portability etc are not that easy problems and something junior developers tend to struggle with

On the original hand, these are easier problems than all the years of math education they have. Once you're relying on simulations to get results to explain natural phenomena, it needs to be put on the same pedestal as mathematics.

Re: Challenge to scientists: does your ten-year-old code still run?

#116

Post by SilasX »

dandelion_lover wrote: post #40
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

As a theoretical physicist doing computer simulations, I am trying to publish all my code whenever possible. However all my coauthors are against that. They say things like "Someone will take this code and use it without citing us", "Someone will break the code, obtain wrong results and blame us", "Someone will demand support and we do not have time for that", "No one is giving away their tools which make their compe…

>"Someone will demand support and we do not have time for that",

Well ... that part isn't nonsense, though I agree it shouldn't be a dealbreaker. And it means we should work towards making such support demands minimal or non-existent via easy containerization.

I note with frustration that even the Docker people, whose entire job is containerization, can get this part wrong. I remember when we containerized our startup's app c. 2015, to the point that you should be able to run it locally just by installing docker and running `docker-compose up`, and it still stopped working within a few weeks (which we found when onboarding new employees), which required a knowledgeable person to debug and re-write.

(They changed the spec for docker-compose so that the new version you'd get when downloading Docker would interpret the yaml to mean something else.)

Re: Challenge to scientists: does your ten-year-old code still run?

#117

Post by ativzzz »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

While you're running experiments, it doesn't matter, but publishing any sort of result or using your code in parts of other publishable code IS production code, and you should treat it as such.

Re: Challenge to scientists: does your ten-year-old code still run?

#118

Post by BeetleB »

core-questions wrote: post #96
BeetleB wrote: post #82

Earlier quoted context omitted.

> Or is quality left up to the primary researchers? Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality. I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

> protested when version control was suggested Academics are strange like this. The root reason is fear: fear that you're complicating their process, that you're going to interrupt their productivity or flow state, that you're introducing complication that has no benefit. They then build up a massive case in their minds for why they shouldn't do this; good luck fighting it. Doubly so if you're IT staff and don't have…

> The root reason is fear: fear that you're complicating their process, that you're going to interrupt their productivity or flow state, that you're introducing complication that has no benefit.

Yes, but how does it compare to all the complicated processes that exist in academic institutions currently? Almost all of which originated from academics themselves, mind you.

Re: Challenge to scientists: does your ten-year-old code still run?

#119

Post by Jabbles »

djaque wrote: post #105
BeetleB wrote: post #87

Earlier quoted context omitted.

> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

How do you know it won't give you results? Maybe it will find a bug that would have resulted in an embarrassing retraction.

Maybe it wouldn't find any bugs, but give confidence to and encourage other users and increasing your citations and "impact".

Maybe it will just save you 48h later on when you need to adapt the code.

Software engineering has generally accepted that unit testing is a good practice and well worth the time taken. Why do you think science is different?

Re: Challenge to scientists: does your ten-year-old code still run?

#120

Post by chriswarbo »

xen0 wrote: post #74
jhrmnn wrote: post #64

Earlier quoted context omitted.

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas h…

My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.

I agree, except that AFAIK "tags" in git are not fixed, they can be deleted and re-created to point at a different commit. Hence I prefer to use (short) commit IDs, since changing them is infeasible.
Post reply on HN