Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

261–270 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#261

Post by Rochus »

O_H_E wrote: post #250
Rochus wrote: post #132

Earlier quoted context omitted.

Yes, I know a couple of Fortran 77 apps and libraries which were developed more than 25 years ago and which are still in use today. My C++ Qt GUI application for NMR spectrum analysis ( https://github.com/rochus-keller/CARA ) runs since 20 years now with continuing high download and citation rates. So obviously C++/Qt or Fortran 77 are very well suited to outlast time.

Nice. Interesting to know that Github starts aren't always a representative metric.

Yes, many of my apps and libs were more than ten years old when I pushed them to github. Some projects started before git was invented.

Re: Challenge to scientists: does your ten-year-old code still run?

#262

Post by jnxx »

therealx wrote: post #252
jnxx wrote: post #84

Earlier quoted context omitted.

As long as it does not use some CUDA hardware which is using tensorflow and Numba which is using a version of llvmlite which does not support Python2 any more..... This isn't a theoretical example.

Then you make a vm or whatnot and install all the old versions of everything. I haven't seen an open source project in a while that doesn't have old versions for download easily. Still, annoying if you can't stand that kind of stuff. (people seem to be in two camps: either they hate it or have almost no problem with it)

Well, where do you download the hardware ? ;-)

Re: Challenge to scientists: does your ten-year-old code still run?

#263

Post by labcomputer »

throwaway7281 wrote: post #145
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science".

In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas.

> how do you expect anyone with the right expertise to assess your findings

I would expect other experts in the field to write their own implementation of the scientific ideas expressed in a paper. If the idea has any merit, their implementations should produce similar results. Which is exactly what they would do if it were a physical experiment.

Re: Challenge to scientists: does your ten-year-old code still run?

#264

Post by garden_hermit »

I favor open code, but like everything, there are issues. For example, the EPA years ago required that research can only inform policy when data is open; open data, however, takes a lot of effort to document and provide. Companies, however, with vested interest in EPA policy can easily produce open (and often very biased) data.

Requirements for open code can lead to similar issues—what happens when a government agency rejects the outcome of a supercomputer simulation because the code wasn't documented well enough? What happens when those with vested interests are the ones best able to produce scientific code?

Scientists already wear many hats. Any shift in policy and norms needs to consider that they have limited time, a fact that can have far-reaching consequences.

Re: Challenge to scientists: does your ten-year-old code still run?

#265

Post by wenc »

The easiest way to preserve code for posterity is the wrap up the runtime environment in a VM. I can boot up a VM from 15 years ago (when I was in grad school) and it will run.

When you're writing code for science, preserving code for posterity is rarely a priority. Your priority is to iterate quickly because the goal is scientific results, not code.

(this is in fact, correct prioritization. Under most circumstances, though not all, most grad students who try to write pristine code find themselves progressing more slowly than those who don't.)

Re: Challenge to scientists: does your ten-year-old code still run?

#266

Post by Bukhmanizer »

stult wrote: post #228
smnrchrds wrote: post #173

Earlier quoted context omitted.

> Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily? A non-native English speaker may make grammatical mistakes when communicating their research in English—it does not in any way invalidate their results or hint that there is anything amiss. It is simply what happens when you are a non-native s…

Journals employ copy editors to address just those sorts of mistakes, why should we not hold software to the same standard as academic language? But more importantly, these software best practices aren't mere "grammatical mistakes," they exist because well-organized, well-tested code has fewer bugs and is easier for third parties to verify. Third-parties validating that the code underlying an academic paper executes…

I’m glad someone else feels this way. It’s an expectation that scientists can share their with other scientists using language. Scientists aren’t always the best writers, but there are standards there. Writing good code is a form of communication. It baffles me that there are absolutely no standards there.

Re: Challenge to scientists: does your ten-year-old code still run?

#267

Post by O_H_E »

roberto wrote: post #236

For my first scientific article, in 2007, I created a Subversion repo with a Makefile. Running `make` would recreate the whole paper: downloading data, running analyses, creating pictures (color or BW, depending on an environment flag) and generating the PDF. I'm going to try to find the repo and see if it still works.

Wow, nice. I will be waiting :D

Re: Challenge to scientists: does your ten-year-old code still run?

#268

Post by searine »

Jabbles wrote: post #146
searine wrote: post #134

Earlier quoted context omitted.

>I am interested to know the distinction between "production-ready" and "science-ready" code. In general, scientists don't care how long it takes or how many resources the code uses. It is not a big deal to run a script for an extra hour, or use up a node of supercomputer. Extravagent solutions or added packages to make the code run smoother or faster is only wasting time. It speed/elegance only really matters when y…

Do you know how you could get to the state that "the code was proven to produce the correct result"? If not by unit tests, code review or formal logic, then what?

>If not by unit tests, code review or formal logic, then what?

Cross referencing independent experiments and external datasets.

Science doesn't work like software. The code can be perfect and still not give results that reflect reality. The code can be logical and not reflect reality. Most scientists I know go in with the expectation that "the code is wrong" and its results must be validated by at least one other source.

Re: Challenge to scientists: does your ten-year-old code still run?

#269

Post by matthewdgreen »

sitkack wrote: post #253
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done. I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it…

All of my 2010 scientific code runs on the then-current edition of Docker. /s

Re: Challenge to scientists: does your ten-year-old code still run?

#270

Post by jbay808 »

improbable22 wrote: post #171
mmmBacon wrote: post #150

Earlier quoted context omitted.

Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…

> Monte-Carlo can and should be deterministic and repeatable I guess it can be made so, but not necessarily easy / fast (if it's parallel, and sensitive to floating point rounding). And sounds like the kind of engineering effort GP is saying isn't worth it. Re-running exactly the same monte-carlo chain does tell you something, but is perhaps the wrong level to be checking. Re-running from a different seed, and gettin…

I guess the best thing would be that it uses a different random seed every time it's run (so that, when re-running the code you'll see similar results which verifies that the result is not sensitive to the seed), but the particular seed that produced the particular results published in a paper is noted.

But still, for code running on different machines, especially for numeric-heavy code that might be running on a particular GPU setup, distributed big data source (where you pull the first available data rather than read in a fixed order), or even on some special supercomputer, it's hard to ask that it be totally reproducible down to the smallest rounding error.

Post reply on HN