Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

161–170 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#161

Post by jnxx »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not understand. What is crucial in the argument here is indeed the distinction between methods employing pseudo-randomness, like Monte Carlo simulation, and non-determinism caused by undefined behavior.

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points.

The person which wrote the linked blog post writes that it was a software engineer at google. Unfortunately, that claim is not falsifiable as the person decided to remain anonymous.

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

The claim is that even with the same random seed for the random generator, the program produces different results, and this is explained by the allegation that it runs non-deterministic (in the sense of undefined behavior) in multiple threads. It claims also that it produces significantly different results depending on which output file format is chosen.

If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined.

Personally, I am very doubtful whether this is true, this would be incredibly sloppy by the imperial college scientists. Some more careful analysis by a recognized programmer might be warranted.

However it underlines well the importance of the main topic that scientific code should be open to analysis.

> What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts.

Fully agree with this. But it should try to document its limitations.

Re: Challenge to scientists: does your ten-year-old code still run?

#162

Post by neutronicus »

mmmBacon wrote: post #150
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…

Then you need to re-imagine the system in such a way that junior scientific programmers (i.e. Grad Students) can at least imagine having enough job security for code maintainability to matter, and for PIs to invest in their students' knowledge with a horizon longer than a couple person-years.

Re: Challenge to scientists: does your ten-year-old code still run?

#163

Post by petschge »

vngzs wrote: post #152
petschge wrote: post #129

Earlier quoted context omitted.

It really helps with debugging if your MC code is deterministic for a given input seed. And then you just run for a sufficient number of different seeds to sample the probability space.

Alternatively: seed the program randomly by default, but allow the user to specify a seed as a CLI argument or function argument (for tests). In the common case, the software behaves as expected (random output), but it is reproducible for tests. You can then publish your RNG seed with the commit hash when you release your code/paper, and others may see your results and investigate that particular code execution.

Sure that works too. But word of advice from real life: Print the random seed at the beginning of the run so you can find out which seed caused it to crash or do stupid things.

Re: Challenge to scientists: does your ten-year-old code still run?

#164

Post by neutronicus »

gowld wrote: post #151
arethuza wrote: post #127

Earlier quoted context omitted.

I would regard (from experience) "science ready" code as something that you run just often enough to get the results to create publications. Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers. This kind of reasoning was one of the many reasons I left academic research - I…

If your experiment is not repeatable, it's an anecdote not data. Any effort to write a paper readable for other people, or document the experiment in any way would probably be seen as wasted effort that could be used to create more results. The "don't show your work" argument only makes sense if you are doing PR, not science.

If it's repeatable by you then it's a trade secret, not an anecdote

Re: Challenge to scientists: does your ten-year-old code still run?

#165

Post by suyjuris »

I wrote a tool to visualise algorithms for binary decision diagrams [1], also in an academic context, where the problem was basically the same: Does the code still run in ten years? In particular, the assumption is that I will not be around then, and no one will have any amount of time to spend on maintenance.

In the end, I chose to write it in C++ with minimal dependencies (only X11, OpenGL and stb_truetype.h), with custom GUI, and packed all resources into a single executable.

A lot of effort, but if it causes the application to survive 5x as long then it is probably worth spending twice the time.

[1] https://github.com/suyjuris/obst

Re: Challenge to scientists: does your ten-year-old code still run?

#166

Post by yummypaint »

Not to disagree with any points in the article, but i would point out that the sciences also have cases of very old code being maintained and used in production successfully. For example we still use a kinematics code written in fortran over half a century ago. In practice parts of it get reimplemented in newer projects, but the original still sees use.

Re: Challenge to scientists: does your ten-year-old code still run?

#167

Post by petschge »

atrettel wrote: post #131
hpcjoe wrote: post #86

Short answer: Yes, my 30 year old Fortran code runs (with a few minor edits between f77 and modern fortran), as did my ancient Perl codes. Watching the density functional theory based molecular dynamics zip along at ~2 seconds per time step on my 2 year old laptop, versus the roughly 6k seconds per time step on an old Sun machine back in 1991. I remember the same code getting down to 60 seconds per time step on my de…

> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianne…

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readability of the data for half a century).

Re: Challenge to scientists: does your ten-year-old code still run?

#168

Post by dmlorenzetti »

Jabbles wrote: post #108
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

Hard-coded file paths for input data. File paths hard-coded to use somebody's Google Drive so that it only runs if you know their password. Passwords hard-coded to get around the above problem.

In-code selection statements like `if( True ) {...}`, where you have no idea what is being selected or why.

Code that only runs in the particular workspace image that contains some function that was hacked out to make things work during a debugging session 5 years ago.

Distributed projects where one person wrote the preprocessor, another wrote the simulation software, and a third wrote the analysis scripts, and they all share undocumented assumptions worked out between the three researchers over the course of two years.

Depending on implementation-defined behavior (like zeroing out of data structures).

Function and variable names, like `doit()` and `hold`, which make it hard to understand the intention.

Files that contain thousands of lines of imperative instructions with documentation like "Per researcher X" every 100 lines or so.

Code that runs fine for 6 hours, then stops because some command-line input had the wrong value.

I've seen all of these over the years. Even as a domain expert who has spoken directly with authors and project leads, this kind of stuff makes it very hard to tease out what the code actually does, and how the code corresponds to the papers written about the results.

Re: Challenge to scientists: does your ten-year-old code still run?

#169

Post by djaque »

mmmBacon wrote: post #150
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…

Hello fellow accelerator physicist!

Yes I understand how seeding PRNGs work and I personally do that for my own code for debugging purposes. My point was that not using a fixed seed doesn't invalidate their result. It's just a cheap shot and, to me, demonstrates that the lockdownskeptics author doesn't have a real understanding of the methods being used.

Also, to be clear, I support open science and have some of my own open-source projects out in the wild (which is not the norm in my own field yet). I'm not arguing against releasing code, I'm arguing against OP arguing against this particular piece of code.

Re: Challenge to scientists: does your ten-year-old code still run?

#170

Post by jnxx »

throwaway7281 wrote: post #145
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

> If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings?

Yes, that should be this way.

Also all cases where some company research team goes to a scientific conference and presents a nifty solution for problem X without telling how it was purportedly done, it should be absolutely required to publish code and data for this.

*And that's also something which is broken about software patents - patents are about open knowledge, software which uses such patents is not open - this combination should not be allowed at all).

Post reply on HN