Challenge to scientists: does your ten-year-old code still run? • Hacker News Board

Re: Challenge to scientists: does your ten-year-old code still run?

#201

Post by jack_h » Mon, Aug 24, 2020, 4:43 PM UTC

BeetleB wrote: post #82

bumby wrote: post #60

Earlier quoted context omitted.

I’m curious, are dedicated software assurance teams a thing in your research area? Or is quality left up to the primary researchers?

> Or is quality left up to the primary researchers? Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality. I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

I think this is a much wider problem than just in academia/research. Really any area where software isn't the primary product tends to have fairly lax software standards. I work in the embedded firmware field and best practices are often looked at with skepticism and even derision by the electrical engineers who are often the ones doing the programming^[1].

I think software development as a field is incredibly vast and diverse. Programming is an amazing tool, but it's a tool that requires a lot of knowledge in a lot of different areas.

^[1] This isn't universally true of course, I'm not trying to be insulting here.

Re: Challenge to scientists: does your ten-year-old code still run?

#202

Post by jnxx » Mon, Aug 24, 2020, 4:44 PM UTC

jonathanstrange wrote: post #110

I think it's unfair to expect from anyone to maintain code forever when the code rot is completely beyond your control, let alone to expect this from scientists who have better things to do. Anything with a GUI is bound to self-destruct, for example, and it's not the programmer's fault. Blame the OS makers and framework/3rd party library suppliers. The damage can be limited by choosing a programming language that pro…

> CommonLisp

Common Lisp has fantastic long-term stability. I think that deserves more recognition, as Common Lisp is often almost as fast as C, but is (by default) not riddled with undefined behavior.

It would be superb if Rust could take C's space in computational science and libraries.

Re: Challenge to scientists: does your ten-year-old code still run?

#203

Post by proverbialbunny » Mon, Aug 24, 2020, 4:45 PM UTC

This seems like a fluff piece because:

1) Prototype code scientists write tends to be written at a high level, so barring imported libraries not up and disappearing, there is a high chance that code written by scientists will run 10 years later. There is a higher chance it will run than production code written at a lower level.

2) The article dives into documentation but scientists code in the Literate Programming Paradigm[0] where the idea is you're writing a book and the code is used as examples to support what you're presenting. Of course scientists write documentation. Presenting findings is a primary goal.

3) Comments here have mentioned unit testing. Some of you may scoff at this but when prototyping, every time you run your code, the output from it teaches you something, and that turns into an iterative feedback loop, so every time you learn something you want to change the code. Unit tests are not super helpful when you're changing what the code should be doing every time you run it. Unit tests are better once the model has been solidified and is being productionized. Having a lack of unit testing does not make 10 year old prototype code harder to run.

[0] https://en.wikipedia.org/wiki/Literate_programming

Re: Challenge to scientists: does your ten-year-old code still run?

#204

Post by qppo » Mon, Aug 24, 2020, 4:47 PM UTC

Jabbles wrote: post #108

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

Disclaimer, I'm a professional engineer and not a researcher.

The kind of code I'll ship for production will include unit testing designed around edge or degenerate cases that arose from case analysis, usually some kind of end to end integration test, aggressive linting and crashing on warnings, and enforcing of style guidelines with auto formatting tools. The last one is more important than people give it credit for.

For research it would probably be sufficient to test that the code compiles and given a set of known valid input the program terminates successfully.

Re: Challenge to scientists: does your ten-year-old code still run?

#205

Post by vharuck » Mon, Aug 24, 2020, 4:47 PM UTC

magv wrote: post #106

An interesting concern is that there often is no single piece of code that has produced the results of a given paper. Often it is a mixture of different (and evolving) versions of different scripts and programs, with manual steps in between. Often one starts the calculation with one version of the code, identifies edge cases where it is slow or inaccurate, develops it further while the calculations are running, does…

>the process it interactive, and not trivially repeatable.

The kind of interaction you're describing should be frowned upon. It requires the audience to trust the manual data edits are no different than rerunning the analysis. But the researcher should just rerun the analysis.

Also, mixing old and new results is a common problem in manually updated papers. It can be avoided by using reproducible research tools like R Markdown.

Re: Challenge to scientists: does your ten-year-old code still run?

#206

Post by izacus » Mon, Aug 24, 2020, 4:48 PM UTC

prionassembly wrote: post #139

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

Institutions need to provide scientists and mathematicians with coders. It's a bit insane to expect them to be software engineers as well.

Noone expects them to be software engineers, but we do expect them to be _scientists_ - to publish results that are reproducible and verifiable. And that has to hold for code as well.

Re: Challenge to scientists: does your ten-year-old code still run?

#207

Post by geoalchimista » Mon, Aug 24, 2020, 4:49 PM UTC

arcanus wrote: post #29

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

> Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. You are blaming scientists but speaking from my personal experience as a computational scientist, this exists because there are few structures in place that incentivize strong programming practices. * Funding agencies d…

Second this. Research code is already hard, and with misaligned incentives from the funding agencies and grad school pipelines, it's an uphill battle. Not to mention that professors with an outdated mindset might discourage graduate students from committing too much time to work on scientific code. "We are scientists, not programmers. Coding doesn't advance your career" is often an excuse for that.

In my opinion, enforcing standards without addressing this root cause is not gonna fix the problem. Worse, students and early career researchers will bear the brunt of increased workload and code compliance requirements from journals. Big, well-funded labs that can afford a research engineer position is gonna have an edge over small labs that cannot do so.

Re: Challenge to scientists: does your ten-year-old code still run?

#208

Post by throwaway287391 » Mon, Aug 24, 2020, 4:50 PM UTC

SilasX wrote: post #191

throwaway287391 wrote: post #172

Earlier quoted context omitted.

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…

You're right about bit-for-bit reproducibility possibly being overkill, but I don't think that invalidates the parent's point that Monte Carlo randomization doesn't obviate reproducibility concerns. It just means that e.g. your results shouldn't be hypersensitive to the details of the randomization. That is, reviewers should be able to take your code, feed it different random data from a similar distribution to what…

That brings up a separate issue that I didn't comment on above: the expectation that the code runs in a completely different development/execution environment (e.g. the one the reviewer is using vs. the one that the researcher used). That means making it run regardless of the OS (Windows/OSX/Linux/...) and hardware (CPU/GPU/TPU, and even within those, which one) the reviewer is using. This would be an extremely difficult if not impossible thing for even a professional software engineer to achieve. It could easily be a full time job. There are daily issues on even the most well-funded projects in machine learning by huge companies (ex: TF, PyTorch) that the latest update doesn't work on GPU X or CUDA version Y or OS Z. It's not a realistic expectation for a researcher even in computer science, let alone researchers in other fields, most of whom are already at the top of the game programming-wise if they would even think to reach for a "script" to automate repetitive data entry tasks etc.

==========

EDIT to reply to BadInformatics' reply below (I'm being rate-limited): I fully agree that a lot of ML code releases could be better about this, and it's even reasonable to expect them to do some of these more basic things like you mention. I don't agree that bit-for-bit reproducibility is a realistic standard that will get us there.

Re: Challenge to scientists: does your ten-year-old code still run?

#209

Post by jnxx » Mon, Aug 24, 2020, 4:50 PM UTC

neuromantik8086 wrote: post #153

Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge ( https://www.nature.com/articles/d41586-019-03296-8 ), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language ( https://www.com…

Konrad Hinsen is an expert in molecular bioinformatics and also has significantly contributed to Numerical Python, for example, and has extensively published around the topic of reproducible science and algorithms - see his blog.

The fact that he might favor different solutions from you does not mean that he is pushing some kind of hidden agenda.

If you think that Common Workflow Language is a better solution, you are free to explain in a blog why you think this.

Are you saying that the reproductive challenge poses a difficulty to Common Workflow Language? If this is so, would that not rather support Hinsen's point - without implying that what he suggests is already a perfect solution?

Re: Challenge to scientists: does your ten-year-old code still run?

#210

Post by dandelion_lover » Mon, Aug 24, 2020, 4:51 PM UTC

SiempreViernes wrote: post #198

dandelion_lover wrote: post #68

Earlier quoted context omitted.

Most of the codes I am developing alone. No one else looks at them ever. My supervisor also develops the code alone and never shows it to anyone (not even members of the group). In other cases, a couple of other researchers may have a look at my code or continue its development. I worked with 4+ research teams and only saw one professional programmer in one of them helping the development. Never heard about a "dedica…

To clarify, nobody sees the code because they aren't allowed, or nobody ever ask to see it?

The second case. However I am hesitating to ask to look at the code of my supervisor. How would I explain why I need it (if it's not needed for my research)? It's also unlikely user-friendly, so it would take a lot of time to understand anything.