Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

271–280 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#271

Post by dekhn »

solatic wrote: post #142
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

As an ex-scientist who used to run lots of simulations, I really fail to see a truly compelling reason why most numerical results (for publication purposes) truly need to publish (and support) deterministic seeding.

We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.

Re: Challenge to scientists: does your ten-year-old code still run?

#272

Post by CoffeeDregs »

Would be a super-useful to have a sciencecode.com service which is a long-term CI system for scientific code and its required artifacts. Journals could include references to sciencecode.com/xyz and sciencecode.com/abc could be derived from sciencecode.com/xyz. Given Github Actions and Forks, the only thing holding this back is scientists doing it (and, possibly, the HN community helping).

And I get that it's not fun to have your code publicly critiqued but it's also not fun to live lives based on (medical, epidemiological) unpublished, unaudited, unverified code...

EDIT: hell, just post a "HELP HN: science code @ github.com/someone/project" and I'd be surprised if you weren't overwhelmed with offers of help.

Re: Challenge to scientists: does your ten-year-old code still run?

#273

Post by yjftsjthsd-h »

labcomputer wrote: post #263
throwaway7281 wrote: post #145

Earlier quoted context omitted.

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

> In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science".

And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that were accurate to +-30° and reactants from a source known for contamination?

Re: Challenge to scientists: does your ten-year-old code still run?

#274

Post by jbay808 »

SilasX wrote: post #191
throwaway287391 wrote: post #172

Earlier quoted context omitted.

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…

You're right about bit-for-bit reproducibility possibly being overkill, but I don't think that invalidates the parent's point that Monte Carlo randomization doesn't obviate reproducibility concerns. It just means that e.g. your results shouldn't be hypersensitive to the details of the randomization. That is, reviewers should be able to take your code, feed it different random data from a similar distribution to what…

It does seem like a valid response to OP's objection to the imperial college's COVID model, though. Doesn't it?

Re: Challenge to scientists: does your ten-year-old code still run?

#276

Post by sitkack »

matthewdgreen wrote: post #269
sitkack wrote: post #253

Earlier quoted context omitted.

> exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done. I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it…

All of my 2010 scientific code runs on the then-current edition of Docker. /s

I made no mention of Docker, VMs or any virtualization system. Those would be an implementation detail and would obviously change over time.

A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

Re: Challenge to scientists: does your ten-year-old code still run?

#277

Post by JoeAltmaier »

therealx wrote: post #245
JoeAltmaier wrote: post #211

Strangely, they were running (some of ) the code on old hardware. That's hardly a useful case, and much easier than 'resurrecting' the code for modern reuse.

Something with non-standard asm?

That sounds like a big issue. And certainly part of getting 10-year-old code resurrected.

Re: Challenge to scientists: does your ten-year-old code still run?

#279

Post by jpeloquin »

The point of being able to run ten-year-old code is the ability to replay an analysis (exact replication). This allows an analysis to be verified after the fact, which increases trust and helps figure out what happened when contradictions appear between experiments. However, if the original work involved physical experimentation or any non-automated steps (as is the case for most science) the ability to run the original code provides only partial replication. Overall the ability to re-run old code is a fairly low priority.

From the perspective of someone who primarily uses computers as a tool to facilitate research, the priority list is closer to:

1. Retain documentation of what was meant to happen. Objectives, experimental design, experimental & analysis protocols, relevant background, etc.

2. Retain documentation of what actually happened, usually in terms of noting deviations from the protocol. This is the purpose of a lab notebook. Pen & paper excels here.

3. Retain raw data files.

4. Retain files produced in the course of analysis.

5. Retain custom source code.

6. Version control all the above.

7. Make everything run in the correct order with one command (i.e, full automation).

Only once all the above is achieved would it be worth ensuring that the software used in the analysis can be re-run in 10 years. Solving the "packaging problem" in a typical scientific context (multiple languages, multiple OSes, commercial software, mostly short scripts) is complex. When the outcome of an analysis is suspect, the easiest and most robust approach is to check the analysis by redoing it from scratch. This takes less time than trying to ensure every analysis will run on demand even as the computing ecosystem changes out from under it.

Most of the time spent writing analysis code is deciding what the code should do, not actually writing the code. There is generally very little code because few people were involved, and they probably weren't programmers. So redoing the work from scratch is generally pretty easy, especially for anyone with the skill to routinely produce fully reproducible computational environments.

Post reply on HN