Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

361–370 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#361

Post by onhn »

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

There is a fundamental reason not to publish scientific code.

If someone is trying to reproduce someone else's results, the data and methods are the only ingredients they need. If you add code into this mix, all you do is introduce new sources of bias.

(Ideally the results would be blinded too.)

Re: Challenge to scientists: does your ten-year-old code still run?

#362

Post by modeless »

Let's not criticize people who release their code. Let's criticize the people who don't release their code instead. We don't need more barriers to releasing code.

I'd much rather fix someone's broken build than reimplement a whole research paper from scratch without the relevant details that seem to always be accidentally omitted from the paper.

Re: Challenge to scientists: does your ten-year-old code still run?

#364

Post by petschge »

iagovar wrote: post #348
petschge wrote: post #167

Earlier quoted context omitted.

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readabil…

Why not dumping into SQLite? It makes everything easy, and we will be able to use sqlite3 for a long time IMO.

Because parallel IO from a lot of different MPI ranks is not supported. And filesystems tend to look unhappy when 100k processes try to open a new file at the same time.

Re: Challenge to scientists: does your ten-year-old code still run?

#365

Post by tanilama »

justinmeiners wrote: post #38
slhck wrote: post #20

The two main problems in academia are that a) few researchers have formal training in best practices of software engineering, and that b) time pressure leads to "whatever worked two minutes before submission deadline" becoming what is kept for posteriority. When I started working as a full-time researcher, I had come from working two years in a software shop, only to find people at the research lab having never used…

Don't you think docker, dependencies, unit test frameworks, etc actually increase the need for ongoing maintenance as opposed to spitting out some C files or python scripts which last "forever"?

No.

Python/C files didn't work in a vacuum. They need dependencies, that is the point of Docker after all.

Capture all necessary dependencies into a single image.

Re: Challenge to scientists: does your ten-year-old code still run?

#366

Post by a_zaydak »

solatic wrote: post #142
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

I do agree with you on publishing seeds for Monte Carlo simulations however the argument against it is also very strong. Usually when you run a monte carlo simulation you are quoting the results in terms of statistics. I think it would be sufficient to say that you can 'reproduce' the results as long as your statistics (over many simulations with different seeds) is consistent with the published results. If you run a single simulation with are particular seed you should get the same results however this might be cherry picking a particular simulation result. This is good for code testing but probably not for scientific results. I think by running the code with new seeds is a better way to test the science.

Re: Challenge to scientists: does your ten-year-old code still run?

#367

Post by throwaway287391 »

dnautics wrote: post #300
throwaway287391 wrote: post #196

Earlier quoted context omitted.

Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).

Most scientific runs are scaled where you run multiple replicates. And not all scientific runs are high-performance in the HPC sense. Even if your code is HPC in the HPC sense, and requires CUDA, and 40,000 cores, you should consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that t…

> consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level.

Ok, that's a reasonable ask :) But yeah as you implied, good luck getting the average scientist, who in the best case begrudgingly uses version control, to care enough to do this.

Re: Challenge to scientists: does your ten-year-old code still run?

#368

Post by j45 »

jpeloquin wrote: post #330
j45 wrote: post #159

Earlier quoted context omitted.

One of the things I come across is scientists who believe they're capable of learning code quickly because they're capable in another field. After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running. For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above? I would have t…

> I would have thought the scientists among us would approach someone with familiarity with software development expertise. Is there a pool of skilled software architects willing to provide consultations at well-below market wages? Or a Q&A forum full of people interested in giving this kind of advice? (StackOverflow isn't useful for this; the allowed question scope is too narrow.) I guess one incentive to publish on…

Are the hiring scientists also paid well-below market wages?

Re: Challenge to scientists: does your ten-year-old code still run?

#369

Post by j45 »

marmaduke wrote: post #326
PeterisP wrote: post #225

Earlier quoted context omitted.

It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development. In any case you'd need to teach them the problem…

> Employing dedicated software developers is simply not a thing that happens This is a really key point that is lost on devs outside of science looking in. In our case, good devs are out of budget by a factor of 2x at least (at an EU public university in a lab doing lots of computational work). The best we get are engineers which are expected to keep the cluster running, order computers, organize seminars.. and event…

Are the hiring scientists also paid well-below market wages by that degree?

Re: Challenge to scientists: does your ten-year-old code still run?

#370

Post by analog31 »

Jabbles wrote: post #294
analog31 wrote: post #285

Earlier quoted context omitted.

I'm a scientist in a group that also includes a software production team. For me, the standard of scientific reproducibility is that a result can be replicated by a reasonably skilled person, who might even need to fill in some minor details themselves. Part of our process involves cleaning up code to a higher state of refinement as it gets closer to entering the production pipeline. I've tested 30 year old code, and…

This sounds great. In your opinion, do you think your team is unusual in those aspects? Do you have any knowledge of the quality of code in other branches of physics or other sciences?

Well, I know the quality of my own code before I got some advice. And I've watched colleagues doing this as well.

My own code was quite clean in the 1980s, when the limitations on the machines themselves tended to keep things fairly compact with minimal dependencies. And I learned a decent "structured programming" discipline.

As I moved into more modern languages, my code kind of degenerated into a giant hairball of dependencies and abstractions. "Just because you can do that, doesn't mean you should." I've kind of learned that the commercial programmers limit themselves to a few familiar patterns, and if you try to create a new pattern for every problem, your code will be hard to hand off.

Scientists would benefit from receiving some training in good programming hygiene.

Post reply on HN