Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

451–460 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#451

Post by Heliosmaster »

We are building Nextjournal[0] exactly for this purpose.

It's a platform for interactive notebooks built on immutable and persistent storage (Datomic) and Docker:

- Changes to the document are automatically versioned to an immutable database (Datomic). Previous versions can be accessed and restored any time.

- Uploaded data or generated result files are automatically versioned in append-only content-addressed storage and can’t be accidentally overwritten.

- Changes to the file system state can be committed as Docker images. Reproducibility is ensured by referencing these images only by their immutable hashes.

[0]https://nextjournal.com

Re: Challenge to scientists: does your ten-year-old code still run?

#452

Post by jnxx »

solatic wrote: post #142
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, [...]

Reproducibility in the scientific sense is different from running the same program with the same input, and getting exactly the same result. Repreducibility means that if you repeat the measurements in another environment, getting somewhat different data, and apply the same theory and methods, you get to the same conclusion.

The property of a computer program that when you run it again with the same input, you get the same output, is nice and very helpful for debugging. But the fact that you can run the same program does not mean that it is bug-free, as much as the fact that you can copy a paper with a mathematical proof does not mean that the proof is correct.

Also, multi-threaded and parallel code is inherently non-deterministic.

> when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation,

That is wrong. Linux for example uses latency measurements from drivers such as HDD drive seek latencies or keyboards to generate entropy. While it might not the best thing to rely on for purposes of cryptography, it is surely not deterministic. If it would matter, you could download real-time astronomical noise measurements and use them to seed your Mersenne Twister generator.

Re: Challenge to scientists: does your ten-year-old code still run?

#453

Post by jnxx »

mmmBacon wrote: post #150
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…

> Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run.

Perhaps if you use only single-threaded computation, you are interested in averages, and the processes you are interested in behave well and mostly linear.

But

- running code in parallel easily introduces non-determinism, even if your result computation is as simple as summing up results from different threads

- the processes one is examining might be highly non-linear - like lightning, weather forecasts, simulation of wildfires, and also epidemic simulations

- especially for all kind of safety research, you might actually be interested not only in averages, but in freak events, like "what is the likelihood that you have two or three hurricanes at the same time in the Gulf of Mexico", or "what happens if your nuclear plant gets struck by freak lightning in the first second of a power failure".

What should be reproducible are the conclusions you come to, not the hashed bits of program output.

> If you aren’t doing that, you aren’t running your Monte- Carlo correctly. That’s a huge red flag.

No, it does not follow from that.

Re: Challenge to scientists: does your ten-year-old code still run?

#454

Post by MaxBarraclough »

ByteJockey wrote: post #443
MaxBarraclough wrote: post #372

Earlier quoted context omitted.

> If it's a one-man-show I would not give too much on code quality This makes me a little uneasy, as I'm not too worried about code quality can easily translate into Yes I know my code is full of undefined behaviour, and I don't care . > PS: quite excited about my first post here Welcome to HN! reddit has more cats, Slashdot has more jokes about sharks and laserbeams, but somehow we get by.

Are we talking actual undefined behavior or just behavior that's undefined by the language standard? The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

> Are we talking actual undefined behavior or just behavior that's undefined by the language standard?

'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity.

> if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

Code should be correct by construction, not correct by coincidence. Results from such code shouldn't be considered publishable. Mathematicians don't get credit for invalid proofs that happen to reach a conclusion which is correct.

Again, this isn't some theoretical quibble. There are plenty of sneaky ways undefined behaviour can manifest and cause trouble. [0][1][2]

In the domain of safety-critical software development in C, extreme measures are taken to ensure the absence of undefined behaviour. If scientists adopt a sloppier attitude toward code quality, they should expect to end up publishing invalid results. Frankly, this isn't news, and I'm surprised the standards seem to be so low.

Also, of all the languages out there, C and C++ are among the most unforgiving of minor bugs, and are a bad choice of language for writing poor-quality code. Ada and Java, for instance, won't give you undefined behaviour for writing int i; int j = i;.

[0] https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...

[1] https://blog.regehr.org/archives/213

[2] https://cryptoservices.github.io/fde/2018/11/30/undefined-be...

See also my longer ramble on this topic at https://news.ycombinator.com/item?id=24264376

Re: Challenge to scientists: does your ten-year-old code still run?

#455

Post by jnxx »

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

> This was used by the Imperial College for COVID-19 predictions. It has race conditions, seeds the model multiple times, and therefore has totally non-deterministic results[0].

>

> [0] https://lockdownsceptics.org/code-review-of-fergusons-model/

This does not looks like a good example at all, as it appears that the blog author there just tries to discredit the program because he does not like the results. He also writes that all epidemiological research should be defunded.

Re: Challenge to scientists: does your ten-year-old code still run?

#456

Post by touristtam »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts. In fact the time spent making it that way is time that a scientist spends doing software engineering instead of science, which isn't very efficient.

If the proof on which the paper is based is in the code that produced the evidence, you absolutely need to be able to let a lambda user run it without specific knowledge to abide to the reproducible principle. Asking a reviewer to fiddle about like a IT professional to get something working is bound to promote lazy reviewing and either will result into dismissing the result or approval without real review.

And by the way producing a paper could be argued it isn't really science either, but if you are working with MSFT Office, you know there is a fair amount of non science work hours that has been put into that as well.

Re: Challenge to scientists: does your ten-year-old code still run?

#457

Post by jnxx »

jnxx wrote: post #174
chrchang523 wrote: post #122

Earlier quoted context omitted.

Nit: implementations of Monte Carlo methods are not necessarily nondeterministic. Whenever I implement one, I always aim for a deterministic function of (input data, RNG seed, parallelism, workspace size).

And it seems that the people from Imperial College have done that with their epidemiological simulation. What critics claim is that their code produces non-deterministic results when given deterministic input and random seeds, i.e. that their code is seriously broken. Which would be a serious issue if true.

To be more specific, the critics claim the code would yield completely different results.

Re: Challenge to scientists: does your ten-year-old code still run?

#458

Post by arethuza »

gowld wrote: post #151
arethuza wrote: post #127

Earlier quoted context omitted.

I would regard (from experience) "science ready" code as something that you run just often enough to get the results to create publications. Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers. This kind of reasoning was one of the many reasons I left academic research - I…

If your experiment is not repeatable, it's an anecdote not data. Any effort to write a paper readable for other people, or document the experiment in any way would probably be seen as wasted effort that could be used to create more results. The "don't show your work" argument only makes sense if you are doing PR, not science.

I specifically got told off by my supervisor for trying to "improve" some of the software we were working on!

Re: Challenge to scientists: does your ten-year-old code still run?

#459

Post by akerro »

therealx wrote: post #244
akerro wrote: post #31

Code written in Oak still works in Java 14. You can still write `public abstract interface BlaBla{}` and it still works. If it doesn't work (due to reflection safety changes in Java9), it sill surely compile with newer compiler. Another thing, are tools used to compile still available? I tried to compile my BCS Android+native OpenCV project and failed quickly. Gradle removed some plugin for native code integration, a…

Or use the old version of Gradle? It sounds like creating a vm/container/whatever with the old versions of everything is the fastest path, although I understand not wanting to do it after some point.

>Or use the old version of Gradle?

Intellij / Android Studio are nagging me to update gradle

Re: Challenge to scientists: does your ten-year-old code still run?

#460

Post by SiempreViernes »

snowwrestler wrote: post #354

The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding. The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding. Concerns about software quality in science are primarily about avoiding…

Yes, this argument, along with the practices of cross checking within one project, is what saves science from the total doom its software practices would otherwise deliver.

However, reproducibility is a precondition to automation, and automation is a real nice thing to have.

Post reply on HN