Challenge to scientists: does your ten-year-old code still run? • Hacker News Board

Re: Challenge to scientists: does your ten-year-old code still run?

#151

Post by gowld » Mon, Aug 24, 2020, 4:15 PM UTC

arethuza wrote: post #127

Jabbles wrote: post #108

Earlier quoted context omitted.

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

I would regard (from experience) "science ready" code as something that you run just often enough to get the results to create publications. Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers. This kind of reasoning was one of the many reasons I left academic research - I…

If your experiment is not repeatable, it's an anecdote not data.

Any effort to write a paper readable for other people, or document the experiment in any way would probably be seen as wasted effort that could be used to create more results.

The "don't show your work" argument only makes sense if you are doing PR, not science.

Re: Challenge to scientists: does your ten-year-old code still run?

#152

Post by vngzs » Mon, Aug 24, 2020, 4:16 PM UTC

petschge wrote: post #129

chrchang523 wrote: post #122

Earlier quoted context omitted.

Nit: implementations of Monte Carlo methods are not necessarily nondeterministic. Whenever I implement one, I always aim for a deterministic function of (input data, RNG seed, parallelism, workspace size).

It really helps with debugging if your MC code is deterministic for a given input seed. And then you just run for a sufficient number of different seeds to sample the probability space.

Alternatively: seed the program randomly by default, but allow the user to specify a seed as a CLI argument or function argument (for tests).

In the common case, the software behaves as expected (random output), but it is reproducible for tests. You can then publish your RNG seed with the commit hash when you release your code/paper, and others may see your results and investigate that particular code execution.

Re: Challenge to scientists: does your ten-year-old code still run?

#153

Post by neuromantik8086 » Mon, Aug 24, 2020, 4:18 PM UTC

Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge (https://www.nature.com/articles/d41586-019-03296-8), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language (https://www.commonwl.org/; currently used in bioinformatics by the Broad Institute via the Cromwell workflow manager).

Hinsen has created his own DSL, Leibniz (https://github.com/khinsen/leibniz ; http://dirac.cnrs-orleans.fr/~hinsen/leibniz-20161124.pdf), which he believes is a better alternative to Common Workflow Language. This reproducibility challenge is in support of this agenda in particular, which is worth keeping in mind; it is not an unbiased thought experiment.

Re: Challenge to scientists: does your ten-year-old code still run?

#154

Post by petschge » Mon, Aug 24, 2020, 4:19 PM UTC

gowld wrote: post #147

petschge wrote: post #126

Earlier quoted context omitted.

One example: My code used to crash for a long time if you set the thermal speed to something greater than the speed if light. Should the code crash? No. And by now I have found the time to write extra code to catch the error and midly insult the user (It says "Faster than light? Please share that trick with me!") Does it matter? No. It didn't run and give plausible-but-wrong results. So that is code that I would call…

Then you publish your work and critics publish theirs and the community decides which claims have proven their merit. This is the fundamental structure of the scientific community. How is "your code has error and I rebuke you" a more painful critique than "you are hiding your methodology and so I rebuke you"?

Nothing limits the field of critics to people who have written their own code and know what they are doing.

Re: Challenge to scientists: does your ten-year-old code still run?

#155

Post by auntienomen » Mon, Aug 24, 2020, 4:19 PM UTC

RandoHolmes wrote: post #130

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> people claiming that their non-software engineering grade code invalidates the results of their study. How exactly is this a bad thing? > I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code. But it should be noted that…

Oh, he didn't say 'accurate science', nice gotcha!

This is exactly the sort of pedantic cluelessness that scientists are seeking to avoid by not publishing their code.

Re: Challenge to scientists: does your ten-year-old code still run?

#156

Post by rudolph9 » Mon, Aug 24, 2020, 4:19 PM UTC

myself248 wrote: post #62

Plenty of actual professional programmers can't manage this, how is it a fair standard to hold scientists to, when the code is just one of the many tools they're trying to use to get their real job done? I think moving away from the cesspool of imported remote libraries that update at random times and can vanish off the internet without warning, would help a lot of both cases.

Professional programmers should adopt package manager that focus on reproducibility like Guix and Nix and make them accessible enough for non programmers to use.

Neither of these are perfect but in my experience they are worlds better than apk, Dockerfiles, and many other commonly used solutions.

http://guix.gnu.org/

https://nixos.org/

Re: Challenge to scientists: does your ten-year-old code still run?

#157

Post by jack_h » Mon, Aug 24, 2020, 4:19 PM UTC

mmmBacon wrote: post #150

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…

Since I have a bit of experience in this area, quasi-Monte Carlo methods also work quite well and ensure deterministic results. They're not applicable for all situations though.

Re: Challenge to scientists: does your ten-year-old code still run?

#158

Post by gowld » Mon, Aug 24, 2020, 4:19 PM UTC

BeetleB wrote: post #82

bumby wrote: post #60

Earlier quoted context omitted.

I’m curious, are dedicated software assurance teams a thing in your research area? Or is quality left up to the primary researchers?

> Or is quality left up to the primary researchers? Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality. I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

"quality" is a subjectit word. Let's be clear what this means:

Individual researchers, and in many disciplines (like physics), there is almost no emphasis on correct results, merely on believable results.

Re: Challenge to scientists: does your ten-year-old code still run?

#159

Post by j45 » Mon, Aug 24, 2020, 4:21 PM UTC

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

One of the things I come across is scientists who believe they're capable of learning code quickly because they're capable in another field.

After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running.

For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above?

I would have thought the scientists among us would approach someone with familiarity with software development expertise. (something abstract and requiring a different set of muscles)

One positive emerging is the variety of low/no-code tooling that can replace a lot of this hornets nest coding.

Re: Challenge to scientists: does your ten-year-old code still run?

#160

Post by gowld » Mon, Aug 24, 2020, 4:21 PM UTC

core-questions wrote: post #96

BeetleB wrote: post #82

Earlier quoted context omitted.

> Or is quality left up to the primary researchers? Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality. I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

> protested when version control was suggested Academics are strange like this. The root reason is fear: fear that you're complicating their process, that you're going to interrupt their productivity or flow state, that you're introducing complication that has no benefit. They then build up a massive case in their minds for why they shouldn't do this; good luck fighting it. Doubly so if you're IT staff and don't have…

I think this is why industry does better science than academia, at least in any area where there are applications. Generally, they get paid for being right, not just for being published, so they put respect and money into people that help get correct results.