Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

431–440 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#431

Post by RandoHolmes »

kalenx wrote: post #413
RandoHolmes wrote: post #394

Earlier quoted context omitted.

> Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".

I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousn…

All you did was restate the argument that I've already rejected.

And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.

Re: Challenge to scientists: does your ten-year-old code still run?

#432

Post by a1369209993 »

kkylin wrote: post #407
hobofan wrote: post #329

Earlier quoted context omitted.

But "rerunning reproducability" is mostly a neccessary requirement for independent reproducability. If you can't even run the original calculations against the original data again how can you be sure that you are not comparing apples to oranges?

This is not clear at all. It depends on the "result" in question. If I wrote a paper describing a super numerical algorithm for inverting matrices, and no one is able to replicate the superior performance of my algorithm despite following the recipe in my paper, then whether they can run my code or not doesn't seem to be of the highest priority. Edit: more careful phrasing.

> whether they can run my code or not doesn't seem to be of the highest priority.

On the contrary; in that case there are four possibilities:

a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements.

b: your algorithm works, but the description in the paper is wrong or incomplete

c: your algorithm works as described, but the replicater implemented it incorrectly

d: other

Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)

Re: Challenge to scientists: does your ten-year-old code still run?

#434

Post by neuromantik8086 »

jnxx wrote: post #209
neuromantik8086 wrote: post #153

Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge ( https://www.nature.com/articles/d41586-019-03296-8 ), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language ( https://www.com…

Konrad Hinsen is an expert in molecular bioinformatics and also has significantly contributed to Numerical Python, for example, and has extensively published around the topic of reproducible science and algorithms - see his blog. The fact that he might favor different solutions from you does not mean that he is pushing some kind of hidden agenda. If you think that Common Workflow Language is a better solution, you ar…

I never answered your last question so here goes:

> Are you saying that the reproductive challenge poses a difficulty to Common Workflow Language?

I don't actually understand how the reproducibility challenge undermines the validity of using CWL / flow-based programming as an approach to promoting reproducible analyses. There certainly wasn't anything in the article that made me think that CWL was challenged, but Hinsen explicitly called out CWL in the abstract, which implies that for some reason he thinks, a priori, that it's a non-solution. He never justifies this implied assumption further, and as near as I can tell, none of the attempted replications used a flow-based language.

If Hinsen really aimed to argue against the viability of CWL/flow-based programming as an approach to reproducibility, he would have done a systematic comparison of historical analyses that used a flow-based system (like National Instruments' Labview or Prograph) vs analyses that are more similar to the approach that he seems to favor (i.e., analyses using Mathematica or Maple).

While I find the challenge interesting to follow, and the retrocomputing geek in me finds it fun, I don't actually understand what it really accomplished other than being a fun diversion. Assuming that an analysis was written in a Turing-complete language and you didn't use non-deterministic algorithms, you should theoretically be able to reproduce the results exactly on modern hardware, and using non-deterministic algorithms I would imagine that a result would be "close enough" within some kind of confidence interval. You may need to go to great lengths (in terms of emulating instruction sets, ripping tapes, etc), but I think a visit to any retrocomputing festival or computer history museum would have made that pretty obvious from the outset.

Re: Challenge to scientists: does your ten-year-old code still run?

#435

Post by jacobwilliamroy »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Our first job as scientists is to make sure we're not fooling ourselves. I wouldn't just use any old scale to take a measurement. I want a calibrated scale, adjusted to meet a specific standard of accuracy. Such standards and calibrations ensure we can all get "the same" result doing "the same" thing, even if we use different equipment from different vendors. The concerns about code are exactly the same. It's even scarier to me because I realize that unlike a scale, most scientists have no idea how to calibrate their code to ensure accurate, reproducible results. Of course with the scales, the calibration is done by a specialized professional who's been trained to calibrate scales. Not sure how we solve this issue with the code.

Re: Challenge to scientists: does your ten-year-old code still run?

#436

Post by Ericson2314 »

rekado wrote: post #378
Ericson2314 wrote: post #280

Glad to see many mentions of Nix in this thread! I wonder if Nix and Guix should standardize the derivation format both share to kick that off as the agreed-upon "thin waste" other projects and the the academy can standardize around.

The derivation format is little more than a compilation artifact (a low-level representation of a build), and I think standardizing on it would not be as useful as it may seem.

Exactly, it's mostly boring, just like IP packets in isolation are boring. Great thing to standardize.

Also, we're working on making them a bit less boring :). I've been working on adding hashing schemes compatible with IPFS's IPLD. Also, we can make derivations that produce derivations, making a sort of recursive Nix that encourages more up-front planning than crude "nix-build inside derivations.

I hope Guix would want some of this stuff too.

Re: Challenge to scientists: does your ten-year-old code still run?

#437

Post by genewitch »

halfdan wrote: post #322
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I have done research on Evolutionary Algorithm and numerical optimization. It was nigh impossible to reproduce poorly described algorithms from state of the art research at the time and researchers would very often not bother to reply to inquiries for their code. Even if you did get the code it would be some arcane C only compatible with a GCC from 1996. Code belongs with the paper. Otherwise we can just continue to…

In 2006 or 2008 a university in England published some fluff about genetic/evolutionary algorithms that were evolving circuits on an fpga, specifically the published stuff regarded an fpga without a clock was able to differentiate between two tones.

I've spent the intervening years trying to find a way to implement this myself, going as far as to buy things like the ice40 fpga because the bitstreams are supposedly unlocked; this is a pre-req for modifying the actual gate/logic on the chip.

I've emailed the professor listed as the headliner in the articles published about it to no avail.

Nearly my entire adult life has been spent reading some interesting article, chasing down the paper, finding out if any code was published, and seeing if I could run the code myself.

It wasn't until machine learning with pytorch became mainstream that I started having luck replicating results. Just some more data points for this discussion.

Re: Challenge to scientists: does your ten-year-old code still run?

#438

Post by CamperBob2 »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

Sure, and that rationale works OK when your code operates in a limited, specialized domain.

But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.

Re: Challenge to scientists: does your ten-year-old code still run?

#439

Post by eru »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

Not so fast. Monte Carlo code turns arbitrary RNG seeds into outputs. That process can, and arguably should be, deterministic.

To do your study, you feed your Monte Carlo code 'random enough' seeds. Coming up with the seeds does not need to be deterministic. But once the seeds are fixed, the rest can be deterministic. Your paper should probably also publish the seeds used, so that people can reproduce everything. (And so they can check whether your seeds are carefully chosen, or really produce typical outcomes.)

Re: Challenge to scientists: does your ten-year-old code still run?

#440

Post by reitzensteinm »

snowwrestler wrote: post #379
sitkack wrote: post #253

Earlier quoted context omitted.

> exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done. I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it…

The history of physics is full of complex, one-off custom hardware. Reviewers have not been expected to take the full technical specs and actually build and run the exact same hardware, just to verify correctness for publication. I doubt any physicist believes we need to get the Tevatron running again just to check decade-old measurements of the top quark. I don't understand why decade-old scientific software code mu…

If you could have spun up a Tevatron at will for $10, would the culture be the same today?

I suspect that software really is different in this way, and treating it like it's complex, one off hardware is cultural inertia that's going to fade away.

Post reply on HN