Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

121–130 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#121

Post by dandelion_lover »

Jabbles wrote: post #108
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

> the distinction between "production-ready" and "science-ready" code

In the first case, you must take into account all (un)imaginable corner cases and never allow the code to fail or hang up. In the second case it needs to produce a reproducible result at least for the published case. And do not expect it to be user-friendly at all.

Re: Challenge to scientists: does your ten-year-old code still run?

#122

Post by chrchang523 »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Nit: implementations of Monte Carlo methods are not necessarily nondeterministic. Whenever I implement one, I always aim for a deterministic function of (input data, RNG seed, parallelism, workspace size).

Re: Challenge to scientists: does your ten-year-old code still run?

#124

Post by TimothyBJacobs »

djaque wrote: post #105
BeetleB wrote: post #87

Earlier quoted context omitted.

> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

This is the same as any other argument against testing. Unless you are actually selling a library, code is not the product. Customers are buying results, not your code base. Yet, we've discovered the importance of testing to make sure customers get the right results without issues.

If you want your results to be usable by others, the quality of the code matters. If all you care is publishing a paper, then I guess sure it doesn't matter if anyone else can build off your work.

Re: Challenge to scientists: does your ten-year-old code still run?

#125

Post by ivanbakel »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily?

Given that such software forms the very foundation of the results of such papers, why shouldn't it fall under scrutiny, even for "minor" points? If you are unable to produce good technical content, why are you qualified to declare what is or isn't minor? Isn't the whole point that scrutiny is best left to technical experts (and not subject experts)?

Re: Challenge to scientists: does your ten-year-old code still run?

#126

Post by petschge »

Jabbles wrote: post #108
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

One example: My code used to crash for a long time if you set the thermal speed to something greater than the speed if light. Should the code crash? No. And by now I have found the time to write extra code to catch the error and midly insult the user (It says "Faster than light? Please share that trick with me!") Does it matter? No. It didn't run and give plausible-but-wrong results. So that is code that I would call "science-ready" but I wouldn't want it criticized by people outside my domain.

Re: Challenge to scientists: does your ten-year-old code still run?

#127

Post by arethuza »

Jabbles wrote: post #108
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

I would regard (from experience) "science ready" code as something that you run just often enough to get the results to create publications.

Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers.

This kind of reasoning was one of the many reasons I left academic research - I personally didn't value publications as deliverables.

Re: Challenge to scientists: does your ten-year-old code still run?

#128

Post by magv »

dunefox wrote: post #102
Sebb767 wrote: post #24

As someone who worked with bits of scientific code: Does the code you write right now work on another machine might be the more appropriate challenge. If seen a lot of hardcoded paths, unmentioned dependencies and monkey-patched libraries downloaded from somewhere; just getting the new code to work is hard enough. And let's not even begin to talk about versioning or magic numbers. Similar to other comments I don't me…

If a scientist needs to write code then it's part of their job. It's as easy as that.

I think the idea that scientific code should be judged by the same standards as production code is a bit unfair. The point when the code works the first time is when an industry programmer starts to refactor it -- because he expects to use and work on it in the future. The point when the code works the first time is when a scientists abandons it -- because it has fulfilled its purpose. This is why the quality is lower: lots of scientific code is the first iteration that never got a second.

(Of course, not all scientific code is discardable, large quantities of reusable code is reused every day; we have many frameworks, and the code quality of those is completely different).

Re: Challenge to scientists: does your ten-year-old code still run?

#129

Post by petschge »

chrchang523 wrote: post #122
djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Nit: implementations of Monte Carlo methods are not necessarily nondeterministic. Whenever I implement one, I always aim for a deterministic function of (input data, RNG seed, parallelism, workspace size).

It really helps with debugging if your MC code is deterministic for a given input seed. And then you just run for a sufficient number of different seeds to sample the probability space.

Re: Challenge to scientists: does your ten-year-old code still run?

#130

Post by RandoHolmes »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> people claiming that their non-software engineering grade code invalidates the results of their study.

How exactly is this a bad thing?

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

But it should be noted that what you didn't say is that you're here to turn out accurate science.

This is the software version of statistics. Imagine if someone took a random sampling of people at a Trump rally and then claimed that "98% of Americans are voting for Trump". And now imagine someone else points out that the sample is biased and therefore the conclusion is flawed, and the response was "Hey, I'm just here to do statistics".

---

Do you see the problem now? The poster above you pointed out that the conclusions of the software can't be trusted, not that the coding style was ugly. Most developers would be more than willing to say "the code is ugly, but it's accurate". What we don't want is to hear "the conclusions can't be trusted and 100 people have spent 10+ years working from those unreliable conclusions".

Post reply on HN