Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

221–230 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#221

Post by neuromantik8086 »

jnxx wrote: post #209
neuromantik8086 wrote: post #153

Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge ( https://www.nature.com/articles/d41586-019-03296-8 ), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language ( https://www.com…

Konrad Hinsen is an expert in molecular bioinformatics and also has significantly contributed to Numerical Python, for example, and has extensively published around the topic of reproducible science and algorithms - see his blog. The fact that he might favor different solutions from you does not mean that he is pushing some kind of hidden agenda. If you think that Common Workflow Language is a better solution, you ar…

I never said that Konrad Hinsen's agenda was hidden; in fact, it's not at all hidden (which is why I linked the abstract). It's just that this context isn't at all clear in the Nature write-up, and it's relevant to take into account.

I haven't taken the time to seriously contemplate the merits of CWL vs Leibniz, although my gut instinct is that we don't really need another domain-specific language for science given the profusion of such languages that already exist (Mathematica, Maple, R, MATLAB, etc). That's the extent of my bias, but again, it's a gut instinct and not a comprehensive well-reasoned argument against Leibniz.

Re: Challenge to scientists: does your ten-year-old code still run?

#222

Post by throwaway287391 »

throwawaygh wrote: post #182
throwaway287391 wrote: post #172

Earlier quoted context omitted.

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…

> or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.

> A lot of open sourced CS research is not reproducible.

I'm not sure if this was meant to be a counter-argument to me, but I completely agree!

> "the code still runs and gives the same output" is not the same as reproducibility.

Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.

Re: Challenge to scientists: does your ten-year-old code still run?

#223

Post by mnw21cam »

detaro wrote: post #19
throwanem wrote: post #12

Earlier quoted context omitted.

The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I…

And that scientists also are rarely supported by programmers, or if they are it's an unstable and unappreciated position.

Being in such a position, I can say that I am appreciated, but not in a manner that results in job stability and promotion. It's a massive problem in academia, and there's an attempt to get the position recognised and call it "Research Software Engineer", with comparable opportunities for promotion and job stability as a researcher. However, it's not going massively well. Academic job progression is still almost completely purely based on the ability to get first or last author papers in top journals. I have lots of papers where I am a middle author, because I wrote the software that did the analysis that was vital for the paper to even exist, but it largely doesn't count. And I'm lucky - many software engineers don't even get put in as a middle author on the paper they contributed to.

Re: Challenge to scientists: does your ten-year-old code still run?

#224

Post by bumby »

gowld wrote: post #158
BeetleB wrote: post #82

Earlier quoted context omitted.

> Or is quality left up to the primary researchers? Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality. I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

"quality" is a subjectit word. Let's be clear what this means: Individual researchers, and in many disciplines (like physics), there is almost no emphasis on correct results , merely on believable results.

There are a few standardized definitions. The most succinct bring “quality is the adherence to requirements”.

As an example, if your science has the requirement of being replicable (as it should) there are a host of best practices that should flow down to the software development requirements. Not implementing those best practices would be indicative of lower quality

Re: Challenge to scientists: does your ten-year-old code still run?

#225

Post by PeterisP »

j45 wrote: post #159
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

One of the things I come across is scientists who believe they're capable of learning code quickly because they're capable in another field. After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running. For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above? I would have t…

It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development.

In any case you'd need to teach them the problem domain, and it's considered cheaper (and simpler from organizational perspective) to get some phd students or postdocs from your domain to spend half a year getting up to speed on coding (and they likely had a few courses in programming and statistics anyway) than to hire an experienced software developer and have them learn the basics of your domain (which may well take a third or half of the appropriate undergraduate bachelor's program).

Re: Challenge to scientists: does your ten-year-old code still run?

#227

Post by solatic »

throwaway287391 wrote: post #172
solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…

Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there.

Re: Challenge to scientists: does your ten-year-old code still run?

#228

Post by stult »

smnrchrds wrote: post #173
ivanbakel wrote: post #125

Earlier quoted context omitted.

Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily? Given that such software forms the very foundation of the results of such papers, why shouldn't it fall under scrutiny, even for "minor" points? If you are unable to produce good technical content, why are you qualified to declare what is or isn…

> Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily? A non-native English speaker may make grammatical mistakes when communicating their research in English—it does not in any way invalidate their results or hint that there is anything amiss. It is simply what happens when you are a non-native s…

Journals employ copy editors to address just those sorts of mistakes, why should we not hold software to the same standard as academic language? But more importantly, these software best practices aren't mere "grammatical mistakes," they exist because well-organized, well-tested code has fewer bugs and is easier for third parties to verify. Third-parties validating that the code underlying an academic paper executes as expected is no different than third-parties replicating the results of a physical experiment. You can be damn sure that an experimental methodology error invalidates a paper, and you can be damn sure that bad documentation of the methodology dramatically reduces the value/reliability of the paper. Code is no different. It's just been the wild west because it is a relatively new and immature field, so most academics have never been taught coding as a discipline nor held to rigorous standards in their own work. Is it annoying that they now have to learn how to use these tools properly? I'm sure it is. That doesn't mean it isn't a standard we should aim for, nor that we shouldn't teach the relevant skills to current students in sciences so that they are better prepared when they become researchers themselves.

Re: Challenge to scientists: does your ten-year-old code still run?

#229

Post by jnxx »

aspaceman wrote: post #199
jnxx wrote: post #161

Earlier quoted context omitted.

edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not understand. What is crucial in the argument here is indeed the distinction between methods employing pseudo-randomness, like Monte Carlo simulation, and non-determinism caused by undefined behavior. > I'm an accelerator physicist and I wouldn't want my code t…

> If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined. That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent. You’re speaking genuine nonsense as far as I’m co…

I am referring to this blog post:

https://lockdownsceptics.org/code-review-of-fergusons-model/

It says, word-by-word:

> Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results.

>

>Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters.

> I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ...

The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213

Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all.

What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work.

These are wildly different things. I do not understand why your comment just adds to the confusion between these two things??

> A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

As said, I don't endorse the critique in the blog. However, critique in a software implementation, as well as in scientific matters, should never carry a call on authority - it should logically explain what is the problem, with concrete points. Unfortunately, the cited blog post remains very vague about this, while claiming:

> My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years.

It would be much better if, instead claiming that there could be race conditions, it could point to lines in the code with actual race conditions, and show how the results of the simulation are different when the race conditions are fixed. Otherwise, it just looks like he claims that the program is buggy, because he is in no position to question the science, and does not like the result.

Re: Challenge to scientists: does your ten-year-old code still run?

#230

Post by kkylin »

dllthomas wrote: post #219
solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

This. I absolutely agree there needs to be more transparency, and scientific code should be as open as possible. But this should not replace replication.
Post reply on HN