Challenge to scientists: does your ten-year-old code still run? • Hacker News Board

Re: Challenge to scientists: does your ten-year-old code still run?

#281

Post by kag0 » Mon, Aug 24, 2020, 5:54 PM UTC

solatic wrote: post #142

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> there is no such thing as truly random number generation on contemporary computers

well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.

Re: Challenge to scientists: does your ten-year-old code still run?

#282

Post by pruthvishetty » Mon, Aug 24, 2020, 5:54 PM UTC

I read it as does your ten-year-old still run code, and was thinking if this was a challenge for scientists to have their kids do better things than coding.

Re: Challenge to scientists: does your ten-year-old code still run?

#283

Post by ajford » Mon, Aug 24, 2020, 5:56 PM UTC

dllthomas wrote: post #219

solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

This! I struggled with this topic in university. I was studying pulsar astronomy, and there was only one or two common tools used at the lower levels of data processing, and had been the same tools used for a couple of decades.

The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software.

I repeatedly brought it up, but I wasn't advanced enough in my studies to be able to do anything about it. By the time I felt comfortable with that, I was on my way out of the field and into an non-academic career.

I have kept up with the field to a certain extent, and there is now a project in progress to create a fully independent replacement for that original code that should help shed some light (in progress for a few years now, and still going strong).

Re: Challenge to scientists: does your ten-year-old code still run?

#284

Post by hpcjoe » Mon, Aug 24, 2020, 5:58 PM UTC

petschge wrote: post #167

atrettel wrote: post #131

Earlier quoted context omitted.

> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianne…

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readabil…

I got into the habit of documenting each file with a file.meta that I could view later on.

I did binary dumps in the past because ascii dumps (remember, 90s) were far more time/space expensive. HDF wasn't quite an option then, either HDF4, or HDF5.

These days I would probably look at something like that, though, to be honest, there is always a danger of choosing something that may not be supported over the long term. This is why I generally prefer open and simple formats for everything. HDF5 is nice and open.

One needs to look carefully at the total risk of using a proprietary format/system for any part of their storage. Chances are you will not be able to even read older data within a small number of decades if any of the format/system dependent technologies goes away.

I've got old word processor files from the mid 80s, that I can't read. What I've written there (mostly college papers) is lost (which may be a net positive for humanity).

My tarballs, and zip files though, are readable 30+ years later. That is pretty amazing.

Simple, documented, and open formats. Picture a time when you can't read/open your pptx/xlsx/docx files any more. Same with data. Simple binary formats are like CSV files, but you do need to maintain metadata on their contents, and document it extensively in the code as to what you are reading/writing, why you are doing this, and how you are doing this.

I think this will get more important over time as we start asking questions on how to maintain open artefact repositories for data and code. The fewer dependencies the better.

And unlike the recent gene renaming snafu in biology[1], you really, never, want your tool to get in the way of the science. Either in terms of formats, or interpretation of data.

[1] https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Re: Challenge to scientists: does your ten-year-old code still run?

#285

Post by analog31 » Mon, Aug 24, 2020, 5:58 PM UTC

Jabbles wrote: post #108

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

I'm a scientist in a group that also includes a software production team. For me, the standard of scientific reproducibility is that a result can be replicated by a reasonably skilled person, who might even need to fill in some minor details themselves.

Part of our process involves cleaning up code to a higher state of refinement as it gets closer to entering the production pipeline.

I've tested 30 year old code, and it still runs, though I had to dig up a copy of Turbo Pascal, and much of it no longer exists in computer readable form but would have to be re-entered by hand. Life was actually simpler back then -- with the exception of the built-ins of Turbo Pascal, it has no dependencies.

My code was in fact adopted by two other research groups with only minor changes needed to suit slightly different experimental conditions. It contained many cross-checks, though we were unaware of modern software testing concepts at the time.

For a result to have broader or lasting impact, replication is not enough. The result has to fit into a broader web of results that reinforce one another and are extended or turned into something useful. That's the point where precise replication of minor supporting results becomes less important. The quality of any specific experiment done in support of modern electromagnetic theory would probably give you the heebie jeebies, but the overall theory is profoundly robust.

The same thing has to happen when going from prototype to production. Also, production requires what I call push-button replication. It has to replicate itself at the click of a mouse, because the production team doesn't have domain experts who can even critique the entirety of their own code, and maintaining their code would be nearly impossible if it didn't adhere to standards that make it maintainable by multiple people at once.

Re: Challenge to scientists: does your ten-year-old code still run?

#286

Post by RandoHolmes » Mon, Aug 24, 2020, 5:58 PM UTC

auntienomen wrote: post #155

RandoHolmes wrote: post #130

Earlier quoted context omitted.

> people claiming that their non-software engineering grade code invalidates the results of their study. How exactly is this a bad thing? > I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code. But it should be noted that…

Oh, he didn't say 'accurate science', nice gotcha! This is exactly the sort of pedantic cluelessness that scientists are seeking to avoid by not publishing their code.

I don't consider accuracy in science to be pedantic, and I suspect most others don't either.

To paraphrase what the other developer said: "I don't want my work to be checked, I'm not here for accuracy, just the act of doing science".

When I was young, the ability to invalidate was the core aspect of science, but apparently that's changed over the years.

Re: Challenge to scientists: does your ten-year-old code still run?

#287

Post by enriquto » Mon, Aug 24, 2020, 6:02 PM UTC

spamizbad wrote: post #242

enriquto wrote: post #215

Earlier quoted context omitted.

> I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code. In what way do idiots making idiotic comments about your correct code invalidate your scientific production? You can still turn out science and let people read and comment freely on it. > As an exam…

> In what way do idiots making idiotic comments about your correct code invalidate your scientific production? You can still turn out science and let people read and comment freely on it. How would a layperson identify a faulty critique? It would be picked up by the media who would do their usual “both sides” thing.

Not that they abstain from doing that shit today, when code is not often published.

An educated and motivated layperson at least would have the chance to learn whether the critique is faulty. Today, with secret code, it is impossible to verify for almost everybody.

Re: Challenge to scientists: does your ten-year-old code still run?

#288

Post by jnxx » Mon, Aug 24, 2020, 6:06 PM UTC

djaque wrote: post #98

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

Specifically, to that point, I want to cite the saying:

"The dogs bark, but the caravan passes."

(There is a more colorful German variant which is, translated: "What does it bother the mighty old oak tree if a dog takes a piss...").

Of course, if you publish your code, you expose it to critics. Some of this will be unqualified. And as we have seen in the case e.g. of climate scientists, some might be even nasty. But who cares? What matters is open discussion which is a core value of science.

Re: Challenge to scientists: does your ten-year-old code still run?

#289

Post by dunefox » Mon, Aug 24, 2020, 6:07 PM UTC

Jabbles wrote: post #119

djaque wrote: post #105

Earlier quoted context omitted.

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

How do you know it won't give you results? Maybe it will find a bug that would have resulted in an embarrassing retraction. Maybe it wouldn't find any bugs, but give confidence to and encourage other users and increasing your citations and "impact". Maybe it will just save you 48h later on when you need to adapt the code. Software engineering has generally accepted that unit testing is a good practice and well worth…

> Why do you think science is different?

It's really not, I guess his focus lies on cranking out irreproducible papers.

Re: Challenge to scientists: does your ten-year-old code still run?

#290

Post by UncleMeat » Mon, Aug 24, 2020, 6:08 PM UTC

jnxx wrote: post #161

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not understand. What is crucial in the argument here is indeed the distinction between methods employing pseudo-randomness, like Monte Carlo simulation, and non-determinism caused by undefined behavior. > I'm an accelerator physicist and I wouldn't want my code t…

Race conditions aren't undefined behavior in C/C++. Data races are. Lots and lots of real systems contain race conditions without catastrophe.