Strangely, they were running (some of ) the code on old hardware. That's hardly a useful case, and much easier than 'resurrecting' the code for modern reuse.
Challenge to scientists: does your ten-year-old code still run?
211–220 of 487 posts
Re: Challenge to scientists: does your ten-year-old code still run?
#212Earlier quoted context omitted.
I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.
I would regard (from experience) "science ready" code as something that you run just often enough to get the results to create publications. Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers. This kind of reasoning was one of the many reasons I left academic research - I…
My experience has been similar.
Still, there's plenty of room to encourage good(/better) practices which cost essentially nothing, e.g. using $PWD rather than /home/bob/foo
Re: Challenge to scientists: does your ten-year-old code still run?
#213Earlier quoted context omitted.
I don't think that would be any problem (why should it?). Code exhibiting undefined behavior is a different kettle of fish...
Which is why I run valgrind on my code (with a parameter file containing physically valid inputs) to get rid of all undefined behavior. But I gave up on running afl-fuzz, because all it found was crashes following from physically invalid inputs. I fixed the obvious once to make the code nicer for new users, but once afl started to find only very creative corner cases I stopped.
Well done!
Re: Challenge to scientists: does your ten-year-old code still run?
#214This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…
The graphics community has started an interesting initiative at this end: http://www.replicabilitystamp.org/ After a paper has been accepted, authors can submit a repository containing a script which automatically replicates results shown in the paper. After a reviewer confirms that the results were indeed replicable, the paper gets a small badge next to its title. While there could certainly be improvements, I think…
But does this badge influence the scientific profile / resume of the researcher in any way?
Re: Challenge to scientists: does your ten-year-old code still run?
#215This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
> I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.
In what way do idiots making idiotic comments about your correct code invalidate your scientific production? You can still turn out science and let people read and comment freely on it.
> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.
I guess you would not need to engage personally with the idiots at "acceleratorskeptics.com", but likely most of their critique would be easily shut off by a simple sentence such as this one. Since most of your readers would not be idiots, they could scrutinize your code and even provide that reply on your behalf. This is called the scientific method.
I agree that you produce science, not merely code. Yet, the code is part of the science and you are not really publishing anything if you hide that part. Criticizing scientific code because it is bad software engineering is like criticizing it because it uses bad typography. You should not feel attacked by that.
Re: Challenge to scientists: does your ten-year-old code still run?
#216Earlier quoted context omitted.
Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…
the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.
This is not correct on several levels. Reproducibility is not achievable in many real world scenarios, but worse it's not even very informative.
Contra your assertion, many people do some sort of regression testing like this but it's isn't terribly useful for verification or validation - but it is good at catching bad patches.
Re: Challenge to scientists: does your ten-year-old code still run?
#217Earlier quoted context omitted.
> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianne…
Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readabil…
Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over the proprietary format that they did end up using. But HDF5 requires an additional library. I did not want to use any external libraries in my particular project (other than MPI), so I tried to look for a solution that achieves close to what HDF5 can achieve but without requiring something as "heavy" as HDF5. I have to admit that perhaps my design choice does not work for more complex situations, but I think it is something people should consider before tying themselves down too much.
Re: Challenge to scientists: does your ten-year-old code still run?
#218Earlier quoted context omitted.
I agree, but that’s similar to saying the data is what matters, not the methodology. In the research germane to this conversation, software is the means by which the scientific data is generated. If the software is flawed, it undermines the confidence in the data and thus the conclusions.
Most researchers would agree with the first statement without significant qualification. Methods are at the end for a reason.
Not disagreeing with your assertion on the opinion of “most researchers” but you’ll often find quite a few people advocating for using the methodology sans data as a means to determine publication worthiness to try and avoid the perverse incentives for novel or meaningful data.
I think it’s too easy to game the data (whether knowingly or not) with poor methodology. I advocate process before product, in other words.
Re: Challenge to scientists: does your ten-year-old code still run?
#219Earlier quoted context omitted.
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes.
I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication.
Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction between experiment and human brain meats that's missing when code is simply reused (whether we consider it apparatus or procedure).
Once we have multiple implementations, if there is a meaningful difference between them, at that point replayability is of tremendous value in identifying why they differ.
But it is not reproducibility, as we want that term to be used in science.
Re: Challenge to scientists: does your ten-year-old code still run?
#220Earlier quoted context omitted.
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…
> But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold
> Plenty of good science got done before modern devops came to be
This isn't as strong of an argument as you think. This is more-or-less the underlying foundation behind the social sciences, which argues that no social sampling can ever be entirely reproduced since no two people are alike, and even the same person cannot be reliably sampled twice as people change with time.
Has there been "good science" done in the social sciences? Sure. I don't think that you're going to find anybody arguing that the state of the social sciences today is about the same as it was in the Dark Ages.
With that said, one of the reasons why so many laypeople look at the social sciences as a kind of joke is because so many contradictory studies come out of these peer-reviewed journals that their trustworthiness is quite low. One of the reasons why there's so much confusion surrounding what constitutes a healthy diet and how people should best attempt to lose weight is precisely because diet-and-exercise studies are more-or-less impossible to reproduce.
> If you can achieve that, great -- it's certainly a useful property to have for debugging
If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.