411–420 of 487 posts
Fortran was the first language I learned, and naming things was a problem.
Guix, mentioned in the fine article, or like it is the answer as long as we keep reading and writing our code. This is why all scientific code should be public, we need continuous review.
Papers without code should be refused.
I'm in the middle of a storm in the west of Ireland. It's windy and wet and hn is still a thing. Happy out
This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
Earlier quoted context omitted.
> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of s…
> Asking a scientist to do better than whole teams of software engineers makes little sense to me. This is not what is being asked, shame on you for the strawman. Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".
The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness".
Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".
This seems like a fluff piece because: 1) Prototype code scientists write tends to be written at a high level, so barring imported libraries not up and disappearing, there is a high chance that code written by scientists will run 10 years later. There is a higher chance it will run than production code written at a lower level. 2) The article dives into documentation but scientists code in the Literate Programming Pa…
> scientists code in the Literate Programming Paradigm I wish. In my career as a computational scientist I have never seen this in practice, either in academia or industry. On unit testing, I half agree. Most unit tests get quickly thrown out as the code changes, so it's a depressing way to write research code. But tests absolutely help someone trying to run old code - they show what parts still work and how to use t…
It is sort of hilarious how many breaking changes there are even between minor versions of these.
Thank gosh I was able to install a 2 year old version of pytorch (0.4.1) today to reproduce some code from within the last year.
Earlier quoted context omitted.
> I would have thought the scientists among us would approach someone with familiarity with software development expertise. Is there a pool of skilled software architects willing to provide consultations at well-below market wages? Or a Q&A forum full of people interested in giving this kind of advice? (StackOverflow isn't useful for this; the allowed question scope is too narrow.) I guess one incentive to publish on…
Are the hiring scientists also paid well-below market wages?
Yes. Well, in engineering anyway. That's why most engineers use academia as a stepping stone to something else. Working in science is, I think, sort of like working at a startup that's perpetually short on cash with no possibility of an exit.
With a little digging you can find published tables of job codes and salaries for many universities, e.g., https://www.udel.edu/content/dam/udelImages/human-resources/...
Faculty are listed separately for some reason: https://www.udel.edu/faculty-staff/human-resources/compensat...
Positions requiring a PhD start being listed at 29E (midrange of $55k) or 30E (midrange of $63,800). You could easily get that with a bachelor's degree in engineering 10 years ago. I suspect you will find the "Information Technology" Job Family salaries particularly amusing.
Earlier quoted context omitted.
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
This reminds me of being in gradschool and the comp-sci people complaining that we don't get bit-wise equal floats when we solve DEs.
Having to re-implement a library from scratch for a project is much more valuable than running the same code in two places. The same way that getting the same results from two different machines is a lot more significant than getting the same result from two cloned machines.
In short: code does not need to be reproducible because scientists know how to average.
Earlier quoted context omitted.
That brings up a separate issue that I didn't comment on above: the expectation that the code runs in a completely different development/execution environment (e.g. the one the reviewer is using vs. the one that the researcher used). That means making it run regardless of the OS (Windows/OSX/Linux/...) and hardware (CPU/GPU/TPU, and even within those, which one) the reviewer is using. This would be an extremely diffi…
I don't think that removes the need to provide enough detail to replicate the original environment though. We write one-off scripts with no expectation that they will see outside usage, whereas research publications are meant for just that! The bar isn't terribly high either: for ML, a requirements.txt + OS version + CUDA version would go a long way, no need to learn docker just for this.
Because I have and I pity anyone who tries and builds a kernel that can run it.
The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding. The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding. Concerns about software quality in science are primarily about avoiding…
Reproducibility is about understanding the result. It is the modern version of "showing your work". One of the unsung and wonderful properties of reproducible workflows is the fact that it can allow science to be salvaged from an analysis that contains an error. If I had made an error in my thesis data analysis (and I did, pre-graduation), the error can be corrected and the analysis re-run. This works even if the aut…
That is something no one outside of highschool cares about. The idea that you can show work in general is ridiculous. Do I need to write a few hundred pages of set theory to start using addition in a physics paper? No. The work you need to show is the work a specialist in the field would find new, which is completely different to what a layman would find new.
Every large lab, the ones that can actually reproduce results, has decades of specialist code that does not interface with anything outside the lab. Providing the source code is then as useful as giving a binary print out of an executable for an OS you've never seen before.