11–20 of 487 posts
The day when code used to produce a paper must also be published can not come soon enough.
You often run into code of the "just get it to work" variety, which has the problem that when it was written, maintainability was bottom of the list of priorities. Often the author has a goal that isn't described in terms of software engineering terms: calculate my option model, work out the hedge amounts, etc. And the people who write this kind of code tend not to think about version control, documentation, dependen…
The "Software Carpentry" movement [1] has in the past decade tried to address this, as I recall. It's very much in the vein of the "basic literacy" course you suggest. I can't say how far they've gotten, and I'm no longer adjacent to academia, but based on what I do still see of academics' code, there's a long way still to go.
https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022...
This was used by the Imperial College for COVID-19 predictions. It has race conditions, seeds the model multiple times, and therefore has totally non-deterministic results[0]. Also, this is the cleaned up repo. The original is not available[1].
A lot of my homework from over 10 years ago still runs (Some require the right Docker container: https://github.com/sumdog/assignments/). If journals really care about the reproducibility crisis, artifact reviews need to be part of the editorial process. Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals.
[0] https://lockdownsceptics.org/code-review-of-fergusons-model/
The day when code used to produce a paper must also be published can not come soon enough.
I mean, Python 2->3 alone is gonna kill this challenge for most people.
IMO there is a big fallacy about the "just get it to work" approach. Most serious scientific code, i.e. supporting months-years of research, is used and modified a lot. It's also not really one-off, it's a core part of a dissertation, or research program, if it fails- you do. I'd argue that (and I found that), using unit-tests, a deployment strategy, etc. ultimately allowed me to do more, and better science because in the long run I didn't spend as much time figuring out why my code didn't run when I tweaked stuff. This is really liberating stuff. I suspect this is all obvious to those who have gone down that path.
Frankly, every reasonably tricky problem benefits from unit-tests as well for another reason. Don't know how to code it, but know the answer? Assert lots of stuff, not just one at a time red-green style. Then code, and see what happens. So powerful for scientific approaches.
You often run into code of the "just get it to work" variety, which has the problem that when it was written, maintainability was bottom of the list of priorities. Often the author has a goal that isn't described in terms of software engineering terms: calculate my option model, work out the hedge amounts, etc. And the people who write this kind of code tend not to think about version control, documentation, dependen…
The fundamental problem here, as you note, is that scientists are rarely also engineers, and don't really share our desiderata. The point is to develop and publish a result, and engineering analysis code for resiliency is of secondary concern at best when that code isn't likely to need to be used again once the paper is finished. The "Software Carpentry" movement [1] has in the past decade tried to address this, as I…
When I started working as a full-time researcher, I had come from working two years in a software shop, only to find people at the research lab having never used VCS, object-oriented programming, etc. Everyone just put together a few text files and Python or MATLAB scripts that output some numbers that went into Excel or gnuplot scripts that got copy-pasted into LaTeX documents with suffixes like "v2_final_modified.tex", shared over Dropbox.
Took a long time to establish some coding standards, but even then it took me a while to figure out that that alone didn't help: you need a proper way to lock dependencies, which, at the time, was mostly unknown (think requirements.txt, packrat for R, …).