Challenge to scientists: does your ten-year-old code still run?

If a scientist needs to write code then it's part of their job. It's as easy as that.

I tend to do the following (some or all, depending on the situation):

- Use known, plaintext formats like LaTeX, Markdown, CSV, JSON, etc. rather than undocumented binary formats like those of Word, Excel, etc.

- Keep sources in git (just a master branch will do)

- Write all of the rendering steps into a shell script of Makefile, so it's just one command with no options

- I go even further and use Nix, with all dependencies pinned (this is like an extreme form of Make)

- Code for generating diagrams, graphs, tables, etc. is kept in git alongside the LaTex/whatever

- Generated diagrams/graphs/tables are not included in git; they're generated during rendering, as part of the shell-script/Makefile/Nix-file; the latter only re-generate things if their dependencies have changed

- All code is liberally sprinkled with assertions, causing a hard crash if anything looks wrong

- If journals/collaborators/etc. want things a certain way (e.g. a zip file containing plain LaTeX, with all diagrams as separate PNGs, or whatever) then the "rendering" should take care of generating that (and make assertions about the result, e.g. that it renders to PDF without error, contains the number of pages we're expecting, that the images have the expected dimensions, etc.)

- I push changes from my working copies into a 'repos' directory, which in turn pushes to my Web server and to github (for backups and redundancy)

- Pushing changes also triggers a build on the continuous integration server (Laminar) running on my laptop. This makes a fresh copy of the repo and tries to render the document (this prevents depending on uncommitted files, the absolute directory path, etc.)

Referencing a particular git commit should be enough to recreate the document (this can also be embedded in the resulting document somewhere, for easy reference). Some care needs to be taken to avoid implicit dependencies, etc. but Nix makes this much easier. Results should also be deterministic; if we need pseudorandom numbers then a fixed seed can be used, or (to prove there's nothing up our sleeves) we can use SHA256 on something that changes on each commit (e.g. the LaTeX source).

For computationally-expensive operations (with relatively small outputs) I'll split this across a few git repos:

1) The code for setting up and performing the experiments/generating the data goes in one repo. This is just like any other software project.

2) The results of each experiment/run are kept in a separate git repo. This may be a bad idea for large, binary files; but I've found it works fine for compressed JSON weighing many MBs. Results are always appended to this repo as new files; existing files are never altered, so we don't need to worry about binary diffs. There should be metadata alongside/inside each file which gives the git commit of the experiment repo (i.e. step 1) that was used, alongside other relevant information like machine specs (if it depends on performance), etc. This could be as simple as a file naming scheme. The exact details for this should be written down in this repo, e.g. in a README and/or a simple script to grab the relevant experiment repo, run it, and store the results+metadata in the relevant place. Results should be as "raw" as possible, so that they don't depend on e.g. post-processing details, or choice of analysis, etc.

3) I tend to put the writeup in a separate git repo from the results, so that those results can be referenced by commit + filename, without a load of unrelated churn from the writeup. This repo will follow the same advice as above, e.g. code for turning the "raw" results into graphs, tables, etc. will be kept here and run as part of the rendering process. Fetching the particular commit from the results repo should also be one of the rendering steps (Nix makes this easy, or you could use a git submodule, etc.)

I don't know what the best advice is w.r.t. large datasets (GBs or TBs), but I've found the above to be robust for about 5 years so far.

An interesting concern is that there often is no single piece of code that has produced the results of a given paper.

Often it is a mixture of different (and evolving) versions of different scripts and programs, with manual steps in between. Often one starts the calculation with one version of the code, identifies edge cases where it is slow or inaccurate, develops it further while the calculations are running, does the next step (or re-does a previous one) with the new version, possibly modifying intermediate results manually to fit the structure of the new code, and so on -- the process it interactive, and not trivially repeatable.

So the set of code one has at the end is not the code the results were obtained with: it is just the code with the latest edge case fixed. Is it able to reproduce the parts of the results that were obtained before it was written? One hopes so, but given that advanced research may take months of computer time and machines with high memory/disk/CPU/GPU/network speed requirements only available in a given lab -- it is not at all easy to verify.

I am interested to know the distinction between "production-ready" and "science-ready" code.

I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

I think it's unfair to expect from anyone to maintain code forever when the code rot is completely beyond your control, let alone to expect this from scientists who have better things to do. Anything with a GUI is bound to self-destruct, for example, and it's not the programmer's fault. Blame the OS makers and framework/3rd party library suppliers.

The damage can be limited by choosing a programming language that provides good long compatibility. Languages like ANSI C, Ada, CommonLisp, and Fortran fit the bill. There are many more. Heck, you could use Chipmunk Basic. Anything fancy and trendy will stop working soon, though, sometimes even within a year.