Challenge to scientists: does your ten-year-old code still run?

The graphics community has started an interesting initiative at this end: http://www.replicabilitystamp.org/

After a paper has been accepted, authors can submit a repository containing a script which automatically replicates results shown in the paper. After a reviewer confirms that the results were indeed replicable, the paper gets a small badge next to its title.

While there could certainly be improvements, I think it's a step in the right direction.

Who says anything about statistical models?

I mean, senior researchers in stable roles don't really do any better. Just to pick the first example off the top of my head - one of the investigators I worked with, during my year as a staff member of an academic institution most of a decade ago, is also one of my oldest friends; he's been a researcher there for what must be well past ten years by now. Despite one of his undergrad degrees being actually in CS, I still find ample reason whenever I see it to give him a hard time about the maintainability of his code.

Like I said before, it's a field in which people really just don't give a damn about engineering. Which is fair! There's little reason why they should, as far as I've ever been able to see.

"Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj... :

> Rule 1: For Every Result, Keep Track of How It Was Produced

> Rule 2: Avoid Manual Data Manipulation Steps

> Rule 3: Archive the Exact Versions of All External Programs Used

> Rule 4: Version Control All Custom Scripts

> Rule 5: Record All Intermediate Results, When Possible in Standardized Formats

> Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds

> Rule 7: Always Store Raw Data behind Plots

> Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected

> Rule 9: Connect Textual Statements to Underlying Results

> Rule 10: Provide Public Access to Scripts, Runs, and Results

... You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo.

... re: [Conda and] Docker container images https://news.ycombinator.com/item?id=24226604 :

> - repo2docker (and thus BinderHub) can build an up-to-date container from requirements.txt, environment.yml, install.R, postBuild and any of the other dependency specification formats supported by REES: Reproducible Execution Environment Standard; which may be helpful as Docker Hub images will soon be deleted if they're not retrieved at least once every 6 months (possibly with a GitHub Actions cron task)

BinderHub builds a container with the specified versions of software and installs a current version of Jupyter Notebook with repo2docker, and then launches an instance of that container in a cloud.

“Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/jou... :

> Rule 6: Present a Sound Data Storage and Preservation Strategy

> Rule 8: Describe How the Data Will Be Disseminated

... DVC: https://github.com/iterative/dvc

> Data Version Control or DVC is an open-source tool for data science and machine learning projects. Key features:

> - Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.

> - Makes projects reproducible and shareable; helping to answer questions about how a model was built.

There are a number of great solutions for storing and sharing datasets.

... "#LinkedReproducibility"

The longest-running code I wrote as a scientist was a sandwich ordering system. I worked for a computer graphics group at UCSF and while taking a year off from grad school while my simulations ran on a supercomputer, and we had a weekly group meeting where everybody ordered sandwiches from a local deli.

It was 2000, so I wrote a cgi-bin in Python (2?) with a MySQL backend. The menu was stored in MySQL, as were the orders. I occasionally check back to see if it's still running, and it is- a few code changes to port to Python3, a data update since they changed vendors, and a mysql update or two as well.

It's not much but at least it was honest work.