Research software code is likely to remain a tangled mess

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change.

I'd say you're confirming the author's theory that writing code is a low-status activity. Papers and citations are high-status, so papers are well refined after the research is "done". Code, however, is not. If the code was considered on the same level as the paper, I think people would refine their code more after they finish the iteration process.

There is no real incentive to organize and clean up the code, even if the scientists involved have the skills to write well-organized software. And organizing this kind of code that often starts in a more exploratory way is a pretty large amount of additional effort. This kind of effort is simply not appreciated, and if spending time on it means you publish fewer papers it's a net negative for your career.

I'd settle for just publishing the code at all, even if it is a tangled mess. This is still not all that common in the natural sciences, though I have a bit of hope this will change.

If you want to write good research software, a good way is to have professional developers implement it.

I worked closely with an NLP researcher for a while on a project that had received a hefty state grant. She knew more or less what her team needed, but she needed someone to implement it cleanly and in a way that would not make users step on each others toes.

The chances of that project being a buggy mess would have been pretty high if it had been written by people who don't write software for a living. And maybe that's OK.

I don't think research code in aggregate is any worse than any other source for code. If we had the same kind of visibility into all the commercially written code, it would be the same pattern of some well structured, and some a complete mess, without any correlation with the companies concerned, but with a lot of correlation to the ability of the author.

The recent example of Citibank´s loan payment interface comes immediately to mind. So does Imperial's Covid model (the one that had timing issues when run on different computers.)

Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot.

Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future?

An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it.

There’s also this part that strikes me,

>Given a tangled mess of source code, I think I could reproduce the results in the associated paper (assuming the author was shipping the code associated with the paper; I have encountered cases where this was not true).

And it strikes me as weird. The main issue to reproduce results is usually data. And depending on the dataset, it’s very hard to get. To be able to reproduce the code, I just need the paper.

The code may have bugs, may stop working, may be in a different language/framework. The source of truth is the paper. This is why the paper was published.