Bad scientific code beats code following "best practices" (2014)

In practice, most research code (including supposedly reusable components) ends up getting written in a slipshod ad-hoc way by grad students with high turnover. It typically has poor choice of basic abstractions, poor documentation, limited testing, regressions from version to version, etc. Researchers make do with what they can, and mainly focus on their written (journal paper) output rather than the quality or project health of the code.

First, scientific code is exploratory: you start building it without knowing which way will work. For example, you end up with 10 different fit functions, all using different approach (multi-gaussian, kde of control region, home-made shape based on a sum of Gumbel and Gaussian fcts, ...), and during your study, you do your test to conclude which one is the best. Because of that, "building for the future the multi-gaussian fit fct" is just plain stupid: you are spending time for something that have a high probability (>75%) of being totally dropped next week. You also need to build for huge flexibility. For example, having some "god object" allows to pivot very quickly when suddenly you discover a new parameter to add to the equation, rather than to refactorise all the function one by one to add a new parameter of a very specific type to all of them. In this context, the person who codes with strong static typing is not being very smart: in research, their code will change very often and they are just building a tool that will waste their time.

Second, the code is the tool, not the product. The scientific conclusion is the product. What needs to be reproducible is the conclusion. You apparently don't understand the goal of reproducibility. Reproducibility is to prove that 2 persons who redo it from scratch obtain the same results, even if they use different tool. By saying that it is important that a scientist B can just blindly run the software of a scientist A, you just screw the reproducibility advantage: if the software has a bug, scientist B will say "I get the same result, so, it is confirmed" while the result is incorrect. At CERN, people from the CMS experiment and people from the Atlas experiment are FORBIDEN to share their code for this reason. Even reading the code can be bad because it can bias the reader. Then, of course, there is a balance and I recommend to share the code with publication, but the reader should be educated enough to know that the code is the last resort to look at if they have a question when trying to reproduce the result.

Third, the way scientists work and collaborate creates to some extend "intrinsic" code review and unit testing. Several teams build software in parallel to check the same thing, and will investigate by comparing their result. And scientists are the first user of their software, intensively: they will never build a feature that they will not themselves use in practice later. So, some of the good practices of software dev are just not well designed for these situations.

At the end, you are right: serving the food on the floor is bad. The software developer "good practice" are serving the food on the floor because they assumed the dish was where it is not.

Finally, one thing to keep in mind is that some software dev people are just full of themselves with big ego. They love to think they are smarter than everyone else. If they are applying good practices themselves, they love feeling superior by thinking the ones not applying them are inferior and that they are so smart. It's a way of rationalising their traditions, and it's then easy to see plenty of ways that confirm this belief.

Don't get me wrong, plenty of good practices are really useful, and plenty of scientific code is just badly done. But you just need to have a balance: instead of blindly applying tradition just because it flatter their ego, people should just take a step back and think of why X and Y is better in some context and see how it translates in another context.

If you want to host the data on Dropbox, Dropbox becomes a part of the network and is hosted by the community and Dropbox or whatever service you like. The problem is that it is very prohibitive to use services like AWS. For instance, downloading all of Arxiv.org from AWS is around $600 last time I checked. Not every student trying to run an experiment is going to have $600 laying around to run the experiment. But with torrent/ipfs it would be free to download.

It only adds. I don't understand how it subtracts.

If you think that it is the core of my complaint, then you did not understood at all.

This element is just there to say that it's easy to not understand the situation of the scientists and to conclude that it's just because they are stupid or they are not doing the things right.

You were the one talking about "poor practices induced by the corrupt incentives of academia", but this is just incompatible with the reality: in region where the "publish or perish" has no impact (and these regions exist, only people who know academia superficially don't realize that), we still see the same software development "mispractice". This is why I'm talking about ego: it is so easy for you to just rationalize different practice as "being somehow forced to do a bad job".

> we agree that reproducibility is important

I don't think you understand what I call "reproducibility". In science, reproducibility means FROM SCRATCH! You know the result, and you rebuild your own experiment. If you copy the experiment of the first author, you will copy the experimental error too.

Reproducibility is one of the reason some of the thing you pretend is a good practice is in fact not ideal in the context of science (as I've said, I myself push for sharing a clean code, but it is just that some of the important goal for usual software development don't exist in the same way in the scientific development context).

> but you don't understand software engineering as a practice

I understand very well why the good practice are indeed very important in the context of the usual software development. Again, you are just rationalising: you see m not agreeing with you, so you invent that I don't understand why good practices are used. It's again an easy way to not challenge what you consider as obvious.

> Good engineering is absolutely not about "blindly applying tradition".

Exactly. People who are jumping on the conclusion "if the scientists are not using these practice, then they are doing a bad job" are bad engineers.

You keep talking about good engineering practice as if these practices are always the correct way to go. Except it is not the case. As given in some example, wasting time on a "clean" architecture on something that the author is smart enough to know that it has a high probability to disappear in one week is not a "good practice". Yet, you are still talking to me as if this practice is "obviously good". It is very good for usual software development, when you have a large set of users and you want to maintain the software for years. But it makes no sense when the software is for the author of the software only and that everyone agrees it would be better to write a proper implementation of the final conclusion rather than to mix the goal and build a robust tool before exploring.

So, yes, you are blindly following the tradition: you just refuse to even consider that maybe what you have adopted in your mind as "good" may not be the best in a different context.