Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research software code is likely to remain a tangled mess

shape-of-code.coding-guidelines.com

Post reply on HN
Print view View original

31–40 of 171 posts

Re: Research software code is likely to remain a tangled mess

#31

Post by WanderPanda »

j-pb wrote: post #4
svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.

In my experience simplicity and generality don‘t go well with performance. If you want to build something that can be used for all kinds of problems and it is simple it will be slow as hell compared to the (dirty) optimised code running hardcoded structures on the GPU

Re: Research software code is likely to remain a tangled mess

#32

Post by orange_tee »

Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points,

1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self contained

2. when I first wrote research software I applied all the programming best practices and was told these weren't any good; turns out using lots of abstraction to increase modularity makes the code much slower, this is language dependent of course

3. you will find it much harder to read research code if you don't understand the math+science behind it

> many of those writing software know very little about how to do it

This is just not true. I found in my experience that people writing research software have a very specific skillset that very very few industry programmers are likely to have. They know how to write good numerics code, and they know how to write fast code for super computers. Not to mention, interpreting the numerics theory correctly in the first place is not a trivial matter either.

Re: Research software code is likely to remain a tangled mess

#33

Post by pca006132 »

bjarneh wrote: post #26
pca006132 wrote: post #18

Earlier quoted context omitted.

I think rather than low status academic activity, it is just not valued in the academia... The code is usually just the by-product of the paper, and higher quality code does not translate to higher quality paper. When your code works, you probably already developed the main part of your paper, and you would have no incentives to improve on your program if what you want is just publication... At least this is what I t…

I've seen many research papers where all they disclose about the software is some pseudo code + some tables with timing results to prove their "performance gains". For those types of papers I agree with your statement. But in many academic scenarios others will want to inspect the source, and the quality of that code is certainly something that will add or subtract from your "status" in the academic world so to speak…

OK, perhaps I should read more papers :)

Re: Research software code is likely to remain a tangled mess

#34

Post by Doctor_Fegg »

asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

> it is never tested, reproduced, or published

This never ceases to amaze me. I regularly read recent papers on shortest-path algorithms. Each one is religiously benchmarked down to the level of saying what C++ compiler was used. But the code itself is almost never published.

Re: Research software code is likely to remain a tangled mess

#35

Post by svalorzen »

Zababa wrote: post #11
svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. I'd…

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment.

At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future research efforts will not benefit from the clean up for the same reasons I outlined in my previous post.

While easing code review for other people is definitely helpful (it can still be done if one really wants to, and clean code does not guarantee that people will look at it anyway), overall the gains are smaller than what "standard" software engineers might assume. And I'm saying this as a researcher that always cleans up and publishes his own code (just because I want to mostly).

Re: Research software code is likely to remain a tangled mess

#36

Post by xgb84j »

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

I think software quality in research has nothing to do with the problems themselves. It's more like article suggests that nobody cares about your software. The only goal is to get published and be cited as many times as possible. Your coding mistakes don't matter if they cannot be found out or hurt your reputability.

How many tests would be written for business software if it had only to run for one meeting and then never be looked at again?

Re: Research software code is likely to remain a tangled mess

#37

Post by asdf_snar »

Dumblydorr wrote: post #27
asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the m…

I don't disagree with you on any points. I have some academic friends who mostly do "a ton of munging and then some basic stat functions", as you say (but with less sensitive data). The problem is that their workflow is prone to human error. Even though the stat functions are simple, the proper labeling of inputs and outputs is less reliable.

I have some research published for which I wrote MATLAB code years ago. I trust the fundamental results but not the values displayed in the tables. I would have personally benefited from rudimentary version control and unit testing.

Re: Research software code is likely to remain a tangled mess

#38

Post by ArtWomb »

There's a huge digital divide forming as well. Between the hardware a junior software engineer at a well funded research institution such as DeepMind has access to. Compared to the postdoc in Theoretical Physics at Princeton. Who is expected not only to write software. But maintain hardware for a proprietary "supercomputer" that was probably cast off ages ago from a government lab or wall street.

We don't expect Aerospace / Mechanical engineering students to learn metalworking. They typically have access to shop technicians for that work. Why not persuade university administrators to similarly invest in in-house software engineering talent. Generalists who can provide services to any problem domain: from digital humanities to deep reinforcement learning?

Re: Research software code is likely to remain a tangled mess

#39

Post by txdv »

asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

Reproducibility is a major principle of the scientific method.

Yet computer scientists consistently fail to achieve reproducibility with a tool that is the most consistent at following instructions - the computer.

Even private business is on the DevOps movement, because they see the positive effects of reproducibility.

If the academic world is truly about science, then there is no more excuse, the tools are out there, they need to use them.

Re: Research software code is likely to remain a tangled mess

#40

Post by speters »

rovr138 wrote: post #19
ellimilial wrote: post #6

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.

Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…

> Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future?

You can include the hash of the commit used for your paper.

Post reply on HN