Research software code is likely to remain a tangled mess • Hacker News Board

Re: Research software code is likely to remain a tangled mess

#61

Post by medstrom » Mon, Feb 22, 2021, 2:32 PM UTC

rovr138 wrote: post #19

ellimilial wrote: post #6

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.

Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…

>The source of truth is the paper. This is why the paper was published.

Speaking as someone who's not the best at math, I find it easier to understand what a paper is saying after I run the code and see all the intermediate results.

When the code doesn't work, it takes me 20 times longer to digest a paper. They could do with only uploading code -- to me it's the shortest and most effective way to express the ideas in the paper.

Re: Research software code is likely to remain a tangled mess

#62

Post by lou1306 » Mon, Feb 22, 2021, 2:33 PM UTC

knuthsat wrote: post #57

I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released. I'm pretty sure it's like that elsewhere. Machine learn…

> I think there's not enough researchers that publish code.

I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers.

Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]).

Others, while not requiring an artifact evaluation, may encourage it by other means. The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available.

[1] http://i-cav.org/2021/artifact-evaluation/

[2] https://etaps.org/2021/call-for-papers

[3] https://www.acm.org/publications/policies/artifact-review-an...

Re: Research software code is likely to remain a tangled mess

#63

Post by burntoutfire » Mon, Feb 22, 2021, 2:36 PM UTC

lmm wrote: post #28

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded. I don't have to imagine it, I'm employed in the software industry. Seriously, nothing you describe sounds any different from normal software development.

The only difference is speed IMO. Sure, new requirements appear and they can wildly change the underlying assumptions of the systems - but usually, in such case we're given months or years to adapt/rewrite the system in a systematic manner. If, for every wild idea the researcher wants to explore, this amount of rigor was applied in its implementation, I'm guessing the research would slow down immensely. BTW most of research code written for chasing dead ends (quickly testing some small hypetheses), and will be discarded without sharing with anyone - so, investing into writing it properly seems especially wasteful.

Re: Research software code is likely to remain a tangled mess

#64

Post by Frost1x » Mon, Feb 22, 2021, 2:38 PM UTC

dariosalvi78 wrote: post #7

I think that incentives play a big role here. Software has near to zero value in academic evaluation and even less its update and maintenance. The only way to make research software survive is to offer packages that other researchers can also use. Maybe.

This is changing drastically. The issue is that more and more science relies heavily on computation. Analytic platforms, computational science, modeling/simulation, etc. There's less "bench" science and more of the scientific process is being embedded in software.

There's a certain degree of naivity in this process that SMEs think it's a trivial step translating their research into software. It's not, not if you demand the rigor science should be operating at. As such, many budgets are astronomically lower than they should be. This has worked in the past but as more science moves into software and it becomes more critical to the process, you must invest in the software and it's not going to be cheap. The shortcuts taken in the past won't cut it.

There's a bigger issue in that as a society we don't want to invest in basic research so it's already cash strapped. Combine that with research scientists who already have to cut corners with the massive cost quality software will take and you're creating a storm where science will either produce garbage or well need to reevaluate how we invest in software systems for science.

Re: Research software code is likely to remain a tangled mess

#65

Post by screye » Mon, Feb 22, 2021, 2:41 PM UTC

> writing software is a low status academic activity

Yep, that's the one liner right there.

The incentives simply do not match the complaints. Researchers already work upwards of 60 hrs/wk on most occasions. Alongside writing code, they also have to do actual research, write papers, give talks and write grants.

All of the latter tasks are primary aspects of their jobs and are commensurately rewarded. The only situation where a well coded tool is rewarded, is when a package blows up, which is quite rare.

Like all fields, the high-level answer to such questions is rather straightforward. The individual contributors align their efforts to the incentives. Find a way to incentivize good research code, and we will see changes overnight.

Re: Research software code is likely to remain a tangled mess

#66

Post by abcc8 » Mon, Feb 22, 2021, 2:43 PM UTC

bjarneh wrote: post #29

abcc8 wrote: post #23

Earlier quoted context omitted.

Agreed. Based on my lab experiences, it might have been more accurate for the author to write that often code is not being written by individuals who are the intellectual drivers of the lab. In many labs the 'thinkers' get far more credit and are more valued than the 'doers'.

> In many labs the 'thinkers' get far more credit and are more valued than the 'doers' In terms of software this never made much sense to me. I would understand if we where talking chemistry or some other discipline where a "new idea" has to be investigated/verified by some "lab-rat" doing mundane tasks for 2 years. In that case the lab-rat would probably get less credit than the person with the actual idea, but this…

Rarely are the developers writing grants, generating hypotheses, planning experiments, composing manuscripts, presenting the lab's work, teaching at the university, etc. Perhaps my choice of words was poor, but this should be more clear.

Re: Research software code is likely to remain a tangled mess

#67

Post by nirse » Mon, Feb 22, 2021, 2:43 PM UTC

bjarneh wrote: post #16

I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's anoth…

>> writing software is a low status academic activity > This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world.

I understand the meaning of 'academic software developers' to mean 'software developers that assist in building software for other, non-CS, fields of research', but you only mention people famous within CS. I don't think this article is meant to apply to CS.

Re: Research software code is likely to remain a tangled mess

#68

Post by nirse » Mon, Feb 22, 2021, 2:49 PM UTC

What doesn't really get mentioned in the article, is that a lot of academic software was written by a single developer. All bigger software projects, academic or not, that were only built and maintained by a single person tends to become messier and messier with time. Perhaps most software suffers from that, that over time it becomes a mess, but having more developers look at code (and enough time, and many other factors) can certainly help to keep things in better shape.

Re: Research software code is likely to remain a tangled mess

#69

Post by jimmyvalmer » Mon, Feb 22, 2021, 2:50 PM UTC

rovr138 wrote: post #19

ellimilial wrote: post #6

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.

Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…

> The source of truth is the paper.

Yes, although truth of the flimsiest kind. A lowly but wise code monkey once said "Talk is cheap. Show me the code."

Re: Research software code is likely to remain a tangled mess

#70

Post by SavageBeast » Mon, Feb 22, 2021, 2:51 PM UTC

I'm currently refactoring a fairly large piece of research code myself. It was written with lean startup thinking in that a little code ought to produce some value in its results. If i was able to eeek some usefulness out of this code, then Id put more energy into it. Otherwise I was perfectly happy to Fail Fast and Fail Cheap.

How did it become such a mess in the first place? Simple - I didn't know my requirements when I started writing it. I built it to do one thing. In running it I learned more things (this is good - why you build stuff like this in the first place). The code changed rapidly to accommodate these lessons.

It wasn't long before I was running into limitations in the design of the underlying libs I was using etc. Of course I could find a way to make it work but it wasn't going to win any Software Design Awards.

Im happy to report that despite ending up a tangled mess, it actually helped me to come to understand and conquer a very specific kind of problem. In doing so I learned the limitations of commercially available tooling, the limitations of commercially available data, not to mention a great deal about the problem domain itself.

This research software has earned its keep and is now being cleaned up into a more organized, near commercial quality kind of project. Im glad I threw out "architecture" when I first started with this. It could have gone the other way where I had a very well built piece of code that didn't in fact perform any useful function.