Research software code is likely to remain a tangled mess • Hacker News Board

Re: Research software code is likely to remain a tangled mess

#21

Post by virgo_eye » Mon, Feb 22, 2021, 12:59 PM UTC

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

Do you think that people doing research at large technical organizations structure their code in the same way as academics? No, although there's always a portion which is active and unstable, they create packages, define interfaces, abstract out pieces which can be reused reliably and depended on. Similarly for other types of researchers in fields where the code is considered an important product. Eg. if you are doing research in compiler design, you're likely to want to create a compiler which can be used by other people. So you make a stable thing with tests, automated builds and so on. And you delimit and instrument the experimental parts.

The real reason is the incentives. Not just are there no incentives to produce good quality code, there are incentives which make people focus on other outputs. Publish or perish means that people put up with technical debt just to get to the next result for the next paper, then do it again and again.

Re: Research software code is likely to remain a tangled mess

#22

Post by childintime » Mon, Feb 22, 2021, 1:00 PM UTC

gmueckl wrote: post #12

j-pb wrote: post #4

Earlier quoted context omitted.

There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.

How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities…

It seems Julia has the answer: https://arstechnica.com/science/2020/10/the-unreasonable-eff...

Re: Research software code is likely to remain a tangled mess

#23

Post by abcc8 » Mon, Feb 22, 2021, 1:00 PM UTC

bjarneh wrote: post #16

I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's anoth…

Agreed. Based on my lab experiences, it might have been more accurate for the author to write that often code is not being written by individuals who are the intellectual drivers of the lab. In many labs the 'thinkers' get far more credit and are more valued than the 'doers'.

Re: Research software code is likely to remain a tangled mess

#24

Post by asdf_snar » Mon, Feb 22, 2021, 1:01 PM UTC

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind being updated after publication.

While others here point out that researchers = bad programmers is a lazy excuse, I think it is important to point out just how steep the learning curve of computer environments can be for the layperson that uses Excel or MATLAB for all their computational work. It can be a huge time investment to get started with tools, such as git or Docker, that we take for granted. I think recognizing this dearth of computer skills is a first step towards training researchers to be computer-competent. Currently, I find the attitude among academics (especially theorists) to be dismissive of the importance of such competencies.

Re: Research software code is likely to remain a tangled mess

#25

Post by milliams » Mon, Feb 22, 2021, 1:07 PM UTC

Disclaimer: I am one of the trustees of the mentioned charity, The Society of Research Software Engineering.

You say that you don't see it having much "difference with regard status and salary". The problem here is two-fold. Firstly, salaries at UK universities are set on a band structure and so an RSE will earn a comparable amount to a postdoc or lecturer. These aren't positions that are known for high wages and historically the reason that people work in research is not for a higher salary.

As for status, I can see that the creation of the Research Software Engineer title (since about 2012) has done great good for improving the status of people with those skills. Before they were "just" postdocs with not many papers but now they can focus on doing what they do best and have career paths which recognise their skills.

My role (at the University of Bristol - https://www.bristol.ac.uk/acrc/research-software-engineering...) is focused almost entirely on teaching. I'm not trying to create a new band of specialists who would identify as RSEs but rather provide technical competency for people working in research so that the code they write is better.

There is a spectrum of RSEs from primarily research-focused postcode who write code to support their work along to full-time RSEs whose job is to support others with their research (almost a contractor-type model). We need to have impact all the way along that spectrum, from training at one end to careers and status at the other.

For more info on the history of the role, there's a great article at https://www.software.ac.uk/blog/2016-08-17-not-so-brief-hist... written by one of the founding members of the Society of Research Software Engineering.

Re: Research software code is likely to remain a tangled mess

#26

Post by bjarneh » Mon, Feb 22, 2021, 1:08 PM UTC

pca006132 wrote: post #18

bjarneh wrote: post #16

I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's anoth…

I think rather than low status academic activity, it is just not valued in the academia... The code is usually just the by-product of the paper, and higher quality code does not translate to higher quality paper. When your code works, you probably already developed the main part of your paper, and you would have no incentives to improve on your program if what you want is just publication... At least this is what I t…

I've seen many research papers where all they disclose about the software is some pseudo code + some tables with timing results to prove their "performance gains".

For those types of papers I agree with your statement. But in many academic scenarios others will want to inspect the source, and the quality of that code is certainly something that will add or subtract from your "status" in the academic world so to speak :-)

Re: Research software code is likely to remain a tangled mess

#27

Post by Dumblydorr » Mon, Feb 22, 2021, 1:09 PM UTC

asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance.

The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the methods section of the article anyway.

So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code.

There's an unwarranted fear of getting your work poached. In modern science, publications are everything, they determine your career. Enabling your direct competitors, those who want the same grants and students and glories, is not common in science.

Re: Research software code is likely to remain a tangled mess

#28

Post by lmm » Mon, Feb 22, 2021, 1:14 PM UTC

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded.

I don't have to imagine it, I'm employed in the software industry.

Seriously, nothing you describe sounds any different from normal software development.

Re: Research software code is likely to remain a tangled mess

#29

Post by bjarneh » Mon, Feb 22, 2021, 1:15 PM UTC

abcc8 wrote: post #23

bjarneh wrote: post #16

I agree that academia produces its fare share of spaghetti code, but I don't think all of his arguments are correct. > writing software is a low status academic activity This is just not true. People like: Stallman, Knuth, Ritchie, Kernighan, Norvig or Torvalds are not considered as people of low status in the academic world. Writing horrible spaghetti code in academia may be considered "low status"; but that's anoth…

Agreed. Based on my lab experiences, it might have been more accurate for the author to write that often code is not being written by individuals who are the intellectual drivers of the lab. In many labs the 'thinkers' get far more credit and are more valued than the 'doers'.

> In many labs the 'thinkers' get far more credit and are more valued than the 'doers'

In terms of software this never made much sense to me. I would understand if we where talking chemistry or some other discipline where a "new idea" has to be investigated/verified by some "lab-rat" doing mundane tasks for 2 years. In that case the lab-rat would probably get less credit than the person with the actual idea, but this just does not apply to software. Developers are not doing mundane tasks on behalf of some great thinker.

Re: Research software code is likely to remain a tangled mess

#30

Post by Dumblydorr » Mon, Feb 22, 2021, 1:18 PM UTC

lmm wrote: post #28

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded. I don't have to imagine it, I'm employed in the software industry. Seriously, nothing you describe sounds any different from normal software development.

In my world, it does sound different, I work with HIPAA data that takes months to get access to. So sharing your code is borderline unacceptable to some orgs, even if it itself doesn't have any privacy data, there's a mass paranoia that you'll accidentally leak patient data, which can lead to fines of 2 million USD.