Research software code is likely to remain a tangled mess • Hacker News Board

Re: Research software code is likely to remain a tangled mess

#71

Post by Robotbeat » Mon, Feb 22, 2021, 2:52 PM UTC

nirse wrote: post #68

What doesn't really get mentioned in the article, is that a lot of academic software was written by a single developer. All bigger software projects, academic or not, that were only built and maintained by a single person tends to become messier and messier with time. Perhaps most software suffers from that, that over time it becomes a mess, but having more developers look at code (and enough time, and many other fac…

Plus it becomes impossible to get multiple developers to work on the code if they can’t understand it because of its messiness, so there’s a bit of survivor’s bias and stronger motivation to clean the code up to be comprehensible to others when you have multiple people working on it.

Also, I feel personally attacked by the headline. :)

It is for this reason I try to keep my code and models pretty simple, only two or three pages of code (or ideally a single page), and I don’t try to do too many things with one program, and I choose implementations and algorithms that are simpler to implement to make concise code feasible (sometimes at the expense of speed or generality).

Re: Research software code is likely to remain a tangled mess

#72

Post by Frost1x » Mon, Feb 22, 2021, 2:53 PM UTC

virgo_eye wrote: post #21

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

Do you think that people doing research at large technical organizations structure their code in the same way as academics? No, although there's always a portion which is active and unstable, they create packages, define interfaces, abstract out pieces which can be reused reliably and depended on. Similarly for other types of researchers in fields where the code is considered an important product. Eg. if you are doin…

>The real reason is the incentives. Not just are there no incentives to produce good quality code, there are incentives which make people focus on other outputs. Publish or perish means that people put up with technical debt just to get to the next result for the next paper, then do it again and again.

I believe this is true and is fueled by a misconception of what software is in research. Software in research is often akin to experimentalist work in the past. It's tacked onto theoretical work projects as an afterthought and not treated as what it really is: forcing the theory to be tested in a computational environment.

If we start treating research software like experimentalism in the past, we might get a bit more rigor out of the development process as well as the respect it really deserves.

Re: Research software code is likely to remain a tangled mess

#73

Post by currymj » Mon, Feb 22, 2021, 2:53 PM UTC

as people are saying, the typical software engineering advice simply wouldn't work in a research context.

one exception is the most basic stuff - people should use version control, do light unit testing, and explicitly track dependencies. These weren't really done in the past but are becoming more and more common, fortunately.

I think if software engineering experts actually sat down, looked at how researchers work with computers, and figured out a set of practices to follow that would work well in the research context, they could do a lot of good. This is really needed. But the standard software engineering advice won't work as it is, it has to be adapted somehow.

Re: Research software code is likely to remain a tangled mess

#74

Post by jimmyvalmer » Mon, Feb 22, 2021, 2:54 PM UTC

stult wrote: post #42

Dumblydorr wrote: post #27

Earlier quoted context omitted.

I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the m…

In well designed software, data ingestion should be easily separable from the core logic of the application. Which is the point the parent comment is making. Some basic best practices would allow you to share your core code without implicating HIPAA. Even if it’s just basic stats, sharing the code makes it easier to reproduce your results and to check your logic. Although I agree with your analysis that enabling comp…

> enabling competitors in science ... really, really should be.

Said someone whose livelihood doesn't depend on said competition.

Re: Research software code is likely to remain a tangled mess

#75

Post by j-pb » Mon, Feb 22, 2021, 2:54 PM UTC

WanderPanda wrote: post #31

j-pb wrote: post #4

Earlier quoted context omitted.

There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.

In my experience simplicity and generality don‘t go well with performance. If you want to build something that can be used for all kinds of problems and it is simple it will be slow as hell compared to the (dirty) optimised code running hardcoded structures on the GPU

Simplicity pretty much excludes generality in a lot of cases, you're only able to port code to the GPU if it wasn't a million LOC to begin with, so you're pretty much making the case for it.

Note that Simple != Easy or Naive

Hardcoded structures is potentially exactly the kind of simplicity needed.

What's not simple is a general "this solves everything and beyond" code-base with every imaginable feature and legacy capability.

Re: Research software code is likely to remain a tangled mess

#76

Post by pydry » Mon, Feb 22, 2021, 2:55 PM UTC

currymj wrote: post #73

as people are saying, the typical software engineering advice simply wouldn't work in a research context. one exception is the most basic stuff - people should use version control, do light unit testing, and explicitly track dependencies. These weren't really done in the past but are becoming more and more common, fortunately. I think if software engineering experts actually sat down, looked at how researchers work w…

Another issue is that the standard software engineering advice doesn't guarantee clean code either.

Re: Research software code is likely to remain a tangled mess

#77

Post by j-pb » Mon, Feb 22, 2021, 3:00 PM UTC

gmueckl wrote: post #12

j-pb wrote: post #4

Earlier quoted context omitted.

There's only one way to solve this: Simplicity. Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.

How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities…

How the heck to you hope to gain any insighfull metrics when you've got a cobbled together mess that you only half understand. For what it's worth you might only be benchmarking random code layout fluctuations.

I've seen research groups drown in their legacy code base.

The issue of juggling too many balls you describe is one you only have to begin with because the state of the art implementations are so shoddy to begin with.

Research suffers as much as everybody else from feature creep. Good experiments keep the number of new variables low.

Re: Research software code is likely to remain a tangled mess

#78

Post by wbl » Mon, Feb 22, 2021, 3:00 PM UTC

lmm wrote: post #28

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> To make a concrete example, imagine writing an application where requirements changed unpredictably every day, and where the scope of those changes is unbounded. I don't have to imagine it, I'm employed in the software industry. Seriously, nothing you describe sounds any different from normal software development.

The program I wrote for my dissertation is as good as it needs to be for a program that had to run once!

Re: Research software code is likely to remain a tangled mess

#79

Post by jimmyvalmer » Mon, Feb 22, 2021, 3:01 PM UTC

einpoklum wrote: post #48

One of the things which has helped derail my own research career [1] is the tendency to not write tangled-mess code, and to publish and maintain much of my research code after I was supposed to be done with it. Annoyingly, more people now know of me due to those pieces of software than for my research agenda. :-( [1] : Not the only thing mind you.

C'mon, no good deed goes unpunished. Everyone knows that.

Re: Research software code is likely to remain a tangled mess

#80

Post by gmueckl » Mon, Feb 22, 2021, 3:01 PM UTC

childintime wrote: post #22

gmueckl wrote: post #12

Earlier quoted context omitted.

How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities…

It seems Julia has the answer: https://arstechnica.com/science/2020/10/the-unreasonable-eff...

I can't quite follow what the article is trying to describe because of the heavy use of analogies.

A Google search makes it look like Julia has a mechanism where you can extent the sets of overloads of a function or method outside the original module. The terminology is different (functions have methods instead of overloads in their speak). I don't see how that feature solves the problem in practice.