Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research software code is likely to remain a tangled mess

shape-of-code.coding-guidelines.com

Post reply on HN
Print view View original

91–100 of 171 posts

Re: Research software code is likely to remain a tangled mess

#91

Post by jonnycomputer »

Zababa wrote: post #52
svalorzen wrote: post #35

Earlier quoted context omitted.

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment. At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future rese…

> At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. I assumed that most code published could be directly useful as an application or a library. Considering what you're saying, this might be only a minority of the code. In that case, I agree with your conclusion about smaller gains.

Most academic code runs once, on one collection of data, on a particular file system.

Academic code can be really bad. But most of the time it doesn't matter, unless they're building libraries, packages, or applications intended for others. That's when it hurts and shows.

I'm a research programmer. I have a master's in CS. I take programming seriously. I think academic programmers could benefit from better practice. But I think software developers make the mistake of thinking that just because academics use code the objective is the same or that best practices should be the same too. Yes, research code should perform tests, though that should mostly look like running code on dummy data and making sure the results look like you expect.

Re: Research software code is likely to remain a tangled mess

#92

Post by jmcdl »

svalorzen wrote: post #35
Zababa wrote: post #11

Earlier quoted context omitted.

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. I'd…

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment. At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future rese…

Shouldn't checking for bugs be of primary importance. How many times have impressive research results turned out to be a mirage built upon a pile of buggy code? I get the sense that is far too common already.

Re: Research software code is likely to remain a tangled mess

#93

Post by analog31 »

SavageBeast wrote: post #70

I'm currently refactoring a fairly large piece of research code myself. It was written with lean startup thinking in that a little code ought to produce some value in its results. If i was able to eeek some usefulness out of this code, then Id put more energy into it. Otherwise I was perfectly happy to Fail Fast and Fail Cheap. How did it become such a mess in the first place? Simple - I didn't know my requirements w…

I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable.

The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up.

Because changing your code could always break it, refactoring also reinforces the habit of writing code that can be readily tested -- also a good thing.

Re: Research software code is likely to remain a tangled mess

#94

Post by vharuck »

Dumblydorr wrote: post #27
asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the m…

As a public health statistician, I am very grateful to all researchers who publish code. More so for those who publish packages that make their techniques easy to use. I am not an expert in the field of statistics, just a grunt applying what you guys devise. It takes a while for me to do enough research and testing to be sure I'm correctly implementing new techniques. Even a basic pseudo-code walkthrough would immensely help.

>My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance.

Is there a standard format for this kind of data? If so, consider using it. That way, others can easily create artificial datasets to test it. Even if you have no control over your data source, you can convert the raw data to the standard as a "pre-munging" step.

>So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code.

Sad but true.

Re: Research software code is likely to remain a tangled mess

#96

Post by jonnycomputer »

ellimilial wrote: post #6

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.

It turns out that maintaining a package is a lot of work, and the career benefit post-publishing said package and accompanying paper is really low.

- writing general purpose software that works on multiple platforms and is bug free is really really hard. So you're just going to be inundated with complaints that it doesn't work on X

- maintaining software is lots of work. Dependencies change, etc.

- supporting and helping an endless number of noobs use your software is a major pita. "I don't know why it wouldn't compile on your system. Leave me alone."

- "oh that was just my grad work"

- its hard to get money to pay for developing it further. great when that happens though.

Re: Research software code is likely to remain a tangled mess

#97

Post by gmueckl »

j-pb wrote: post #77
gmueckl wrote: post #12

Earlier quoted context omitted.

How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations. Simplicity is a nice dream. The realities…

How the heck to you hope to gain any insighfull metrics when you've got a cobbled together mess that you only half understand. For what it's worth you might only be benchmarking random code layout fluctuations. I've seen research groups drown in their legacy code base. The issue of juggling too many balls you describe is one you only have to begin with because the state of the art implementations are so shoddy to beg…

Research code is not only written to measure runtime. Reducing the argument to only that aspect is not helping the discussion.

And you say it yourself: good experiments change a single variable at a time. So how do you check that a series of potential improvements that you are making is sound?

Re: Research software code is likely to remain a tangled mess

#98

Post by deklund »

orange_tee wrote: post #32

Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points, 1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self cont…

As someone who's worked for a large part of my career as a sort of bridge between academia and industry (working with researchers to implement algorithms in production), both you and the original author are right to an extent.

On one hand, academics I've worked with absolutely undervalue good software engineering practices and the value of experience. They tend to come at professional code from the perspective of "I'm smart, and this abstraction confuses me, so the abstraction must be bad", when really there's good reason to it. Meanwhile they look at their thousands of lines of unstructured code, and the individual bits make sense so it seems good, but it's completely untestable and unmaintainable.

On the other side, a lot of the smartest software engineers I've known have a terrible tendency to over-engineer things. Coming up with clever designs is a fun engineering problem, but then you end up with a system that's too difficult to debug when something goes wrong, and that abstracts the wrong things when the requirements slightly change. And when it comes to scientific software, they want to abstract away mathematical details that don't come as easily to them, but then find that they can't rely on their abstractions in practice because the implementation is buried under so many levels of abstraction that they can't streamline the algorithm implementation to an acceptable performance standard.

If you really want to learn about how to properly marry good software engineering practice with performant numerical routines, I've found the 3D gaming industry to be the most inspirational, though I'd never want to work in it myself. They do some really incredible stuff with millions of lines of code, but I can imagine a lot of my former academia colleagues scoffing at the idea that a bunch of gaming nerds could do something better than they can.

Re: Research software code is likely to remain a tangled mess

#99

Post by f6v »

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time.

Oh, boy, how many times have I heard this working at a startup. There is some truth to it, it's hard to organise code in the first weeks of a new project. But if you work on something for 3+ months, it becomes a matter of making a conscious effort to clean things up.

> To make a concrete example, imagine writing an application where requirements changed unpredictably every day,

Welcome to working with product managers at any early stage-company. Somehow I managed to apply TDD and good practices most of the time. Moreover, I went back to school after 7+ years developing software full-time. I guarantee that most of the low-quality research code is a result of a lack of discipline and experience in writing maintainable software.

Re: Research software code is likely to remain a tangled mess

#100

Post by bjarneh »

abcc8 wrote: post #66
bjarneh wrote: post #29

Earlier quoted context omitted.

> In many labs the 'thinkers' get far more credit and are more valued than the 'doers' In terms of software this never made much sense to me. I would understand if we where talking chemistry or some other discipline where a "new idea" has to be investigated/verified by some "lab-rat" doing mundane tasks for 2 years. In that case the lab-rat would probably get less credit than the person with the actual idea, but this…

Rarely are the developers writing grants, generating hypotheses, planning experiments, composing manuscripts, presenting the lab's work, teaching at the university, etc. Perhaps my choice of words was poor, but this should be more clear.

I guess this differs from place to place, but at my university (Oslo), we did all that..
Post reply on HN