Research software code is likely to remain a tangled mess • Hacker News Board

Re: Research software code is likely to remain a tangled mess

#51

Post by throwaway6734 » Mon, Feb 22, 2021, 2:02 PM UTC

svalorzen wrote: post #35

Zababa wrote: post #11

Earlier quoted context omitted.

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. I'd…

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment. At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future rese…

I am under the impression that most authors do not even publish functioning code when publishing ML/DL papers which I find to be absurd. The paper is describing software. Imo the code is more important than the written word.

Re: Research software code is likely to remain a tangled mess

#52

Post by Zababa » Mon, Feb 22, 2021, 2:06 PM UTC

svalorzen wrote: post #35

Zababa wrote: post #11

Earlier quoted context omitted.

> The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus giving no opportunity to structure the code in advance so it is easy to change. I'd…

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment. At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future rese…

> At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else.

I assumed that most code published could be directly useful as an application or a library. Considering what you're saying, this might be only a minority of the code. In that case, I agree with your conclusion about smaller gains.

Re: Research software code is likely to remain a tangled mess

#53

Post by Bukhmanizer » Mon, Feb 22, 2021, 2:07 PM UTC

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible.

If the problem was only unpredictability, then projects with a clear and defined end goal (eg, a website to host results) would be of substantially higher quality. But they’re not. Well defined projects tend to end up basically just as crappy as exploratory projects.

The problem is evaluation and incentives. There’s literally no evaluation of software or software development capability in the industry. I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money. Usually there are grant updating mechanisms, and reports, but he bsed his way through that knowing there’s a 0.0000000% chance that any granting agency is going to look through his code. The fraud was only found because he got fired for unrelated activities.

I once looked up older web projects on a grant. 4/6 were completely offline less than 2 years after their grants completed. For 2 of those 4, it’s unclear whether the site ever completed in the first place.

Re: Research software code is likely to remain a tangled mess

#54

Post by User23 » Mon, Feb 22, 2021, 2:15 PM UTC

leecarraher wrote: post #9

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

Yes, not pointing out the difference between coding some novel technique and a well defined software project, completely misses the reason the code is often not well organized. Suggesting that researchers are bad programmers is just a lazy excuse, somewhat damaging, and by no means the rule. I wrote a large complex framework for my research and the very nature of it causes me to add modules and techniques for parts I…

This sounds like my software development methodology when I was in my early teens. I was certainly able to get things done and explore all kinds of things (I was doing game dev of course), but the code was a mess and I didn't even have a mature understanding that it was. I just thought that was how programming was and you just had to be really smart to keep things straight in your head.

Re: Research software code is likely to remain a tangled mess

#55

Post by taeric » Mon, Feb 22, 2021, 2:18 PM UTC

orange_tee wrote: post #32

Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points, 1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self cont…

Your points apply to industry, too. I heretically push flatter code all the time. I'm not against abstraction, but it is easy to fall into the trap of building a solution machine, but missing the solution you need.

Re: Research software code is likely to remain a tangled mess

#56

Post by acmj » Mon, Feb 22, 2021, 2:18 PM UTC

orange_tee wrote: post #32

Well as somebody who has written research software, I don't agree that research software is a "tangled mess". A couple of points, 1. often when I read read software written by profession programmers I find it very hard to read because it is too abstract, almost every time I try to figure out how something works, it turns out I need to learn a new framework and api, by contrast research code tends to be very self cont…

Quite a few professional programmers evaluate the quality of code by "look": presence of tests, variable length, function length etc. However, what makes great code is really the code structure and logical flows behind. In my experience, good industrial programmers are as rare as good academic programmers. Many industrial programmers make a fuss about coding styles but are not really good at organizing structured code for a medium sized project.

Re: Research software code is likely to remain a tangled mess

#57

Post by knuthsat » Mon, Feb 22, 2021, 2:21 PM UTC

I think there's not enough researchers that publish code.

For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released.

I'm pretty sure it's like that elsewhere. Machine learning and deep learning for some reason has a lot of code in the open but that's not the norm.

I'd prefer the code to be open first. Once that's abundant then I might prefer the code to also be well designed.

Re: Research software code is likely to remain a tangled mess

#58

Post by galangalalgol » Mon, Feb 22, 2021, 2:25 PM UTC

Zababa wrote: post #52

svalorzen wrote: post #35

Earlier quoted context omitted.

Yes... and no. It is true that after a result is obtained, one could clean up the code for publication. And it is true that coding is not seen add first class at the moment. At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. Reproducing results can be done with ugly code, and future rese…

> At the same time, you need to consider that such a clean up is only realistically helpful for other people to check whether there are bugs in the original results, and not much else. I assumed that most code published could be directly useful as an application or a library. Considering what you're saying, this might be only a minority of the code. In that case, I agree with your conclusion about smaller gains.

Yeah my standard quote about research code is that it is not the product, so it is ok thta it is bad. The results are the product and those need to be good. Someday someone will take those results (in the form of some data or a paper) and make a software product, and that should be good.

Re: Research software code is likely to remain a tangled mess

#59

Post by qmmmur » Mon, Feb 22, 2021, 2:31 PM UTC

Bukhmanizer wrote: post #53

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible. If the problem was only unpredictability, then projects with a clear and defined end goa…

I can tell you why the sites went offline, because the funding stopped. I don't know what you're research background is but its painful to even get 5 GBP a month to host a droplet on digital ocean in a pretty lucrative department with liberal internal funding.

Re: Research software code is likely to remain a tangled mess

#60

Post by Frost1x » Mon, Feb 22, 2021, 2:31 PM UTC

txdv wrote: post #39

asdf_snar wrote: post #24

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind be…

Reproducibility is a major principle of the scientific method. Yet computer scientists consistently fail to achieve reproducibility with a tool that is the most consistent at following instructions - the computer. Even private business is on the DevOps movement, because they see the positive effects of reproducibility. If the academic world is truly about science, then there is no more excuse, the tools are out there…

This is really an artifact if unreasonable expectations and modern software ecosystems. When I say unreasonable expectations, the issue is that people assume they can use the latest greatest trendy library and get reproducible results. Good luck on the level of determinism you're looking for.

You need to step back and look at more mature, simple codebases and what you can do in those sorts of environments when you want reproducibility. You can't cobble together a bunch of async services in the cloud and hope your Frankenstein tool gives you perfect results. It will give you good enough results for certain aspects if you focus on those specific aspects (banking does a good job of this with transactional processing and making sure values are consistent because it's their entire business, maybe your account or their web interface is skrewy but that's fine, that can fail).