Research software code is likely to remain a tangled mess • Hacker News Board

Re: Research software code is likely to remain a tangled mess

#141

Post by tryonenow » Mon, Feb 22, 2021, 9:05 PM UTC

Not all "research" code is equal. I would imagine that research code closer to hard sciences and cutting edges is more difficult to keep clean. The trouble is when you're breaking new ground in applied science, you don't necessarily know how well the new tech will work, and exactly what you'll be able to do with it. As development progresses your expectations must be adjusted, and it is impossible to forecast the direction of highly experimental research since progress is typically incremental and constantly dependent on the most recent results.

There are many paths to scaling a mountain, so to speak, and sometimes for any of a multitude of reasons you end up on another peak long after you've started climbing.

Re: Research software code is likely to remain a tangled mess

#142

Post by analog31 » Mon, Feb 22, 2021, 9:12 PM UTC

amelius wrote: post #119

analog31 wrote: post #93

Earlier quoted context omitted.

I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable. The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up. Because changing your code coul…

Makes you wonder why most languages don't come with good refactoring tools.

I should have added that I am probably abusing the term refactoring if it has a precise definition. What I'm talking about is working on improving the readability of my code, but also improving its structure. Today, "spaghetti" probably doesn't refer to a tangled mess of code sequences because we've gotten rid of the GOTO, but tangled interactions between modules, many of which are vestigial.

A lot of my code interacts with hardware configurations that will cease to exist when a project is done, but I mainly look at the stuff that's potentially reusable, and making it worth re-using.

I'm using Python, and there are a lot of tools for enforcing coding styles and flagging potential errors. I try to remove all of the red and yellow before closing any program file. I don't trust myself with too much automation! "Walk before you run."

Re: Research software code is likely to remain a tangled mess

#143

Post by brakus127 » Mon, Feb 22, 2021, 9:14 PM UTC

bordercases wrote: post #139

brakus127 wrote: post #138

Structure is great for a well understood problem space, but this is not usually the case when working working something novel. As a researcher your focus should be on learning and problem solving, not creating a beautiful code base. Imposing too many constraints early on can negatively impact your project later on. In the worse case, your code starts to limit the way you think about your research. I agree that there…

Results need to be refined so that the way they were first formulated doesn't get in the way of their replication. At scale, this too becomes a cost to industry. In the small, this isn't different from taking a lab notebook and making it clearer and better summarized so that it can be passed on to the poor sucker who has to do what you did after you move on to another project. Furthermore, software projects that are…

Agreed that there should be a minimum bar for "completed" research code such as reproducibility and a clear summary, but engineers shouldn't expect the first version of a new algorithm to be easy to understand without additional material or ready for production without a complete rewrite.

Re: Research software code is likely to remain a tangled mess

#144

Post by amelius » Mon, Feb 22, 2021, 9:18 PM UTC

realusername wrote: post #134

amelius wrote: post #119

Earlier quoted context omitted.

Makes you wonder why most languages don't come with good refactoring tools.

I don't think refactoring tools are that useful for refactoring, most of the time you are doing non-obvious refactoring tools can't help with anyways. Depends what we call "refactoring", tools are mostly useful for what I would call "housekeeping".

Well, every refactoring can be seen as a series of correctness-preserving housekeeping operations.

Re: Research software code is likely to remain a tangled mess

#145

Post by yudlejoza » Mon, Feb 22, 2021, 9:28 PM UTC

This topic is near and dear to my heart and at a quick glance, I pretty much agree with all/most of this post.

I gained multiple years of industry software engineering experience before joining academia (non-CS, graduate-level). And I was flabbergasted at the way software and programming is treated in research setting where the "domain" is not CS or software itself. It took me a few years just to get a hint of what on earth these people (my collaborators who program side-by-side with me) are thinking, and what kind of mindset do they come from.

Then I took a short break and went to the industry. Software engineering, hardcore CS; no domain, no BS. I was expecting that it would feel like an oasis. It didn't. Apart from a handful of process improvements, like use of version control, issue tracking, deadline-management, the quality of the tangled mess of the code was only slightly better.

Initially I took away the lesson that it's the same in academia and industry. But on further reflection there are two big differences:

- The codebase I worked on in the industry was at least 10x bigger. Despite that, the quality was noticeably better.

- More importantly, I could connect with the my coworkers in the industry. If I raised a point about some SwE terminology like test-driven dev, agile, git, whatever, I could have a meaningful discussion. Whereas in academia, not only most domain experts knew jack about 90% of software-engineering concepts and terminology, they were expert at hiding their ignorance, and would steer the conversation in a way that you wouldn't know if they really didn't know or knew too much. I never got over that deceitful ignorance mixed with elitist arrogance.

In the end, I do think that, despite enormous flaws, the industry is doing way better than academia when it comes to writing and collaborating on software and programming, and that the side-by-side comparison of actual codebases is a very small aspect of it.

Re: Research software code is likely to remain a tangled mess

#146

Post by xkriva11 » Mon, Feb 22, 2021, 9:40 PM UTC

disgruntledphd2 wrote: post #140

grahamlee wrote: post #135

Earlier quoted context omitted.

Refactoring tools were invented in Smalltalk and worked just fine.

This has always surprised me, since I learned it. What are the features of Smalltalk that allowed this to happen? Conversely, what is stopping this from existing in more modern dynamic languages?

Smalltalk has simple and strong reflective features. Moreover, it does not make a difference between the developed program and IDE. This means that doing things like that are very natural and well established in the Smalltalk cultural background.

Re: Research software code is likely to remain a tangled mess

#147

Post by paulclinger » Mon, Feb 22, 2021, 9:43 PM UTC

Bukhmanizer wrote: post #53

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible. If the problem was only unpredictability, then projects with a clear and defined end goa…

> I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money.

I wonder if a whistleblower payout similar to the one that SEC is doing for 1M+ fines (10-30%) would help in cases like this. The host organization would potentially be on the hook as well, so there is going to be a significant incentive to not let that happen (especially with all the associated reputational damage).

Re: Research software code is likely to remain a tangled mess

#148

Post by lazyjeff » Mon, Feb 22, 2021, 10:02 PM UTC

Bukhmanizer wrote: post #128

matthewdgreen wrote: post #113

Earlier quoted context omitted.

>I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money. I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about. Intentional academic fraud is a very small percentage of what happens in academia. Partly this is because it's so stupid: academia pays poorly compared t…

> I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about This isn’t a rare edge case, this is very common in software projects. I’ve heard of it because I was part of the team brought in to fix the situation. Intentional fraud only is rare when it’s recognized as fraud. P-hacking was incredibly widespread (and to some extent still is) because it wasn’t recogn…

Just curious, what kind of 3-year informatics grant not being completed ends up with a team brought in to fix the situation? Multi-million dollar grants don't sound big enough to be a dependency for any major customer (like defense or pharma), so I imagine if fraud was detected, they would just demand a reimbursement and ban the PI.

But I think you're both right in some sense. The cases of intentional major fraud is probably a rare edge case and they make the news when they're uncovered. But there's a lot of grey-ish area like p-hacking as you mentioned, plus funding agencies know there needs to be some flexibility in the proposed timeline due to realities. Realities like you don't necessary get the perfect student for the project right when the grant starts, as the graduate student cycle is annual, plus the research changes over time and it isn't ideal to have students work on an exact plan as if they are an employee.

But I totally agree that maintaining software that people are using should be funded and rewarded by the academic communities. A possible way to do this is have a supplement so that after a grant is over, people who have software generated from the grant that is used by at least 10 external parties without COI, should be funded 100K/yr for however many years they are willing to maintain and improve it. Definitions of what this means needs to be carefully constructed, of course.

Re: Research software code is likely to remain a tangled mess

#149

Post by grahamlee » Mon, Feb 22, 2021, 10:02 PM UTC

xkriva11 wrote: post #146

disgruntledphd2 wrote: post #140

Earlier quoted context omitted.

This has always surprised me, since I learned it. What are the features of Smalltalk that allowed this to happen? Conversely, what is stopping this from existing in more modern dynamic languages?

Smalltalk has simple and strong reflective features. Moreover, it does not make a difference between the developed program and IDE. This means that doing things like that are very natural and well established in the Smalltalk cultural background.

Indeed. Having the whole system in front of you, and knowing how patterns like MVC or Thing-Model-View-Editor encapsulate parts of it, makes it very easy to "reason about" wholesale changes to the system.

Re: Research software code is likely to remain a tangled mess

#150

Post by geebee » Mon, Feb 22, 2021, 10:03 PM UTC

svalorzen wrote: post #3

I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…

I've worked with a lot of research code. I agree with you that tangled code is somewhat intrinsic to the kind of code written for research.

Here's the thing. Sometimes, there's no code - I mean, they'll find something, but nobody can say, with certainty, that it is the code that generated the data or results you're trying to recreate. There's often no data - and by that, I mean, nothing, not even a dummy file so you can tell if it even runs or understand what structure the data needs to be in. No build, no archive history, no tests. And when I say no tests, I'm not talking about red/green bar integration and unit tests, I mean, ok, the code ran... was this what it was supposed to produce?

Many of these projects are far, far more messed up than the intrinsic nature of research would explain - though I will again agree that research code may be unusually likely to descend into entropy.