This is actually the opportunity for our startup. I think there is generally a great opportunity to be the Databricks of a lot of academic software. We're starting in a big research area in biology :)
Research software code is likely to remain a tangled mess
121–130 of 171 posts
Re: Research software code is likely to remain a tangled mess
#122Earlier quoted context omitted.
I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable. The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up. Because changing your code coul…
Makes you wonder why most languages don't come with good refactoring tools.
Safe automatic refactoring requires the ability to do static analysis of the code. Many refactorings are harder in loosely-typed languages.
Re: Research software code is likely to remain a tangled mess
#123Earlier quoted context omitted.
I like Brooks' "plan to throw one away; you will, anyhow.": This [first] system acts as a "pilot plan" that reveals techniques that will subsequently cause a complete redesign of the system. However, in practice I'm not confident enough in my understanding, and fear losing all that hard-won work, so I refactor too. A rewrite from scratch is probably more viable when the project is small enough to keep in your head at…
Good architecture is pretty much just about slicing things up so that rewrites / refactors can happen incrementally, rather than all at once. This can actually go both bottom-up (these functions are easy to re-arrange and don't need a rewrite) and top-down (these functions suck, but I don't have to rearrange anything to replace them). "Good architecture is the one that allows you to change."
What if you slice it up wrong?
Re: Research software code is likely to remain a tangled mess
#124I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released. I'm pretty sure it's like that elsewhere. Machine learn…
> I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers. Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for i…
This feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well.
I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.
Re: Research software code is likely to remain a tangled mess
#125Earlier quoted context omitted.
Good architecture is pretty much just about slicing things up so that rewrites / refactors can happen incrementally, rather than all at once. This can actually go both bottom-up (these functions are easy to re-arrange and don't need a rewrite) and top-down (these functions suck, but I don't have to rearrange anything to replace them). "Good architecture is the one that allows you to change."
What if you slice it up wrong?
Then you're gonna have a shit time of it, and may need to do a total rewrite, instead of an incremental one.
Re: Research software code is likely to remain a tangled mess
#126Keep in mind that there're different kinds of research software. Take Seurat[1] as an example. There's CI, issue tracking, etc. It might not be the prettiest code you ever seen, but it absolutely has to be maintainable as it's being actively developed. Such projects are rare, but the low quality is often an indication of a software that isn't used by anyone. 1. https://github.com/satijalab/seurat
Also things like EISPAC, BLAS, LINPACK and so on, for FORTRAN. Back in the 70s my dad worked a bit with them when he was employed for UTHERCC: The University of Texas Health, Education, and Research Computer Center. You can find references to UTHERCC in papers from that era.
Come to think of it, something like UTHERCC might be exactly what is needed to help the current situation.
Re: Research software code is likely to remain a tangled mess
#127I’m an engineer from the other side of researches or science, but somehow interested in the topic. Recently, I’ve learned about great work done by Grigori Fursin and entire community of reserach engineers with the goal to make research software more applicable to the industries by doing it with some kind of framework inside. I want to leave some links here, if you don’t mind to watch it – the talk is called ” Reproducing 150 Research Papers and Testing Them in the Real World”:ACM page with webcast https://event.on24.com/wcc/r/2942043/9C904C7AE045B5C92AAB2CF...
Also, source docs available here: https://zenodo.org/record/4005773?fbclid=IwAR1JGaAj4lwCJDrkJ...
And, their solution product https://cknowledge.io/ and source code https://github.com/ctuning/ck
I guess it should be helpful to the researchers community.
Re: Research software code is likely to remain a tangled mess
#128Earlier quoted context omitted.
As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible. If the problem was only unpredictability, then projects with a clear and defined end goa…
>I know of a researcher that held a multimillion dollar informatics grant for 3 years. In that 3 years they literally did nothing except collect money. I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about. Intentional academic fraud is a very small percentage of what happens in academia. Partly this is because it's so stupid: academia pays poorly compared t…
> I hate that every HN post about academia ends with an anecdote describing some rare edge-case they've heard about
This isn’t a rare edge case, this is very common in software projects. I’ve heard of it because I was part of the team brought in to fix the situation.
Intentional fraud only is rare when it’s recognized as fraud. P-hacking was incredibly widespread (and to some extent still is) because it wasn’t recognized as a form of fraud. Do you really think not delivering on a software project has any consequences? Who is going to go in and say what’s fraud, what’s incompetence, and what’s bad luck?
The problem is that the bar for getting software grants isn’t high, it’s nonsensical. As far as I can tell, ability to produce or manage software development isn’t factored in at all. As with everything else, it’s judged on papers, and the grant application. In some cases, having working software models and preexisting users end up being detrimental to the process, since it shows less of a “need” for the money. You get “stars” in their field, who end up with massive grants and no idea of how to implement their proposals. Conversely, plenty of scientists who slave away on their own time on personal projects that hundreds of other scientists depend on get no funding whatsoever.
Re: Research software code is likely to remain a tangled mess
#129I was reminded that there are research packages like LINPACK, BLAS, and EISPACK for FORTRAN (and some other languages) that have been maintained since the 70s and are still in use.
Back in the 70s my dad was working for an organization called UTHERCC, the University of Texas Health, Education, and Research Computer Center, and these libraries were some of the code he worked with.
You can find references to UTHERCC in papers from the time, although I don't think it exists under that name. Maybe institutions need something like UTHERCC as an ongoing department now.
Re: Research software code is likely to remain a tangled mess
#130Earlier quoted context omitted.
I believe that of all the lessons to come from contemporary software development, constant refactoring may be the most valuable. The spaghetti monster looms large when you're in the heat of battle. But we've all got some idle time for whatever reason. I spend some time every week doing a couple of things: 1) Reading about good techniques. 2) Working through old code and cleaning it up. Because changing your code coul…
Makes you wonder why most languages don't come with good refactoring tools.
At least in the languages I work primarily in (JS and Java), I find my IDE to be pretty good at analyzing a lot of it.
Refactoring is kind of subjective, because there is rarely One Right Way to solve a problem, and you need context, so I could see why it’s not something that languages themselves take strong opinions on.