I once interviewed for a programming job that was a bit of bait and switch. The hiring manager showed me a foot-high stack of green bar paper that was Fortran code written by optical scientists that I was expected to convert to C. He was somewhat surprised when I declined and ended the interview. I pity whoever got stuck with that task.
Research software code is likely to remain a tangled mess
81–90 of 171 posts
Re: Research software code is likely to remain a tangled mess
#82Earlier quoted context omitted.
Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…
>The source of truth is the paper. This is why the paper was published. Speaking as someone who's not the best at math, I find it easier to understand what a paper is saying after I run the code and see all the intermediate results. When the code doesn't work, it takes me 20 times longer to digest a paper. They could do with only uploading code -- to me it's the shortest and most effective way to express the ideas in…
>Speaking as someone who's not the best at math, I find it easier to understand what a paper is saying after I run the code and see all the intermediate results.
As long as you understand the paper after, that's okay.
> When the code doesn't work, it takes me 20 times longer to digest a paper.
What if the data isn't available? That's another issue. I see where you're coming from, but that's why the paper itself is the source of truth. Not the implementation.
Another case, what if the implementation makes assumptions on the data? Or on the OS it's being run on?[0][1]
> They could do with only uploading code -- to me it's the shortest and most effective way to express the ideas in the paper.
In my opinion, no. The math and algorithm behind it is more important than an implementation and better for longevity.
[0] https://science.slashdot.org/story/19/10/12/1926252/python-c...
[1] https://arstechnica.com/information-technology/2019/10/chemi...
Re: Research software code is likely to remain a tangled mess
#83I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…
> imagine writing an application where requirements changed unpredictably every day
Imagine?
Re: Research software code is likely to remain a tangled mess
#84Earlier quoted context omitted.
Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…
> Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? You can include the hash of the commit used for your paper.
oh, that's good. Or even a tag
Re: Research software code is likely to remain a tangled mess
#85Earlier quoted context omitted.
As someone who has been on both the research and industry software end, there’s really not that much difference. Requirements change, you build that into your plans. Frankly, a lot of best practice software development that gets totally ignored by academia (e.g. OOP) can handle this exact case, and makes things way more flexible. If the problem was only unpredictability, then projects with a clear and defined end goa…
I can tell you why the sites went offline, because the funding stopped. I don't know what you're research background is but its painful to even get 5 GBP a month to host a droplet on digital ocean in a pretty lucrative department with liberal internal funding.
Is N years of opex not part of the budget in grant applications?
Re: Research software code is likely to remain a tangled mess
#86Earlier quoted context omitted.
Same. But it could be an issue with the sample. 213 in a span of 14 years is not a lot. Also, a question. If you publish a paper with a repo, what would be the best way to handle the version in the paper matching the repo in the future? An opinion, there is such a thing as software being ‘done’ and ‘as is’. Software solves a need. After that’s meet, that’s it. There’s also this part that strikes me, >Given a tangled…
> The source of truth is the paper. Yes, although truth of the flimsiest kind. A lowly but wise code monkey once said "Talk is cheap. Show me the code."
Here's some code. Data is proprietary. There's no paper explaining the data, prep, steps to gather, caveats, assumptions, etc.
What now?
Re: Research software code is likely to remain a tangled mess
#87I don't really agree with the reasons given, even though my conclusions are the same. The main reason why research code becomes a tangled mess is due to the intrinsic nature of research. It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time. Moreover, you have no idea on advance where your experiments are going to take you, thus…
>It is highly iterative work where assumptions keep being broken and reformed depending on what you are testing and working on at any given time.
This is describing infinitely fast and efficient p-hacking (i.e. research that is likely to produce invalid results).
If your assumptions are broken then that should ideally be reported as part of your research.
When you do research, you ideally start out with fixed assumptions, and then test those assumptions. The code required to do this can be buggy (and can therefore get fixed), and you can re-purpose earlier code, but the assumptions/brief shouldn't change in the middle of the coding it up.
If you aren't following the original brief, you've rejected your original research concept and you're now doing a different piece of research than you started out - and this is no longer a sound piece of research.
Research should be highly dissimilar to a web design project in this respect.
The reason these projects often become a tangled mess is because researchers don't have the coding skill to program any other way (in my opinion, and nor do institutions invest sufficiently in people who do have this skill).
Re: Research software code is likely to remain a tangled mess
#88>>> writing software is a low status academic activity; it is a low status activity in some companies, but those involved don’t commonly have other higher status tasks available to work on.
If measured by compensation, then research is a low status activity. Perhaps more precisely, researchers have low bargaining power. But I don't think that academics actually analyze activities in such detail. The PI might not even know how much programming is being done.
The researcher is programming, not because they see it as a way to raise (or lower) their status, but because it's a force multiplier for making themselves more productive overall. Though I work in industry, I'm a "research" programmer for all intents and purposes. I program because I need stuff right away, and I do the kind of work that the engineers hate. Reacting to rapidly changing requirements on a moment's notice disrupts their long term planning. Communicating requirements to an engineer who doesn't possess domain knowledge or math skills is painful. Often, a working piece of spaghetti code that demonstrates a process is the best way to communicate what I need. They can translate it into fully developed software if it threatens to go into a shipping product. That's a good use of their time and not of mine.
>>> Why would a researcher want to invest in becoming proficient in a low status activity?
To get a better job. I sometimes suspect that anybody who is good enough at programming to get paid for it, is already doing so.
>>> Why would the principal investigator spend lots of their grant money hiring a proficient developer to work on a low status activity?
Because they don't know how to manage a developer. Software development is costly in terms of both time and effort, and nobody knows how to manage it. Entire books have been written in this topic, and it has been discussed at length on HN. A software project that becomes an end unto itself or goes entirely off the rails can eat you alive. Finding a developer who can do quantitative engineering is hard, and they're already in high demand. It may be that the PI has a better chance managing a researcher who happens to know how to translate their own needs into "good enough" code, than to manage a software project.
Re: Research software code is likely to remain a tangled mess
#89Earlier quoted context omitted.
> The source of truth is the paper. Yes, although truth of the flimsiest kind. A lowly but wise code monkey once said "Talk is cheap. Show me the code."
Here's some code. Data is proprietary. There's no paper explaining the data, prep, steps to gather, caveats, assumptions, etc. What now?
No need for the reductionist strawman. Some experiments cannot be reproduced
for proprietary data. Those that can should be.
Re: Research software code is likely to remain a tangled mess
#90Earlier quoted context omitted.
I can tell you why the sites went offline, because the funding stopped. I don't know what you're research background is but its painful to even get 5 GBP a month to host a droplet on digital ocean in a pretty lucrative department with liberal internal funding.
Is N years of opex not part of the budget in grant applications?
In research no and it would depend entirely on your institution. For example, I looked at a job putting together a portal for people to freely examine the research put together for a research team. The project had secured a connection with the british museum, and so that website would live on under that. However, if the project had asked to host it themselves even for 60$ a year for 10 years the answer would be no. Funding grants see small opex that extend beyond the life of the project to be open to corruption or just too facile to fund, wrongly or rightly.