Depends on context and competition
Challenge to scientists: does your ten-year-old code still run?
461–470 of 487 posts
Re: Challenge to scientists: does your ten-year-old code still run?
#462Earlier quoted context omitted.
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code. Sure, and that rationale works OK when your code operates in a limited, specialized domain. But if you're modeling climate change or infectious diseases, and you expec…
> Sure, and that rationale works OK when your code operates in a limited, specialized domain.
There are a lot of domains which do have a deep impact on society but are completely underfunded. For example research on ecology and declining insect populations. Or research on education. And domains like epidemiology, climate change research, cancer research or such are not known to earn better salaries to scientists as on average. Most scientists earn a pity.
> But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.
What one can expect from scientists, in whatever subject they work, is honesty, integrity, a full account of their findings. What you can't expect is that they just turn into expert software engineers or make their working code beautiful. You can't expect them to work for free. What the academic system demands from them is that they work on their next paper instead, so if you want pretty code, you need at least in part to change the system.
Re: Challenge to scientists: does your ten-year-old code still run?
#463Earlier quoted context omitted.
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
As a software developer, I tgi k maybe you misunderstand scientific reproducibility.
Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome.
That an identical experiment leads to an identical outcome fails to validate the hypothesis, because the casual factors may have been misidentified.
Precise reproducibility still matters if generalized reproduction fails, however, because the differences in the experimental implementation may lead to new and more accurate hypothesis about causality.
Re: Challenge to scientists: does your ten-year-old code still run?
#464Earlier quoted context omitted.
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
As a software developer, I tgi k maybe you misunderstand scientific reproducibility. Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. That an identical experiment leads to an identical outcome fails to validate the hypothesis, because th…
> That an identical experiment leads to an identical outcome [..]
So what happens if/when an identical experiment fails to lead to an identical outcome?
Re: Challenge to scientists: does your ten-year-old code still run?
#465Earlier quoted context omitted.
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
As a software developer, I tgi k maybe you misunderstand scientific reproducibility. Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. That an identical experiment leads to an identical outcome fails to validate the hypothesis, because th…
> Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome.
The problem here is a step before this: The results of these "identical" experiments are so wildly different there is nothing valid to propose, let alone for a recreation to compare against.
Re: Challenge to scientists: does your ten-year-old code still run?
#466Earlier quoted context omitted.
I was reacting to the part of your post I quoted. The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousn…
All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.
I don't even understand the point you are making then (apart from me apparently arguing solely with sophisms, which is kind of a prowess).
How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms?
Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".
Re: Challenge to scientists: does your ten-year-old code still run?
#467Just as a quick bit of context here, Konrad Hinsen has a specific agenda that he is trying to push with this challenge. It's not clear from this summary article, but if you look at the original abstract soliciting entries for the challenge ( https://www.nature.com/articles/d41586-019-03296-8 ), it's a bit clearer that Hinsen is using this to challenge the technical merits of Common Workflow Language ( https://www.com…
There seem to be some misunderstanding here.
CWL is intended for stringing together other programs. It is useful for reproducibility in that it attempts to provide a fairly specific description of the runtime environment needed to execute a program, and also abstracts site-specific details such as file system layout or batch system in use. CWL platforms such as Arvados also generate comprehensive provenance traces which are vital for going back and reviewing how a data result was produced.
Leibniz seems to be a numerical computing language for describing equations, which is more similar to something like NumPy or R. It seems like an apples-and-oranges comparison.
The original call-out is weird, because CWL did not exist 10 years ago so you can't yet answer the question yet of whether it facilitates running 10 year old workflows.
Re: Challenge to scientists: does your ten-year-old code still run?
#468Earlier quoted context omitted.
Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…
> there is no such thing as truly random number generation on contemporary computers well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.
There is a shortage of truly random easily sampled noise though.
Re: Challenge to scientists: does your ten-year-old code still run?
#469Earlier quoted context omitted.
Are the hiring scientists also paid well-below market wages by that degree?
In many fields industry pays noticeably better than academia, but the difference is not that meaningful for the actual scientists; a principal investigator gets a reasonable amount of money but also significant degree of freedom and influence which helps job satisfaction even if the pay itself is lower. The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and…
Academia definitely can place software creators at a lower payscale to keep other higher. I wonder why such a caste system exists?
Re: Challenge to scientists: does your ten-year-old code still run?
#470Earlier quoted context omitted.
It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development. In any case you'd need to teach them the problem…
As a grad student in physics, I not only wrote code, but also designed my own (computer controlled) electronics, mechanics, optics, vacuum systems, etc. I was my own machinist and millwright. Today I work in a small R&D team within a larger business, and still do a lot of those things myself when needed. There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team.…
There's no problem with not leaving programming to programmers, its about how to encourage anyone picking up programming to have healthier habits so others can participate in the creation in the future.