CMake Error at /usr/share/cmake-3.18/Modules/FindQt4.cmake:1314 (message):
Found unsuitable Qt version "5.15.0" from /usr/bin/qmake, this code
requires Qt 4.x
Challenge to scientists: does your ten-year-old code still run?
351–360 of 487 posts
Re: Challenge to scientists: does your ten-year-old code still run?
#352Earlier quoted context omitted.
In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…
No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe…
I don't entirely disagree, but haven't there also been cases of experimental results being invalidated due to subtle mechanical, electrical, chemical, etc complications with the test equipment, when none of the people involved in the experiment were experts in those fields?
I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is hard and we need real replication of everything before reaching important conclusions, and over-focusing on one specific type of errors isn't all that helpful.
Re: Challenge to scientists: does your ten-year-old code still run?
#353Earlier quoted context omitted.
Or Fortran.
Oh gosh yes, the amount of `just works` Fortran in science is one of those things akin to COBOL in business. I just know some people are thinking 10 years - ha, be some instances of 40 and possible 50 years for some. Heck, the sad part is many will have computer systems older than 10 years just as it links to this bit of kit and the RS232 just works with the DOS software fine as and the updated version had issues whe…
I know two fresh PhDs from two different schools whose favorite language is fortran. I think it's rather different from cobol in that way -- yes, the old stuff still works, but newer code cuts down on the boilerplate and is much more readable. And yeah, the ability to link to 50 year-old battle-tested code is quite a feature.
Re: Challenge to scientists: does your ten-year-old code still run?
#354The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding.
The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding.
Concerns about software quality in science are primarily about avoiding experimental error at the time of publication, not the durability of the results. If you did the experiment correctly, it doesn't matter if your code can run 10 years later. Someone else can run their own experiment, write their own code, and find the same thing you did.
And if you did the experiment incorrectly, it also doesn't matter if you can run your code 10 years later; running wrong code a decade later does not tell you what the right answer is. Again--conducting new research to explore the same phenomenon would be better.
When it comes to hardware, we get this. Could you pick up a PCR machine that's been sitting in a basement for 10 years and get it running to confirm a finding from a decade ago? The real question is, why would you bother? There are plenty of new PCR machines available today, that work even better.
And it's the same for custom hardware. We use all sorts of different telescopes to look at Jupiter. Unless the telescope is broken, it looks the same in all of them. Software is also a tool for scientific observation and experimentation. Like a telescope, the thing that really matters is whether it gives a clear view of nature at the time we look through it.
Re: Challenge to scientists: does your ten-year-old code still run?
#355The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding. The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding. Concerns about software quality in science are primarily about avoiding…
> running wrong code a decade later does not tell you what the right answer is.
It can tell, however, exactly where the error lies (if the error is in software at all). Like a math teacher that can circle where the student made a mistake in an exam.
Re: Challenge to scientists: does your ten-year-old code still run?
#356The gold standard for a scientific finding is not whether an particular experiment can be repeated, it is whether a different experiment can confirm the finding. The idea is that you have learned something about how the universe works. Which means that the details of your experiment should not change what you find... assuming it's a true finding. Concerns about software quality in science are primarily about avoiding…
Reproducibility is about understanding the result. It is the modern version of "showing your work".
One of the unsung and wonderful properties of reproducible workflows is the fact that it can allow science to be salvaged from an analysis that contains an error. If I had made an error in my thesis data analysis (and I did, pre-graduation), the error can be corrected and the analysis re-run. This works even if the authors are dead (which I am not :) ).
Reproducibility abstracts the analysis from data in a rigorous (and hopefully in the future, sustainable) fashion.
Re: Challenge to scientists: does your ten-year-old code still run?
#357This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…
The criticisms of the code from Imperial College are strange to me. Non-deterministic code is the least of your problems when it comes to modeling the spread of a brand new disease. Whatever error is introduced by race conditions or multiple seeds is completely dwarfed by the error in the input parameters. Like, it's hard to overstate how irrelevant that is to the practical conclusions drawn from the results.
Skeptics could have a field day tearing apart the estimates for the large number of input parameters to models like that, but they choose not to? I don't get it.
Re: Challenge to scientists: does your ten-year-old code still run?
#358I visited my first employer recently (a local government) and found that the first MySQL/PHP database I created, an internal app, had been in continuous use for nearly 18 years.
Re: Challenge to scientists: does your ten-year-old code still run?
#359Earlier quoted context omitted.
> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…
But "rerunning reproducability" is mostly a neccessary requirement for independent reproducability. If you can't even run the original calculations against the original data again how can you be sure that you are not comparing apples to oranges?
In some simulations, each rerun produces different results as you’re simulating random events (like lightning formation) or using a non-deterministic algorithm (like Monte Carlo sampling). Just “saving the random seed” might not be sufficient to make it deterministic either, as if you do parallelized or concurrent actions in your code (common in scientific code) the same pseudorandom numbers may be used in different orders each time you run it.
But repeating the simulation a large number of times, with different random seeds, should produce statistically similar output if the code is rigorous. So even if each simulation is not reproducible, as long as the statistical distribution of outputs is reproducible, that should be sufficient.
Re: Challenge to scientists: does your ten-year-old code still run?
#360Earlier quoted context omitted.
You're right about bit-for-bit reproducibility possibly being overkill, but I don't think that invalidates the parent's point that Monte Carlo randomization doesn't obviate reproducibility concerns. It just means that e.g. your results shouldn't be hypersensitive to the details of the randomization. That is, reviewers should be able to take your code, feed it different random data from a similar distribution to what…
It does seem like a valid response to OP's objection to the imperial college's COVID model, though. Doesn't it?
Reviewing the original comment, I think so (that the original comment is overcritical). For purpose of reproducibility, it's enough that you can validate that you can run the model with different random data and see that their results aren't due to pathological choices of initial conditions. If the race conditions and non-determinism just transform the random data into another set of valid random data, that doesn't compromise reproducibility.