I'm a grad student in biophysics - even if I wrote perfect code, it would almost certainly go obsolete in 10 years because the hardware that it interfaces with would go obsolete.
Challenge to scientists: does your ten-year-old code still run?
381–390 of 487 posts
Re: Challenge to scientists: does your ten-year-old code still run?
#382Earlier quoted context omitted.
> If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined. That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent. You’re speaking genuine nonsense as far as I’m co…
I am referring to this blog post: https://lockdownsceptics.org/code-review-of-fergusons-model/ It says, word-by-word: > Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results. > >Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters. > I’ll illustrate with a few bugs. In issue…
There is something I need to add, it is a subtle but important point:
Non-determinism can be caused by
a) random seeds derived from hardware, such as seek times in a HDD controller, which is fed into pseudo random number (PRNG) generation. This is not a problem. For debugging, or comparison, it can make sense to switch it off, though.
b) data race conditions, which is a form of undefined behavior. This not only can dramatically change results of a program run, but also invalidates the program logic, in languages such as C and C++. This is what he blog post in "lockdownskeptics.org" suggests. For the application area and its consequences, this would be a major nightmare.
c) What I had forgotten is that parallel execution (for example in LAM/MPI, map/reduce or similar frameworks) is inherently non-deterministic and, in combination with properties of floating-point computation, can yield different but valid results.
Here an example:
A computation is carried out on five nodes and they return the values 1e10, 1e10, 1e-20, -1e10, -1e10, in random order. The final result is computed by summing these up.
Now, the order of computation could be:
((((1e10 + 1e10) + 1e-20) + -1e10) + -1e10)
or it could be:
(((1e10 + -1e10) + 1e-20) + (+1e10 + -1e10))
In the first case, the result would be zero, in the second case, 1e-20, because of the finite length of floating point representation.
_However_... if the numerical model or simulation or whatever is stable, this should not lead to a dramatic qualitative difference in the result (otherwise, we have a stability problem with the model).
Finally, I want to cite one last paragraph from the post on lockdownskeptics.org:
> Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.
Re: Challenge to scientists: does your ten-year-old code still run?
#383This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…
A seasoned software developer encountering scientific code can be a jarring experience. So many code smells. Yet, most of those code smells are really only code smells in application development. Most scientific programming code only ever runs once, so most of the axioms of software engineering are inapplicable or a distraction from the business at hand.
Scientists, not programmers, should be the ones spear-heading the development of standards and rules of thumb.
Still, there are real problematic practices that an emphasis on sharing scientific code would discourage. One classic one is the use of a single script that you edit each time you want to re-parameterize a model. Unless you copy the script into the output, you lose the informational channel between your code and its output. This can have real consequences. Several years ago I started up a project with a collaborator to follow up on their unpublished results from a year prior. Our first task was to take that data and reproduce the results they obtained before, because the person no longer had access to the exact copy of the script that they ran. We eventually determined that the original result was due to a software error (which we eventually identified). My colleague took it well, but the motivation to continue the project was much diminished.
Re: Challenge to scientists: does your ten-year-old code still run?
#384Earlier quoted context omitted.
Making this mandatory might have bad downstream effects like prohibiting publication of some research at all (GPT-X I am looking at you)
Closed source research isn't publication, it's advertisement.
So R&D is not a thing, but A&D is? That would be new to me
Re: Challenge to scientists: does your ten-year-old code still run?
#385Earlier quoted context omitted.
I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…
Monte-Carlo can and should be deterministic and repeatable. It’s a matter of correctly initializing you random number generators and providing a known/same random seed from run to run. If you aren’t doing that, you aren’t running your Monte-Carlo correctly. That’s a huge red flag. Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their studen…
> Monte-Carlo can and should be deterministic and repeatable.
That's a nitpick, but if the computation is executed in parallel threads (e.g. on multicore, or on a multicomputer), and individual terms are, for example, summed in a random order, caused by the non-determinism introduced by the parallel computation, then the result is not strictly deterministic. This is a property of floating-point computation, more specifically, the finite accuracy of real floating-point implementations.
So, it is not deterministic, but that should not cause large qualitative differences.
Re: Challenge to scientists: does your ten-year-old code still run?
#386Earlier quoted context omitted.
One of the things I come across is scientists who believe they're capable of learning code quickly because they're capable in another field. After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running. For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above? I would have t…
It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development. In any case you'd need to teach them the problem…
As a grad student in physics, I not only wrote code, but also designed my own (computer controlled) electronics, mechanics, optics, vacuum systems, etc. I was my own machinist and millwright. Today I work in a small R&D team within a larger business, and still do a lot of those things myself when needed.
There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team. The first is keeping them occupied. There was programming to be done, but not full time. And it had to be done in an extremely agile fashion, with requirements changing constantly, often at the location where the problem is occurring, not where their workstation happens to be set up. Many developers hate this kind of work.
Second is just managing software development. Entire books have been written about the topic, and it's not a solved problem how to keep software development from eating you alive and taking ownership of your organization. Nobody knows how to estimate the time and effort. You never know if you're going to be able to recover your source code and make sense of it, if your programmer up and quits.
With apologies to Clemenceau, programming is too important to be left to the programmers. ;-)
Re: Challenge to scientists: does your ten-year-old code still run?
#387Earlier quoted context omitted.
edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not understand. What is crucial in the argument here is indeed the distinction between methods employing pseudo-randomness, like Monte Carlo simulation, and non-determinism caused by undefined behavior. > I'm an accelerator physicist and I wouldn't want my code t…
Race conditions aren't undefined behavior in C/C++. Data races are. Lots and lots of real systems contain race conditions without catastrophe.
> Race conditions aren't undefined behavior in C/C++. Data races are.
You are right with the distinction, I had data race conditions in mind.
Race conditions can well happen in a correct C/C++ multi-threaded program in the sense that the order of specific computation steps is sometimes random. And for operations such as floating-point addition, where order of operations does matter, the exact result can be random as a consequence. But the end result should not depend dramatically on it (which is what the poster at lockdownskeptics.org claims).
Re: Challenge to scientists: does your ten-year-old code still run?
#388Earlier quoted context omitted.
Sounds like simplicity for the win. The complex house of cards we currently stand on seems fragile by comparison.
We also benefit, for that old stuff, from enthusiasts that build cool stuff. Like DosBox, Floppy Emulators, etc. I doubt there are going to be folks nostalgic for the complex mess we have now.
Indeed. I participate in Atariage.com and the level of dedication is amazing.
Are there groups for Win 3.1, Win95?
Re: Challenge to scientists: does your ten-year-old code still run?
#389Earlier quoted context omitted.
>I am interested to know the distinction between "production-ready" and "science-ready" code. In general, scientists don't care how long it takes or how many resources the code uses. It is not a big deal to run a script for an extra hour, or use up a node of supercomputer. Extravagent solutions or added packages to make the code run smoother or faster is only wasting time. It speed/elegance only really matters when y…
Do you know how you could get to the state that "the code was proven to produce the correct result"? If not by unit tests, code review or formal logic, then what?
Not all scientific code is amenable to unit testing. From my own experience from a PhD in condensed matter physics, the main issue was that how important equations and quantities “should” behave by themselves was often unknown or undocumented, so very often each such component could only be tested as part of a system with known properties.
You can then use unit testing for low-level infrastructure (e.g. checking that your ODE solver works as expected), but do the high-level testing via scientific validation. The first line of defense is to check that you don’t break any laws of physics, e.g. that energy and electric charge is conserved in your end results. Even small implementation mistakes can violate these.
Then you search for related existing publications of a theoretical or numerical nature, trying to reproduce their results; the more existing research your code can reproduce, the more certain you can be that it is at least consistent with known science. If this fails, you have something to guide your debugging; or if you’re very lucky, something interesting to write a paper about :).
The final validation step is of course to validate against experiments. This is not suited for debugging though, since you can’t easily say whether a mismatch is due to a software bug, experimental noise, neglected effects in the mathematical model, etc.
Re: Challenge to scientists: does your ten-year-old code still run?
#390Earlier quoted context omitted.
As someone coming from the computing side of things, I found nix to be quite difficult to grok enough to write a package spec, and guix was pretty close, at least in part because of the whole "packages are just side-effects of a functional programming language" idea. At least nix also suffers from a lot of "magic"; if you're trying to package, say, an autotools package then the work's done for you - and that's great,…
> guix was pretty close, at least in part because of the whole "packages are just side-effects of a functional programming language" idea This must be a misunderstanding. One of the big visible differences of Guix compared to Nix is that packages are first-class values.
You're right; on further reading I can see guix making packages the actual output of functions. I do maintain that the use of a whole functional language to build packages raises the barrier to entry, but my precise criticism was incorrect.