The longest-running code I wrote as a scientist was a sandwich ordering system. I worked for a computer graphics group at UCSF and while taking a year off from grad school while my simulations ran on a supercomputer, and we had a weekly group meeting where everybody ordered sandwiches from a local deli. It was 2000, so I wrote a cgi-bin in Python (2?) with a MySQL backend. The menu was stored in MySQL, as were the or…
Challenge to scientists: does your ten-year-old code still run?
421–430 of 487 posts
Re: Challenge to scientists: does your ten-year-old code still run?
#422Earlier quoted context omitted.
Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on n…
> But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold > Plenty of good science got done before modern devops came to be This isn't…
0: specifically, for results claiming a 95% confidence level (ponce. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).
Re: Challenge to scientists: does your ten-year-old code still run?
#423As someone who worked with bits of scientific code: Does the code you write right now work on another machine might be the more appropriate challenge. If seen a lot of hardcoded paths, unmentioned dependencies and monkey-patched libraries downloaded from somewhere; just getting the new code to work is hard enough. And let's not even begin to talk about versioning or magic numbers. Similar to other comments I don't me…
> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).
Theoretical Physicists (literal conversation I had):
>Yeah, this looked like it simplifies to 1-ish and Smart John said it's probably right.
Experimental physicists (another literal conversation):
>Yeah, we build it with duck-tape and there's hot glue holding the important bits that kept falling off. Don't put anything metal in that, we use it as a tea heater, but there's 1000A running through it so it's shoots spoons out when we turn the main machine on.
Re: Challenge to scientists: does your ten-year-old code still run?
#424Earlier quoted context omitted.
It seems like there can be a balance between "the results are unverifiable because no one else can touch the data" and "effectively open-source the dataset"? Something like: "To make it easier to verify the code behind this paper, we've used to generate a synthetic dataset with the same fields as the original and included it with the source code. The isn't comfortable with publishing the full dataset, but they did ag…
> but they did agree to provide the same data to groups working on verification studies as long as they're willing to sign a data privacy agreement. Send a query to ..." This would be administrative overhead, it will be shut down 9 times out of 10. I understand why this might seem easy but it really is not, you can have multiple hospital that each have their committee that agreed to give the researcher their data. Th…
I didn't say it was easy--I just said it struck a balance relative to trying to openly publish the whole dataset. Yes. Obviously comes with administrative overhead. So did dealing with the initial researcher. If the institution can manage the one, it can manage the other.
> As for the synthetic datasets that's basically just having tests
An appropriate synthetic dataset would inevitably be part of a great test suite, but it's also pretty simple to write narrow unit-tests that embed rather than stretch the same assumptions and biases that are also in the code (i.e., simple enough that even people who feed themselves with code do it).
An independent project/practice for synthesizing sample datasets from the real dataset lowers the bar and clarifies the best-practice for releasing a dataset that a verifier could actually use to spot simple bugs, edge-cases, and algorithm issues. Ideally, yes, this practice nudges the researchers to bother running their program over generated sample datasets as well, and to pay attention to whether the results make sense.
Re: Challenge to scientists: does your ten-year-old code still run?
#425Earlier quoted context omitted.
The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time
If the code you base your work on is horrible it definitely makes me question your results. That's why it's called the reproducibility crisis. Writing some tests, using a linter, commenting your code, and learning about best programming practices doesn't take long and pays off - even for yourself when writing the code or you need to touch the code again. "48 hours writing unit tests" is a ridiculous comparison.
This is just complaining that science is too hard because you can't be bother to replicate an experiment.
Re: Challenge to scientists: does your ten-year-old code still run?
#426Earlier quoted context omitted.
Guix is one of several solutions that has been touted as a solution. Another one that is quite popular in HPC circles is Spack ( https://spack.readthedocs.io/en/latest/ ). At my institute, we actually tried out Spack for a little bit, but consistently felt like it was implemented more as a research project rather than something that was production-level and maintainable. In large part, this was due to the dependency…
> Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable. I would be quite interested to learn more what these problems are, in your experience. I've only tried Guix (on top of Debian and Arch) and while it is definitively more resource-hungry (especially in terms of disk space), I don't percive it as impractical.
In theory this isn't an issue with Spack (assuming that you have a largely homogeneous set of hardware or don't use CPU family-specific instruction sets), since you can set up cached, pre-compiled binaries on a mirror server (similar to a yum repo) and have people install from there.
Spack, however, has a lot of power/complexity. A lot of untamed power that means that bugs can sometimes be more likely than in other, more mature (or mature-ish) package managers. Namely, Spack allows you to not only specify the version number of a package, but also the compiler that you use to make that package, specific versions of dependencies that you want to use, which implementation of an API you want to use (i.e., MPICH or OpenMPI for MPI), and compiler flags for that package. When you run an install command / specify what you want to install, Spack then performs dependency resolution and "concretizes" a DAG that fulfills all of the constraints.
The issue that I ran into was that if you don't specify everything, Spack makes decisions for you about which version of a dependency, which compiler, etc to use (i.e., it fills in free variables in a space with a lot of dimensions). This would be great and dandy normally, although the version of Spack that I used occasionally constructed totally different graphs for the same "spack install gcc" command (if I recall correctly; take all of this with a grain of salt b/c I might be misremembering). This meant that it wouldn't use cached versions of gcc that had already been built, and ended up rebuilding minor variants of gcc with options I didn't care about.
At National Labs and larger outfits, the trade-offs between this kind of complexity/power and the accompanying bugginess (Spack has yet to hit 1.0) seem to favor complexity/power while accepting these sorts of bugs, but I don't work at a larger outfit and my group didn't need that level of power/control over dependencies and rather needed something that "just worked" and would allow researchers to be able to install packages independently of us (IT people). conda (mostly) fit the bill for this. I still think that Spack is the future and it has a special place in my heart, but it will have to be more stable for me to want to use it in production.
Re: Challenge to scientists: does your ten-year-old code still run?
#427Earlier quoted context omitted.
> Employing dedicated software developers is simply not a thing that happens This is a really key point that is lost on devs outside of science looking in. In our case, good devs are out of budget by a factor of 2x at least (at an EU public university in a lab doing lots of computational work). The best we get are engineers which are expected to keep the cluster running, order computers, organize seminars.. and event…
Are the hiring scientists also paid well-below market wages by that degree?
The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and an external developer coming from the industry - no matter how experienced or skilled - can usually be offered only a junior position with pay appropriate for that because they do not meet the criteria required for non-junior positions (no PhD, often not even a masters', no relevant publications, etc). From that perspective the only difference between a grad student who just started and a seasoned software developer is that the grad student can be employed as a part-time research assistant while a 'pure' developer could be full-time; but the hourly rate and conditions would be pretty much the same, targeted at less experienced employees. We can hire skilled mid-level individual contributors with reasonable pay for post-doc positions, but post-doc positions are limited to candidates who have a PhD. And it's not a that big limitation, since it's expected that everybody who's working "in the field" will get a PhD during their first few years of practical work experience as a grad student, the concept of "experienced/skilled but no degree" is not considered by the system as such people are rare in academia, and they stay rare due to the existing system.
So the disparity in evaluation criteria means that it's tricky to transfer between the different "career paths" - if you come from an environment where degrees mostly don't matter to an environment where a PhD is almost table stakes (to be a "hiring scientist", PhD is mandatory but nowhere near sufficient), then "getting your worth" is possible only if you are willing to put in quite some time and effort to fit the criteria used to evaluate scientists, even if you're there just to do software development.
Re: Challenge to scientists: does your ten-year-old code still run?
#428Plenty of actual professional programmers can't manage this, how is it a fair standard to hold scientists to, when the code is just one of the many tools they're trying to use to get their real job done? I think moving away from the cesspool of imported remote libraries that update at random times and can vanish off the internet without warning, would help a lot of both cases.
Professional programmers should adopt package manager that focus on reproducibility like Guix and Nix and make them accessible enough for non programmers to use. Neither of these are perfect but in my experience they are worlds better than apk, Dockerfiles, and many other commonly used solutions. http://guix.gnu.org/ https://nixos.org/
The next guy to come along didn't understand it and threw it all away.
Re: Challenge to scientists: does your ten-year-old code still run?
#429Earlier quoted context omitted.
Don't you think docker, dependencies, unit test frameworks, etc actually increase the need for ongoing maintenance as opposed to spitting out some C files or python scripts which last "forever"?
No. Python/C files didn't work in a vacuum. They need dependencies, that is the point of Docker after all. Capture all necessary dependencies into a single image.
I can build a docker file from 5 years ago because all the links are dead.
Re: Challenge to scientists: does your ten-year-old code still run?
#430Earlier quoted context omitted.
> scientists code in the Literate Programming Paradigm I wish. In my career as a computational scientist I have never seen this in practice, either in academia or industry. On unit testing, I half agree. Most unit tests get quickly thrown out as the code changes, so it's a depressing way to write research code. But tests absolutely help someone trying to run old code - they show what parts still work and how to use t…
It's even more common today. eg, Jupyter Lab or Jupyter Notebooks.