Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

441–450 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#441

Post by ISL »

konjin wrote: post #420
ISL wrote: post #356

Earlier quoted context omitted.

Reproducibility is about understanding the result. It is the modern version of "showing your work". One of the unsung and wonderful properties of reproducible workflows is the fact that it can allow science to be salvaged from an analysis that contains an error. If I had made an error in my thesis data analysis (and I did, pre-graduation), the error can be corrected and the analysis re-run. This works even if the aut…

>Reproducibility is about understanding the result. It is the modern version of "showing your work". That is something no one outside of highschool cares about. The idea that you can show work in general is ridiculous. Do I need to write a few hundred pages of set theory to start using addition in a physics paper? No. The work you need to show is the work a specialist in the field would find new, which is completely…

I heartily disagree -- reproducible analysis is essential for the intercomparison of analyses between specialist research groups.

Here is my thesis work, minus the data, which is too large to store within a GitHub repo. By calling `make`, it goes from raw data to final document in a single shot. The entire workflow, warts and all, can be audited. If you see a bug or concern, please let me know: https://github.com/4kbt/PlateWash

Re: Challenge to scientists: does your ten-year-old code still run?

#442

Post by sitkack »

matthewdgreen wrote: post #299
sitkack wrote: post #276

Earlier quoted context omitted.

I made no mention of Docker, VMs or any virtualization system. Those would be an implementation detail and would obviously change over time. A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

My point, which I guess I did not make clearly enough, is that container systems don't necessarily exist or remain supported over the ten-year period being discussed. The idea of ironing over long-term compatibility issues using a container environment seems like a great one! (For the record, .tgz -- the "standard" format for scientific code releases in 2010, does not solve these problems at all .) But the "implement…

Of course of course. I am not trying to boil the ocean here, or we would have a VM like wasm and a execution env like wasi and run all our 1000 year code inside of that.

The first step is just having your code, data and deps in an archive. Depending on the project and the age, more stuff makes it into the archive. I have been on projects where the source to the compiler toolchain was checked into the source repo and the first step was to boostrap the tooling (from a compiler binary checked into the repo).

We aren't even to the .tar.gz stage yet.

Re: Challenge to scientists: does your ten-year-old code still run?

#443

Post by ByteJockey »

MaxBarraclough wrote: post #372
martingab wrote: post #310

Earlier quoted context omitted.

That's exactly the philosophy we follow e.g. in particle physics and its a common excuse to dismiss all guidelines made in the article. However, this kind of validation/falsification is often done between different research groups (maybe using different but formally equivalent approaches) while people within the same group have to deal with the 10 years old code base. I myself had very bad experience with extending t…

> If it's a one-man-show I would not give too much on code quality This makes me a little uneasy, as I'm not too worried about code quality can easily translate into Yes I know my code is full of undefined behaviour, and I don't care . > PS: quite excited about my first post here Welcome to HN! reddit has more cats, Slashdot has more jokes about sharks and laserbeams, but somehow we get by.

Are we talking actual undefined behavior or just behavior that's undefined by the language standard?

The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

Re: Challenge to scientists: does your ten-year-old code still run?

#444

Post by dllthomas »

improbable22 wrote: post #334
ajford wrote: post #283

Earlier quoted context omitted.

This! I struggled with this topic in university. I was studying pulsar astronomy, and there was only one or two common tools used at the lower levels of data processing, and had been the same tools used for a couple of decades. The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software. I rep…

This is almost an argument for not publishing code. If you publish all the equations, then everybody has to write their own implementation from that. Something like this is the norm in some more mathematical fields, where only the polished final version is published, as if done by pure thought. To build that, first you have to reproduce it, invariably by building your own code -- perhaps equally awful, but independen…

Maybe gate release of the code by some number of attempted replications.

Re: Challenge to scientists: does your ten-year-old code still run?

#445

Post by sriku »

dllthomas wrote: post #219
solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

I would also place as a requirement that the code be comprehensible to someone familiar with the domain - .i.e. a "peer".

Re: Challenge to scientists: does your ten-year-old code still run?

#447

Post by lambdatronics »

Fishysoup wrote: post #392

As a scientist I've written massive amounts of shitty code that turned out to be reproducible by lucky accident. Part of the problem are the tools: depending on the field, scientists either use Matlab, C++, Fortran or some other framework that needs to die. They base their code on other ancient code that runs for unknown reasons, and use packages written by other scientists with the same problems. As someone who's tr…

Julia could be a big win, not just b/c of the notation, but the dependency control is a first-class language feature. Also, the Lispy-ness of Julia allows to do things like Latexify expressions.

To quote someone upthread: https://news.ycombinator.com/item?id=24260590

> Everyone just put together a few text files and Python or MATLAB scripts that output some numbers that went into Excel or gnuplot scripts that got copy-pasted into LaTeX documents with suffixes like "v2_final_modified.tex", shared over Dropbox.

It would be amazing to have an environment that could handle the entire workflow. Not everybody has time to make an executable thesis like this person did:

https://github.com/4kbt/PlateWash

Re: Challenge to scientists: does your ten-year-old code still run?

#448

Post by selectionbias »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Sorry to be pedantic, but although Monte Carlo simulations are based on pseudo-randomness, I still think it is good practice that they have deterministic results (i.e., use a given seed) so that the exact results can be replicated. If the precise numbers can be reproduced then a) it helps me as a reviewer see that everything is kosher with their code and b) it means that if I tweak the code to try something out my results will be fully compatible with theirs.

Re: Challenge to scientists: does your ten-year-old code still run?

#449

Post by martingab »

jnxx wrote: post #395
martingab wrote: post #310

Earlier quoted context omitted.

That's exactly the philosophy we follow e.g. in particle physics and its a common excuse to dismiss all guidelines made in the article. However, this kind of validation/falsification is often done between different research groups (maybe using different but formally equivalent approaches) while people within the same group have to deal with the 10 years old code base. I myself had very bad experience with extending t…

> After all, I'm wasting 2 months of my PhD for the marriage of my own results with known results which -in principle- could have been done within one day if the code base would allow for it. Sounds like it is quite good science to do that, because it puts the computation on a pair of independent feet. Otherwise, it could just be that the code you are using as a bug and nobody notes until it is too late.

I see your and MaxBarraclough concerns. In my case, there exist 5-6 codes which do -at their core- the same thing as ours does and they all have been cross-checked against each other within either theoretical or numerical precision (where possible). That's the spirit that sjburt was referring to, I guess, and which triggered me because it is only true to a certain extend.

The cross-checking is anyways good scientific practise, not only because of bugs in the code (that's actually a sub-leading problem imho), but because of the degree of difficulty of the problems and the complexity of their solutions (and their reproducibility). In that sense, cross-checking should discover both, scientific "bugs" and programming-bugs. The "debugging" is partly also done at the community level - at least in our field of research.

However, it is also a matter of efficiency. I -and many others too- need to re-implement not because of bug-hunting/cross-checking but simply because we do not understand the "ugly" code of our colleagues and instead of taking the risk to break existing code we simply write new one which is extremely inefficient (others may take the risk and then waste months on debugging and reverse-engineering which is also inefficient). So my point on writing "good code" is not so much about avoiding bugs but about being kind to you colleagues, saving them nerves and time (which they can then spend on actual science) and thus also saving taxpayers money...

Re: Challenge to scientists: does your ten-year-old code still run?

#450

Post by jhrmnn »

xen0 wrote: post #74
jhrmnn wrote: post #64

Earlier quoted context omitted.

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas h…

My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.

I should have mentioned that I of course use Git. But the need to manually keep track of the calculation~commit pairing is tiresome and error-prone.
Post reply on HN