Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

341–350 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#341

Post by hobofan »

closeparen wrote: post #241
djsumdog wrote: post #16

Earlier quoted context omitted.

You can always run old Python2 stuff in a Docker container, so long as the dependencies haven't disappeared.

Most of the "requirements.txt" I come across in the real world do not actually lock down all deps to Python 2.7 compatible versions. I've been able to get most of them running again, but it's a long porcess looking through changelogs to find the last 2.7-compatible version of each dependency.

Yes, because the "requirements.txt" is a dependency requirements file and not a lockfile. It took the Node.js ecosystem an embarrassingly long time to arrive at that insight, and I feel like the Python ecosystem/community still isn't there yet (though finally it's easily usable with Poetry).

Re: Challenge to scientists: does your ten-year-old code still run?

#342

Post by [deleted] »

djaque wrote: post #105
BeetleB wrote: post #87

Earlier quoted context omitted.

> their job is not coding To me, that's like a theoretical physicist saying "My job is not to do mathematics" when asked for a derivation of a formula he put in the paper. Or an experimental physicist saying "My job is not mechanical engineering" when asked for details of their lab equipment (almost all of which is typically custom built for the experiment).

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

[deleted]

Re: Challenge to scientists: does your ten-year-old code still run?

#343

Post by allenofthehills »

ajford wrote: post #283
dllthomas wrote: post #219

Earlier quoted context omitted.

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

This! I struggled with this topic in university. I was studying pulsar astronomy, and there was only one or two common tools used at the lower levels of data processing, and had been the same tools used for a couple of decades. The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software. I rep…

> The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software.

This is the difference between reproducibility and replicability [1]. Reproducibility is the ability to run the same software on the same input data to get the same output; replication would be analyzing the same input data (or new, replicated data following the original collection protocol) with new software and getting the same result.

I've experienced the same lack of interest with established researchers in my field, but I can at least ensure that all my studies are both reproducible and replicable by sharing my code and data.

[1] Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Front Neuroinform. 2018;11:76.

Re: Challenge to scientists: does your ten-year-old code still run?

#344

Post by hobofan »

slhck wrote: post #20

The two main problems in academia are that a) few researchers have formal training in best practices of software engineering, and that b) time pressure leads to "whatever worked two minutes before submission deadline" becoming what is kept for posteriority. When I started working as a full-time researcher, I had come from working two years in a software shop, only to find people at the research lab having never used…

requirements.txt is not a lockfile

Re: Challenge to scientists: does your ten-year-old code still run?

#345

Post by BadInformatics »

dllthomas wrote: post #313
BadInformatics wrote: post #234

Earlier quoted context omitted.

Conversely though, it is often impossible to obtain the original code to replay and identify differences once that step is reached without some sort of strong incentive or mandate for researchers to publish it. When the only copy is lost in the now-inaccessible home folder of some former grad student's old lab machine, there is a strong disincentive to try replicating at all because one has little to consult on wheth…

And so we find ourselves in the same situation as the rest of the scientific process, throughout history. When I try to replicate your published paper and I fail, it's completely unclear whether it's "your fault" or "my fault" or pure happenstance, and there's a lot of picking apart that needs to be done with usually no access to the original experimental apparatus and sometimes no access to the original experimenter…

Of course, but the flip side is that same confluence of attributes has also exacerbated issues of reproducibility. Just as science and the methods/mediums by which we conduct/disseminate it have changed, so too should the standard of what is considered acceptable to reproduce. This is especially relevant given how much broader the societal and policy implications have become.

More concretely, it is 100% fair (and I might argue necessary) to demand more of our institutions and work to improve their failures. I'm sure many researchers have encountered publications of the form "we applied (not explained) to (partially explained) after and obtained SOTA results!" in a reputable venue. Sure, this might be even less reproducible 200 years ago than now, but the authors would also be less likely to be competing with you for limited funding! Debating about the traditional definition of reproducibility has its place, but we should also be doing as much as possible to give reviewers and replicators a leg up. This is often flies in the face of many incentives the research community faces, but shifting blame to institutions by default (not saying you're doing this, but I've seen many who do) is taking the easy road out and does little to help the imbalanced ratio of discussion:progress.

Re: Challenge to scientists: does your ten-year-old code still run?

#346

Post by kalenx »

RandoHolmes wrote: post #293
labcomputer wrote: post #263

Earlier quoted context omitted.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe…

> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software

We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me.

Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.

Re: Challenge to scientists: does your ten-year-old code still run?

#348

Post by iagovar »

petschge wrote: post #167
atrettel wrote: post #131

Earlier quoted context omitted.

> Whats been really awesome about that has been the fact that I've written some binary data files on big endian machines in the early 90s, and re-read them on the laptop (little endian) adding a single compiler switch. I want to second the idea of just dumping your floating point data as binary. It's basically the CSV of HPC data. It doesn't require any libraries, which could break or change, and even if the endianne…

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readabil…

Why not dumping into SQLite? It makes everything easy, and we will be able to use sqlite3 for a long time IMO.

Re: Challenge to scientists: does your ten-year-old code still run?

#349

Post by pthread_t »

dandelion_lover wrote: post #40
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

As a theoretical physicist doing computer simulations, I am trying to publish all my code whenever possible. However all my coauthors are against that. They say things like "Someone will take this code and use it without citing us", "Someone will break the code, obtain wrong results and blame us", "Someone will demand support and we do not have time for that", "No one is giving away their tools which make their compe…

> "No one is giving away their tools which make their competitive advantage"

This hits close to home. Back in college, I developed software, for a lab, for a project-based class. I put the code up on GitHub under the GPL license (some code I used was licensed under GPL as well), and when the people from the lab found out, they lost their minds. A while later, they submitted a paper and the journal ended up demanding the code they used for analysis. Their solution? They copied and pasted pieces of my project they used for that paper and submitted it as their own work. Of course, they also completely ignored the license.

Re: Challenge to scientists: does your ten-year-old code still run?

#350

Post by ryandrake »

smnrchrds wrote: post #173
ivanbakel wrote: post #125

Earlier quoted context omitted.

Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily? Given that such software forms the very foundation of the results of such papers, why shouldn't it fall under scrutiny, even for "minor" points? If you are unable to produce good technical content, why are you qualified to declare what is or isn…

> Doesn't it concern you that it would be possible for critics to look at your scientific software and find mistakes (some of which the OP mentioned are not "minor") so easily? A non-native English speaker may make grammatical mistakes when communicating their research in English—it does not in any way invalidate their results or hint that there is anything amiss. It is simply what happens when you are a non-native s…

On the contrary: If I'm (in industry) doing a code review and see simple, obvious mistakes like infinite loops, obvious null pointer exceptions, ignored compiler warnings, etc., in my mind it casts a good deal of doubt over the entire code. If the author is so careless with these obvious errors, what else is he/she being careless about?

Same with grammatical or spelling errors. I don't review research but I do review resumes, and I've seen atrocious spelling on resumes. Here's the candidate's first chance to make an impression. They have all the time in the world to proofread, hone, and have other eyes edit it. Yet, they still miss obvious mistakes. If hired, will their work product also be sloppy?

Post reply on HN