Challenge to scientists: does your ten-year-old code still run? • Hacker News Board

Re: Challenge to scientists: does your ten-year-old code still run?

#391

Post by jnxx » Mon, Aug 24, 2020, 9:53 PM UTC

yjftsjthsd-h wrote: post #273

labcomputer wrote: post #263

Earlier quoted context omitted.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

> In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that…

In many parts of scientific research, researchers are, to stay in your metaphor, more travelers using a map, than map makers.

Of course, it is a difference whether you make a clinical study on drugs, and use a pocket calculator to compute a mean, or whether you research in numerical analysis, or are presenting a paper in how to use Coq to more efficiently prove the four-color theorem or Fermat's last theorem.

In short, much of science is not computer science, and for it, computation is just a tool.

Re: Challenge to scientists: does your ten-year-old code still run?

#392

Post by Fishysoup » Mon, Aug 24, 2020, 9:53 PM UTC

As a scientist I've written massive amounts of shitty code that turned out to be reproducible by lucky accident. Part of the problem are the tools: depending on the field, scientists either use Matlab, C++, Fortran or some other framework that needs to die. They base their code on other ancient code that runs for unknown reasons, and use packages written by other scientists with the same problems.

As someone who's transitioning into industry, I can tell you that scientists will never adopt software engineering principles to any significant extent. It takes too much time to do things like write tests and thorough documentation, learn Git, etc., and software engineering just isn't interesting to most of them.

So the only alternative I see is changing the tools to stuff that's still easy to hack around with but where it's harder to mess up (or it's more obvious when you do so). That doesn't leave a ton of options (that I can see). Some I can think of are:

- Make your code look more like math and less like mathlib.linalg.dot(x1, x2).reshape(a, b).mean().euclidean_distance((x3, x4)) + (other long expression) or whatever: Use a language like Julia

- Your language/environment gets angry when you write massive hairballs, loads of nested for-loops and variables that keep getting changed: Use a language like Rust, and/or write more modular code with a functional-leaning language like Rust or Julia.

- You're forced to make your code semi-understandable to you and others more than an hour after writing it: Forcing people to write documentation isn't gonna work (a lot). Forcing sensible variable names is slightly more realistic. More likely, you need some combination of the above two things that just make your code more legible.

How do you make that happen? No idea.

Re: Challenge to scientists: does your ten-year-old code still run?

#393

Post by dllthomas » Mon, Aug 24, 2020, 9:55 PM UTC

hobofan wrote: post #329

dllthomas wrote: post #219

Earlier quoted context omitted.

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

But "rerunning reproducability" is mostly a neccessary requirement for independent reproducability. If you can't even run the original calculations against the original data again how can you be sure that you are not comparing apples to oranges?

Very interesting. I was thinking of software as most similar to apparatus, and secondarily to procedure. You raise a third possible comparison: calculations, which IIUC would be expected to be included in the paper.

There are some kinds of code (a script that controls a sensor or an actuator) where I think that doesn't match up well at all. There are plenty of kinds of code where they are, in fact, simply crunching numbers produced earlier. For the latter, I'm honestly not sure the best way to treat it, except to say that we should be sure that enough information is included in some form that replication should be possible, and that we keep in mind the idea that replication should involve human interaction.

Re: Challenge to scientists: does your ten-year-old code still run?

#394

Post by RandoHolmes » Mon, Aug 24, 2020, 9:56 PM UTC

kalenx wrote: post #346

RandoHolmes wrote: post #293

Earlier quoted context omitted.

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe…

> How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of s…

> Asking a scientist to do better than whole teams of software engineers makes little sense to me.

This is not what is being asked, shame on you for the strawman.

Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".

Re: Challenge to scientists: does your ten-year-old code still run?

#395

Post by jnxx » Mon, Aug 24, 2020, 9:56 PM UTC

martingab wrote: post #310

sjburt wrote: post #180

Earlier quoted context omitted.

The findings really should be independent of the code. Reproduction should occur by taking the methodology and re-implementing the software and running new experiments.

That's exactly the philosophy we follow e.g. in particle physics and its a common excuse to dismiss all guidelines made in the article. However, this kind of validation/falsification is often done between different research groups (maybe using different but formally equivalent approaches) while people within the same group have to deal with the 10 years old code base. I myself had very bad experience with extending t…

> After all, I'm wasting 2 months of my PhD for the marriage of my own results with known results which -in principle- could have been done within one day if the code base would allow for it.

Sounds like it is quite good science to do that, because it puts the computation on a pair of independent feet.

Otherwise, it could just be that the code you are using as a bug and nobody notes until it is too late.

Re: Challenge to scientists: does your ten-year-old code still run?

#396

Post by Fiahil » Mon, Aug 24, 2020, 10:00 PM UTC

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

My work position was created because scientists are not engineers. I had to explain -to my disappointment- why non-deterministic algorithms are bad, how to write tests, and how to write SQL queries, more than once.

However, when working as equals scientists and engineers can create truly transformative projects. Algorithms accounts for 10% of the solution. The code, infrastructure and system design accounts for 20% of the final result. The remaining 70% of the value, is directly coming from its impact. A projects that nobody uses is a failure. Something that perfectly solves a problem that nobody cares about is useless.

Re: Challenge to scientists: does your ten-year-old code still run?

#397

Post by RandoHolmes » Mon, Aug 24, 2020, 10:03 PM UTC

ufmace wrote: post #352

RandoHolmes wrote: post #293

Earlier quoted context omitted.

No one is saying that code is the science. If I'm given bad information and I act on that information, then problems can occur. Similarly, if the software is giving the scientist bad information, problems can occur. How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe…

I don't entirely disagree, but haven't there also been cases of experimental results being invalidated due to subtle mechanical, electrical, chemical, etc complications with the test equipment, when none of the people involved in the experiment were experts in those fields? I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is h…

If they're setting up experiments whose correct results require electrical expertise, then yes, they should either get better training or bring in someone who has it.

It's not clear to me why you think I would argue that inaccuracies should be avoided in software but accept that they're ok for electrical systems.

Re: Challenge to scientists: does your ten-year-old code still run?

#398

Post by andrewprock » Mon, Aug 24, 2020, 10:24 PM UTC

solatic wrote: post #142

djaque wrote: post #98

Earlier quoted context omitted.

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes

This is definitely not correct. The experiment as a whole needs to be reproducible independently. This is very different, and more robust, from requiring that a particular portion of a previous version of the experiment to be reproducible in isolation.

Re: Challenge to scientists: does your ten-year-old code still run?

#399

Post by oliver101 » Mon, Aug 24, 2020, 10:24 PM UTC

djaque wrote: post #98

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

Why is "doing software engineering" not "doing science"?

Anybody who has conducted experimental research will say they spent 80% of the time using a hammer or a spanner. Repairing faulty lasers or power supplies. This process of reliable and repeatable experimentation is the basis of science itself.

Computational experiments must be held to the same standards as physical experiments. They must be reproducible and they should be publicly available (if publicly funded).

Re: Challenge to scientists: does your ten-year-old code still run?

#400

Post by pbalau » Mon, Aug 24, 2020, 10:26 PM UTC

djaque wrote: post #98

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

> people claiming that their non-software engineering grade code invalidates the results of their study.

But that's exactly the problem.

Are you familiar with that bug in early Civ games where an overflow was making Ghandi nuke the crap out of everyone? What if your code has a similar issue?

What if you have a random value right smack in the middle of your calculations and you just happened to be lucky when you run your code?

I'm not that familiar with Monte Carlo, my understanding is that this is just a way to sample the data. And I won't be testing your data sampling, but I will expect that given the same data to your calculations part (eg, after the sampling happens), I get exactly the same results every time I run the code and on any computer. And if there are differences I expect you to be able to explain why they don't matter, which will show you were aware of the differences in the first place and you were not just lucky.

And then there is the matter of magic values that plaster research code.

Researchers should understand that the rules for "software engineering grade code" are not there just because we want to complicate things, but because we want to make sure the code is correct and does what we expect it to do.

/edit: The real problem is not getting good results with faulty code, is ignoring good solutions because faulty code.