Challenge to scientists: does your ten-year-old code still run? • Hacker News Board

Re: Challenge to scientists: does your ten-year-old code still run?

#291

Post by jnxx » Mon, Aug 24, 2020, 6:08 PM UTC

dekhn wrote: post #271

solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

As an ex-scientist who used to run lots of simulations, I really fail to see a truly compelling reason why most numerical results (for publication purposes) truly need to publish (and support) deterministic seeding. We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.

If nothing else, it helps debugging code which tries to reproduce your findings.

Re: Challenge to scientists: does your ten-year-old code still run?

#292

Post by dunefox » Mon, Aug 24, 2020, 6:08 PM UTC

magv wrote: post #128

dunefox wrote: post #102

Earlier quoted context omitted.

If a scientist needs to write code then it's part of their job. It's as easy as that.

I think the idea that scientific code should be judged by the same standards as production code is a bit unfair. The point when the code works the first time is when an industry programmer starts to refactor it -- because he expects to use and work on it in the future. The point when the code works the first time is when a scientists abandons it -- because it has fulfilled its purpose. This is why the quality is lowe…

That's not the point, though. If you obtain your results by writing and executing code then code quality matters - to reproduce and validate them.

Re: Challenge to scientists: does your ten-year-old code still run?

#293

Post by RandoHolmes » Mon, Aug 24, 2020, 6:09 PM UTC

labcomputer wrote: post #263

throwaway7281 wrote: post #145

Earlier quoted context omitted.

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

No one is saying that code is the science.

If I'm given bad information and I act on that information, then problems can occur.

Similarly, if the software is giving the scientist bad information, problems can occur.

How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well?

There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life.

---

This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.

Re: Challenge to scientists: does your ten-year-old code still run?

#294

Post by Jabbles » Mon, Aug 24, 2020, 6:10 PM UTC

analog31 wrote: post #285

Jabbles wrote: post #108

Earlier quoted context omitted.

I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

I'm a scientist in a group that also includes a software production team. For me, the standard of scientific reproducibility is that a result can be replicated by a reasonably skilled person, who might even need to fill in some minor details themselves. Part of our process involves cleaning up code to a higher state of refinement as it gets closer to entering the production pipeline. I've tested 30 year old code, and…

This sounds great. In your opinion, do you think your team is unusual in those aspects? Do you have any knowledge of the quality of code in other branches of physics or other sciences?

Re: Challenge to scientists: does your ten-year-old code still run?

#295

Post by abdullahkhalids » Mon, Aug 24, 2020, 6:10 PM UTC

Sebb767 wrote: post #24

As someone who worked with bits of scientific code: Does the code you write right now work on another machine might be the more appropriate challenge. If seen a lot of hardcoded paths, unmentioned dependencies and monkey-patched libraries downloaded from somewhere; just getting the new code to work is hard enough. And let's not even begin to talk about versioning or magic numbers. Similar to other comments I don't me…

Lots of people saying, it is the scientist's job to produce reproducible code. It is, and the benefits of reproducible code are many. I have been a big proponent of it in my own work.

But not with the current mess of software frameworks. If I am to produce reproducible scientific code, I need an idiot-proof method of doing it. Yes, I can put in the 50-100 hours to learn how to do it [1], but guess what, in about 3-5 years a lot of that knowledge will be outdated. People comparing it with math, but the math proofs I produce will still be readable and understandable a century from now.

Regularly used scientific computing frameworks like matlab/R/Python ecosystem/mathematica need a dumb guided method of producing releasable and reproducable code. I want to go through a bunch of next buttons, that help me fix the problems you indicate, and finally release a final version that has all the information necessary for someone else to reproduce the results.

[1] I have. I would put myself in the 90th percentile of physicists familiar with best practices for coding. I speak for the 50% percentile.

Re: Challenge to scientists: does your ten-year-old code still run?

#296

Post by yjftsjthsd-h » Mon, Aug 24, 2020, 6:11 PM UTC

emerged wrote: post #141

Any scientist with good foresight would've implemented their code in 6502 for the NES. The emulators are nearly flawless and will probably be around until the end of time.

I once had a thought, that if I wanted to write something that would last forever and run anywhere, I should write it to target DOS, and make sure to test it on FreeDOS in a VM and on DOSBox. That way it would run on a stable ABI with loads of emulators, and via DOSBox it will happily run on all modern desktop OSs (and some non-desktops; IIRC there's at least an Android port).

Re: Challenge to scientists: does your ten-year-old code still run?

#297

Post by PeterisP » Mon, Aug 24, 2020, 6:11 PM UTC

TimothyBJacobs wrote: post #124

djaque wrote: post #105

Earlier quoted context omitted.

The point is that as a scientist your code is a tool to get the job done and not the product. I can't spend 48 hours writing unit tests for my library (even though I want to) if it's not going to give me results. It's literally not my job and is not an efficient use of my time

This is the same as any other argument against testing. Unless you are actually selling a library, code is not the product. Customers are buying results, not your code base. Yet, we've discovered the importance of testing to make sure customers get the right results without issues. If you want your results to be usable by others, the quality of the code matters. If all you care is publishing a paper, then I guess sur…

But the results are usable by others, in most fields of science the code is not part of these results and is not needed to enjoy, use and build upon the research results.

The only case where the code would be used (which is a valid reason why it should be available somehow) is to assert that your particular results are flawed or fraudulent; otherwise the quality of the code (or its availability, or even existence - perhaps you could have had a bunch of people do all of it on paper without any code) is simply irrelevant if you want your results to be usable by others.

Re: Challenge to scientists: does your ten-year-old code still run?

#298

Post by woah » Mon, Aug 24, 2020, 6:14 PM UTC

djaque wrote: post #98

djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

I’m very puzzled by this attitude. As an accelerator physicist, would you want you accelerator to be held together by duct tape, and producing inconsistent results? Would you complain that you’re not a professional machinist when somebody pointed it out? Why is software any different than hardware in this respect?

Re: Challenge to scientists: does your ten-year-old code still run?

#299

Post by matthewdgreen » Mon, Aug 24, 2020, 6:15 PM UTC

sitkack wrote: post #276

matthewdgreen wrote: post #269

Earlier quoted context omitted.

All of my 2010 scientific code runs on the then-current edition of Docker. /s

I made no mention of Docker, VMs or any virtualization system. Those would be an implementation detail and would obviously change over time. A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

My point, which I guess I did not make clearly enough, is that container systems don't necessarily exist or remain supported over the ten-year period being discussed. The idea of ironing over long-term compatibility issues using a container environment seems like a great one! (For the record, .tgz -- the "standard" format for scientific code releases in 2010, does not solve these problems at all.)

But the "implementation detail" of which container format you use, and whether it will still be supported in 10 years, is not an implementation detail at all -- since this will determine whether containerization actually solves the problem of helping your code run a decade later. This gets worse as the number, complexity and of container formats expands.

Of course if what you mean is that researchers should provide perpetual maintenance for their older code packages, moving them from one obsolete platform to a more recent one, then you're making a totally different and very expensive suggestion.

Re: Challenge to scientists: does your ten-year-old code still run?

#300

Post by dnautics » Mon, Aug 24, 2020, 6:15 PM UTC

throwaway287391 wrote: post #196

dnautics wrote: post #183

Earlier quoted context omitted.

the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.

Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).

Most scientific runs are scaled where you run multiple replicates. And not all scientific runs are high-performance in the HPC sense. Even if your code is HPC in the HPC sense, and requires CUDA, and 40,000 cores, you should consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level.

I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.