Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

231–240 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#231

Post by petschge »

atrettel wrote: post #217
petschge wrote: post #167

Earlier quoted context omitted.

Counter argument: Binary dumps are horrible because usually the documentation that allows you to read the data is missing. Using a self-documenting format such as HDF5 is far superior. It will tell you of the bit are floating point numbers in single or double precision, which endianess and what the layout of the 3d array was. (No surprise that HDF was invented for the Voyager mission where they had to ensure readabil…

Your argument raises a lot of good points. I actually agree that binary does lose all of the metadata and documentation that goes with it. That is a big problem. That is why I think it is also important to include some sort of documentation like an Xdmf file [1]. That is what I use to tie everything together in my particular project. HDF5 is fine. In fact, I would have strongly preferred my colleagues using HDF5 over…

Having an Xdmf file alongside is nice, but the breaking changes between v2 and v3 are very annoying. And I understand the want to have few external dependencies, but at least HDF5 is straight forward to compile and available as a pre-compiled module on all supercomputers that I have ever seen.

Re: Challenge to scientists: does your ten-year-old code still run?

#232

Post by bumby »

dandelion_lover wrote: post #210
SiempreViernes wrote: post #198

Earlier quoted context omitted.

To clarify, nobody sees the code because they aren't allowed, or nobody ever ask to see it?

The second case. However I am hesitating to ask to look at the code of my supervisor. How would I explain why I need it (if it's not needed for my research)? It's also unlikely user-friendly, so it would take a lot of time to understand anything.

I think you touched on something important. Researchers are most concerned with “getting things working”.

One of my favorite points from the book Clean Code was that professional developers aren’t satisfied with “working code”, they aim to make it maintainable. Which may mean writing it in a way that is more clear and concise than we are used to

Re: Challenge to scientists: does your ten-year-old code still run?

#233

Post by mnw21cam »

detaro wrote: post #34
non-entity wrote: post #23

Earlier quoted context omitted.

Ive seen job listings for "scientific programmers" where what they're asking for is a scientist who happens to know a little programming.

Yeah - who then likely doesn't have that much software experience, and worse, if they want to stay a scientist such a role is often a bad career move, because they help others get ahead with their research instead of publishing their own work. Even if they build some really great domain-specific software tool in that role, it often doesn't count as much. Or it's an informal thing done by some student as a side-gig. W…

Simple answer for that. University pay scales tend to be fairly inflexible in terms of which grades you are eligible for without a PhD, if you are counted as academic staff. If you're non-academic staff (like the cleaner, the receptionist, and the central IT sysadmin) then you can be paid a fair wage based upon your experience, but if you are academic staff, then you have a hard ceiling without a PhD. An individual research group with a grant may only be able to hire academic staff, but they want a sysadmin, so in order to be able to pay them more than a pittance they would have to have a PhD.

Re: Challenge to scientists: does your ten-year-old code still run?

#234

Post by BadInformatics »

dllthomas wrote: post #219
solatic wrote: post #142

Earlier quoted context omitted.

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard . Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same insta…

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction…

Conversely though, it is often impossible to obtain the original code to replay and identify differences once that step is reached without some sort of strong incentive or mandate for researchers to publish it. When the only copy is lost in the now-inaccessible home folder of some former grad student's old lab machine, there is a strong disincentive to try replicating at all because one has little to consult on whether/how close the replicated methods are to the original ones.

Re: Challenge to scientists: does your ten-year-old code still run?

#235

Post by bumby »

throwaway287391 wrote: post #144
bumby wrote: post #60

Earlier quoted context omitted.

I’m curious, are dedicated software assurance teams a thing in your research area? Or is quality left up to the primary researchers?

> I’m curious, are dedicated software assurance teams a thing in your research area? Are these a thing in any research area? I've heard of exactly one case of an academic lab (one that was easily 99th+ percentile in terms of funding) hiring one software engineer not directly involved in leading a research effort, and when I tell other academics about this they're somewhat incredulous. (I admittedly have a bit of trou…

>Are these a thing in any research area

I can say there are some that have the explicit intent but it can often fall to the wayside due to cost pressure. For example, government funded research from large organizations (think DoD or NASA) have these quality requirements but they can often be hand-waved away or just plain ignored due to cost concerns

Re: Challenge to scientists: does your ten-year-old code still run?

#236

Post by roberto »

For my first scientific article, in 2007, I created a Subversion repo with a Makefile. Running `make` would recreate the whole paper: downloading data, running analyses, creating pictures (color or BW, depending on an environment flag) and generating the PDF.

I'm going to try to find the repo and see if it still works.

Re: Challenge to scientists: does your ten-year-old code still run?

#237

Post by mnw21cam »

xen0 wrote: post #74
jhrmnn wrote: post #64

Earlier quoted context omitted.

In all my papers the results were produced on multiple days (spanning months), with multiple versions of the code, and they are computationally too expensive to reproduce with the final version of the code. I'm trying to keep track of all the used versions, but given that there is no automated framework for this (is there?) and research involves lots of experiments, it's never perfect. Given this context, any ideas h…

My first thought: Demand the journals provide hosting for a code repo that is part of your paper. For every numerical result, specify the version (e.g. a git tag) used to generate your result. And if that means scientists need to learn about version control, well... they should if they're writing code.

For a paper I recently submitted, the journal demanded a github release of the software.

Re: Challenge to scientists: does your ten-year-old code still run?

#238

Post by BadInformatics »

throwaway287391 wrote: post #208
SilasX wrote: post #191

Earlier quoted context omitted.

You're right about bit-for-bit reproducibility possibly being overkill, but I don't think that invalidates the parent's point that Monte Carlo randomization doesn't obviate reproducibility concerns. It just means that e.g. your results shouldn't be hypersensitive to the details of the randomization. That is, reviewers should be able to take your code, feed it different random data from a similar distribution to what…

That brings up a separate issue that I didn't comment on above: the expectation that the code runs in a completely different development/execution environment (e.g. the one the reviewer is using vs. the one that the researcher used). That means making it run regardless of the OS (Windows/OSX/Linux/...) and hardware (CPU/GPU/TPU, and even within those, which one) the reviewer is using. This would be an extremely diffi…

I don't think that removes the need to provide enough detail to replicate the original environment though. We write one-off scripts with no expectation that they will see outside usage, whereas research publications are meant for just that! The bar isn't terribly high either: for ML, a requirements.txt + OS version + CUDA version would go a long way, no need to learn docker just for this.

Re: Challenge to scientists: does your ten-year-old code still run?

#239

Post by Kenji »

_wldu wrote: post #103

IMO, this is why ISO standard programming languages are so important and will be around forever. One can always compile with --std=c++11 (or whatever) and be certain it will work.

Hahaha you would be surprised. Compiling complex C++ projects is incredibly difficult.
Post reply on HN