Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

481–487 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#481

Post by PeterisP »

j45 wrote: post #469
PeterisP wrote: post #427

Earlier quoted context omitted.

In many fields industry pays noticeably better than academia, but the difference is not that meaningful for the actual scientists; a principal investigator gets a reasonable amount of money but also significant degree of freedom and influence which helps job satisfaction even if the pay itself is lower. The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and…

Academia definitely can place software creators at a lower payscale to keep other higher. I wonder why such a caste system exists?

It does not place software creators at a lower payscale - all the software creators I know in academia are at the payscale level where they should be given their experience, however, all of them have a PhD or are in the process of getting one very soon.

My point is that it places outsiders (no matter if they're going to do software development or something else) on a lower payscale until they catch up on all the academia-specific factors of evaluation.

It's not a caste system between different types of activities, but rather a barrier of entry - in some sense, you have to start from 'level 1' no matter how much experience you have in other fields, so inexperienced people can join easily, but for senior/experienced people doing it is possible but costly.

Re: Challenge to scientists: does your ten-year-old code still run?

#482

Post by robocat »

PeterisP wrote: post #225
j45 wrote: post #159

Earlier quoted context omitted.

One of the things I come across is scientists who believe they're capable of learning code quickly because they're capable in another field. After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running. For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above? I would have t…

It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development. In any case you'd need to teach them the problem…

> research labs overwhelmingly have the coding done by researchers and involved students

This is a general problem we all have, whenever we should employ a professional to do necessary work. The right professional will take a tenth of the time and the job will be done some multiple better. But how do you pick the right person?

I have two experiences with post-grad work that I think are relevant:

1. A friend needed some work done in a statistics package that used a language that felt like it was from the 80's. I was able to complete the work in a few hours, but I don't think a student could have done it (complicated need combined with a crappy language and IDE).

2. Another postdoc engineering friend needed to do some heavy duty data analysis, and she was recommended to learn C++. I suspect she wasted years learning C++, time which should have been spent on investigating different forms of analysis. She wanted to listen to her engineering fellows, not some practicing software engineer, so wasted her life not achieving much...

Re: Challenge to scientists: does your ten-year-old code still run?

#483

Post by MaxBarraclough »

ByteJockey wrote: post #480
MaxBarraclough wrote: post #454

Earlier quoted context omitted.

> Are we talking actual undefined behavior or just behavior that's undefined by the language standard? 'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity. > if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code. Code should be correct by construction, not correct by coincide…

> Code should be correct by construction, not correct by coincidence. Glad we agree, if you're aware of how your compiler handles these things, you can construct it to be correct in this way. It won't be portable at all (even to the next patch version of the compiler), I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

> if you're aware of how your compiler handles these things, you can construct it to be correct in this way.

I presume we agree but I'll do my usual rant against UB: Deliberately introducing undefined behaviour into your code is playing with fire, and trying to outsmart the compiler is generally a bad idea. Unless the compiler documentation officially commits to a certain behaviour (rollover arithmetic for signed types, say), then you should take steps to avoid undefined behaviour. Otherwise, you're just going with guesswork, and if the compiler generates insane code, the standards documents define it to be your fault.

It might be reasonable to make carefully disciplined and justified exceptions, but that should be done very cautiously. JIT relies on undefined behaviour, for instance, as ultimately you're treating an array as a function pointer.

> It won't be portable at all (even to the next patch version of the compiler)

Right, doing this kind of thing is extremely fragile. Does it ever crop up in real-life? I've never had cause to rely on this kind of thing.

It would be possible to use a static assertion to ensure my code only compiles on the desired compiler, preventing unpleasant surprises elsewhere, but I've never seen a situation where it's helpful.

This isn't the same thing as relying on 'ordinary' compiler-specific functionality, such as GCC's fixed-point functionality. Such code will simply refuse to compile on other compilers.

> I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

Disagree. It should be possible to independently reproduce the experiment. Robust code helps with this. Code shouldn't depend on an exact compiler version, there's no good reason code should.

Re: Challenge to scientists: does your ten-year-old code still run?

#484

Post by avasthe »

labcomputer wrote: post #263
throwaway7281 wrote: post #145

Earlier quoted context omitted.

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

You are missing the point.

How many actually try to reproduce the results by writing corresponding code themselves? Apparently lot of papers with slightly wrong findings because code errors have passed the peer review (all of us in the SWE bubble know how often bugs occur), at least in less prestigious journals.

There is nothing wrong with mandating the code to be supplied with the paper. Because, many time code is somewhere between the experimental setup and proof / result.

Re: Challenge to scientists: does your ten-year-old code still run?

#485

Post by martingab »

snowwrestler wrote: post #379
sitkack wrote: post #253

Earlier quoted context omitted.

> exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done. I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it…

The history of physics is full of complex, one-off custom hardware. Reviewers have not been expected to take the full technical specs and actually build and run the exact same hardware, just to verify correctness for publication. I doubt any physicist believes we need to get the Tevatron running again just to check decade-old measurements of the top quark. I don't understand why decade-old scientific software code mu…

They didn't rebuild the Tevatron but still were able to rediscover the top within a different experimental environment (i.e. LHC with tons of different discovery channels) and have lots of fits for it properties from indirect measurements (LEP, Belle). Physics is not an exact science. If you have only one measurement (no matter if its software- or hardware-based), no serious physicist would fully trust in the result as long as it wasn't confirmed by an independent research group (by doing more than just rebuilding/copying the initial experiment but maybe using slightly different approximations or different models/techniques). I'm not so much in computer science, but I guess here it might be a bit different ones a prove is based on rigorous math. However even if so, I guess, it's sometimes questionable if the prove is applicable to real-world systems and then one might be in a similar situation.

Anyways, in physics they always require several experimental proves for our theory. They also have several "software experiments" for e.g. predicting the same observables. Therefore, researchers need to be able to compile and run the code of their competitors in order to compare and verify the results in detail. In this place, bug-hunting/fixing is sometimes also taking place - of course. So applying the articles suggestions would have the potential to accelerate scientific collaboration.

btw; I know some people who do still work with the data taken at the LEP experiment which was shut down almost 20 (!) years ago and they have a hard time in combining old detector-simulations, monte-carlos etc. with new data-analysis techniques for the exact same reasons mentioned in the article. For large-scale experiments it is a serious problem which nowadays has much more attention than at LEP ages, since LHC has anyways obvious big-data problems to solve before their next upgrade, including also software-solutions.

Re: Challenge to scientists: does your ten-year-old code still run?

#486

Post by spott »

djaque wrote: post #98
djsumdog wrote: post #13

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f40022... This was used by the Imperial College…

I am all for open science, but you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study. I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor…

>when that is the entire point of Monte Carlo methods and doesn't change their result.

Two nitpicks: a) it shouldn't change the conclusions, but MC calculations will get different results depending on the seed. and b) it is considered good practice in reproducible science to fix the seed so that the results of subsequent runs give exactly the same results.

Ultimately, I think there is a balance: really poor code can lead to incorrect conclusions... but you don't need production ready code for scientific exploration.

Re: Challenge to scientists: does your ten-year-old code still run?

#487

Post by rudolph9 »

konjin wrote: post #428
rudolph9 wrote: post #156

Earlier quoted context omitted.

Professional programmers should adopt package manager that focus on reproducibility like Guix and Nix and make them accessible enough for non programmers to use. Neither of these are perfect but in my experience they are worlds better than apk, Dockerfiles, and many other commonly used solutions. http://guix.gnu.org/ https://nixos.org/

I've had the pleasure of setting up a guix system in production. The next guy to come along didn't understand it and threw it all away.

That’s very unfortunate.
Post reply on HN