Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenge to scientists: does your ten-year-old code still run?

nature.com

Post reply on HN
Print view View original

471–480 of 487 posts

Re: Challenge to scientists: does your ten-year-old code still run?

#471

Post by j45 »

UweSchmidt wrote: post #373
jpeloquin wrote: post #330

Earlier quoted context omitted.

> I would have thought the scientists among us would approach someone with familiarity with software development expertise. Is there a pool of skilled software architects willing to provide consultations at well-below market wages? Or a Q&A forum full of people interested in giving this kind of advice? (StackOverflow isn't useful for this; the allowed question scope is too narrow.) I guess one incentive to publish on…

I'll state the obvious and answer with No. There are not enough skilled software architects to go around and many who consider themselves skilled are not actually producing good code themselves, probably including many confident posters here in this forum. The idiosyncrasies and tastes of many 'senior' software engineers would likely make the code unreadable and unmaintainable for the average scientist and possibly d…

I think the suggestion to keep the codebase as simple as possible for scientists applies as well to the software creators.

Life is different when you might have the a relationship with a single code base for 2-5 years, or even more. Complexities will happen on their own, no need to add them in.

Re: Challenge to scientists: does your ten-year-old code still run?

#472

Post by marmaduke »

j45 wrote: post #469
PeterisP wrote: post #427

Earlier quoted context omitted.

In many fields industry pays noticeably better than academia, but the difference is not that meaningful for the actual scientists; a principal investigator gets a reasonable amount of money but also significant degree of freedom and influence which helps job satisfaction even if the pay itself is lower. The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and…

Academia definitely can place software creators at a lower payscale to keep other higher. I wonder why such a caste system exists?

It’s not a caste system, rather a capitalist imitation where the capital is impact factor and grants and first authorship in nature and science. In this system software creators are just means to an end, and good software engineers are an irrational cost given the PhD student that can churn out working code for same impact factor for less money.

Re: Challenge to scientists: does your ten-year-old code still run?

#473

Post by RandoHolmes »

kalenx wrote: post #466
RandoHolmes wrote: post #431

Earlier quoted context omitted.

All you did was restate the argument that I've already rejected. And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.

I don't even understand the point you are making then (apart from me apparently arguing solely with sophisms, which is kind of a prowess). How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms? Also, please do not rephrase this last sentence as "oh so since every…

> Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".

That's exactly what I'm going to do. Point out that we can demand better even in the face of a lack of perfection.

There are two problems here with your stance.

1. The assumption that all bugs are created equal, and 2. The assumption that the truth isn't the overriding concern of science.

It's real easy to define the set of bugs that are unacceptable in science. Any bug that would render the results inaccurate is unacceptable.

The fact that some jackass web developer wrote a bug that deleted an entire database in no way obviates that responsibility of the scientists.

Re: Challenge to scientists: does your ten-year-old code still run?

#474

Post by analog31 »

j45 wrote: post #470
analog31 wrote: post #386

Earlier quoted context omitted.

As a grad student in physics, I not only wrote code, but also designed my own (computer controlled) electronics, mechanics, optics, vacuum systems, etc. I was my own machinist and millwright. Today I work in a small R&D team within a larger business, and still do a lot of those things myself when needed. There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team.…

There's no problem with not leaving programming to programmers, its about how to encourage anyone picking up programming to have healthier habits so others can participate in the creation in the future.

Indeed, and one thing that's lacking is any kind of coaching or training. Those of us doing it can't necessarily coach the next generation, because stuff has gotten ahead of us.

Re: Challenge to scientists: does your ten-year-old code still run?

#475

Post by kazagistar »

labcomputer wrote: post #263
throwaway7281 wrote: post #145

Earlier quoted context omitted.

That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In GIS, there's a saying "the map is not the terrain". It seems like HN is in a little SWE bubble, and needs to understand "the code is not the science". In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas. > how do you expect anyone with the right expertise to assess your findings I would expect other experts in the fiel…

Mathematicians are expected to publish their proofs. Not so that people can do the proof again independently, but so that other mathematicians can find and point out if they have a tangible error in their proof that tangibly invalidates the result.

Sure, some people might point out spurious bugs and "design issues" or whatever, boo hoo. But others might actually find flaws in the code that meaningfully affect science itself: true bugs.

Sure, they could do this by doing a full replication in a lab and then custom coding everything from scratch. But even then, all you have is two conflicting results, with no good way yet to determine which one is more right or why they disagree. Technically, you can use the scientific progress to eventually find bugs in the scientific process, but why waste so much time when publishing the code will allow for reviews to find bugs so much faster. Its pure benefit to science to not obscure its proofs and rigor.

Re: Challenge to scientists: does your ten-year-old code still run?

#476

Post by kazagistar »

MaxBarraclough wrote: post #454
ByteJockey wrote: post #443

Earlier quoted context omitted.

Are we talking actual undefined behavior or just behavior that's undefined by the language standard? The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

> Are we talking actual undefined behavior or just behavior that's undefined by the language standard? 'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity. > if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code. Code should be correct by construction, not correct by coincide…

I think its poor practice, but undefined behavior shouldn't instantly invalidate results. In fact, this mindset is what keeps people from publishing the code in the first place.

Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

You would drive yourself mad pointing out places in math proofs where some steps, even seemingly important ones, were skipped. But the papers are not retracted unless such a gap actually holds a flaw that invalidates the rest of thr proof.

Let thdm publish their gross, awful, and even buggy code. Sometimes the bugs don't effect the outcomes.

Re: Challenge to scientists: does your ten-year-old code still run?

#477

Post by kkylin »

a1369209993 wrote: post #432
kkylin wrote: post #407

Earlier quoted context omitted.

This is not clear at all. It depends on the "result" in question. If I wrote a paper describing a super numerical algorithm for inverting matrices, and no one is able to replicate the superior performance of my algorithm despite following the recipe in my paper, then whether they can run my code or not doesn't seem to be of the highest priority. Edit: more careful phrasing.

> whether they can run my code or not doesn't seem to be of the highest priority. On the contrary; in that case there are four possibilities: a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements. b: your algorithm works, but the description in the paper is wrong or incomplete c: your algorithm works as described, but the replicater implemented it incor…

I agree with the case analysis, but disagree with the implication that the code needs to be runnable (which is seems to be the point of the discussion at hand). In many cases having source code, even if it no longer runs, should be sufficient.

Re: Challenge to scientists: does your ten-year-old code still run?

#478

Post by MaxBarraclough »

kazagistar wrote: post #476
MaxBarraclough wrote: post #454

Earlier quoted context omitted.

> Are we talking actual undefined behavior or just behavior that's undefined by the language standard? 'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity. > if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code. Code should be correct by construction, not correct by coincide…

I think its poor practice, but undefined behavior shouldn't instantly invalidate results. In fact, this mindset is what keeps people from publishing the code in the first place. Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the re…

> undefined behavior shouldn't instantly invalidate results

Granted, it's not a guarantee that the results are wrong, but it's a serious issue with the experiment. I agree it wouldn't generally make sense to retract a publication unless it can be determined that the results are invalid. It should be possible to independently investigate this, if the source-code and input data are published, as they should be.

(It isn't universally true that reproduction of the experiment should be practical given that the source and data are published, as it may be difficult to reproduce supercomputer-powered experiments. iirc, training AlphaGo cost several million dollars of compute time, for instance.)

> this mindset is what keeps people from publishing the code in the first place

As I explained in [0], this attitude makes no sense at all. It has no place in modern science, and it's unfortunate the publication norms haven't caught up.

Scientific publication is meant to enable critical independent review of work, not to shield scientists from criticism from their peers, which is the exact opposite.

> Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

I'm not sure what to make of 'professionals', but otherwise I agree, go ahead and publish the binaries too, as much as applicable. Could be a valuable addition. (In some cases it might not be possible/practical to publish machine-code binaries, such as when working with GPUs, or Java. These platforms tend to be JIT based, and hostile to dumping and restoring exact binaries.)

I agree with your final two paragraphs.

[0] https://news.ycombinator.com/item?id=24264376

Re: Challenge to scientists: does your ten-year-old code still run?

#479

Post by kkylin »

kkylin wrote: post #477
a1369209993 wrote: post #432

Earlier quoted context omitted.

> whether they can run my code or not doesn't seem to be of the highest priority. On the contrary; in that case there are four possibilities: a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements. b: your algorithm works, but the description in the paper is wrong or incomplete c: your algorithm works as described, but the replicater implemented it incor…

I agree with the case analysis, but disagree with the implication that the code needs to be runnable (which is seems to be the point of the discussion at hand). In many cases having source code, even if it no longer runs, should be sufficient.

I complete agree with your case analysis, but disagree with the conclusion that the code needs to be runnable for it to be useful -- I thought this was the point of the discussion at hand? In most situations, having source code, even if it no longer runs, would be sufficient to conduct the analysis you describe.

I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility.

Edit: rewrote to clarify my position.

Re: Challenge to scientists: does your ten-year-old code still run?

#480

Post by ByteJockey »

MaxBarraclough wrote: post #454
ByteJockey wrote: post #443

Earlier quoted context omitted.

Are we talking actual undefined behavior or just behavior that's undefined by the language standard? The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

> Are we talking actual undefined behavior or just behavior that's undefined by the language standard? 'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity. > if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code. Code should be correct by construction, not correct by coincide…

> Code should be correct by construction, not correct by coincidence.

Glad we agree, if you're aware of how your compiler handles these things, you can construct it to be correct in this way.

It won't be portable at all (even to the next patch version of the compiler), I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

Post reply on HN