Bad scientific code beats code following "best practices" (2014) • Hacker News Board

Re: Bad scientific code beats code following "best practices" (2014)

#161

Post by mike_hearn » Sat, Jan 06, 2024, 1:17 PM UTC

collyw wrote: post #157

mike_hearn wrote: post #138

Earlier quoted context omitted.

> I do not know how bad it was, but it cannot be good the code was bad. I do know because I reviewed the code and its issue tracker extensively. I then wrote an article summarizing its problems that went viral and melted the server hosting it. The Imperial College code wasn't merely "bad". It was unusable. It produced what were effectively random numbers distributed in a way that looked right to the people who wrote…

Aside from the code quality, his models have never been close to accurate on anything.

Right. That's not unique to Ferguson, epidemiology doesn't understand respiratory virus dynamics and doesn't seem particularly curious to learn anymore (I read papers from the 80s which were very different and much more curious than modern papers, not sure though it that's indicative of a trend or just small sample size).

Other models I checked didn't have the same software quality issues though. They tended to use R rather than C and be much simpler. None of them produced correct predictions either, and there were often serious issues of basic scientific validity too, but at least the code didn't contain any obvious bugs.

Re: Bad scientific code beats code following "best practices" (2014)

#162

Post by kolinko » Sat, Jan 06, 2024, 1:19 PM UTC

ohlokkru wrote: post #112

astrobe_ wrote: post #73

This is partly because, in my opinion, some "best practices" are superstitions. Some practice was best because of some issue with 80s era computing, but is now completely obsolete; problem has been solved in better ways or has completely disappeared thanks e.g. to better tooling or better, well, practices. e.g. Hungarian notation. Yet it is still passed down as a best practice and followed blindly because that's what…

[flagged]

It was the same in communist countries, even more so. If anything, we’re having less of this now that we switched to capitalism than before.

Re: Bad scientific code beats code following "best practices" (2014)

#163

Post by op00to » Sat, Jan 06, 2024, 1:19 PM UTC

jakobnissen wrote: post #141

I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author. I've found the problems that biologists cause are mostly: * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months * Writing completely unreadable code, even to themselves, making it imp…

This was my exact experience working in biomedical hpc.

Re: Bad scientific code beats code following "best practices" (2014)

#164

Post by civilized » Sat, Jan 06, 2024, 1:24 PM UTC

jakobnissen wrote: post #141

I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author. I've found the problems that biologists cause are mostly: * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months * Writing completely unreadable code, even to themselves, making it imp…

These patterns appear in many fields. I take it as a sign that the tooling in the field is underdeveloped.

This leads to a split between domain problem solvers, who are driven to solve the field's actual problems at all costs (including unreliable code that produces false results) and software engineers, who keep things tidy but are too risk-averse to attempt any real problems.

I encourage folks with interests in both software and an area of application to look at what Hadley Wickham did for tabular data analysis and think about what it would look like to do that for your field.

Re: Bad scientific code beats code following "best practices" (2014)

#165

Post by dimask » Sat, Jan 06, 2024, 1:24 PM UTC

9dev wrote: post #62

Oh look, another tiring craftsmanship debate that other disciplines long figured out! A, say, physicist writing bad code could equally well be building a pergola for his garden. He doesn’t really know woodworking but god be damned if he couldn’t calculate the forces acting on the beams, and then add some screws - how hard can it be! And probably, he’ll even get the thing up, and it doesn’t look too bad even. Now get…

Research software engineering is a hybrid field and fixed roles do not work well. It requires one having a grasp (at some level) of a lot of different stuff, eg software engineerin, the relevant scientific theories, statistics _and_, quite importantly, the culture of scientific practices in a field, in order to make something good. So it gathers a lot of people who, no matter where they started from, often have to sort of converge by learning stuff outside their own discipline.

The problem is not that "physicists write code". Scientists end up learning a lot of stuff and getting good at it (software engineers the same). The problem is that writing software is often left as a job for the occasional phd, postdoc or research assistant, ie people with temporary positions, and in general to people who see building software as a side duty at best, annoyance at worst. This results in no generational knowledge building, being hard to find mentors, less learning and reflecting on practices on how to build software, rediscovering the wheel constantly, too much effort put into . It is not that one graduates as "software engineer" or as a "scientist" and then they know the best practices and everything of their respective fields. People get to learn stuff. Software engineering practices should be in the culture of scientific software building, not simply carrying them from the software engineering world to science, but adapting them and taking it each own idiosyncrasies as a field.

Re: Bad scientific code beats code following "best practices" (2014)

#166

Post by movpasd » Sat, Jan 06, 2024, 1:27 PM UTC

jakobnissen wrote: post #141

I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author. I've found the problems that biologists cause are mostly: * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months * Writing completely unreadable code, even to themselves, making it imp…

I work in an R&D environment with a lot of people from scientific backgrounds who have picked up some programming but aren't software people at heart. I couldn't agree more with your assessment, and I say that without any disrespect to their competence. (Though, perhaps with some frustration for having to deal with bad code!)

As ever, the best work comes when you're able to have a tight collaboration between a domain expert and a maintainability-minded person. This requires humility from both: the expert must see that writing good software is valuable and not an afterthought, and the developer must appreciate that the expert knows more about what's relevant or important than them.

Re: Bad scientific code beats code following "best practices" (2014)

#167

Post by dillydogg » Sat, Jan 06, 2024, 1:28 PM UTC

sseagull wrote: post #155

jampekka wrote: post #126

Earlier quoted context omitted.

I'm an ex-software developer/engineer and current scientist. In my experience TFA makes a good point, even though it's quite strawmanish. Most scientific code is horrible from any sane software developer perspective. The quality is so bad that I think a huge proportion of published results are plain wrong due to bugs in the analysis. These apply to my much of my code as well. But a lot of "software engineering" code…

I basically agree (except I will take static typing over dynamic typing any day). I work in computational chemistry, and scientists here don’t necessarily have problems actually coding (we’ve been doing it for 70+ years). But the “other stuff” is taking more and more time. Before, you wrote Fortran, put the files on disks or whatever, and sent them around. Now, you need to know: C++ and Python (and maybe Fortran too)…

> Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so.

This is how I see it as well. I'm an immunologist, and it feels impossible for me to keep on top of my field of research and just about anything else. I don't have to produce quality software, but it seems difficult to keep your research cutting edge while maintaining software. It's hard enough keeping the research up to date!

Re: Bad scientific code beats code following "best practices" (2014)

#168

Post by 9dev » Sat, Jan 06, 2024, 1:33 PM UTC

sidlls wrote: post #143

9dev wrote: post #128

Earlier quoted context omitted.

Don’t think so, no. A physicist has other stuff to learn and spend their time on. Instead, they should partner with a carpenter to do their woodworking from a rough sketch.

If your point is that the physicist should partner with someone who is a "professional programmer" ("carpenter") to do the coding, I couldn't disagree more, speaking as a former research physicist who wrote many programs while I was in academia. A "rough sketch" is not enough for a "carpenter" to go off of for the programs a physicist using computational techniques is writing. They'd need to have a sophisticated unde…

Now we’re lost in metaphors. A carpenter should definitely be able to build a pergola from a rough paper sketch, and a software engineer should be able to build a machine learning system from an algorithm paper by a data scientist.

Re: Bad scientific code beats code following "best practices" (2014)

#169

Post by caddemon » Sat, Jan 06, 2024, 1:44 PM UTC

dimask wrote: post #165

9dev wrote: post #62

Oh look, another tiring craftsmanship debate that other disciplines long figured out! A, say, physicist writing bad code could equally well be building a pergola for his garden. He doesn’t really know woodworking but god be damned if he couldn’t calculate the forces acting on the beams, and then add some screws - how hard can it be! And probably, he’ll even get the thing up, and it doesn’t look too bad even. Now get…

Research software engineering is a hybrid field and fixed roles do not work well. It requires one having a grasp (at some level) of a lot of different stuff, eg software engineerin, the relevant scientific theories, statistics _and_, quite importantly, the culture of scientific practices in a field, in order to make something good. So it gathers a lot of people who, no matter where they started from, often have to so…

I agree, a lot of unmaintained code is due to turnover of staff and few considering the software as an important output in its own right. But I don't think it would be that hard to find grad students/postdocs who would care about code, if that were something the field properly incentivized. Not only does software contribution not check the right boxes for career progression, it is also often looked down on.

I find this especially laughable in biology... I've seen some (faculty) PhD committee members object to the student having a thesis chapter related to software contributions, because this is not "intellectual". But they are perfectly fine with one of the chapters being a wet lab paper where the student was a 3rd author who contributed purely through helping run experiments designed by the 1st author (e.g. handling mice, pipetting shit). There are PIs that simultaneously hold these two views, which to me signals a real misunderstanding of the challenges in and importance of writing decent software.

Re: Bad scientific code beats code following "best practices" (2014)

#170

Post by sidlls » Sat, Jan 06, 2024, 1:45 PM UTC

9dev wrote: post #168

sidlls wrote: post #143

Earlier quoted context omitted.

If your point is that the physicist should partner with someone who is a "professional programmer" ("carpenter") to do the coding, I couldn't disagree more, speaking as a former research physicist who wrote many programs while I was in academia. A "rough sketch" is not enough for a "carpenter" to go off of for the programs a physicist using computational techniques is writing. They'd need to have a sophisticated unde…

Now we’re lost in metaphors. A carpenter should definitely be able to build a pergola from a rough paper sketch, and a software engineer should be able to build a machine learning system from an algorithm paper by a data scientist.

That really depends on what you mean by "machine learning system." For a business, implementing some researched algorithm and essentially using a template? Sure, it's possible. The research that goes into producing the algorithm in the first place? Probably not.

There may be some classes of scientific programming which have simple enough models for a software engineer to implement. Scientific programming that relies on deep understanding of the domain and mathematical models employed to study it don't fit in that category.