Bad scientific code beats code following "best practices" (2014) • Hacker News Board

Re: Bad scientific code beats code following "best practices" (2014)

#141

Post by jakobnissen » Sat, Jan 06, 2024, 12:31 PM UTC

I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author.

I've found the problems that biologists cause are mostly:

* Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months

* Writing completely unreadable code, even to themselves, making it impossible to maintain. This means they always restart from zero, and projects grow into folders of a hundred individual scripts with no order, depending on files that no longer exists

* Foregoing any kind of testing or quality control, making real and nasty bugs rampant.

IMO the main issue with the software people in our field (of which I am one, even though I'm formally trained in biology) is that they are less interested in biology than in programming, so they are bad at choosing which scientific problems to solve. They are also less productive when coding than the scientists because they care too much about the quality of their work and not enough about getting shit done.

Re: Bad scientific code beats code following "best practices" (2014)

#142

Post by j-bos » Sat, Jan 06, 2024, 12:31 PM UTC

nohuck13 wrote: post #114

9dev wrote: post #62

Oh look, another tiring craftsmanship debate that other disciplines long figured out! A, say, physicist writing bad code could equally well be building a pergola for his garden. He doesn’t really know woodworking but god be damned if he couldn’t calculate the forces acting on the beams, and then add some screws - how hard can it be! And probably, he’ll even get the thing up, and it doesn’t look too bad even. Now get…

"It’s the same story, really: It is a software engineer’s job to build quality software. A scientists job is to solve problems." That's not the distinction. Good software engineers solve problems. That's what the paycheck is for. The distinction is whether code has to be maintained. It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper'…

> Who cares if the metaphorical wood rots next winter, the paper's been published.

Isn't this why the replication crisis was able to be kept hidden for so long?

Re: Bad scientific code beats code following "best practices" (2014)

#143

Post by sidlls » Sat, Jan 06, 2024, 12:34 PM UTC

9dev wrote: post #128

madsbuch wrote: post #101

Earlier quoted context omitted.

If the bad wood working would jepordize the results of his, professional salary earning, work then he should probably consider learning wood working, no?

Don’t think so, no. A physicist has other stuff to learn and spend their time on. Instead, they should partner with a carpenter to do their woodworking from a rough sketch.

If your point is that the physicist should partner with someone who is a "professional programmer" ("carpenter") to do the coding, I couldn't disagree more, speaking as a former research physicist who wrote many programs while I was in academia. A "rough sketch" is not enough for a "carpenter" to go off of for the programs a physicist using computational techniques is writing. They'd need to have a sophisticated understanding of the physics and the mathematical model involved: which almost no "carpenters" have.

Re: Bad scientific code beats code following "best practices" (2014)

#144

Post by sidlls » Sat, Jan 06, 2024, 12:37 PM UTC

logicchains wrote: post #35

jurschreuder wrote: post #12

This is so true I don't think I ever read something so true. It's not even scientists vs software developers. It's people who are really into software development and clean code. They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files. Ever since I make sure to repeat my mantra every week to developers: How maintainable code is is measured i…

>They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files. Anyone who in 2023 still thinks inheritance is a good idea for anything other than a few very specialised use-cases is not somebody who seriously cares about the craft of software development, not somebody who's put any effort to study programming theory and move beyond destructive 19…

Rust and Go both provide features to implement code in an OO-ish way: traits and interfaces. I just code in Rust as a hobby but I code in go professionally. The go codebases I work on at work are bloated messes of abstractions and duck typing, often meant to enforce some absurd standard of unit-testing. They can easily be as bad as any "enterprise Java" stereotype you wish to invoke.

Re: Bad scientific code beats code following "best practices" (2014)

#145

Post by LeonardoTolstoy » Sat, Jan 06, 2024, 12:38 PM UTC

9dev wrote: post #131

mihaaly wrote: post #100

Earlier quoted context omitted.

It is very true people should stick to the domain they know because otherwise they will have higher than average chance to f up. But that 'clear boundary' thing is a naive bollocks! No such thing! Both domain experts need to understand things beyond this imaginary and when precisely drawn then highly arbitrary boundary that is more like a gradient than a line normally (also not something relevant in a final good prod…

Perhaps I phrased this badly. My point wasn’t a clear boundary between professions, because you are right, that is difficult to impossible to draw. However, there’s a clear boundary between the goals of the code written by scientists vs. software engineers. Where a scientist aims to prove something, a software engineer builds code to produce business value. Both are trained very differently towards these goals.

This probably highlights why so many software engineers in scientific R&D are "bad" in the strictest sense: they often don't understand that business value doesn't come from production-perfect solutions, but instead 100% of the business value comes from doing stuff fast, good enough, and understandable (so that researchers, not software engineers, can read it in a paper and iterate off of it).

Re: Bad scientific code beats code following "best practices" (2014)

#146

Post by quickthrower2 » Sat, Jan 06, 2024, 12:41 PM UTC

c048 wrote: post #113

quickthrower2 wrote: post #104

I agree. Been doing devops recently but back at some coding at work and I wrote the function as simple as I could, adding complexity but only as needed. So it started as a MVC controller function that was as long as your arm. Then it got split up into separate functions, and eventually I moved those functions to another file. I had some genuine need for async, so added some stuff to deal with that, timeouts, error ha…

Congrats, you used design patterns.

Still have the self made pat on my back

Re: Bad scientific code beats code following "best practices" (2014)

#147

Post by ben_w » Sat, Jan 06, 2024, 12:43 PM UTC

mike_hearn wrote: post #138

graemep wrote: post #90

Earlier quoted context omitted.

If you cannot model using decent code is it worth writing models at all? What if bugs mean the model is simply wrong? It has consequences too. There has been a lot of argument about how much impact the poor code quality of the Imperial college covid epidemiology model (which was the basis of British government policy during the pandemic) had on its accuracy. I do not know how bad it was, but it cannot be good the cod…

> I do not know how bad it was, but it cannot be good the code was bad. I do know because I reviewed the code and its issue tracker extensively. I then wrote an article summarizing its problems that went viral and melted the server hosting it. The Imperial College code wasn't merely "bad". It was unusable. It produced what were effectively random numbers distributed in a way that looked right to the people who wrote…

Eek! That makes the worst code I've ever seen, seem good in comparison.

Re: Bad scientific code beats code following "best practices" (2014)

#148

Post by hytfyiv3j » Sat, Jan 06, 2024, 12:47 PM UTC

jltsiren wrote: post #124

zelphirkalt wrote: post #118

Earlier quoted context omitted.

> It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper's been published. Sounds a little like cargo-culting than proper reproducible research. But this is a pest in academia definitely. Many papers do not provide all required data, all required model parameters etc. to get to the exact same result. Admittedly, they might nowadays need…

Cargo-culting is about focusing on the process without fully understanding its purpose. Such as bureaucratic requirements for providing all data, software, parameters etc so that someone can reproduce exactly the same numbers with minimal effort. Proper reproducible research is not like that. It's about providing sufficient details that other people in the field can extrapolate the rest. That they can use similar met…

In software I have found getting the exact details and parameters very useful even if I don't intend to use them. Because when I try to do the same thing my own way and fail, then I can reference the original ones and gradually make my own version more and more similar to that, and see when it starts working. Or make their version more and more similar to mine and see when it breaks. This allows quickly easily identifying the critical difference.

Re: Bad scientific code beats code following "best practices" (2014)

#149

Post by Beldin » Sat, Jan 06, 2024, 12:47 PM UTC

j-bos wrote: post #142

nohuck13 wrote: post #114

Earlier quoted context omitted.

"It’s the same story, really: It is a software engineer’s job to build quality software. A scientists job is to solve problems." That's not the distinction. Good software engineers solve problems. That's what the paycheck is for. The distinction is whether code has to be maintained. It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper'…

> Who cares if the metaphorical wood rots next winter, the paper's been published. Isn't this why the replication crisis was able to be kept hidden for so long?

Not as far as I know. Much more problematic was the fact that replication is very hard to publish. That's because either you more or less confirm previous findings and therefore contribute little to the scientific record (or so reviewers seem to think), or your findings counter the original results and now it's on you to explain the discrepancy. Even if you satisfy the reviewers that you're right, they may not consider your result of sufficient caliber to accept for publication in this particular venue [1].

[1] I've seen this happen to a paper that proved that a theoretical framework for constructing proofs about RFID protocols was neither sound nor complete.

Re: Bad scientific code beats code following "best practices" (2014)

#150

Post by mike_hearn » Sat, Jan 06, 2024, 12:59 PM UTC

niemandhier wrote: post #133

Scientific programming and industry programming are distinct disciplines. For scientific code certain things are just not important, hence you do not deal with them: - Observability: You just care for the result of the run, not for the state of the running system - Security: Your code is running in isolation, used by yourself The result looks horrible to a normal programmer, even if it’s well maintainable, but it is…

No it isn't and this idea needs to die.

Sure, you can ignore security if all you're doing is processing local text files, granted. But things looking horrible to programmers isn't just about security bugs, it's about the whole span of correctness bugs. And scientists need to write code that is both correct and maintainable. The frequency with which they don't is partly why results so often can't be replicated, making the money spent on academia wasted.

The idea that science code doesn't need to be maintainable or that they have some magic way to do it that looks wrong, isn't right either. It's not uncommon to find model "codes" that scientists have been hacking on for decades. The results have become completely untrustworthy many years earlier, but they deny/obfuscate/ignore, attack or even sue people who point out concrete problems. Sadly, often with the acquiescence of the media who are supposed to be ferreting out coverups.

Scientists need to collectively get a grip on this situation. They will happily attack anyone outside their institutions as being non-expert conspiracy theorists, but when it comes to software they suddenly know everything and don't need to hire professionals. Paper-invalidating bugs are constantly being covered up and the only reason the problem hasn't reached criticality yet is that many people don't want to hear about it. But the unreliability of academic output is now becoming a political problem and a divisive culture war issue, when it really shouldn't be. A good first step to solving the replication crisis would be for scientists to stop pretending it's OK to quickly knock together a program themselves instead of assigning a ticket to a trained full time SWE. Yes it would cost more (a lot more), and that's OK. Generate fewer papers but get them right!