Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bad scientific code beats code following "best practices" (2014)

yosefk.com

Post reply on HN
Print view View original

271–280 of 333 posts

Re: Bad scientific code beats code following "best practices" (2014)

#271

Post by mike_hearn »

IlliOnato wrote: post #247
mike_hearn wrote: post #138

Earlier quoted context omitted.

> I do not know how bad it was, but it cannot be good the code was bad. I do know because I reviewed the code and its issue tracker extensively. I then wrote an article summarizing its problems that went viral and melted the server hosting it. The Imperial College code wasn't merely "bad". It was unusable. It produced what were effectively random numbers distributed in a way that looked right to the people who wrote…

Wow. It's pretty unbelievable. It there a place where I can read the whole article? The one I found the funniest/crziest is "Bug reports were blown off by saying that they didn't matter because the "scientists" just ran their simulation lots of times and took the average", because this is exactly how some scientists I know think. This thinking is not limited to software. My father was by trade involved in building ex…

Yeah confusion between simulation and reality can be observed all over the place. Multiple runs can be needed if you're doing measurements of the natural world, but for a simulation that doesn't make sense (you can do Monte Carlo style stuff, but that's still replicable).

You could see the lines being blurred in other ways. Outputs of simulations would be referred to as "findings", for example, or referenced in ways that implied empirical observation without it being clear where they came from unless you carefully checked citations.

Here are some of the articles I wrote about what happened (under a pseudonym)

https://dailysceptic.org/2020/05/06/code-review-of-fergusons...

https://dailysceptic.org/2020/05/09/second-analysis-of-fergu...

https://dailysceptic.org/2020/06/11/how-replicable-is-the-im...

After that people started sending me non-Imperial models to look at, which had some similar problems:

https://dailysceptic.org/2020/08/08/schools-paper/

I don't write for that website anymore, by the way. Back then it was called Lockdown Sceptics and was basically the only forum that would publish any criticism of COVID science. Nowadays it's evolved to be a more general news site.

Re: Bad scientific code beats code following "best practices" (2014)

#272

Post by mglz »

I just handed in my PhD in computer science. Our department teaches "best practices" but adherence to them is hardly possible in research:

1) Requirements change constantly, since... it's research. We don't know where exactly we're going and what problems we encounter.

2) Buying faster hardware is usually an option.

3) Time spent on documentation, optimization or anything else that does not directly lead to results is directly detrimental to your progress. The published paper counts, nothing else. If a reviewer ask about reproducibility, just add a git repository link.

4) Most PhD students never worked in industry, and directly come from the Master's to the PhD. Hence there is no place where they'd encounter the need to create scalable systems.

I guess Nr. 3 is has the worst impact. I would love to improve my project w.r.t. stability and reusability, but I would shoot myself into the foot: It's no publishable, I can't mention it a lot in my thesis, and the professorship doesn't check.

Re: Bad scientific code beats code following "best practices" (2014)

#273

Post by noobermin »

civilized wrote: post #164
jakobnissen wrote: post #141

I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author. I've found the problems that biologists cause are mostly: * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months * Writing completely unreadable code, even to themselves, making it imp…

These patterns appear in many fields. I take it as a sign that the tooling in the field is underdeveloped. This leads to a split between domain problem solvers, who are driven to solve the field's actual problems at all costs (including unreliable code that produces false results) and software engineers, who keep things tidy but are too risk-averse to attempt any real problems. I encourage folks with interests in bot…

May be biology (or really, may be not) but honestly it's just the nature of the beast. Literally fortran is the oldest language, it's just the attitude and spirit is different than that of software development.

Re: Bad scientific code beats code following "best practices" (2014)

#274

Post by mike_hearn »

collyw wrote: post #156
graemep wrote: post #90

Earlier quoted context omitted.

If you cannot model using decent code is it worth writing models at all? What if bugs mean the model is simply wrong? It has consequences too. There has been a lot of argument about how much impact the poor code quality of the Imperial college covid epidemiology model (which was the basis of British government policy during the pandemic) had on its accuracy. I do not know how bad it was, but it cannot be good the cod…

Given Ferguson's track record of being out by orders of magnitude on absolutely everything beforehand, it almost seems like he was chosen to give an over the top estimate.

It would be comforting to believe that because it'd mean there were other epidemiologists who were right but ignored. Go read the works by his counterparts though, and they're all out by similar orders of magnitude.

Re: Bad scientific code beats code following "best practices" (2014)

#275

Post by mike_hearn »

dash2 wrote: post #257
graemep wrote: post #90

Earlier quoted context omitted.

If you cannot model using decent code is it worth writing models at all? What if bugs mean the model is simply wrong? It has consequences too. There has been a lot of argument about how much impact the poor code quality of the Imperial college covid epidemiology model (which was the basis of British government policy during the pandemic) had on its accuracy. I do not know how bad it was, but it cannot be good the cod…

How are people supposed to do science without running statistical models?

This is asked in good faith of course, but that question really gets to the heart of what's been corrupting science.

Statistical techniques can be very useful (ChatGPT!) but they aren't by themselves science. Science is about building a theoretical understanding of the natural world, where that theory can be expressed in precise language and used to produce new and novel hypotheses.

A big part of why so much science doesn't replicate is that parts of academia have lost sight of that. Downloading government datasets and regressing them against each other isn't actually science even though it's an easy way to get published papers, because it doesn't yield a theoretical understanding of the domain. It often doesn't even let you show causality, let alone the mechanisms behind that causality.

If you look at epidemiology, part of why it's lost its way is that it's become dominated by what the media calls "mathematicians"; on HN we'd call them data scientists. Their papers are essentially devoid of theorizing beyond trivial everyday understandings of disease (people get sick and infect each other). Thousands of papers propose new models which are just a simple equation overfitted to a tiny dataset, often just a single city or country. The model's predictions never work but this doesn't invalidate any hypothesis because there weren't any to begin with.

How do you even make progress in a field if there's nothing to be refuted or refined? You can fit curves forever and get nowhere.

In psychology this problem has at least been recognized. "A problem in theory" discusses it:

https://www.nature.com/articles/s41562-018-0522-1

Re: Bad scientific code beats code following "best practices" (2014)

#276

Post by thethimble »

jayd16 wrote: post #265
astrobe_ wrote: post #73

This is partly because, in my opinion, some "best practices" are superstitions. Some practice was best because of some issue with 80s era computing, but is now completely obsolete; problem has been solved in better ways or has completely disappeared thanks e.g. to better tooling or better, well, practices. e.g. Hungarian notation. Yet it is still passed down as a best practice and followed blindly because that's what…

Such as what? I don't really know of any such superstitions that are based on nothing. I see a lot of opinion/taste presented as something more, but I really can't think of superstitions.

OOP madness? XML? Web scale databases?

Perhaps not superstition but certainly fundamentalist/hype-based thinking.

Re: Bad scientific code beats code following "best practices" (2014)

#277

Post by noobermin »

laserbeam wrote: post #41

I agree with the feelings of the author, most software is overengineered (including most of my software). That being said, most scientific code I've encountered doesn't compile/run. It ran once at some point, it produced results, it worked for the authors and published a paper. The goal for that code was satisfied and than that code somehow rusted out (doesn't work with other compilers, hadn't properly documented how…

Yeah, well gnome 2 also doesn't compile or run on my machine. It ran once at some point, but one is considered a "worse" class of software.

Re: Bad scientific code beats code following "best practices" (2014)

#278

Post by Xeamek »

Nevermark wrote: post #234
Xeamek wrote: post #92

Earlier quoted context omitted.

>Writing clean and maintainable code should be the best practice and writing obfuscated code for performance and security should be reserved for "few specialised cases" Except, we can (fairly objectively) reason about performance and security, while 'clean code' and 'maintainability' are arbitrary, with vague guidelines at best. Throwing out those first characteristics in name of the latter ones is just irrational. (…

> Except, we can (fairly objectively) reason about performance and security, while 'clean code' and 'maintainability' are arbitrary, with vague guidelines at best. Ok, how does one best reason about performance and security with messy unmaintainable code? You barely need to try even shallow reasoning about a code base at all, before it's clean vs. messy, and maintainable vs. unmaintainable status will feel very objec…

>Ok, how does one best reason about performance and security with messy unmaintainable code?

The same way one does it with 'clean' code: Using profiling tools. Security is a bit less straight forward, but still.

>will feel very objective.

Keyword: Feel. And while most people could probably agree on terrible code being terrible, the 'less terrible' the code is, the more this argument becomes a feeling. And then we hit up point where it's no longer possible to discuss things using objective arguments - how will a senior java developer, who is used to heavy OOP style coding reason with a senior C developer for whom such OOP heavy style is the oposite of 'clean'?

Ofcourse, the example is (too) simplistic, but even in this thread You have people arguing about "big functions" vs splitting things up. And unlike the performance, which You can alaways just point to the raw numbers, here You can't rely on any sorts of 'objectivity'

Re: Bad scientific code beats code following "best practices" (2014)

#279

Post by cauch »

lynndotpy wrote: post #230

Scientist and programmer here, and my experiences are the opposite. I value keeping things "boringly simple", but I desperately wish there was any kind of engineering discipline. First is the reproducibility issue. I think I've spent about as much time simply _trying_ to get the dependencies of research code to run as I have done writing or doing research in my PhD. The simple thing is to write a requirements.txt fil…

My problem with this discussion is that a lot of people just say "I'm a scientist (or I'm working with scientists) and I'm observing X so I can say 'scientists blahblahblah'".

Different scientific research fields are using widely different computer software environment, and have their own habits and traditions. The way a biologist uses programming has no reason to be similar to the way an astrophysicist does: they have not at all experienced the same software environment. It may even be useless to talk about "scientist" in the same field as two different labs working in the same field may have very different approaches (but it's more difficult if there are shared framework).

So, I'm not at all surprised that you observe opposite experience. The same way I'm not surprised to see someone saying they had the opposite experience if someone says "European people are using a lot of 'g' and 'k' in their words" just because they observed what happened in Germany.

Re: Bad scientific code beats code following "best practices" (2014)

#280

Post by noobermin »

logicchains wrote: post #35
jurschreuder wrote: post #12

This is so true I don't think I ever read something so true. It's not even scientists vs software developers. It's people who are really into software development and clean code. They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files. Ever since I make sure to repeat my mantra every week to developers: How maintainable code is is measured i…

>They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files. Anyone who in 2023 still thinks inheritance is a good idea for anything other than a few very specialised use-cases is not somebody who seriously cares about the craft of software development, not somebody who's put any effort to study programming theory and move beyond destructive 19…

I hate to say this but this attitude of "it's 2023, inheritance is so 2008" is telling. How do you know the attitudes regarding best practices today are not going to be as bad as 20 layers of inheritance?
Post reply on HN