Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

121–130 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#121

Post by Balgair »

csours wrote: post #104
chrisseaton wrote: post #87

Earlier quoted context omitted.

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

The Replication Crisis is much larger in scope than just software reproducibility. But yes, it does include that too.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#122

Post by 0xdeadbeefbabe »

chrisseaton wrote: post #87
ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

> maybe these things aren't actually as important as we think they are in professional software development?

Maybe for professional professionals they are.

It is funny that Hinsen's insight is so mundane and obvious: "the code itself is merely a means to this end".

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#123

Post by noelsusman »

tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

You're correct to focus on the effect of parameter choices over code quality. It's been a little funny to watch a bunch of software engineers freak out about unit tests while ignoring everything else that has a much larger impact on the output of the model. I would bet large sums of money that this code is producing the correct output according to the model/parameter specifications.

All I can say is welcome to epidemiology. The spread of a disease is highly dependent on a host of factors that we have very little insight into. Even simple things like hospitalization rate or fatality rate can be difficult if not impossible to estimate accurately. Epidemiologists are open about this, but few people ever want to listen. Humans just aren't good at truly conceptualizing uncertainty.

The theory behind disease spread models is relatively sound, but they're highly dependent on accurate estimates of input parameters, and governments have not prioritized devoting resources toward improving those estimates. I sat in on discussions between epidemiologists and government officials about COVID models. The response to nearly every question was "we don't know, but here's our best guess". I listened to them beg officials for random testing of the population to improve their parameter estimates. That testing never happened.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#124

Post by vonmoltke »

rch wrote: post #113
flankstaek wrote: post #12

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

For this to happen there would need to be a nationally accepted PE certification for 'software engineers' on par with other engineering disciplines. It's unreasonable to expect non-experts, and experts from other fields, to perform credentialing on a case by case basis.

There is no "nationally accepted" PE license for any discipline. It's a state-level credential and reciprocity is not universal or automatic.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#125

Post by Balgair »

ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

Academics are not software people, by and large. The code is custom built for a specific graph or statistical run. Typically the people writing the software are grad students with little to no interest in code; they just want the answer. Their interest is in the thing they are studying, not code. They may not even have coded before grad school, nor know matrix algebra or calculus. Excel is a big step for them.

I have experience in this issue. A fellow grad student of mine asked for some help with his code. I said sure, but it'll cost you a 12 pack of beer. After the 11th nested 'if' statement in Matlab, I upped it to a 24 pack. We never did get it working right.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#126

Post by JamesBarney »

noobaccount wrote: post #30
JamesBarney wrote: post #10

Earlier quoted context omitted.

I don't think the government is taking drastic measures based solely on this code. It's just a way to model facts we already know. We know that viruses spread exponentially, and we know what exponential growth looks like this just lets use model different assumptions to see how they affect that model.

The model is way more granular than that. Have you read it?

Sorry I guess I wasn't clear. My point is that if this software was destroyed a year ago our response to the coronavirus would be substantially the same.

Basically that we are not relying on this model exclusively or even substantially to determine policy.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#128

Post by silveraxe93 »

lucideer wrote: post #106
silveraxe93 wrote: post #69

Earlier quoted context omitted.

Where in my comment did you see me proposing that "being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)" Go back and re-read it please. I'm saying you need to be capable of using the tools for your job. If I was working as a software engineer, I'd not be asked to perform heart surgery. A more realistic example would be a software engineer be…

Sorry but there's a pretty big gap between understanding the domain for which your software is intended, and being qualified to work as a professional in that domain. The latter is what you're asking of academics.

I guess we are in agreement then ;)

If academics are writing predictive modelling software, I ask that they are qualified to do so.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#129

Post by friday99 »

klyrs wrote: post #97
SpicyLemonZest wrote: post #38

Earlier quoted context omitted.

If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.

> it seems only fair to say that academics optimize for prestige and publication in good journals. On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time. On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles…

Authors aren't actually incentivized to find accurate results, but rather publishable results, which typically means novel. Researchers also cite articles based on their impact, not their veracity. There are plenty of instances of retracted results continuing to be cited as if they are still accurate.

There are issues with both industrial and academic research, but I do think that industrial research is more transparent in its motivations.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#130

Post by rjsw »

fsh wrote: post #110
rjsw wrote: post #74

Earlier quoted context omitted.

Not the OP, but I know of at least one University that runs "Programming for scientists" courses at low or no cost.

These courses are usually short introductions into python or Java. Most scientists simply don't have time in their curriculum for a full software engineering course.

I didn't claim they were good. My father is doing a Python one right now and I'm not very impressed. He is retired from the University but is still eligible to take the course.
Post reply on HN