Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

141–150 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#141

Post by chasd00 »

veddox wrote: post #49
chasd00 wrote: post #25

they need to defer to the experts of software development. I, as a software engineer, wouldn't try to design an epidemic model, i would defer to experts. Epidemic experts shouldn't be writing the implementation of their model, they should defer to the experts. Also, "It’s you, the software engineering community, that is responsible for tools like C++ that look as if they were designed for shooting yourself in the foo…

> Epidemic experts shouldn't be writing the implementation of their model, they should defer to the experts. Believe me, a lot of scientists would love to. But as pointed out in the OP: it is next to impossible to get funding for a paid software developer position in a research team. It's a problem the scientific community is increasingly aware of, but changing funding guidelines takes a long time. (Source: computati…

That's a fair point. Also, passing judgement on source code is a very easy thing to do. May he who writes/maintains perfect code cast the first stone.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#142

Post by csours »

chrisseaton wrote: post #135
csours wrote: post #104

Earlier quoted context omitted.

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.

> Are you sure that writing tests makes science better? Or are you just assuming that?

It is perfectly acceptable to write code without tests. Proof-of-Concept or Minimum Viable Product are a great place to write code without tests.

It is less acceptable if other people will run or use that code. It is even less acceptable if anyone (including oneself) ever updates or extends the code.

---

You could take this analogy to scientific instruments. Imagine you make a novel particle detector. You get a scientific result with your detector.

A colleague uses your detector, but they don't clean it properly before use, and they use a power supply with lower voltage. They don't detect any particles! Was your science bad? Would the "science be better" if there were clearer instructions and pre-requisites?

Now imagine another scientist makes a copy of your detector from the description in your paper. They get some stuff wrong because your description was ambiguous. Was the science bad?

---

By the way, all of these things are real problems with scientific investigations, and not just in the software realm.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#143

Post by tentboy »

tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

This reminds me of the Drake equation. A sound formula for the probability extra terrestrial life..but half the parameters are wild guesses that can differentiate in orders of magnitude.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#144

Post by csours »

Balgair wrote: post #121
csours wrote: post #104

Earlier quoted context omitted.

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

The Replication Crisis is much larger in scope than just software reproducibility. But yes, it does include that too.

Certainly. I think the "it works on my machine" attitude is reminiscent of the problems associated with the replication crisis.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#145

Post by lukeex »

Bridges are designed, built, tested, and verified as fit for purpose. Planes are designed, built, tested, and verified as fit for purpose. Why should software models that inform policy be excluded from those requirements? What is the expected cost of failure - it seems that critical risk controls are missing from academia - in the case that they are wrong.

The case in hand - is problematic because it highlights that lack of controls that are present and that should probably be present when building models and simulations to inform policies.

Quality control is important in nearly every other industry - the lack of quality controls in academia appears to be the root cause to me. Rather than particular language/build choices.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#146

Post by claytonjy »

disgruntledphd2 wrote: post #131
veddox wrote: post #116

Earlier quoted context omitted.

It's easier to dip your toes into, but it is a terrible language to learn software development with. It's such a hodge-podge of ill-thought-out and ill-fitting components with random names and no overall sense of structure that you're never going to learn core CS concepts from it, let alone good programming practice.

That's not really its purpose though, right? R is "an environment for statistical computing and graphics". Note that it doesn't talk about software engineering at all. I agree with you that R has lots of rough edges, but please remember that it's a 90's era clone of a 70's era language (S) and a lot of those rough edges and corners are legacies from that time. I completely agree that the naming conventions (i.e. the…

I really appreciate this sentiment as a data person who learned R first, now works in python, and helps non-engineer scientists write R programs for research. I've had great luck teaching scientists to think in a function-driven way, where functions are pure-as-possible and inputs are never mutated (which doesn't need to be discussed as R makes the opposite quite unnatural). I can't imagine teaching these same folks about python, custom classes, and why you need pd.DataFrame.copy() all over the place.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#147

Post by downerending »

chasd00 wrote: post #141
veddox wrote: post #49

Earlier quoted context omitted.

> Epidemic experts shouldn't be writing the implementation of their model, they should defer to the experts. Believe me, a lot of scientists would love to. But as pointed out in the OP: it is next to impossible to get funding for a paid software developer position in a research team. It's a problem the scientific community is increasingly aware of, but changing funding guidelines takes a long time. (Source: computati…

That's a fair point. Also, passing judgement on source code is a very easy thing to do. May he who writes/maintains perfect code cast the first stone.

Most scientific software isn't of subtly poor quality. Most scientific software is a stinking, flaming dumpster fire that looks like it was written by packs of drunk kindergartners.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#148

Post by mcv »

lucideer wrote: post #102
mcv wrote: post #98

Earlier quoted context omitted.

> "The private sector's typical response to the problems in the article would be to hire qualified software engineers to assist researchers. Funding dictates this is impossible, so they make do." But that is a bizarre conclusion. Is it really cheaper to let scientists focus on something they know nothing about, than to hire an expert to do it? Do you cut costs by letting scientists mop the floor, clean the toilet and…

> Is it really cheaper to let scientists focus on something they know nothing about, than to hire an expert to do it? Do you cut costs by letting scientists mop the floor, clean the toilet and guard the entrance? Believe it or not, mean software engineer salaries are marginally higher than those of janitors and door security.

But easily worth it if it saves the scientist a lot of time and makes the implementation of their model more accurate and reliable.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#149

Post by lucideer »

silveraxe93 wrote: post #128
lucideer wrote: post #106

Earlier quoted context omitted.

Sorry but there's a pretty big gap between understanding the domain for which your software is intended, and being qualified to work as a professional in that domain. The latter is what you're asking of academics.

I guess we are in agreement then ;) If academics are writing predictive modelling software, I ask that they are qualified to do so.

> Where in my comment did you see me proposing that "being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)"

> I ask that they are qualified to do so.

You need to make up your mind on this one. Are you asking this or aren't you?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#150

Post by tripletao »

tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

I partially agree with the comment above, but I also think it misunderstands how numerical models are often used. At least where I've built them (not epidemiology), the goal wasn't necessarily to gather the most accurate set of inputs and produce the most accurate prediction of the output. The goal was often to help a highly skilled operator explore the parameter space and guide their intuition on the problem, to help that person and simulation together reach some decision.

So code quality mattered less then usual. If there's a significant bug, then the operator will probably notice, and if there's an insignificant bug then no one cares. The large number of input parameters also doesn't matter. The operators are fully aware that they could artificially manipulate the output to wherever they wanted, but to do so would be cheating only themselves.

It feels to me like Ferguson's model was built with similar intent, and probably served that purpose well. The problem came only when the media portrayed the model as a source of authority apart from the people operating it, perhaps to create a feeling of objectivity behind the decisions driven from that. That created an expectation of rigor that either didn't exist (in the software engineering), or fundamentally can't exist given our current knowledge of the science (in the input assumptions).

Post reply on HN