Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

41–50 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#41

Post by silveraxe93 »

rjtobin wrote: post #19

I'm somewhat skeptical. Firstly, I don't think the language is the problem with scientific code. You can write messy code in any language. So the warning then has to be about writing software in general. In that case, I think a warning like "don't try to write software unless you have years of training" is a bit much. Many people with no training learn to write nice code. Many projects made by amateurs might have ugl…

This model didn't just influence decisions in healthcare. It single-handedly changed the UK government's strategy over this pandemic.

From what I understand the UK was planning on beating COVID by creating herd immunity, similarly to Sweden. Then this model came out and everyone started yelling that Boris wanted to kill your grandma.

The problem is that it's impossible to have an intelligent discussion over this. This pandemic became a partisan issue. We're not discussing whether one of the most impactful decisions made by a government this generation should be based over absolute trash code. You're either uncritical of the lockdown or "anti-science".

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#42

Post by wbhart »

tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong.

Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll give up on data science entirely.

It's already quite bad that people think they can choose their reality by finding numbers that agree with them and ignoring the ones that don't.

I do understand the point you are making, which is like the epicycles argument. But in global warming and epidemics alike, more parameters are actually needed to model reality.

I do agree that those parameters should be based on actual data, not guesses though. But what value of R would you pick? Is that actually well-constrained?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#43

Post by lucideer »

silveraxe93 wrote: post #14

I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…

This is such a ridiculous comment I don't know where to start.

Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)?

If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, would your argument be the same?

The private sector's typical response to the problems in the article would be to hire qualified software engineers to assist researchers. Funding dictates this is impossible, so they make do.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#44

Post by raverbashing »

What we're missing is an "open letter" of scientists criticizing software engineers evaluation of this model.

Because it seems no one of these critics have ever coded anything to do with stochastic forecasting or simulations.

"The code produces different results between runs" well, I would be very worried if it didn't, unless your RNG has a fixed seed, and all randomness in your model derives from it (so, if you use multiple threads, there goes your repeatability - which is fine).

Also, you want multiple runs to be different, so you can establish best/worse case scenarios (also tuning some of the parameters).

Does the model pass basic sniff tests? (for example, no more infected people than the population of a place? does it follows known pandemic curves? Can the parameters be reasonably tuned to fit its behaviour in existing places?) And yes the model was checked against a different model from what I remember.

In essence, if you take Population x (percentage for herd immunity) x IFR you can get a very good estimate of the worse case scenario for deaths. Then you can see if your model goes to that value given the known parameters.

Yes, the code is ugly. Yes you probably made worse code at a point in your career.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#45

Post by andi999 »

htk wrote: post #3

I get what he’s saying, but the “non-experts” using code to represent their models should also let outside help come when the need arises. When your model predicts an apocalyptical scenario and government is taking drastic measures based on it, it’s a good time to expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look.

There is probably no funding for outside help... So they get a project fund a PhD student (or post doc) with that, this guy has to produce new results (fast) and more results, then these guys leave, next guys come..

Generally for a simulation group you would need an experienced programmer maintaining the code base, (helping) the other guys cleaning up there ideas before handover, and this as a permanent position. Experienced c++ also means not cheap. Go to you University admins and mention both shocking words (not cheap and permanent without the magic word 'professor') in one sentence, maybe he will recover from his shock in a few days.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#46

Post by enitihas »

So government can spend billions and trillions on implementing covid19 related policies, but can't spend any money on getting good data for the same? The fact that government policy is driven by some unpublished, undocumented, non repeatable code, affecting the lives of an entire nation, should be treated as a national defence issue. Anyone saying the person writing this was not a software engineer is giving a poor excuse. Government shouldn't appoint random people to do random jobs and in case of poor results say they weren't a trained XYZ. The people deserve a better process.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#47

Post by toolz »

rscho wrote: post #35
klenwell wrote: post #22

Earlier quoted context omitted.

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…

Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?

Everyone working for a living optimizes their work towards making more money. That doesn't change for people funded with public money. It's pretty widely accepted that the people in charge of public funding (politicians) sometimes act outside of the publics interest for self-gain.

I make no claim as to which achieves better results for the public because it's such a complicated problem, but I think it's rather naive to just assume publicly funded incentives are more aligned with social health than private incentives.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#48

Post by Nursie »

klenwell wrote: post #22
flankstaek wrote: post #12

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…

Yeah that's a weird set of conclusions there. I would think the papers based on it want another look though.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#49

Post by veddox »

chasd00 wrote: post #25

they need to defer to the experts of software development. I, as a software engineer, wouldn't try to design an epidemic model, i would defer to experts. Epidemic experts shouldn't be writing the implementation of their model, they should defer to the experts. Also, "It’s you, the software engineering community, that is responsible for tools like C++ that look as if they were designed for shooting yourself in the foo…

> Epidemic experts shouldn't be writing the implementation of their model, they should defer to the experts.

Believe me, a lot of scientists would love to. But as pointed out in the OP: it is next to impossible to get funding for a paid software developer position in a research team. It's a problem the scientific community is increasingly aware of, but changing funding guidelines takes a long time. (Source: computational biologist)

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#50

Post by SpicyLemonZest »

wrsh07 wrote: post #24
bJGVygG7MQVF8c wrote: post #7

this is quite off the mark. the author sets the bar too low for himself by criticizing the most easily (and to be fair legitimately) dismissed criticisms of the Imperial College model. here's a better laid-out critique that the OP doesn't speak to: The Imperial College modelers released the source code a couple of days ago to the model that shut down the world economy. It's not the original model code but was rather…

It's worth noting that in a pandemic situation, waiting before taking action is "making a decision" Shutting down the economy is costly, yes, but allowing a virus to continue its exponential growth with all of the unknowns that entails is much more costly in expectation (we can only speak in expectation given the unknowns) Whether or not this model should have been the basis for such a decision I don't know, but it's…

The government wasn't waiting before taking action. The Imperial report was impactful because it said that the government's actions were insufficient, that if they continued on with only moderate social distancing people would die in the streets for lack of hospital beds.
Post reply on HN