Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Imperial College London have released their Covid-19 epidemic simulation

github.com

Post reply on HN
Print view View original

71–80 of 82 posts

Re: Imperial College London have released their Covid-19 epidemic simulation

#71

Post by thu2111 »

spatulon wrote: post #45
abainbridge wrote: post #7

This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer…

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfec…

Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.

Re: Imperial College London have released their Covid-19 epidemic simulation

#72

Post by thu2111 »

creato wrote: post #62
jfnixon wrote: post #53

Earlier quoted context omitted.

If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.

It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read b…

The non-determinism that page talks about is coming from bugs like memory corruptions, floating point inaccuracies, initialisation order bugs. It's not an intentional part of the model. Averaging corrupted data is meaningless, it doesn't magically fix the corruption.

I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?

Re: Imperial College London have released their Covid-19 epidemic simulation

#73

Post by thu2111 »

te_chris wrote: post #16
lbeltrame wrote: post #10

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212

That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest. Carmack is OK with it, he's put his name to reviewing it. That's not to excuse…

It's too late. The early buggy code was already used to drive decisions, and they refused to release it for a month and a half whilst they tried to fix the most embarrassing errors.

Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.

Re: Imperial College London have released their Covid-19 epidemic simulation

#74

Post by mping »

thu2111 wrote: post #71
spatulon wrote: post #45

Earlier quoted context omitted.

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfec…

Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.

Academia is geared towards paper publishing, so why would they bother with code quality? Even if in the long run it would benefit them, its at best a nice to have. Most academia doesn't share code or data anyway.

It's like saying that programmers are professional English writers, so comments and such should be written in perfect English.

IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.

Re: Imperial College London have released their Covid-19 epidemic simulation

#75

Post by thu2111 »

mping wrote: post #74
thu2111 wrote: post #71

Earlier quoted context omitted.

Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.

Academia is geared towards paper publishing, so why would they bother with code quality? Even if in the long run it would benefit them, its at best a nice to have. Most academia doesn't share code or data anyway. It's like saying that programmers are professional English writers, so comments and such should be written in perfect English. IMHO the only valid criticism is if the the simulation modelling is sound and im…

At every place I've worked correct spelling and grammar in comments is in fact expected of developers, as is high quality documentation.

(yes I know I've been lucky)

Re: Imperial College London have released their Covid-19 epidemic simulation

#76

Post by curl_e »

Johnjonjoan wrote: post #33
devit wrote: post #27

Is this thing actually being used for policy decisions? Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for m…

Metawards is also being used. https://github.com/ldanon/MetaWards

There is also a more user friendly version of the MetaWards at https://metawards.org/ ( GH repo: https://github.com/chryswoods/MetaWards ) implemented in Python which identically matches the results of the C implementation, verified by tests.

Re: Imperial College London have released their Covid-19 epidemic simulation

#77

Post by Gibbon1 »

lbeltrame wrote: post #52
Gibbon1 wrote: post #41

Earlier quoted context omitted.

HN is one of the increasing fewer places online where covid19 deniers can congregate.

I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts. Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism. Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or…

I started using the term deniers because all of them are studiously ignoring something that doesn't pencil out. It doesn't matter if they are loony deniers, sciency deniers, I have a big brain tech bro deniers. The result is all the same. Garbage.

Dangerous garbage.

Re: Imperial College London have released their Covid-19 epidemic simulation

#78

Post by creato »

thu2111 wrote: post #72
creato wrote: post #62

Earlier quoted context omitted.

It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read b…

The non-determinism that page talks about is coming from bugs like memory corruptions, floating point inaccuracies, initialisation order bugs. It's not an intentional part of the model. Averaging corrupted data is meaningless, it doesn't magically fix the corruption. I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any leve…

> The non-determinism that page talks about is coming from bugs like memory corruptions, ..., initialisation order bugs.

Source? I haven't seen these specifically cited anywhere.

> floating point inaccuracies,

Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK.

In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible.

I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand.

I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment...

> To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model.

> With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage

Re: Imperial College London have released their Covid-19 epidemic simulation

#79

Post by jarym »

spatulon wrote: post #45
abainbridge wrote: post #7

This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer…

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfec…

Review by someone claiming to be a ex-Googler here: https://lockdownsceptics.org/code-review-of-fergusons-model/

If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’

This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.

Re: Imperial College London have released their Covid-19 epidemic simulation

#80

Post by jarym »

For a long time academics would only publish formulas and results as part of their papers - omitting source code and arguing that ‘the formulas are the only thing that matter’.

Perhaps the real reason is the code typically smells of Swiss cheese.

From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)

Post reply on HN