Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Imperial College London have released their Covid-19 epidemic simulation

github.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Imperial College London have released their Covid-19 epidemic simulation

#51

Post by SiempreViernes »

heisenzombie wrote: post #37
cataphract wrote: post #31

Earlier quoted context omitted.

I've seen a lot of code coming out of academia that exhibits a similar complete lack of structure and completely ignores the most basic software development best practices (such as having tests). There are exceptions, and research software engineers are a thing, but unsurprisingly most academics focus their careers on the science rather than the code.

It’s the same reason that if you go into a working Physics laboratory, most things will seem to be wrapped in kitchen aluminium foil. There’s better ways of making thermal insulation, EMF shielding, grounding, dust protection... but the foil is /right there/ and the experiment only really needs to work /once/

Hey now, how could anything be better than foil? You can reliably steal enough of it from any nearby kitchen, and it does the job well enough, what's the downside? ;)

Re: Imperial College London have released their Covid-19 epidemic simulation

#52

Post by lbeltrame »

Gibbon1 wrote: post #41
JauntyHatAngle wrote: post #23

Earlier quoted context omitted.

I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.

HN is one of the increasing fewer places online where covid19 deniers can congregate.

I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts.

Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism.

Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match).

I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.

Re: Imperial College London have released their Covid-19 epidemic simulation

#53

Post by jfnixon »

SiempreViernes wrote: post #50
lbeltrame wrote: post #10

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst pr…

If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.

Re: Imperial College London have released their Covid-19 epidemic simulation

#54

Post by lbeltrame »

SiempreViernes wrote: post #50
lbeltrame wrote: post #10

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst pr…

> Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly.

Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow.

That said, that "personal level" statement is absolutely out of the line.

Re: Imperial College London have released their Covid-19 epidemic simulation

#55

Post by SiempreViernes »

masonic wrote: post #43
lbeltrame wrote: post #38

Earlier quoted context omitted.

I once had to debug a crash in a very popular program used in genomics (for those in the know, it was an early version of "samtools" IIRC) and I found it really hard to read. (In the end the crash was due to corrupted input data, so I fixed the data and stopped debugging)

the crash was due to corrupted input data Bad data shouldn't cause a crash . That's just sloppy.

Proper input validation is really hard, and a reliable crash on incorrect input is frankly much better than other alternatives such as soldiering on and output trash, or even worse seemingly fine results.

Of course, it sucks if it takes a few hours before you get your crash, that usually tricks you into thinking the software did something wrong and not you. Best is if it dies immediately upon reading the input, and if it does that with an error or a crash isn't crucial, though of course getting a clear error message is always better.

As a comparison, a collaboration internal tool I need to use will print a helpful usage message, and then segfault if you call it without any arguments...

Re: Imperial College London have released their Covid-19 epidemic simulation

#56

Post by jfnixon »

tasogare wrote: post #47
notkaiho wrote: post #42

Earlier quoted context omitted.

Having seen a discussion about this on a different forum, someone who used to work in academia mentioned that repeatability and consistency are key to academic code, which means often readability and convention may get sacrificed - it works for the author and if someone wants to repeat the experiment it should get the same results without wangling around with many external packages or modules that may change or becom…

Yep. Also, speaking from academic point of view, code won’t make you advance on your career by itself (only papers based on it will), so the minimum is done until the thing is usable for its goal. So forget about CI/CD, proper test suits, documentation (I hate this point personally and document my projects a bit more than the average researcher) and engineering best practices.

Then public policy should heavily discount results based on academic code, if it shown to be poorly engineered. I'd go further and say you can't trust the papers based on the results of badly engineered simulations. As a poster said earlier "I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies."

Re: Imperial College London have released their Covid-19 epidemic simulation

#57

Post by jfnixon »

beshrkayali wrote: post #32

Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.

I'll be "that guy" and point out we know the climate simulations have similar issues.

Re: Imperial College London have released their Covid-19 epidemic simulation

#58

Post by SiempreViernes »

lbeltrame wrote: post #54
SiempreViernes wrote: post #50

Earlier quoted context omitted.

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst pr…

> Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly. Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow. That said, that "personal level" statement is abs…

Sure, but for an inherently stochastic model the main results are the average behaviour, and the reproducibility of these are the main concern. Reproducibility of individual runs is more a convenience.

Re: Imperial College London have released their Covid-19 epidemic simulation

#59

Post by goldforever »

zimpenfish wrote: post #48
unyttigfjelltol wrote: post #44

Earlier quoted context omitted.

13+ years ago the same academic modeler predicted 200 million deaths from a bird flu outbreak that killed 282 people in total.[1]. That's a pretty enormous error-- alarmist publicity-seeking behavior then and now, but the damage was felt more acutely now. If this modeler really wanted a competent simulation it would have been better constructed. That it was not well-constructed is another datapoint indicating how egr…

That's an amazingly disingenuous comment. https://www.theguardian.com/world/2005/sep/30/birdflu.jamess... He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad. And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public…

That Ferguson kept getting his predictions/forecasts wrong time after time - good riddance.

Re: Imperial College London have released their Covid-19 epidemic simulation

#60

Post by mytailorisrich »

spatulon wrote: post #45
abainbridge wrote: post #7

This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer…

I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfec…

If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release".

The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.

Post reply on HN