Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

131–140 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#131

Post by disgruntledphd2 »

veddox wrote: post #116
disgruntledphd2 wrote: post #89

Earlier quoted context omitted.

It's much more accessible, as you need to know less to use it. Consider - using Eigen in a C++ project, or calling install.packages("eigen") in R.

It's easier to dip your toes into, but it is a terrible language to learn software development with. It's such a hodge-podge of ill-thought-out and ill-fitting components with random names and no overall sense of structure that you're never going to learn core CS concepts from it, let alone good programming practice.

That's not really its purpose though, right?

R is "an environment for statistical computing and graphics". Note that it doesn't talk about software engineering at all.

I agree with you that R has lots of rough edges, but please remember that it's a 90's era clone of a 70's era language (S) and a lot of those rough edges and corners are legacies from that time.

I completely agree that the naming conventions (i.e. the absence of same) are super annoying, but again it's a tradeoff for the decades of statistical computing knowledge embedded in the system of R.

I find your disdain for R a little annoying, and while I'm probably not a computer scientist, it was my first language and I learned about closures, higher order functions, OOP, testing and interacting with API's from it, and it definitely gave me insight into how computers worked.

I'm sorry that you have to deal with horrible legacy R code, but that's no reason to throw the baby out with the bathwater.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#132

Post by disgruntledphd2 »

silveraxe93 wrote: post #115
disgruntledphd2 wrote: post #93

Earlier quoted context omitted.

I honestly don't think R and Python would have worked here. While I was able to clone the repo and run it for Ireland, the UK requires at least 26Gb of RAM, which is not common on most personal computers. The US requires much, much more. And given that it's pretty slow when written in C++, imagine how slow it would have been in R or Python? I agree with you in principle with respect to this stuff being better, but th…

Any serious attempt at modelling this over python would use the pydata stack (numpy, pandas, etc), which run on top of C++ anyways.

Yeah of course, apologies if that wasn't clear.

The best solution here would probably be to package up the core routines into a library and use this from either R or Python.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#133

Post by klenwell »

lern_too_spel wrote: post #37
klenwell wrote: post #22

Earlier quoted context omitted.

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…

That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones. The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.

I agree. Moreover, the cited bug ("predictions varied by around 80,000 deaths after 80 days") doesn't really seem to impact the over all policy implications: no lockdown means exceptional number (400 to 480k) of deaths.

Unless the entire simulation is bogus, it comes off as nitpicking.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#134

Post by Joeboy »

ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability

Have you actually checked to what extent the Ferguson codebase is guilty of these things? It seems to me there are people spreading misinformation, and too many people taking their word for it.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#135

Post by chrisseaton »

csours wrote: post #104
chrisseaton wrote: post #87

Earlier quoted context omitted.

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle.

Is this brittleness stopping the scientists achieving what they need to achieve?

Are you sure that writing tests makes science better? Or are you just assuming that?

They aren't idiots and they aren't ignorant of how professional software developers work.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#136

Post by pjc50 »

silveraxe93 wrote: post #41
rjtobin wrote: post #19

I'm somewhat skeptical. Firstly, I don't think the language is the problem with scientific code. You can write messy code in any language. So the warning then has to be about writing software in general. In that case, I think a warning like "don't try to write software unless you have years of training" is a bit much. Many people with no training learn to write nice code. Many projects made by amateurs might have ugl…

This model didn't just influence decisions in healthcare. It single-handedly changed the UK government's strategy over this pandemic. From what I understand the UK was planning on beating COVID by creating herd immunity, similarly to Sweden. Then this model came out and everyone started yelling that Boris wanted to kill your grandma. The problem is that it's impossible to have an intelligent discussion over this. Thi…

> beating COVID by creating herd immunity, similarly to Sweden

The big problem here is that herd immunity requires that either you have a vaccine or you get some large fraction of the population infected, over 50%.

The death rate is about 1%, plus further people suffering long-term complications.

So achieving herd immunity in the UK would require about 300,000 dead.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#137

Post by andi999 »

w0utert wrote: post #91
andi999 wrote: post #45

Earlier quoted context omitted.

There is probably no funding for outside help... So they get a project fund a PhD student (or post doc) with that, this guy has to produce new results (fast) and more results, then these guys leave, next guys come.. Generally for a simulation group you would need an experienced programmer maintaining the code base, (helping) the other guys cleaning up there ideas before handover, and this as a permanent position. Exp…

I find this a pretty weak excuse for a high-impact epidemiological model like this. Universities are full of the brightest people you will find anywhere in society, and most universities have faculties and courses in computer science and/or software engineering. How about enforcing multi-disciplinary efforts, having computer science students work with microbiologists, and vice versa? Exchange programs between univers…

It wasnt meant as an excuse, just to point out what effectively would have to be done to solve this is the future.

I am a bit sceptical of your suggestion of putting more non permanent staff (even from CS) into the pot.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#138

Post by rch »

vonmoltke wrote: post #124
rch wrote: post #113

Earlier quoted context omitted.

For this to happen there would need to be a nationally accepted PE certification for 'software engineers' on par with other engineering disciplines. It's unreasonable to expect non-experts, and experts from other fields, to perform credentialing on a case by case basis.

There is no "nationally accepted" PE license for any discipline. It's a state-level credential and reciprocity is not universal or automatic.

I included the "nationally accepted" bit since I'd prefer to see some incremental improvement over current models, but I don't have confidence that an international standard would be accepted in all relevant contexts. It could still be administered at the state level.

I'm also ignoring the requirement that candidates must start with a degree from an ABET-accredited institution, which seems to feature prominently in proposals from IEEE and others. Ideally I think there should be some way around that, but alternatives I'm familiar with aren't great either (e.g. FINRA).

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#139

Post by datastoat »

wbhart wrote: post #42
tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong. Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll gi…

I would pick a value of R that shows itself to have good predictive accuracy.

The way to test predictive models is always to look for their predictive accuracy on holdout data. Machine learning has this ingrained. Classic statistics does this too -- AIC is used to compare models, and it's (asymptotically) leave-one-out cross validation [1].

There's nothing intrinsically wrong with models that have millions of parameters; they might overfit in which case they will have poor predictive accuracy on holdout data, or they might predict well.

I agree with the original article that software engineer scrutiny isn't appropriate for this sort of code -- but I would argue instead that it needs a general-purpose statistician or data scientist or ML expert to evaluate its predictive accuracy. You can't possibly figure this out from a simulator codebase.

At the time the model was published, and acted on by the UK government, there was very little data on which to test predictive accuracy. That's fine -- all it means is that the predictions should have been presented with gigantic confidence intervals.

[1] http://www.stats.ox.ac.uk/~ripley/Nelder80.pdf

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#140

Post by 0xdeadbeefbabe »

zozbot234 wrote: post #118
veddox wrote: post #111

Earlier quoted context omitted.

Oh we teach programming alright, but software development is another pair of shoes entirely. Unfortunately, that's where the problem lies...

That's the whole problem with the "Learn to code" movement, tbh. Plus Python and Java are terrible languages when it comes to those broader swdev/softeng concerns.

What is better than Java?
Post reply on HN