To software engineers criticizing Neil Ferguson’s epidemics simulation code • Hacker News Board

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#161

Post by chrisseaton » Mon, May 18, 2020, 6:11 PM UTC

mcv wrote: post #151

chrisseaton wrote: post #135

Earlier quoted context omitted.

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.

If the results aren't reproducible, they can't be assumed to be true. Then they're only useful if you only care about publication and not about whether the results are actually true. And yes, this is a serious problem in science.

> If the results aren't reproducible, they can't be assumed to be true.

I don't understand why having brittle code would mean that the results are not reproducible? You don't need to modify the program to do a reproducibility study.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#162

Post by btrettel » Mon, May 18, 2020, 6:11 PM UTC

As a researcher, if I want to find a software engineer willing to review my code for free (I have no budget for this), how should I find one?

The article says

> We can’t ask software experts for a code review every time we do something important.

but I think there are people who'd be willing to give at least 15 minutes of their time to review scientific software once.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#163

Post by chrisseaton » Mon, May 18, 2020, 6:14 PM UTC

lalaithion wrote: post #94

chrisseaton wrote: post #87

Earlier quoted context omitted.

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

The problem is, we need confidence that the software is correct in order to trust the scientific results.

Do you need tests to do that? Are tests the only way?

Are you doing formal verification of your software? Why not? If you're not bothering to do that why are you criticising researchers for not bothering to use tests?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#164

Post by silveraxe93 » Mon, May 18, 2020, 6:31 PM UTC

kwhitefoot wrote: post #152

silveraxe93 wrote: post #41

Earlier quoted context omitted.

This model didn't just influence decisions in healthcare. It single-handedly changed the UK government's strategy over this pandemic. From what I understand the UK was planning on beating COVID by creating herd immunity, similarly to Sweden. Then this model came out and everyone started yelling that Boris wanted to kill your grandma. The problem is that it's impossible to have an intelligent discussion over this. Thi…

> creating herd immunity, similarly to Sweden. > ... > The problem is that it's impossible to have an intelligent discussion over this. As far as I can tell the Swedish government never had this plan. It was mentioned in an interview and dismissed as unworkable, journalists misunderstood. On the other hand the UK government appears to have had no plans whatever until jolted into action by the fear that public opinion…

It seems the government was following this document at the start: https://assets.publishing.service.gov.uk/government/uploads/...

The reason it seemed they were doing nothing are these passages:

ii. Minimise the potential impact of a pandemic on society and the economy by:

• Supporting the continuity of essential services, including the supply of medicines, and protecting critical national infrastructure as far as possible.

• Supporting the continuation of everyday activities as far as practicable.

• Upholding the rule of law and the democratic process.

• Preparing to cope with the possibility of significant numbers of additional deaths.

• Promoting a return to normality and the restoration of disrupted services at the earliest opportunity.

There's way more, but I've honestly not read it all. But there was a plan, drafted before this epidemic.

Public opinion was turning against the government, but it actually kept course for some time. Something I was honestly impressed with. What made it drop the plan was Neil Ferguson's study.

There are many reasons for criticising the plan. This article is pretty good. https://www.theguardian.com/politics/2020/mar/29/uk-strategy...

What really gets me is that if the lockdown was the correct decision, we arrived there for the wrong reasons.

This paper had such an outsized impact that it should be held to a higher standard. And it's scary (but not really unexpected) that the government is making decisions of this magnitude based on such a shaky foundation.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#165

Post by SJetKaran » Mon, May 18, 2020, 6:38 PM UTC

claytonjy wrote: post #146

disgruntledphd2 wrote: post #131

Earlier quoted context omitted.

That's not really its purpose though, right? R is "an environment for statistical computing and graphics". Note that it doesn't talk about software engineering at all. I agree with you that R has lots of rough edges, but please remember that it's a 90's era clone of a 70's era language (S) and a lot of those rough edges and corners are legacies from that time. I completely agree that the naming conventions (i.e. the…

I really appreciate this sentiment as a data person who learned R first, now works in python, and helps non-engineer scientists write R programs for research. I've had great luck teaching scientists to think in a function-driven way, where functions are pure-as-possible and inputs are never mutated (which doesn't need to be discussed as R makes the opposite quite unnatural). I can't imagine teaching these same folks…

` and why you need pd.DataFrame.copy() all over the place.` Thanks for saying this!

Also, in the past decade, the R data analysis/munging ecosystem has matured far more than the equivalent in Python. These days, writing (tabular) data transformation code in R is often far cleaner, clearer and less error prone than any other platform imo.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#166

Post by silveraxe93 » Mon, May 18, 2020, 6:44 PM UTC

disgruntledphd2 wrote: post #132

silveraxe93 wrote: post #115

Earlier quoted context omitted.

Any serious attempt at modelling this over python would use the pydata stack (numpy, pandas, etc), which run on top of C++ anyways.

Yeah of course, apologies if that wasn't clear. The best solution here would probably be to package up the core routines into a library and use this from either R or Python.

Sorry if I came out a bit snippy out there. But yeah I assumed you meant python without numpy, etc.

A lot of the criticism I saw was because the core routines did not need to be packaged up. There were a lot of common data structures reimplemented, etc.

I don't think the model had many novel routines. It could be built just using industry standard and tested tools in python, R, Julia (if you really want speed) etc. But it reinvented the whole ecosystem in one big ball of C.

tbh, this should have been built on STAN or similar. There's so many variables and assumptions that the output is completely dominated by the parameters chosen. Seeing the distribution of outcomes instead of a point estimate would be actually useful.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#167

Post by pushingice » Mon, May 18, 2020, 7:37 PM UTC

chrisseaton wrote: post #87

ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

As a former academic, the point is to get the research published. Whether the code continues to work after that, is irrelevant.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#168

Post by datastoat » Mon, May 18, 2020, 7:46 PM UTC

rainforest wrote: post #153

datastoat wrote: post #139

Earlier quoted context omitted.

I would pick a value of R that shows itself to have good predictive accuracy. The way to test predictive models is always to look for their predictive accuracy on holdout data. Machine learning has this ingrained. Classic statistics does this too -- AIC is used to compare models, and it's (asymptotically) leave-one-out cross validation [1]. There's nothing intrinsically wrong with models that have millions of paramet…

The model isn't predictive though - it's a simulator. If we'd waited until we had enough data to make predictions with it (which I doubt you could given the sheer number of parameters) it'd be too late to use any of the interventions. How would you ethically collect training data for the interventions?

The outputs of the model _were_ being treated as predictions.

The Ferguson paper from 16 March used the language of prediction: "In the (unlikely) absence of any control measures [...] given an estimated R0 of 2.4, we predict 81% of the GB and US populations would be infected over the course of the epidemic." [1]. The news coverage also used that language: "Imperial researchers model likely impact of public health measures" [2]. And look at the rest of the comments in this discussion, and count how many types "predict" appears!

> If we'd waited until we had enough data to make predictions with it

This is like the drunk looking for their keys under a streetlight. "Did you lose the keys here?" "No, but the light is much better here." -- "How confident are you in your model's predictions?" "I have no idea, but it's the model I have."

Also -- the Ferguson model made predictions, based on the parameters they picked. You don't need to wait for data to make predictions; you only need data to validate your predictions.

> How would you ethically collect training data for the interventions?

You don't. You (as a scientist who influences public policy) should publish validated confidence intervals for your predictions. You (as a government) should understand that there is a huge margin of uncertainty in the predictions, and accept that sometimes you just have to make decisions in the absence of knowledge. You (both the scientist and the government) do not go around spouting "Our decisions are led by science".

[1] https://spiral.imperial.ac.uk:8443/bitstream/10044/1/77482/1...

[2] https://www.imperial.ac.uk/news/196234/covid19-imperial-rese...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#169

Post by veddox » Mon, May 18, 2020, 8:30 PM UTC

disgruntledphd2 wrote: post #131

veddox wrote: post #116

Earlier quoted context omitted.

It's easier to dip your toes into, but it is a terrible language to learn software development with. It's such a hodge-podge of ill-thought-out and ill-fitting components with random names and no overall sense of structure that you're never going to learn core CS concepts from it, let alone good programming practice.

That's not really its purpose though, right? R is "an environment for statistical computing and graphics". Note that it doesn't talk about software engineering at all. I agree with you that R has lots of rough edges, but please remember that it's a 90's era clone of a 70's era language (S) and a lot of those rough edges and corners are legacies from that time. I completely agree that the naming conventions (i.e. the…

> I learned about closures, higher order functions, OOP, testing and interacting with API's from it, and it definitely gave me insight into how computers worked.

Well done, I'm honestly impressed :D

> remember that it's a 90's era clone of a 70's era language

That's not an excuse. Lisp is two decades older.

> I find your disdain for R a little annoying ... I'm sorry that you have to deal with horrible legacy R code

This has nothing to do with legacy code - I rarely have to touch that. (Scientific code is rarely reused...). R does offer a lot of great features, I'll grant you that. It has a huge library with great functions for doing statistical analysis and really neat graphics - exactly what it was designed for. As a biologist, I cannot imagine not using it.

But, and this is a big but, as a programming language it is absolutely horrible. I've worked with half a dozen languages, and none of them are anywhere near as big a pain in the backside as R. Most of that has to do with the terrible incongruity of a plethora of mutually-incompatible data types. You know, you have one function that will only work on a data frame, but somehow your data ended up as a list; or you want strings or numbers but have factors; that kind of thing. In no other language do I have to do as much googling while coding, because even basic operations are so willy-nilly idiosyncratic in their details that I keep forgetting how to do them properly. The "tidyverse" and related packages do a lot to make the experience less painful, but pure R remains a linguistic nightmare.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#170

Post by mcv » Mon, May 18, 2020, 8:50 PM UTC

chrisseaton wrote: post #161

mcv wrote: post #151

Earlier quoted context omitted.

If the results aren't reproducible, they can't be assumed to be true. Then they're only useful if you only care about publication and not about whether the results are actually true. And yes, this is a serious problem in science.

> If the results aren't reproducible, they can't be assumed to be true. I don't understand why having brittle code would mean that the results are not reproducible? You don't need to modify the program to do a reproducibility study.

Reproducibility is one of the issues mentioned earlier in the thread. And being able to audit the code and understand what it actually does seems rather important too.