Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

171–180 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#171

Post by mturmon »

SpicyLemonZest wrote: post #117
rscho wrote: post #78

Earlier quoted context omitted.

That's also true. Now the million $ question: Whom would you like our society to rely on to generate quality work? I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions. There is no smart solution to a stupid problem. But the truth is th…

In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good result…

> In most fields, our society relies on private industry to generate quality work ...

This is just not true. Military, police, courts? What does "most fields" even mean?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#172

Post by SpicyLemonZest »

mturmon wrote: post #171
SpicyLemonZest wrote: post #117

Earlier quoted context omitted.

In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good result…

> In most fields, our society relies on private industry to generate quality work ... This is just not true. Military, police, courts? What does "most fields" even mean?

My job, food, apartment, utilities, entertainment, all come from private companies.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#173

Post by brigandish »

pushingice wrote: post #167
chrisseaton wrote: post #87

Earlier quoted context omitted.

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

As a former academic, the point is to get the research published. Whether the code continues to work after that, is irrelevant.

This is, indeed, the whole truth and nothing but the truth. My own anecdote: through happenstance I'm co-author on a paper, a large number of my friends are PIs/PhDs etc and one of them needed help after the software written for them by the CS student they'd been assigned didn't work (he got a masters for it, I can't believe anyone in the CS department looked at his work). I asked about publishing my code after the paper was published and no one seemed bothered.

I've also seen my friends on the mill of, shall we call it paper chasing? That seems most apt. It reminded me of borrowing someone's homework and making minor changes to get it past teacher - except the homework was the same paper being constantly rejigged for submission to a new journal. Not much truly new work seemed to be going on.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#174

Post by brigandish »

Luc wrote: post #86

John Carmack has reviewed the code and didn't seem to find it all that bad, so there's that. https://twitter.com/ID_AA_Carmack/status/1258192134752145412 https://twitter.com/ID_AA_Carmack/status/1244302925855326209

Really?

> Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby programming is so extensive that even Microsoft were unable to make it run right.

That's from the first analysis[1]. There's a follow up[2]:

> Sadly it shows that Imperial have been making some false statements.

and

> It’s clear that the changes made over the past month and a half have radically altered the predictions of the model. It will probably never be possible to replicate the numbers in Report 9.

[1] https://lockdownsceptics.org/code-review-of-fergusons-model/

[2] https://lockdownsceptics.org/second-analysis-of-fergusons-mo...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#175

Post by Joeboy »

brigandish wrote: post #174
Luc wrote: post #86

John Carmack has reviewed the code and didn't seem to find it all that bad, so there's that. https://twitter.com/ID_AA_Carmack/status/1258192134752145412 https://twitter.com/ID_AA_Carmack/status/1244302925855326209

Really? > Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby p…

Yes. John Carmack says the code is OK, some anonymous person on the lockdownsceptics.org website says it's a flaming heap of garbage.

For what it's worth I looked into some of the tickets linked in those articles and concluded the author is, broadly speaking, full of shit. I am nobody in particular though.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#176

Post by kwhitefoot »

mcv wrote: post #100
TeMPOraL wrote: post #82

Earlier quoted context omitted.

> software quality is not valued in academia. And in general, it shouldn't. Code written by scientists is almost always throwaway prototypes. So it absolutely doesn't matter how decoupled, or easy to extend it is. The only aspect of quality that matters here is the simplicity, in the sense "it obviously has no bugs" (and related practices like testing). This is something that could be taught, but I think it should be…

It doesn't have to be decoupled or easy to extend, but it should be readable enough to ensure it does what it's supposed to, and well-tested enough to verify it does what it's supposed to.

I think Blaise Pascal's (or whoever really said it) remark is apposite here:

    If I Had More Time, I Would Have Written a Shorter Letter
https://quoteinvestigator.com/2012/04/28/shorter-letter/

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#177

Post by rainforest »

datastoat wrote: post #168
rainforest wrote: post #153

Earlier quoted context omitted.

The model isn't predictive though - it's a simulator. If we'd waited until we had enough data to make predictions with it (which I doubt you could given the sheer number of parameters) it'd be too late to use any of the interventions. How would you ethically collect training data for the interventions?

The outputs of the model _were_ being treated as predictions. The Ferguson paper from 16 March used the language of prediction: "In the (unlikely) absence of any control measures [...] given an estimated R0 of 2.4, we predict 81% of the GB and US populations would be infected over the course of the epidemic." [1]. The news coverage also used that language: "Imperial researchers model likely impact of public health me…

How do you validate the predictions for the number of infected cases in May for scenarios that don't happen?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#178

Post by thu2111 »

noelsusman wrote: post #123
tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

You're correct to focus on the effect of parameter choices over code quality. It's been a little funny to watch a bunch of software engineers freak out about unit tests while ignoring everything else that has a much larger impact on the output of the model. I would bet large sums of money that this code is producing the correct output according to the model/parameter specifications. All I can say is welcome to epidem…

I would bet large sums of money that this code is producing the correct output according to the model/parameter specifications.

I'll take that money off you then.

The code has various memory safety bugs in it and originally had a typo in a random number generator constant. Amongst other problems.

There's really no reason to believe it produces correct outputs, in fact, we know it didn't and probably still doesn't given how it was written.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#179

Post by thu2111 »

chrisseaton wrote: post #135
csours wrote: post #104

Earlier quoted context omitted.

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.

Clearly not, but that's because what they need to achieve is publication of something interesting, not something correct.

Academics have a lot of excuses for writing terrible code. It's really shocking. So far I've seen:

- It doesn't matter if the code is buggy because the results don't need to be accurate

- We can't afford to write good code.

- We aren't properly trained to write good code.

- We just average the results and that fixes the bugs (!)

- We aren't computer scientists so what do you expect.

- Science is special which means we don't need to write tests to get correct results. We just eyeball them and know if they're right (so why bother writing a model at all then?)

- You can't criticise or judge us because mere software developers can't understand science.

- Nobody told us it's easy to screw up memory management in C

And a whole lot of other baffling and insulting nonsense. How many of these excuses would be accepted if a private company produced a dangerous product due to bad code, and produced this litany of BS in a courtroom? None.

They aren't idiots and they aren't ignorant of how professional software developers work.

It's apparent from some of the responses to this fiasco that they are totally ignorant of how professional software developers work. And they're proud of it, which in my view makes them idiots too. You can't both tell governments and whole societies to "follow the science" and then blow off any suggestion of working to professional standards.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#180

Post by disgruntledphd2 »

silveraxe93 wrote: post #166
disgruntledphd2 wrote: post #132

Earlier quoted context omitted.

Yeah of course, apologies if that wasn't clear. The best solution here would probably be to package up the core routines into a library and use this from either R or Python.

Sorry if I came out a bit snippy out there. But yeah I assumed you meant python without numpy, etc. A lot of the criticism I saw was because the core routines did not need to be packaged up. There were a lot of common data structures reimplemented, etc. I don't think the model had many novel routines. It could be built just using industry standard and tested tools in python, R, Julia (if you really want speed) etc. B…

True, I think that's worth noting. To be fair though, this codebase is pretty old, and it's unlikely that the technology landscape looked much like today (especially in terms of R and Python), so I can see how they ended up here.

I'd love if this was written in R, Python or Stan so I could contribute, but that's probably not the researchers focus ;)

While Stan is amazing, I shudder to think as to how long this model would have taken to run using MCMC (1 week plus maybe?).

Post reply on HN