Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

181–189 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#181

Post by thu2111 »

lern_too_spel wrote: post #37
klenwell wrote: post #22

Earlier quoted context omitted.

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…

That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones. The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.

Seriously?

Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct.

Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#182

Post by thu2111 »

lucideer wrote: post #43
silveraxe93 wrote: post #14

I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

There's plenty of funding in academia. Come on. It's a huge segment of society, governments spend billions on it.

They don't hire software engineers to write their models because they know they can get published without that and they'd rather hire more students into their department and publish more papers. For academia to claim they can't afford to produce programs that work is absurd and damning - why should anyone believe anything in a scientific paper if this attitude is so widespread? Some people here are arguing it's ridiculous to expect academics to produce work that's correct, even putting assumptions to one side.

All it takes to fix this is multi-disciplinary teams. Of the type found in almost any industry but apparently, not academia. Why is this idea so strongly rejected by academia when it's just common sense everywhere else?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#183

Post by thu2111 »

rainforest wrote: post #81
munchbunny wrote: post #15

Earlier quoted context omitted.

should also let outside help come when the need arises. expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look. 1. Software engineers are expensive. Hiring them to write your code is how you end up needing even more money to do your science, and I think the software engineering world if anything can appreciate prioritizing being scrappy to get more done.…

I believe this model is quite old (at least in some form) so there have been opportunities to review it. I confess I haven't looked, but I haven't heard any defects have actually been identified. If the model stood up to peer review I assume the results it produces are at least consistent with the expectations of the people who wrote the mathematical model. Hopefully this will be a watershed moment that makes it easi…

Many defects have been identified. A small collection are linked to from here:

https://lockdownsceptics.org/second-analysis-of-fergusons-mo...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#184

Post by btrettel »

lbeltrame wrote: post #23
flankstaek wrote: post #12

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

Back in 2008-2009 my then PI tried this: it worked for a while, but then it proved to be untenable. Not because there wasn't enough money (there was) but because the university did not like the idea to get someone on board just to help researchers develop software, and thus there were so many roadblocks at one point that it was impossible to go on.

I'm interested in hearing more details on this, if you're able to discuss them.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#185

Post by lbeltrame »

btrettel wrote: post #184
lbeltrame wrote: post #23

Earlier quoted context omitted.

Back in 2008-2009 my then PI tried this: it worked for a while, but then it proved to be untenable. Not because there wasn't enough money (there was) but because the university did not like the idea to get someone on board just to help researchers develop software, and thus there were so many roadblocks at one point that it was impossible to go on.

I'm interested in hearing more details on this, if you're able to discuss them.

I don't honestly know how much I can disclose (FTR, I was not the one doing that job - I was working as a postdoc there at the time), but I'll try to summarize it briefly:

- PI got a couple of EU funded grants

- PI wanted to build a program / programs out of some ideas he had before moving to the institution he was currently employed at

- PI hired a software developer (actually a software engineer) to do this job

- Statistician drafted algorithms, developer created the software

- I used the software for my Ph.D. thesis, got hired at the lab of the PI as a postdoc, new requirements arose due to the way I used it

- Developer made a preliminary version of a new version of the software following discussions with me

- University made it very hard to keep developer on the team, due to bureaucracy and kind of hostility towards this kind of employment

- At some point (I can't recall the details exactly, but it was something that spanned almost one year), the form of keeping the developer on board was no longer possible

- PI offers an alternative contract, but it is financially wasteful to the developer (not the fault of the PI, but the way certain things work in my country)

- Developer leaves the project

Also the university, to my knowledge, complained that the developer cost a lot (IIRC, the project was paid at market rate, so in line with other, non academic software projects).

I can't comment on the quality of the software we used (sadly it was never open sourced) as it was in Java and I only have a passing understanding of the language, but the approach of having a dedicated developer IMO worked (and also net quite a number of publications over that period).

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#186

Post by brigandish »

Joeboy wrote: post #175
brigandish wrote: post #174

Earlier quoted context omitted.

Really? > Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby p…

Yes. John Carmack says the code is OK, some anonymous person on the lockdownsceptics.org website says it's a flaming heap of garbage. For what it's worth I looked into some of the tickets linked in those articles and concluded the author is, broadly speaking, full of shit. I am nobody in particular though.

> John Carmack says the code is OK

He doesn't say it's okay, he engages in a weird kind of whataboutery like "Heck, professional software engineering struggles mightily with just making completely reproducable builds". I struggle to note one part of the article by the "retired software engineer" (as if that has any relevance either) that he deals with specifically.

But since it's John Carmack we must let him wave his hand and say it is so. The Github issues are also far more enlightening than Carmack's tweets on this, but again, who cares for precision and points argued with evidence when we have a name giving their opinion?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#187

Post by luord »

> Consider what you, as a client, expect from engineers in other domains. You expect cars to be safe to use by anyone with a driver’s license. You expect household appliances to be safe to use for anyone after a cursory glance at the instruction manuals. It is reasonable then to expect your clients to become proficient in your work just to be able to use your products responsibly? Worse, is it reasonable to make that expectation tacitly?

That doesn't seem like a proper analogy. I certainly would expect a civil engineer to think me an idiot if I tried to do his job for him. And that's how most code written by non software engineers/computer scientists ends up looking.

Software engineers don't produce programming languages, but programs. Languages (and libraries, etc) are our tools, not our end products. Much like an architect's job is not to produce rulers and pencils, but plans.

It just so happens that (more so for computer scientists than for software engineers) we create our own tools.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#188

Post by Joeboy »

brigandish wrote: post #186
Joeboy wrote: post #175

Earlier quoted context omitted.

Yes. John Carmack says the code is OK, some anonymous person on the lockdownsceptics.org website says it's a flaming heap of garbage. For what it's worth I looked into some of the tickets linked in those articles and concluded the author is, broadly speaking, full of shit. I am nobody in particular though.

> John Carmack says the code is OK He doesn't say it's okay, he engages in a weird kind of whataboutery like "Heck, professional software engineering struggles mightily with just making completely reproducable builds". I struggle to note one part of the article by the "retired software engineer" (as if that has any relevance either) that he deals with specifically. But since it's John Carmack we must let him wave his…

> The Github issues are also far more enlightening than Carmack's tweets on this

Totally agree. The "lockdown skeptics" articles significantly misrepresent the github issues. They imply that there are mysterious uncertainties creeping into the results, when the actual issues relate to things like failures to set RNG seeds consistently, or a checksum failing in a test due to floating point rounding differences in Cray supercomputers' native instructions. Most readers aren't going to investigate the github issues though.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#189

Post by lern_too_spel »

thu2111 wrote: post #181
lern_too_spel wrote: post #37

Earlier quoted context omitted.

That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones. The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.

Seriously? Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct. Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?

What does reading uninitialized memory or reusing variables that weren't set to zero have to do with seeding issues? Read my comment again and reply to its content instead of making up a comment that you would like to reply to.
Post reply on HN