Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

71–80 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#72

The author makes it sound like a warning label is missing from the C++ tin. Maybe. But what tool should he have used? I haven't seen this code but is there any doubt it would look just as bad in Java or Python, maybe with fewer segfaults? Or FORTRAN.

Well, according to some reviews the model did run into problems specific to amateurish use of c/c++. Specifically, it produced different results when running on CPUs with different number of cores and even single core but different CPUs

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#73
Whether the Imperial code is good or bad doesn't actually matter: you can derive the headline numbers analytically, without any simulation at all [1].

(The real problem with the worst case estimate is that it assumes people don't individually change their behavior in the face of a pandemic.)

A better critique of the software engineering criticism is [2].

[1] https://twitter.com/trvrb/status/1258879531022082049

[2] https://philbull.wordpress.com/2020/05/10/why-you-can-ignore...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#74
post #52
post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. Could you elaborate on what you mean by that?

Not the OP, but I know of at least one University that runs "Programming for scientists" courses at low or no cost.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#75

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

> experts

Being an expert means you know your tools well. If you can't code well and coding is a critical part of your toolset, then you're not an expert yet. You don't get a free pass because you allegedly are strong in other parts of your craft, especially if the part of your craft you're weak in can be this problematic.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#76
What a collection of terrible arguments. I'm afraid I can't help myself here. I've got to eviscerate this USENET-style:

> "The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers. And the senior or former scientists who decided to give tax-payer money to this research group are probably even more ignorant of the importance of code for science. Otherwise they would surely have attributed money for software development, and verified the application of best practices."

Let's start with the observation that this is indeed the main problem here. Planning research that requires software to be developed, without accounting for software development, is a terrible idea. There's a good reason why scientific programmers exist: to help researchers write the code they need for their scientific projects. Hire one if you need one. Don't blame someone else if you forget to do so.

> "It’s you, the software engineering community, that is responsible for tools like C++ that look as if they were designed for shooting yourself in the foot. It’s also you, the software engineering community, that has made no effort to warn the non-expert public of the dangers of these tools."

Excuse me? The internet is riddled with jokes about how easy it is to shoot yourself in the foot with C++. Of course you can shoot yourself in the foot with any programming language, but C++ excels at it. Many programmers avoid it because they don't want to have to manage their own memory. Follow their example and use something that focuses on the problem area you want to focus on.

> "You know, the kind of warning that every instruction manual for a microwave oven starts with: don’t use this to dry your dog after a bath."

Anyone who needs that kind of warning is a danger to themselves and others. A scientist who lacks this level of common sense should seek guidance from someone who has it.

> "A clear message saying “Unless you are willing to train for many years to become a software engineer yourself, this tool is not for you.”"

Software engineer is a serious profession that requires lots of training. Should any idiot expect to whip up their own epidemic simulation without knowing what they're doing and expect reasonable results?

> "But power comes with responsibility. If you want scientists to construct reliable implementations of models that matter for public health decisions, the best you can do is make good tools for that task, but the very least you must do is put clear warning signs on tools that you do not want scientists to use"

Is it unreasonable to expect someone who needs tools, to either research what tools are suitable for their needs, or otherwise ask advice from an expert on those tools? Nobody grabs just a random tool from their toolbox to solve a specific problem. If it's a nail, you grab a hammer, if it's a screw, you use a screw driver.

> "scientists are not software engineers, and have neither the time nor the motivation to become software engineers."

Then hire one. Don't blame your lack of motivation on others.

> "Consider what you, as a client, expect from engineers in other domains. You expect cars to be safe to use by anyone with a driver’s license."

Yeah, with a driver's license. We don't let random idiots drive off in a car, we expect them to learn how to drive first. If you want to use C++, you're going to have to learn about memory management. If you don't want to do that, get a different language.

> "You expect household appliances to be safe to use for anyone after a cursory glance at the instruction manuals."

Is that cursory glance going to be enough to tell them not to put their dog in a microwave? At least C++ is not going to kill your dog.

> "It is reasonable then to expect your clients to become proficient in your work just to be able to use your products responsibly?"

They can use our products just fine, but if they want to use our tools, they need to learn how to use them.

Should toolboxes now contain warning labels not to build your own car from scratch? Or not to use these tools to repair a nuclear reactor? It's absolutely valuable to learn how to use tools, but if you just grab random power tools without knowing what you're doing, and without being willing to learn to use them, you're likely to lose a limb.

Finally, if you're really looking for a programming language to let scientists play around with, try Python. It's designed to be easy to learn, and it's also very suitable for all sorts of scientific modeling. There's a good reason it's popular in research. I still recommend putting some effort in learning to use it.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#77
post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

The flip side to having lots of parameters is that you have lots of knobs to tune beyond a basic lockdown.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#78
post #35

Earlier quoted context omitted.

Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?

If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.

That's also true. Now the million $ question:

Whom would you like our society to rely on to generate quality work?

I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions.

There is no smart solution to a stupid problem. But the truth is that _as an institution_ the NHS is the only actor whose mission is to optimize towards public health.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#79
post #42
post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong. Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll gi…

"But what value of R would you pick?"

I don't know -- and until we can agree on the answer to your simple question with a high degree of confidence, I think complex models based on specific assumed values of R obscure more than they reveal.

A little bit of modeling is useful because humans are intuitively bad at exponential math and we need scary graphs to jolt us awake sometimes. But when we don't even know the basic parameters (transmission/hospitalization/fatality) with a high degree of precision, complex models with myriad parameters create a false sense of confidence.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#80

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…

That last part is mindblowing given the fact that the health insurance industry in the US is trying to argue it shouldn’t have to pay for COVID-19 treatments because it’s part of a pandemic and not part of normal medical treatments.

I guess if your model is:

   if pandemic and COVID:
       is_covered = False
You can have a very clean pandemic model.
Post reply on HN