Look, the software engineering community has switched to Rust by now. If you're still using C++, that's your problem. Don't blame us.
To software engineers criticizing Neil Ferguson’s epidemics simulation code
31–40 of 189 posts
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#32Publicly funded science should in almost all cases make the software underpinning it publicly available. Otherwise, how are results reproducible, verifiable or open to peer review?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#33I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does.
Here's one params file that specifies some of the inputs to a run of the model:
https://github.com/mrc-ide/covid-sim/blob/master/data/param_...
Here's another one:
https://github.com/mrc-ide/covid-sim/blob/master/data/admin_...
There are hundreds of constants in there. A lot of them appear to be wild-ass guesses. Presumably, all of them affect the output of the model in some way.
When a model has enough parameters for which you can make unsubstantiated guesses, you have a ton of wiggle room to generate whatever particular output you want. I'd like to see policy and public discussion focus more on the key parameters (R-naught, hospitalization rate, fatality rate) and less on overly-sophisticated models.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#34Absolute cringe, this was hard to read. Seems like a blatant shift of responsibility. At what point do people have to take responsibility and stand by their creations, regardless of what has been spoon-fed to them?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#35This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?
The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…
Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health?
What could go wrong?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#36I think the reason I find this so objectionable is that he's focused on a tool for doing the job of software engineering, and then blamed software engineers for not making it easy for him.
There's a spot-on comment on there at the moment - this isn't the car that's ready to use by anyone with a license. This is the welding gear we use to put bits of car together. You walked into the machine shop here, used some of our tools and made something that wasn't safe.
If scientists are not software engineers, and do not have the time or the motivation to become software engineers, perhaps they shouldn't play at being software engineers?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#37This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?
The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…
That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones.
The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#38Earlier quoted context omitted.
The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best…
Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?
If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#39Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge problem in academia. The problems with the Ferguson model and codebase are just the latest prominent example of this.
The tools that academics have access to are excellent. If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. I think the OP is wrong to suggest otherwise.
The problems with the Ferguson model are an opportunity to educate more members of the community about the fact that good software development practices are not optional for good science, and that senior members of the community jeopardize their own reputation by not paying attention to them.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#40Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…
I'm not sure I would describe R as "easier" than C++ tbh.