Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

151–160 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#151

Post by mcv »

chrisseaton wrote: post #135
csours wrote: post #104

Earlier quoted context omitted.

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.

If the results aren't reproducible, they can't be assumed to be true. Then they're only useful if you only care about publication and not about whether the results are actually true.

And yes, this is a serious problem in science.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#152

Post by kwhitefoot »

silveraxe93 wrote: post #41
rjtobin wrote: post #19

I'm somewhat skeptical. Firstly, I don't think the language is the problem with scientific code. You can write messy code in any language. So the warning then has to be about writing software in general. In that case, I think a warning like "don't try to write software unless you have years of training" is a bit much. Many people with no training learn to write nice code. Many projects made by amateurs might have ugl…

This model didn't just influence decisions in healthcare. It single-handedly changed the UK government's strategy over this pandemic. From what I understand the UK was planning on beating COVID by creating herd immunity, similarly to Sweden. Then this model came out and everyone started yelling that Boris wanted to kill your grandma. The problem is that it's impossible to have an intelligent discussion over this. Thi…

> creating herd immunity, similarly to Sweden. > ... > The problem is that it's impossible to have an intelligent discussion over this.

As far as I can tell the Swedish government never had this plan. It was mentioned in an interview and dismissed as unworkable, journalists misunderstood.

On the other hand the UK government appears to have had no plans whatever until jolted into action by the fear that public opinion would turn against them.

What Sweden has done is similar to Norway, where I live, which relies largely on voluntary changes in behaviour and temporary closure of institutions and businesses that require close contact between employees and customers. But Sweden took longer to implement those measures and also Swedish society is different from Norway, anecdotally Swedes seem to me to be more urban people than Norwegians and more gregarious.

Exactly why Sweden has a much higher death rate, 36/100k inhabitants versus 4.3/100k in Norway, is unclear at the moment partly because of different definitions but also because of differing conditions, and the epidemic being at different stages in the two countries.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#153

Post by rainforest »

datastoat wrote: post #139
wbhart wrote: post #42

Earlier quoted context omitted.

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong. Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll gi…

I would pick a value of R that shows itself to have good predictive accuracy. The way to test predictive models is always to look for their predictive accuracy on holdout data. Machine learning has this ingrained. Classic statistics does this too -- AIC is used to compare models, and it's (asymptotically) leave-one-out cross validation [1]. There's nothing intrinsically wrong with models that have millions of paramet…

The model isn't predictive though - it's a simulator. If we'd waited until we had enough data to make predictions with it (which I doubt you could given the sheer number of parameters) it'd be too late to use any of the interventions.

How would you ethically collect training data for the interventions?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#154

Post by rainforest »

chrisseaton wrote: post #135
csours wrote: post #104

Earlier quoted context omitted.

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]" 0 - https://en.wikipedia.org/wiki/Replication_crisis

> Without those qualities, your code is brittle, your deploys are brittle, changes are brittle. Is this brittleness stopping the scientists achieving what they need to achieve? Are you sure that writing tests makes science better? Or are you just assuming that? They aren't idiots and they aren't ignorant of how professional software developers work.

Move fast and break things has its downsides: work that influenced public policy/political debate was retracted after an error in the model was discovered [1], errors have been found in widely cited databases [2]. These are things that should be avoided I'm sure you'll agree.

Science has this additional problem that its memory is short - mistakes seem to be discovered when work is in the long tail of the citation curve, once it's out of the news. Even if you retract a paper, there is no easy way to trace the contagion to the work that uses it. That's before you consider mistakes that might be deliberate [3].

I have no doubts that more software/data rigour would make science more accurate, but the cost would be substantial, and it would no doubt slow down discovery until the benefit of open source kicked in.

[1] : https://theconversation.com/the-reinhart-rogoff-error-or-how... [2] : https://www.the-scientist.com/news-opinion/mistaken-identiti... [3] : https://retractionwatch.com/2016/09/26/yes-power-pose-study-...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#155

Post by web007 »

lucideer wrote: post #43
silveraxe93 wrote: post #14

I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

If you write software for your research then yes.

Just as I would expect you to have a solid basis in statistics, you need more than a 101-level competency in programming / software engineering if that's what you use to do your job. Same goes no matter what tools you're using, you need something beyond basic competence or advice from someone who does.

Nothing that I do day-to-day is remotely like surgery or in any way healthcare related, so that's a stupid straw man. Asking the same question of the academic _studying healthcare_ would be more apropos, but equally stupid and nonsensical.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#156

Post by silveraxe93 »

lucideer wrote: post #149
silveraxe93 wrote: post #128

Earlier quoted context omitted.

I guess we are in agreement then ;) If academics are writing predictive modelling software, I ask that they are qualified to do so.

> Where in my comment did you see me proposing that "being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)" > I ask that they are qualified to do so. You need to make up your mind on this one. Are you asking this or aren't you?

Being -- capable of writing quality software -- should be a base requirement for -- Anyone developing software used for academic research -- in any field -- in which the conclusions are based upon the correctness of this software.

Not all academic research depends on writing software.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#157

Post by Nursie »

commandlinefan wrote: post #84
llarsson wrote: post #6

Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…

> it is exceedingly hard to know that there are no mistakes But that's true of all software, no matter how well engineered. Usually when we discuss issues of code quality, we're looking at long-term maintainability and overall efficiency - that is, targets that are achievable. I'd love to see some effort toward quality metrics around provability of correctness, but I'm not even sure that's possible.

Without formal provability, yes, it's impossible to know there are no mistakes. Even then as most formal methods appear to operate on rules and requirements rather than on the actual code, it's still not possible.

However - with well formatted code that's had a review process of some sort, it's going to be easier to cut down on the glaring errors even if there are less obvious problems lurking.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#158

Post by mcv »

rjtobin wrote: post #19

I'm somewhat skeptical. Firstly, I don't think the language is the problem with scientific code. You can write messy code in any language. So the warning then has to be about writing software in general. In that case, I think a warning like "don't try to write software unless you have years of training" is a bit much. Many people with no training learn to write nice code. Many projects made by amateurs might have ugl…

Yeah, requiring years of training to write anything is nonsense. Everybody should learn to write code, and there's tons of interesting stuff you can do without knowing software engineering best-practices. Not every scientific model has to scale to industrial scale or be maintainable by many people over many years.

My problem is entirely with the article that blames the tool they chose and the software engineering community that didn't put big warning stickers on that tool.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#159

Post by mcv »

hpoe wrote: post #26
rjtobin wrote: post #19

I'm somewhat skeptical. Firstly, I don't think the language is the problem with scientific code. You can write messy code in any language. So the warning then has to be about writing software in general. In that case, I think a warning like "don't try to write software unless you have years of training" is a bit much. Many people with no training learn to write nice code. Many projects made by amateurs might have ugl…

I'd like to point out I've met a lot of software engineers that subscribe to two in reverse. "This domain area of expertise is much easier than programming, ergo I am qualified to solve it."

I think it's true of many experts, that they see their own area of expertise as the most important one, and the others as relatively minor.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#160

Post by mcv »

dirtydroog wrote: post #4
cheerlessbog wrote: post #2

The author makes it sound like a warning label is missing from the C++ tin. Maybe. But what tool should he have used? I haven't seen this code but is there any doubt it would look just as bad in Java or Python, maybe with fewer segfaults? Or FORTRAN.

I'm genuinely surprised it wasn't an Excel spreadsheet.

I haven't seen the code, but from what I read, it may well have been better to have done this in an Excel spreadsheet.

Of course we all would have been just as horrified at that.

Post reply on HN