Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

91–100 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#91

Post by w0utert »

andi999 wrote: post #45
htk wrote: post #3

I get what he’s saying, but the “non-experts” using code to represent their models should also let outside help come when the need arises. When your model predicts an apocalyptical scenario and government is taking drastic measures based on it, it’s a good time to expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look.

There is probably no funding for outside help... So they get a project fund a PhD student (or post doc) with that, this guy has to produce new results (fast) and more results, then these guys leave, next guys come.. Generally for a simulation group you would need an experienced programmer maintaining the code base, (helping) the other guys cleaning up there ideas before handover, and this as a permanent position. Exp…

I find this a pretty weak excuse for a high-impact epidemiological model like this. Universities are full of the brightest people you will find anywhere in society, and most universities have faculties and courses in computer science and/or software engineering. How about enforcing multi-disciplinary efforts, having computer science students work with microbiologists, and vice versa? Exchange programs between universities that specialize in one or the other?

University graduates need to do internships and a thesis anyway, so in terms of 'funding' you could argue they are basically 'free' in terms of assigning them to multi-disciplinary research.

I know I would have been delighted if someone at my university had offered me an interesting multi-disciplinary microbiology project when I was studying computer science. Instead we mostly just got boring run-of-the-mill exercises in parroting established computer-science theory...

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#92

Post by asddsfgdfsh »

tkiley wrote: post #79
wbhart wrote: post #42

Earlier quoted context omitted.

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong. Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll gi…

"But what value of R would you pick?" I don't know -- and until we can agree on the answer to your simple question with a high degree of confidence, I think complex models based on specific assumed values of R obscure more than they reveal. A little bit of modeling is useful because humans are intuitively bad at exponential math and we need scary graphs to jolt us awake sometimes. But when we don't even know the basi…

We have a model, we can run some sensitivity analysis, then we can go out and collect data to better estimate the parameters to which we are sensitive to. Important but not glamorous work and hence underfunded.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#93

Post by disgruntledphd2 »

motohagiography wrote: post #59

I'm trying to be charitable on this code base issue, and the institutions will need to accept they need to be able to bear more scrutiny on the inputs to policy recommendations. Both R and Python are taught in highschool and undergraduate courses for scientific data analysis. Ferguson and other scientists did not need software engineers. A free co-op student, or to have spent the three weeks learning a language that…

I honestly don't think R and Python would have worked here.

While I was able to clone the repo and run it for Ireland, the UK requires at least 26Gb of RAM, which is not common on most personal computers. The US requires much, much more.

And given that it's pretty slow when written in C++, imagine how slow it would have been in R or Python?

I agree with you in principle with respect to this stuff being better, but the incentives skew otherwise at the moment.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#94

Post by lalaithion »

chrisseaton wrote: post #87
ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

The problem is, we need confidence that the software is correct in order to trust the scientific results.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#95

Post by Mikhail_K »

motohagiography wrote: post #59

I'm trying to be charitable on this code base issue, and the institutions will need to accept they need to be able to bear more scrutiny on the inputs to policy recommendations. Both R and Python are taught in highschool and undergraduate courses for scientific data analysis. Ferguson and other scientists did not need software engineers. A free co-op student, or to have spent the three weeks learning a language that…

Python is unsuitable for writing Monte-Carlo simulations due to its abysmal performance and built-in unfixable inability for parallelism. In addition, it is not a good language in general.

Complex simulations should be written in multiple-dispatch high-perfomance languages like Fortran or Julia.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#97

Post by klyrs »

SpicyLemonZest wrote: post #38
rscho wrote: post #35

Earlier quoted context omitted.

Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?

If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.

> it seems only fair to say that academics optimize for prestige and publication in good journals.

On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time.

On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles on the basis of their veracity (or, if incorrect, as punching bags). Of course that's a fallible process and just as vulnerable to cronyism, but when researchers are caught cooking the books they're discredited, not rewarded.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#98

Post by mcv »

lucideer wrote: post #43
silveraxe93 wrote: post #14

I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

> "The private sector's typical response to the problems in the article would be to hire qualified software engineers to assist researchers. Funding dictates this is impossible, so they make do."

But that is a bizarre conclusion. Is it really cheaper to let scientists focus on something they know nothing about, than to hire an expert to do it? Do you cut costs by letting scientists mop the floor, clean the toilet and guard the entrance?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#99

Post by Nursie »

disgruntledphd2 wrote: post #93
motohagiography wrote: post #59

I'm trying to be charitable on this code base issue, and the institutions will need to accept they need to be able to bear more scrutiny on the inputs to policy recommendations. Both R and Python are taught in highschool and undergraduate courses for scientific data analysis. Ferguson and other scientists did not need software engineers. A free co-op student, or to have spent the three weeks learning a language that…

I honestly don't think R and Python would have worked here. While I was able to clone the repo and run it for Ireland, the UK requires at least 26Gb of RAM, which is not common on most personal computers. The US requires much, much more. And given that it's pretty slow when written in C++, imagine how slow it would have been in R or Python? I agree with you in principle with respect to this stuff being better, but th…

Given we're talking about something that it seems people say does not have good code quality, I would guess that whether the C++ implementation is more efficient due to language choice would be a wide open question.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#100

Post by mcv »

TeMPOraL wrote: post #82
pbourke wrote: post #64

Earlier quoted context omitted.

> Funding dictates this is impossible, so they make do. Put another way: software quality is not valued in academia. Let’s say the code in question was in great shape. Would it have mattered at all in the trajectory of anyone’s career? No. This is probably the one academic code base in a million that has received any negative reputational hit due to its quality.

> software quality is not valued in academia. And in general, it shouldn't. Code written by scientists is almost always throwaway prototypes. So it absolutely doesn't matter how decoupled, or easy to extend it is. The only aspect of quality that matters here is the simplicity, in the sense "it obviously has no bugs" (and related practices like testing). This is something that could be taught, but I think it should be…

It doesn't have to be decoupled or easy to extend, but it should be readable enough to ensure it does what it's supposed to, and well-tested enough to verify it does what it's supposed to.
Post reply on HN