Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

101–110 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#101

Post by zozbot234 »

chrisseaton wrote: post #87
ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

> Or maybe these things aren't actually as important as we think they are in professional software development?

There's something to this. Most academic software is a simple one-off development, with little consideration for long-term maintenance. OTOH it's not likely that undocumented, untested and unfixable software can produce actual "useful scientific results".

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#102

Post by lucideer »

mcv wrote: post #98
lucideer wrote: post #43

Earlier quoted context omitted.

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

> "The private sector's typical response to the problems in the article would be to hire qualified software engineers to assist researchers. Funding dictates this is impossible, so they make do." But that is a bizarre conclusion. Is it really cheaper to let scientists focus on something they know nothing about, than to hire an expert to do it? Do you cut costs by letting scientists mop the floor, clean the toilet and…

> Is it really cheaper to let scientists focus on something they know nothing about, than to hire an expert to do it? Do you cut costs by letting scientists mop the floor, clean the toilet and guard the entrance?

Believe it or not, mean software engineer salaries are marginally higher than those of janitors and door security.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#103

Post by Majromax »

enitihas wrote: post #53
Majromax wrote: post #28

Earlier quoted context omitted.

> Any such model ought to have been independently reviewed before it is ever used for real policy decisions. Policy analysis is awash in models but no one ever really checks them. Going forward, health policy makers should ask for and disclose independent validation of any model before using its results to make recommendations of any consequence. That's ignoring the time-limited nature of a virus response. "No decisi…

The quickest decision is random number generation. Should people go with that? Governments across the world had loads and loads of time. > Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/…

>If you have a once in a century crisis, and still are thinking about saving some millions, while being sure your crisis response will be above billions, your policy is already flawed and not much can be done to help you.

By the time you're in the crisis, it's too late to make software bulletproof. Not only does that process take time, but it also needs to be integrated with the whole of the research effort up to that point.

So you can't pick and choose topics: you need the funding to make bulletproof every potential policy-related model you might need in an emergency. That's where the costs add up, since we don't have a time machine to go back and fund exactly the lines of research we in fact need at the moment.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#104

Post by csours »

chrisseaton wrote: post #87
ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. Or maybe these things aren't actually as important as we think they are in professional software development? If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle.

It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]"

0 - https://en.wikipedia.org/wiki/Replication_crisis

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#105

Post by ookdatnog »

silveraxe93 wrote: post #14

I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…

> If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours.

It certainly _can_ be the manufacturer's fault, if the manufacturer's salespeople lied to you about the learning required to master the tool.

I think the point of the article is that collectively, as a community of experts, we have been misguiding non-experts into thinking that C++ is a good tool for their job.

A non-expert cannot make an informed decision on which tool to use, they _have_ to rely on advice from experts to pick their first tool, so if they feel like we're recommending C++ (and I agree with the author that this is not a good tool for non-experts), then that's, in a sense, on us.

(That being said I did not inspect the code, so I don't know if the author's implication is right that C++ is the core problem of the code base)

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#106

Post by lucideer »

silveraxe93 wrote: post #69
lucideer wrote: post #43

Earlier quoted context omitted.

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

Where in my comment did you see me proposing that "being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)" Go back and re-read it please. I'm saying you need to be capable of using the tools for your job. If I was working as a software engineer, I'd not be asked to perform heart surgery. A more realistic example would be a software engineer be…

Sorry but there's a pretty big gap between understanding the domain for which your software is intended, and being qualified to work as a professional in that domain. The latter is what you're asking of academics.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#107

Post by veddox »

Well, it looks like one of the biggest problems in computational biology has finally blown up in our face.

I work in a very similar field to the one discussed here - ecosystem modelling. And much of the code I see is probably of a similar quality to Neil Ferguson's model. (Although I haven't had a detailed look at his work.) For all you angry devs out there, I have three comments:

1. Yes, we have a problem with code quality, and no, this should not be acceptable where policy-relevant decisions of such magnitude are concerned. (Although one should bear in mind that all science is flawed and its achieved reliability will always be limited by constraints of time/budget/etc.)

2. However, changing the status quo is really hard. Specifically, the two greatest changes that are needed are to teach proper software development practices to students in the natural sciences (just like we teach lab technique), and/or make it easier to get funding for software developer positions in a research team. This sounds easy, and ought to be easy, but both run counter to some pretty entrenched ideas held by the "old folks at the top" in universities and funding bodies. (Believe me, I'm speaking from experience :-/ )

3. But yes, we are working on it. The debate is growing, and there's a new generation of computational biologists with much closer ties to computer science who are trying to shake things up a bit. In some sense, we are several decades behind the wider CS world in the techniques we use, but we're catching up. And at the same time, we're starting to develop some of our own methods of quality control (such as pattern-oriented modelling).

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#108

Post by zanew101 »

w0utert wrote: post #66
llarsson wrote: post #6

Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…

Exactly. The C++ code in the GitHub repo is absolutely frightening, but I'm 100% sure if would be pretty much the same in e.g. Python, probaly even worse. From my own experience code quality is inversely related to barrier of entry, which means I see a lot more terrible MATLAB and Python code than terrible C++ code. The only conclusion you can draw from a repo like this, is the conclusion I've drawn countless times w…

I work in an research setting with access to lots of skilled software engineers and I've come to a very similar conclusion. The problem I often see is that there is very little incentive for the researcher to bring a software engineer onto their project unless it is absolutely necessary. It is just now becoming commonplace to release code when you publish and plenty of researchers still don't.

If you think no one is going to see your code then it makes it much harder to care about its quality. One way we've been fighting this internally is by trying to get researchers less silo'd and more open to code reviews. Once a code review is part of the process then bringing on an "expert coder" to address some of the issues that come up in the reviews has more tangible results! We've seen some success in improving our code quality with this strategy.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#109

Post by veddox »

veddox wrote: post #107

Well, it looks like one of the biggest problems in computational biology has finally blown up in our face. I work in a very similar field to the one discussed here - ecosystem modelling. And much of the code I see is probably of a similar quality to Neil Ferguson's model. (Although I haven't had a detailed look at his work.) For all you angry devs out there, I have three comments: 1. Yes, we have a problem with code…

If you're interested, here is some further reading:

* DeAngelis, D. L., & Grimm, V. (2014). Individual-based models in ecology after four decades. F1000prime Reports, 6(June), 39. https://doi.org/10.12703/P6-39

* Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221, 2760–2768. https://doi.org/10.1016/j.ecolmodel.2010.08.019

* Grimm, V., & Railsback, S. F. (2011). Pattern-oriented modelling: A ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society B, 367(1586), 298–310. https://doi.org/10.1098/rstb.2011.0180

* Nowogrodzki, A. (2019). How to support open-source software and stay sane. Nature, 571(7763), 133–134. https://doi.org/10.1038/d41586-019-02046-0

* Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2016). Good Enough Practices in Scientific Computing. 1–30. http://arxiv.org/abs/1609.00037

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#110

Post by fsh »

rjsw wrote: post #74
veddox wrote: post #52

Earlier quoted context omitted.

> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. Could you elaborate on what you mean by that?

Not the OP, but I know of at least one University that runs "Programming for scientists" courses at low or no cost.

These courses are usually short introductions into python or Java. Most scientists simply don't have time in their curriculum for a full software engineering course.
Post reply on HN