Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To software engineers criticizing Neil Ferguson’s epidemics simulation code

blog.khinsen.net

Post reply on HN
Print view View original

81–90 of 189 posts

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#81

Post by rainforest »

munchbunny wrote: post #15
htk wrote: post #3

I get what he’s saying, but the “non-experts” using code to represent their models should also let outside help come when the need arises. When your model predicts an apocalyptical scenario and government is taking drastic measures based on it, it’s a good time to expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look.

should also let outside help come when the need arises. expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look. 1. Software engineers are expensive. Hiring them to write your code is how you end up needing even more money to do your science, and I think the software engineering world if anything can appreciate prioritizing being scrappy to get more done.…

I believe this model is quite old (at least in some form) so there have been opportunities to review it. I confess I haven't looked, but I haven't heard any defects have actually been identified. If the model stood up to peer review I assume the results it produces are at least consistent with the expectations of the people who wrote the mathematical model.

Hopefully this will be a watershed moment that makes it easier to cost a research software engineer onto a grant in the future.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#82

Post by TeMPOraL »

pbourke wrote: post #64
lucideer wrote: post #43

Earlier quoted context omitted.

This is such a ridiculous comment I don't know where to start. Are you actually proposing that being a fully experienced & knowledgeable software engineer should be a base requirement for all academic research (in any field)? If you, working as a software engineer, were told tomorrow by your manager that you needed to perform heart surgery, and that that now fell under your responsibilities in your current role, woul…

> Funding dictates this is impossible, so they make do. Put another way: software quality is not valued in academia. Let’s say the code in question was in great shape. Would it have mattered at all in the trajectory of anyone’s career? No. This is probably the one academic code base in a million that has received any negative reputational hit due to its quality.

> software quality is not valued in academia.

And in general, it shouldn't. Code written by scientists is almost always throwaway prototypes. So it absolutely doesn't matter how decoupled, or easy to extend it is. The only aspect of quality that matters here is the simplicity, in the sense "it obviously has no bugs" (and related practices like testing). This is something that could be taught, but I think it should be also addressed from the other end of the spectrum - more pressure put on verification and reproducibility of results.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#83

Post by CharlesW »

wbhart wrote: post #42
tkiley wrote: post #33

I poked at the github repo for a bit. The ugliness of the code doesn't bother me, but the quantity of parameters does. Here's one params file that specifies some of the inputs to a run of the model: https://github.com/mrc-ide/covid-sim/blob/master/data/param_... Here's another one: https://github.com/mrc-ide/covid-sim/blob/master/data/admin_... There are hundreds of constants in there. A lot of them appear to be wild…

The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong. Unless all important factors are accounted for, they are going to result in incorrect information for someone. Public policy will then be based on incorrect predictions. People will grow tired of the predictions being wrong and they'll gi…

> The problem is, unsophisticated models do not predict anything. You apply them in one country and they do ok, and apply them in another and they get it totally and completely wrong.

That's the nature of all models, "sophisticated" or not. Relatively simple models may or may not be useful for a particular case, just as relative complex models may be.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#84

Post by commandlinefan »

llarsson wrote: post #6

Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…

> it is exceedingly hard to know that there are no mistakes

But that's true of all software, no matter how well engineered. Usually when we discuss issues of code quality, we're looking at long-term maintainability and overall efficiency - that is, targets that are achievable. I'd love to see some effort toward quality metrics around provability of correctness, but I'm not even sure that's possible.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#87

Post by chrisseaton »

ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability.

Or maybe these things aren't actually as important as we think they are in professional software development?

If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#88

Post by tensor »

w0utert wrote: post #66
llarsson wrote: post #6

Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…

Exactly. The C++ code in the GitHub repo is absolutely frightening, but I'm 100% sure if would be pretty much the same in e.g. Python, probaly even worse. From my own experience code quality is inversely related to barrier of entry, which means I see a lot more terrible MATLAB and Python code than terrible C++ code. The only conclusion you can draw from a repo like this, is the conclusion I've drawn countless times w…

How do you propose identifying these "skilled software engineers"? University doesn't teach "professional software engineering" so all these hotshots coming out of school into industry can't be what you're looking for.

There is no standards body defining the skills one needs to be considered professional. There is no responsibility on practitioners unlike other engineering professions. If a mechanical engineer screws up and causes death, it's going to be bad for them. Software routinely screws up with no consequence.

Hell, take two "software engineers" from FAANG and give them a piece of code and they won't agree on whether it's good or bad.

None of this is to excuse poorly written scientific code, but if "professional software engineers" want to throw rocks, maybe they should fix their own glass house first.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#89

Post by disgruntledphd2 »

zozbot234 wrote: post #40
llarsson wrote: post #6

Not a comment on the specific repo in question, but I just want to note that I have seen utter monstrosities of academic code written in Python, MATLAB, and R - languages that are ostensibly "easier" than C++. I so not think that poor code quality is due to the many footguns C++ admittedly gives you. I am sure that the main research is not in the implemented code. But with unclear code, it is exceedingly hard to know…

I'm not sure I would describe R as "easier" than C++ tbh.

It's much more accessible, as you need to know less to use it.

Consider - using Eigen in a C++ project, or calling install.packages("eigen") in R.

Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code

#90

Post by silveraxe93 »

ak217 wrote: post #39

Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…

I couldn't agree more. We're in the middle of a reproducibility crisis. It's extremely important that researchers start putting more effort into quality research instead of quantity. "Publish or Perish" is killing academia.
Post reply on HN