Everything is fine. You get what you pay for.
To software engineers criticizing Neil Ferguson’s epidemics simulation code
51–60 of 189 posts
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#52Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge prob…
> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great.
Could you elaborate on what you mean by that?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#53this is quite off the mark. the author sets the bar too low for himself by criticizing the most easily (and to be fair legitimately) dismissed criticisms of the Imperial College model. here's a better laid-out critique that the OP doesn't speak to: The Imperial College modelers released the source code a couple of days ago to the model that shut down the world economy. It's not the original model code but was rather…
> Any such model ought to have been independently reviewed before it is ever used for real policy decisions. Policy analysis is awash in models but no one ever really checks them. Going forward, health policy makers should ask for and disclose independent validation of any model before using its results to make recommendations of any consequence. That's ignoring the time-limited nature of a virus response. "No decisi…
The quickest decision is random number generation. Should people go with that? Governments across the world had loads and loads of time.
> Adding research capacity to ensure that governments have a stable of well-researched, thoroughly-vetted models for emergencies would be a great thing, but it would also be quite expensive. Keeping any single model up to spec might be the job of 1FTE (so an additional ~$150k/grant/yr -- large for a research budget but small for a government), but that would have to be multiplied by every area where the government might possibly want research-informed decisionmaking on short notice.
If you have a once in a century crisis, and still are thinking about saving some millions, while being sure your crisis response will be above billions, your policy is already flawed and not much can be done to help you.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#54Laughable nonsense, and especially glib when there are lives on the line due to the question of competence/lack of it.
One thing I'm surprised not to have seen yet - as a MSc Geographic Information Science grad - is criticism of the methdology from a GI science aspect. And the question that I have is: where is the proof that this model's results are in any way representative of reality? Surely Professor Ferguson - before releasing results that have the potential to turn economies upside down, and with thousands of lives at stake - would have run this against some real world data to validate the model. Or not? If not, why should we listen to anything he says ever again?
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#55That is a reasonable problem statement and proposed solution, but the problems with software engineering go very deep. That is why so many software projects fail outright. Just as we acknowledge that aviation engineering has different risks and constraints from basic tool building so we all need to understand that at least for now and likely for some time software development will be inherently messy and risky for all who dare it.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#56I chuckled when I saw the author of this article. Many years ago I was a scientist (PhD student) writing code in Python, but wanted to use features in Mathematica like symbolic integration. Turns out Mathematica has a C API that lets you send Math expressions to it and evaluate them. I wrote a clunky Python interface and shared it with Konrad Hinsen, who looked at it and suggested an elegant recursive object representation of Math objects in Python that led to automatic conversion between Python and Mathematica, massively simplifying the code and making it more elegant. I got slightly better at software engineering that day.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#57Earlier quoted context omitted.
should also let outside help come when the need arises. expose your “non-expert code” to the software engineering communities (and all other associated fields) to take a look. 1. Software engineers are expensive. Hiring them to write your code is how you end up needing even more money to do your science, and I think the software engineering world if anything can appreciate prioritizing being scrappy to get more done.…
I will personally pay for a professional review of the next model the British government uses to guide lockdowns. Cost problem solved.
While I appreciate the sentiment, we can probably both see why one person offering to solve the problem for one study isn't a solution to the systemic problem.
Private funding poses all sorts of issues. Surmountable issues, but still issues once individuals are paying for reviews at scale.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#58I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…
Well, even if, the tool doesn't come with a warning. Real-life tools come with them.
(The question is, what kind of warning would be useful on a programming language? C/C++ perhaps should come with a booklet about pointers, but other than that, I can't think of any but one: "WARNING: this is a tool for extreme clarification of thought; if you're confused about your subject matter, your project will produce wrong results. As you write code, take steps to verify your understanding of your problem domain.")
More importantly, our industry is spending a lot of time and marketing money on convincing that everyone can code, that it's just a matter of picking up a JavaScript tutorial and pressing F12 in the browser, and you too are now a software developer. These two views are in direct opposition to each other.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#59I'm trying to be charitable on this code base issue, and the institutions will need to accept they need to be able to bear more scrutiny on the inputs to policy recommendations.
Both R and Python are taught in highschool and undergraduate courses for scientific data analysis. Ferguson and other scientists did not need software engineers. A free co-op student, or to have spent the three weeks learning a language that met the needs of the level of abstraction he was working at would have sufficed.
The culture that enables that code as described to be acceptable for policymaking is one that intentionally produces complex black boxes that obfuscate risk and attribution, and to launder decision accountability through technology. I've seen this in other institutional code as well.
However, while the policy recommendations that resulted from his model may (or may not) have saved tens of thousands of lives, it did so at the risk of losing the credibility to do it again.
Deflecting blame to nebulous software engineers is disingenuous and serves mainly to exacerbate the suspicions of reasonable people, and further polarize those most harmed by the policy response.
Re: To software engineers criticizing Neil Ferguson’s epidemics simulation code
#60I disagree so strongly with this that I had a visceral reaction while reading it. C++ is a tool, not an end product. If you're not qualified to use a tool correctly it's not the manufacturer's fault, it's yours. Why do so many people believe that good software development is not part of their job? If you write code then you're a developer, no matter your job title. If you write shit software, saying you're a research…
Good software development is missing from most academic promotion tracks. "Research Software Engineers" are a recently new thing, which is an admission that academics are not very good at software engineering and supports the notion that they can't reasonably be expected to be experts in two disciplines.
There is a large amount of software being written by non-software engineers (most of it is Excel) so I think it's probably constructive to say that the solution lies elsewhere other than expecting academics to become software engineers. If that sort of requirement is in place it would be hard to argue against software engineering being a chartered engineering discipline with liabilities and qualifications which would be very disruptive to a lot of the industry.
It is worth remembering that academic code sharing is rare in and of itself. This reaction is likely to cause other academics to shy away from publishing their code. Perhaps giving them an opportunity to make their mistakes public so that more skilled engineers could help before the credibility of their results is used to justify drastic government policies would be better?