61–70 of 82 posts
Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.
I'll be "that guy" and point out we know the climate simulations have similar issues.
Earlier quoted context omitted.
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors... Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions. The worst pr…
If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
Earlier quoted context omitted.
I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team. Given your concerns, I would like to mention two things: 1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfec…
If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release". The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.
Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.
Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.
I'll be "that guy" and point out we know the climate simulations have similar issues.
Earlier quoted context omitted.
If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome. Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read b…
Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear.
To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
If this has been flagged off the front page I despair for HN. This should be an apex story for us: open source release of a computer model that has incredible relevance to loads of people globally. Quite sad.
Earlier quoted context omitted.
Yep. Also, speaking from academic point of view, code won’t make you advance on your career by itself (only papers based on it will), so the minimum is done until the thing is usable for its goal. So forget about CI/CD, proper test suits, documentation (I hate this point personally and document my projects a bit more than the average researcher) and engineering best practices.
Then public policy should heavily discount results based on academic code, if it shown to be poorly engineered. I'd go further and say you can't trust the papers based on the results of badly engineered simulations. As a poster said earlier "I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor…
This code is great fun. Start here to explore the horror: https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi... I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing. But it gets worse - the pointer…
I've seen a lot of code coming out of academia that exhibits a similar complete lack of structure and completely ignores the most basic software development best practices (such as having tests). There are exceptions, and research software engineers are a thing, but unsurprisingly most academics focus their careers on the science rather than the code.