All models are wrong, but some are completely wrong • Hacker News Board

Re: All models are wrong, but some are completely wrong

#51

Post by spaceman_2020 » Sun, Apr 05, 2020, 7:33 PM UTC

heurifk wrote: post #22

puranjay wrote: post #10

I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or…

We have recent data from Wuhan. We have old data from 1918. This is a white swan, not a black one. More importantly, the mortality, while much higher than flu, it's still relatively low. Now imagine a virus as contagious as this one, but with 10% mortality over all age groups. That would be unprecedented and probably cause society meltdown.

How do you reckon social media will impact your models? Rumors and information - real or fake - has never been easier to spread in the world (Wuhan's data becomes less relevant here). We've already seen calls for defiance of lockdowns in right wing circles. In my country, old videos are being circulated to spread misinformation about the treatment.

Again, our data for all older epidemics is applicable to the epidemic in isolation. But there is no way to accurately model how the epidemic interacts with people simply because the way people live has changed drastically from past epidemics.

Re: All models are wrong, but some are completely wrong

#52

Post by darkerside » Sun, Apr 05, 2020, 7:38 PM UTC

tartoran wrote: post #48

hinkley wrote: post #29

Earlier quoted context omitted.

It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept , people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hil…

The low hanging fruit metaphor meaning do the simplest or easiest work first is not really applicable to fruit growing but seems to have been coopted by the busieness world instead, then we started using it too in our contexts. Not crazy about this metaphor either, but it could be useful in some contexts and while also misunderstood in others. The same goes for most jargon.

Low hanging fruit is a point in time tactic, not a generalized recommendation

Re: All models are wrong, but some are completely wrong

#53

Post by kgwgk » Sun, Apr 05, 2020, 7:41 PM UTC

> The FT chose to run with an inflammatory headline, assuming an extreme value of ρ that most researchers consider highly implausible.

The headline "Coronavirus may have infected half of UK population — Oxford study” is not really out of line with the preprint the article talks about. What's questionable is that it was a good idea to write about the preprint at all.

"Importantly, the results we present here suggest the ongoing epidemics in the UK and Italy started at least a month before the first reported death and have already led to the accumulation of significant levels of herd immunity in both countries."

"Our overall approach rests on the assumption that only a very small proportion of the population is at risk of hospitalisable illness. [...] Three different scenarios under which the model closely reproduces the reported death counts in the UK up to 19/03/2020 are presented in Figure 1 . [...] [In two of those scenearios] By 19/03/2020, approximately 36% (R 0 =2.25) and 40% (R 0 =2.75) of the population would have already been exposed to SARS-CoV-2. [...] [The third scenario] suggests that 68% would have been infected by 19/03/2020."

The secondary headline “New epidemiological model shows vast majority of people suffer little or no illness” was much worse as that's the assumption in the model and not the result. It was changed to "New epidemiological model shows urgent need for large-scale testing" in the amended article.

> Since its publication, hundreds of scientists have attacked the work, forcing the original authors to state publicly that they were not trying to make a forecast at all.

"Attacked" sounds as if the criticism was unwarranted.

Re: All models are wrong, but some are completely wrong

#54

Post by galaxyLogic » Sun, Apr 05, 2020, 7:42 PM UTC

Maybe the best thing about models is not the numbers they produce but the insight into how different variables affect each other, how steeply.

But this information is already contained in the model itself. Therefore people should be reading the models, not their results.

The models should of course be verified by comparing their results to empirical data. But that does not often exist with global things like pandemics and climtae change.

Re: All models are wrong, but some are completely wrong

#55

Post by discreteevent » Sun, Apr 05, 2020, 7:43 PM UTC

hinkley wrote: post #29

grawprog wrote: post #9

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not suppo…

It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept , people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hil…

I doubt many people think of it from the perspective of a fruit grower. I think of it as picking wild fruit, in which case the analogy mostly works.

Re: All models are wrong, but some are completely wrong

#56

Post by rossdavidh » Sun, Apr 05, 2020, 7:43 PM UTC

So, I basically agree with everything this article says, but it seems to miss a basic point. If journalists do what this paper suggests, they make less money.

Journalists, and the newsmedia corporations and organizations that employ them, don't run with the most inflammatory headline possible as an accidental fluke of a mistake that they were too careless to catch. Even public sector newsmedia organizations use measures of how widely read their articles are, as a figure of merit to how well they are performing. Private sector newsmedia are rewarded financially in more or less direct proportion to how widely read (or at least clicked on) their articles are, not how well informed the reader is after they're done reading it (if they even do read past the headline).

If there is one less to be learned from this whole Covid-19 debacle (and I'm sure there are several), it is that our entire news ecosystem, public and private, is fundamentally structured wrong for doing what is supposed to be its purpose, which is to make people better informed. It's not bad at it by mistake, it's bad at it as an inevitable consequence of its design.

Re: All models are wrong, but some are completely wrong

#57

Post by yk » Sun, Apr 05, 2020, 7:47 PM UTC

puranjay wrote: post #10

I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or…

If you look at the models, they have something like an order of magnitude uncertainty. And the reason for that is precisely what you are stating, they rely on future behavior, which we just don't know yet.

However, the utility of the models is to give us a sense of how the different parameters interact. There are parts of the model were we can have a lot of trust, for example that people with severe conditions will need hospitalization, or that people will react quite similar as they reacted yesterday. So for the short term, the models give us quite good guidance, and for the long term, they help to map out scenarios.

So if you actually look at the report in question, you will see that they are actually trying to estimate the impact of various non medical interventions, like encouraging social distancing, by comparing different countries. It is just that newspapers as usual just run with the most immediately digestible number, independent wether that number is important or useful.

The study in question:

https://www.imperial.ac.uk/media/imperial-college/medicine/m...

Some overview video from Dr. Campbell on youtube: (in general, I think his youtube channel is quite good)

https://www.youtube.com/watch?v=c1aoULlMpn0

Re: All models are wrong, but some are completely wrong

#58

Post by oivey » Sun, Apr 05, 2020, 7:50 PM UTC

YZF wrote: post #14

martingoodson wrote: post #3

Author here: happy to take comments or criticism

I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.

Don’t discount domain experts. Models are meant to predict the real world. In order for them to be accurate, the model itself needs to capture how the real world works, and the math underlying the model has to be correct. Domain experts are the most likely to have experience in both areas. Drawing an example, a physicist models the universe and knows the math and model behind electromagnetism. A mathematician probably knows the math but maybe not the model.

Re: All models are wrong, but some are completely wrong

#59

Post by not2b » Sun, Apr 05, 2020, 8:00 PM UTC

guscost wrote: post #50

martingoodson wrote: post #43

Earlier quoted context omitted.

I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.

You will have nothing to add if the “2.2 million deaths in the US” scenario, which was blasted across every newspaper front page a few weeks ago, turns out to have been impossible all along? If that scenario was “completely wrong” too, it seems like it would serve as a perfect example of the consequences of this kind of (still hypothetical) misinformation.

We can't rerun the experiment with a control version of the US in which nothing was shut down and we continued to have crowded sports events and night life. So it won't be possible to determine that the 2.2 million deaths scenario is impossible, especially if it's interpreted as 2.2 million extra deaths from either COVID-19 or other causes that could have been treatable by a medical system that wasn't completely overwhelmed.

Re: All models are wrong, but some are completely wrong

#60

Post by zero_shift » Sun, Apr 05, 2020, 8:01 PM UTC

rossdavidh wrote: post #56

So, I basically agree with everything this article says, but it seems to miss a basic point. If journalists do what this paper suggests, they make less money. Journalists, and the newsmedia corporations and organizations that employ them, don't run with the most inflammatory headline possible as an accidental fluke of a mistake that they were too careless to catch. Even public sector newsmedia organizations use measu…

A small (but important) correction: generally, journalists do not write headlines; sub-editors do. Most subs have been journalists but most journalists do not become subs.