Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All models are wrong, but some are completely wrong

rssdss.design.blog

Post reply on HN
Print view View original

41–50 of 216 posts

Re: All models are wrong, but some are completely wrong

#41

Post by garmaine »

martingoodson wrote: post #16
YZF wrote: post #14

Earlier quoted context omitted.

I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.

I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.

Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand.

Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use.

The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.

Re: All models are wrong, but some are completely wrong

#43

Post by martingoodson »

guscost wrote: post #31
martingoodson wrote: post #3

Author here: happy to take comments or criticism

If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?

I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.

Re: All models are wrong, but some are completely wrong

#44

Post by carapace »

DoofusOfDeath wrote: post #17

Regarding the "all models are wrong" maxim... Is that statement 100% true for the low-level models that physicists use and develop? In particular, I'm curious if quantum-physics models are 100% right, just not 100% precise.

The Standard Model has no known contradictions in the real world. https://en.wikipedia.org/wiki/Standard_Model

In "QED" Feynman states that the predictions are as precise as being able to measure the distance between (points in) New York and Los Angeles accurately to within the width of a human hair.

https://en.wikipedia.org/wiki/QED:_The_Strange_Theory_of_Lig...

Re: All models are wrong, but some are completely wrong

#45

Post by svat »

martingoodson wrote: post #3

Author here: happy to take comments or criticism

This is a great start on how scientists (and journalists) should communicate to the public! Thanks for writing this, and the world would be a better place if everyone remembered to follow these principles.

I think in fact it would be better to go even further: not speak from a position of superiority (even if one knows more), but try to acknowledge the audience and persuade effectively. Here's a recent article on the topic: https://undark.org/2020/03/19/coronavirus-myths/ and here's one of my favourites (in a different field of science) from three years ago: https://deansforimpact.org/why-mythbusting-fails-a-guide-to-...

Re: All models are wrong, but some are completely wrong

#46

Post by martingoodson »

garmaine wrote: post #41
martingoodson wrote: post #16

Earlier quoted context omitted.

I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.

Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand. Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug.…

That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year."

https://thehill.com/changing-america/well-being/prevention-c...

Re: All models are wrong, but some are completely wrong

#47

Post by gadders »

martingoodson wrote: post #43
guscost wrote: post #31

Earlier quoted context omitted.

If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?

I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.

Why do you think the Oxford model compares poorly to the Imperial one? Both are created by eminent people in their field. Neither has been peer-reviewed.

Re: All models are wrong, but some are completely wrong

#48

Post by tartoran »

hinkley wrote: post #29
grawprog wrote: post #9

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not suppo…

It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept , people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hil…

The low hanging fruit metaphor meaning do the simplest or easiest work first is not really applicable to fruit growing but seems to have been coopted by the busieness world instead, then we started using it too in our contexts. Not crazy about this metaphor either, but it could be useful in some contexts and while also misunderstood in others. The same goes for most jargon.

Re: All models are wrong, but some are completely wrong

#49

Post by qchris »

DoofusOfDeath wrote: post #17

Regarding the "all models are wrong" maxim... Is that statement 100% true for the low-level models that physicists use and develop? In particular, I'm curious if quantum-physics models are 100% right, just not 100% precise.

At the level that you're talking about, you'll start running into the unresolved questions of modern physics.

One of the best ways to look at this is through the Standard Model of particle physics, which essentially defines how the fundamental particles of the universe are related. Between the number of observations and large-scale experiments dealing with high-energy collision products, astrophysics, neutrino detectors, etc., some people consider the Standard Model to be the most thoroughly-tested and verified framework in all of science. That's a pretty grand claim, but hey.

But it still falls short in some ways--for one, it starts breaking down past a certain scale. Classical field theory as defined by general relativity (another model that has had enormous success under test both theoretically and experimentally) and particle physics don't get along. Neither one fully explains reality, and the interface between those two models of reality hasn't been found. That's why people research things like string theory--they're attempting to find a mathematical framework that can resolve those two frameworks, among other things.

So while each of them describes the universe extremely accurately in their own domain (check the sigma values and number of observations of experiments run on the LHC), they're not 100% right, since they can't be correctly extended to cover all scales and frames. The models remain just that--models which provide a useful framework to interpret reality, but don't fully describe the physical reality itself.

Re: All models are wrong, but some are completely wrong

#50

Post by guscost »

martingoodson wrote: post #43
guscost wrote: post #31

Earlier quoted context omitted.

If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?

I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.

You will have nothing to add if the “2.2 million deaths in the US” scenario, which was blasted across every newspaper front page a few weeks ago, turns out to have been impossible all along?

If that scenario was “completely wrong” too, it seems like it would serve as a perfect example of the consequences of this kind of (still hypothetical) misinformation.

Post reply on HN