All models are wrong, but some are completely wrong • Hacker News Board

Re: All models are wrong, but some are completely wrong

#21

Post by YZF » Sun, Apr 05, 2020, 6:44 PM UTC

smitty1e wrote: post #2

One feels that, irrespective of the models, the data in the covid-19 case may be unusually bad. It may be time to add a third error category[1] I. False positive II. False negative III. Deliberately skewed off the map for propaganda reasons. [1] https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Ty...

This is driving me crazy ;) the poor quality of data.

You'd want health organizations around the world to be publishing every possible detail (anonymized) so that the disease can be better understood. Yet three months in, with over a million cases worldwide, we still have experts disagreeing about things like asymptomatic transmission, use of masks, droplets vs. aerosol, how much distance one should stand from another, viability on surfaces, etc. etc. Even for treatment options rather than insisting on randomized double blind trials start by using the natural experiments that are already happening.

We should have the data to answer a lot of these questions (or at least draw out some probability distributions), or at least someone has it. This stuff is going to be critical in informing exit strategies.

Re: All models are wrong, but some are completely wrong

#22

Post by heurifk » Sun, Apr 05, 2020, 6:45 PM UTC

puranjay wrote: post #10

I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or…

We have recent data from Wuhan.

We have old data from 1918.

This is a white swan, not a black one.

More importantly, the mortality, while much higher than flu, it's still relatively low.

Now imagine a virus as contagious as this one, but with 10% mortality over all age groups. That would be unprecedented and probably cause society meltdown.

Re: All models are wrong, but some are completely wrong

#23

Post by xg15 » Sun, Apr 05, 2020, 6:45 PM UTC

scribu wrote: post #4

Reading the examples, where people jumped on a single mistake to discredit an entire report points to some sad conclusions: 1. Scientific literacy is super low in the general population. 2. Motivated reasoning is rampant. People will believe anything that enables them to do what they wanted to do anyway.

> 2. Motivated reasoning is rampant.

This is an important point. Scientific scrutiny is extremely important, but there is still a difference between a judge that is stern but fair - and one that actively wants you to fail.

Motivated reasoners have no problems holding opposing parties to impossibly high standards while accepting claims without any evidence as valid arguments for their side.

Today, climate scientists have learned the lessons and improved communication and modeling considerably, even to the point where we now how "attribution science" we we can discuss climate change in the context of particular weather events. We also start seeing changes in weather patterns that are hard to ignore even for laymen.

Nevertheless we are still having the same discussions as before.

Re: All models are wrong, but some are completely wrong

#24

Post by tomrod » Sun, Apr 05, 2020, 6:46 PM UTC

DoofusOfDeath wrote: post #17

Regarding the "all models are wrong" maxim... Is that statement 100% true for the low-level models that physicists use and develop? In particular, I'm curious if quantum-physics models are 100% right, just not 100% precise.

I keep this distinction in mind:

- Models are deliberate simplifications of reality, in order to guide thinking and otherwise pull in only important information

- Formulations (formalizations) are encapsulation of principles into a mathematical framework

While there is significant overlap, the two categories do not overlap 100%. I see formalizations of physics as the latter, and we use the former to help keep our understanding of the latter clear.

Re: All models are wrong, but some are completely wrong

#25

Post by magoghm » Sun, Apr 05, 2020, 6:47 PM UTC

YZF wrote: post #21

smitty1e wrote: post #2

One feels that, irrespective of the models, the data in the covid-19 case may be unusually bad. It may be time to add a third error category[1] I. False positive II. False negative III. Deliberately skewed off the map for propaganda reasons. [1] https://en.m.wikipedia.org/wiki/Type_I_and_type_II_errors#Ty...

This is driving me crazy ;) the poor quality of data. You'd want health organizations around the world to be publishing every possible detail (anonymized) so that the disease can be better understood. Yet three months in, with over a million cases worldwide, we still have experts disagreeing about things like asymptomatic transmission, use of masks, droplets vs. aerosol, how much distance one should stand from anothe…

My experience with data is: you never get good data. Always, the closer you look at it the more problems you find in how it was collected.

Re: All models are wrong, but some are completely wrong

#26

Post by [deleted] » Sun, Apr 05, 2020, 6:48 PM UTC

scribu wrote: post #4

Reading the examples, where people jumped on a single mistake to discredit an entire report points to some sad conclusions: 1. Scientific literacy is super low in the general population. 2. Motivated reasoning is rampant. People will believe anything that enables them to do what they wanted to do anyway.

[deleted]

Re: All models are wrong, but some are completely wrong

#27

Post by guscost » Sun, Apr 05, 2020, 6:51 PM UTC

martingoodson wrote: post #7

dwohnitmok wrote: post #6

Earlier quoted context omitted.

This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment. You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties. Climate science still hasn't solved this problem despite trying to imple…

I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.

I don’t think there’s value in doing this, unless the second opinion disagrees with the original model. It’s not hard to find a second “expert” to agree with just about anything, if you look hard enough.

The number of people who agree with something tells you, as a rational person, practically nothing - it reminds me of the absurd compilation argument “One Hundred Authors Against Einstein”: https://archive.org/details/HundertAutorenGegenEinstein/mode...

Einstein’s famous reply: “If I were wrong, then one would have been enough!”

Re: All models are wrong, but some are completely wrong

#28

Post by magoghm » Sun, Apr 05, 2020, 6:51 PM UTC

martingoodson wrote: post #8

cassiet wrote: post #5

The headline really cuts off the points nose.

Yeah - I was trying to play off the George Box quote but maybe it's too obscure https://en.wikipedia.org/wiki/All_models_are_wrong

The reference to the George Box quote in the title is what originally got my attention.

Re: All models are wrong, but some are completely wrong

#29

Post by hinkley » Sun, Apr 05, 2020, 6:51 PM UTC

grawprog wrote: post #9

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not suppo…

It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept, people don’t want to talk about it any more.

We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning.

My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.

Re: All models are wrong, but some are completely wrong

#30

Post by puranjay » Sun, Apr 05, 2020, 7:01 PM UTC

ajuc wrote: post #11

puranjay wrote: post #10

I fail to understand how you can reasonably model an unprecedented event in modern history. We have no data on how people will act under a weeks, even months long lockdown. Will they stay indoors and follow guidelines? Maybe. Will they watch their livelihoods get affected, their mental health deteriorate, or get careless over time and break quarantine? Maybe. We just don't know because we just don't have the data or…

There's very little unprecedented events in modern history, and pandemic certainly isn't one.

Complete lockdowns across multiple countries and cessation of all economic activity certainly is unprecedented.