Simpson’s Paradox (2016) • Hacker News Board

Re: Simpson’s Paradox (2016)

#21

Post by rwilson4 » Sat, Mar 02, 2019, 3:34 PM UTC

AnthonyMouse wrote: post #19

rwilson4 wrote: post #8

Earlier quoted context omitted.

Yes absolutely! Random assigment along with statistical power and significance considerations does indeed allow one to draw causal conclusions. It’s the gold standard for causal inference.

> Yes absolutely! The problem with these cases is generally that people want to use data that didn't come from a controlled experiment to begin with. You have a nice, fat data set of all the people who have been treated for kidney stones -- you could never afford to do a controlled experiment at that scale. But because the treatments weren't randomized (and neither was anything else), the conclusions are erroneous. T…

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit.

[0] Causality, Judea Pearl

[1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens and Donald Rubin

Re: Simpson’s Paradox (2016)

#22

Post by gwern » Sat, Mar 02, 2019, 3:36 PM UTC

sopooneo wrote: post #5

In simples case at least, such as with the kidney stones, can we reduce our risk of reaching wrong conclusions by increasing our sample size of patients and randomizing which receive each treatment?

Yes, but it won't help with other problems like measuring the wrong metric.

For example, the YouTube latency example linked at the bottom was a randomized A/B test ("launched an opt-in to a fraction of our traffic"), but it was measuring per-user latency metrics when the distribution of 'user' had changed radically thanks to the improvements; for this, he would've needed to instead be monitoring some more global long-term effect like user retention or total traffic (then he would've seen a result like 'latency got a lot worse, but we're getting a ton more users and they're coming back much more frequently, so, that's good overall but why is latency up and who are all these new users...? aha!'). You have a Simpson's paradox on the level of metrics here, instead of individuals.

Re: Simpson’s Paradox (2016)

#23

Post by currymj » Sat, Mar 02, 2019, 3:36 PM UTC

Judea Pearl’s explanations of this in terms of causality are the only way it really makes sense, in my view.

https://ftp.cs.ucla.edu/pub/stat_ser/r414.pdf

Re: Simpson’s Paradox (2016)

#24

Post by says_you » Sat, Mar 02, 2019, 3:40 PM UTC

knappa wrote: post #12

The sex-discrimination lawsuit against UC Berkley seems to be a kind of academic urban myth; the administration was apparently afraid of such a lawsuit and the study was done in response to those administrative fears.

Some people would reliah the chance to disregard a narrative that fails to align with their ideology. An advantage is obtained with selective acknowlegement of reality.

Now, how to go about the rationalization of ignoring it?

Re: Simpson’s Paradox (2016)

#25

Post by GolDDranks » Sat, Mar 02, 2019, 3:42 PM UTC

clircle wrote: post #20

Iirc, you can guard against simpson's paradox by designing/collecting balanced data

I thought the same; at least in the kidney stone story, the data wasn't balanced: treatment A was assigned a lot more "harder cases". Either the trial wasn't randomized or the data set size wasn't big enough.

Re: Simpson’s Paradox (2016)

#26

Post by lettergram » Sat, Mar 02, 2019, 3:42 PM UTC

Idk why the 2016 needs to be in the title here. I understand for date relevant content, but this is not.

Re: Simpson’s Paradox (2016)

#27

Post by naasking » Sat, Mar 02, 2019, 3:44 PM UTC

throway88989898 wrote: post #14

Neatly phrased: Trends which appear in slices of data may disappear or reverse when the groups are combined.

Or perhaps even more succinctly: slicing data can introduce bias.

Re: Simpson’s Paradox (2016)

#28

Post by mcguire » Sat, Mar 02, 2019, 4:03 PM UTC

I'd like to say that the author has been reading The Book of Why, but it seems that he hasn't because he missed the punch line of the section on the paradox: you need a causal model to separate the two branches of the paradox. It's as easy to construct examples where the overall view is correct as it is so construct examples where the separate views are.

Re: Simpson’s Paradox (2016)

#29

Post by AnthonyMouse » Sat, Mar 02, 2019, 4:04 PM UTC

rwilson4 wrote: post #21

AnthonyMouse wrote: post #19

Earlier quoted context omitted.

> Yes absolutely! The problem with these cases is generally that people want to use data that didn't come from a controlled experiment to begin with. You have a nice, fat data set of all the people who have been treated for kidney stones -- you could never afford to do a controlled experiment at that scale. But because the treatments weren't randomized (and neither was anything else), the conclusions are erroneous. T…

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit. [0] Causality, Judea Pearl [1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens an…

The trouble is you can't fix large numbers of statistical confounders with more data because there is a limit for how many factors you can control for before the measurement error overwhelms the signal.

To do statistical controls, you essentially sort the data by category, so that you're not just comparing black people with white people, you're comparing middle class 18 year old black female college applicants with college educated parents to middle class 18 year old white female college applicants with college educated parents.

But every one of those factors is a chance to have measured something wrong. Your group of middle class 18 year old black female college applicants with college educated parents will have a couple of people who were misidentified as middle class, a couple of people who were misidentified as black, a couple of people who were misidentified as female, a couple of people who were misidentified as 18 and a couple of people who were misidentified as having college educated parents. And they don't cancel out exactly because the original correlations with the primary factor existed to begin with, so the measurement error compounds in proportion to the strength of the correlation of the primary factor with each confounder.

Meanwhile the size of each subcategory shrinks each time you bisect it further. So the more things you try to control for, the higher the percentage of the sample in each subcategory is measurement error.

Re: Simpson’s Paradox (2016)

#30

Post by rwilson4 » Sat, Mar 02, 2019, 4:13 PM UTC

AnthonyMouse wrote: post #29

rwilson4 wrote: post #21

Earlier quoted context omitted.

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit. [0] Causality, Judea Pearl [1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens an…

The trouble is you can't fix large numbers of statistical confounders with more data because there is a limit for how many factors you can control for before the measurement error overwhelms the signal. To do statistical controls, you essentially sort the data by category, so that you're not just comparing black people with white people, you're comparing middle class 18 year old black female college applicants with c…

Lots of practical challenges for sure!