Live data from Hacker News

Simpson's Paradox

vudlab.com

51–52 of 52 posts

Re: Simpson's Paradox

#51
post #46

Earlier quoted context omitted.

That's a great reference. I'm going to use it from now on too.

I won't claim to have fully internalized all of it, but Judea Pearl's book Causality is incredible.

I've read a lot of his papers and I own Causality, but unfortunately it ended up at my parents and I always forget to grab it when I'm there. I really would like to get further into the book.

But this is a great, fast example of why his work is important.

Re: Simpson's Paradox

#52

Judea Pearl has formalized a resolution to Simpson's Paradox: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.... An oversimplification of his idea would be to say that given assumptions about the causal independence of variables, it becomes clear which way you should group the data. Although it's not always possible to make these independence assumptions, much of the time they are obvious and uncontrover…

Well, it certainly makes more sense to look at the by department data rather than the aggregate data. But I would say that the fact that we see such large disparities in which departments are applied to is already enough to refute the idea that male and female applicants are pretty much the same, and that therefore it would be pretty reckless to conclude any sex discrimination based on the difference in acceptance ra…

And Pearl's point is that the "real world" logic you just applied is both important and not statistical—you need to augment your analysis with these causal assumptions in order to translate probability into meaningful causal statements.
Post reply on HN