For me this was far better explained in the following text (also posted on HN not so long ago): http://www.michaelnielsen.org/ddi/if-correlation-doesnt-impl...
Simpson's Paradox
21–30 of 52 posts
Re: Simpson's Paradox
#22Anyone interested in this stuff should definitely watch the lectures from the Stats 110 class from Harvard that's up on iTunes U (and perhaps other places).
Lecture 6 talks about this paradox and if I recall correctly he might even talk about this exact case:
https://itunes.apple.com/us/course/statistics-110-probabilit...
Re: Simpson's Paradox
#23Earlier quoted context omitted.
If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.
> If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
I'm optimistic enough to believe in a few wild cards trying to do good science. A whole bunch of the rest have to put a little bit of effort into competing with them.
It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
Re: Simpson's Paradox
#24The point of an interactive illustration like this is that it should make it more intuitive and easier to understand than the core concept, not more confusing. What is the meaning of the green and purple lines on the graph, why do they have different gradients and why can't I adjust them? Why does the Simpson's Paradox apply sometimes and not others? Why are there so many bars and donut charts? What does the gray cir…
It doesn't help things aren't labelled very well. I had to fiddle with things to work it all out. But at least I could, which gives it a leg up over an equivalently labelled static graph.
In the doughnut charts, blue is accepted rate, red is rejected rate (as labelled earlier in the article).
The gray circle indicates which has a higher percentage of acceptance between male and female.
The y-axis on the line graph is %age admitted combined over both departments. As you drag the women slider, you'll see the %age accepted rate for women combined over both department matches the y value of the women line in the line chart.
The illustration is to show how the distribution of applicants between the easy and hard departments (the lurking explanatory) effects the combined acceptance percentages (the explained). This is the focal point, so all other variables need to be fixed.
The fixed values are: in the easy department, 62% of men are accepted and 80% of women are accepted. In the hard department, 26% of men are accepted and 27% of women are accepted. In both departments women are slightly favoured over men, so both always have a gray circle around the women's charts.
There are 1,835 women applicants and 1,362 men applicants, making a total of 2,691.
The combined accept rates is derived from this static data and the user controlled variable of the distribution of applicants between the departments. You can pick some arbitrary inputs and trace the maths through to calculate how many apply to each department, and how many of those get accepted, then add that up to find the combined value (be warned the actual percentages aren't the nice rounded ones they label them as).
While the point on the purple line is below the point on the green line in the y axis, the combined acceptance percentage for women is larger than the men. There is no Simpson's paradox here because taking the lurking explanation into account - that the distribution of applicants between departments has an effect - does not change the outcome. Women are favoured in all three measurements; the gray circle is around all the women's charts.
When the point on the purple line is above the point in the green line in the y axis, the combined acceptance percentage for women flips to being smaller than men. Here we have the Simpson's paradox - taking into account the lurking explanation changes the outcome. Women are favoured in the breakdown by department, but men appear favoured in the combined statistic.
I found the interactive chart helped me figure out this explanation, but it wasn't helped by poor labelling and the fact they seem to have used real (read messy) statistics to perform the underlying calculations with rather than the nice rounded "62%" etc. they label.
Re: Simpson's Paradox
#25The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do t…
Surely some studies account for this? If you look for people who do X and people who don't and just analyze their lives, yes this problem is likely to exist. But if you take two randomized samples of a the population and say to group A, "do X," and to group B "don't do X," you have an effective control group. At least I think so. Don't some dietary studies even provide the participants with custom food regimens to tr…
That's one way to do it (the best way). You can still draw "causal" conclusion on observational studies (where you don't get to explicitly assign people to group A and B, but where those assignment are given to you), but it is much harder and you have to be very careful with what you are doing. If you are interested in that topic I suggest you google "Rubin Causal Framework", or check out some of the papers by Donald Rubin (arguably the pioneer of the field of Causal Inference).
Re: Simpson's Paradox
#26The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do t…
Surely some studies account for this? If you look for people who do X and people who don't and just analyze their lives, yes this problem is likely to exist. But if you take two randomized samples of a the population and say to group A, "do X," and to group B "don't do X," you have an effective control group. At least I think so. Don't some dietary studies even provide the participants with custom food regimens to tr…
Epidemiologist here. Many studies account for it and we are acutely aware of this problem and have some sophisticated methods for dealing with unmeasured propensities. And randomized trial are "epidemiologic" studies too. Not everything in epidemiology is observational. Simpson's paradox is taught in 1st year intro to epi courses. There is also a lesser known "reverse-simpson" which comes from overgeneralizing specific results.
Re: Simpson's Paradox
#27Judea Pearl has formalized a resolution to Simpson's Paradox:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34....
An oversimplification of his idea would be to say that given assumptions about the causal independence of variables, it becomes clear which way you should group the data. Although it's not always possible to make these independence assumptions, much of the time they are obvious and uncontroversial.
Taking the Berkeley gender bias case as an example: We know it's possible that biological gender influences which department graduates apply to, but that it's impossible that the department a graduate applies to influences their biological gender. This fact alone tells us that we need to look at the data by department rather than in aggregate, resolving the paradox.
Re: Simpson's Paradox
#28The point of an interactive illustration like this is that it should make it more intuitive and easier to understand than the core concept, not more confusing. What is the meaning of the green and purple lines on the graph, why do they have different gradients and why can't I adjust them? Why does the Simpson's Paradox apply sometimes and not others? Why are there so many bars and donut charts? What does the gray cir…
It's a bit confusing, but overall, I really love the presentation of this article. I wish more serious research was presented like this, with lots of knobs for fiddling and experimentation. (See also: Bret Victor.)
Re: Simpson's Paradox
#29Earlier quoted context omitted.
If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? That is, if they can control for it, they can make assertions about smaller effects.
> If it was a huge deal, wouldn't epidemiologists be slobbering over the statistical power they would get from using it? Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
That may get you written about in popular-science magazines, but your work will have a low impact factor- this is largely measured in how many of your fellow researchers cite your work, and its crucial to an academic researcher's career.
If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
Re: Simpson's Paradox
#30The Omitted Variable Problem is part of my mental framework that causes me to not believe most epidemiological studies, especially ones that confirm a popular belief. The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do t…
Don't throw the baby out with the bathwater. There are ways of calculating whether a hidden variable is more likely to cause the observational data than the independent variables of your model.
A simple example would be a wet lawn. We know rain causes a wet lawn, and our observation shows indeed that rain and wet lawns are strongly associated. However, observing a case where a given lawn is wet, and yet there's no associated rain is a clear signal that a latent cause hasn't been accounted for (namely sprinklers). This principle still applies in noisy observational data or probabilistic rather than deterministic causal relations, though you do need a bigger sample to reach the same confidence.
We also can look at measures of model fitness. If a variation of the model that hypothesizes a latent causal variable is more likely to generate the observed data than the model without it, we know we've missed something. The general case of this is that we learn the model itself from the observed data.
This page from Kevin Murphy is a reasonable survey of the methods: http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html I also recommend his textbook.