Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

1–10 of 111 posts

Re: Simpson's paradox

#2

Post by roenxi »

https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error to creep in when populations are created in a biased way.

These two effects explain a lot of the stupid decisions that come out of "data driven" processes. It is common for data to suggest the opposite of the truth.

Re: Simpson's paradox

#3

Post by r_thambapillai »

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold had almost doubled, from 10% to nearly 20%.

The finance team had asked me to double check the marketing team's numbers, to see if there'd been some funny math in the reporting. But the marketing team were totally right, marketing spend across the three main categories - games, beauty, and nutrition had all fallen (~15% to ~10%, ~30% to ~25%, and ~50% to ~30% respectively). However, the mix of these product categories had shifted massively, with nutrition growing from roughly 10% of our total sales to now nearly 50%.

In net that meant that whilst the marketing team had gotten more cost-efficient at selling every individual product category, the growth in the nutrition industry had vastly outstripped the growth in all other categories, and since that was the highest individual category, the aggregate marketing costs % had gone up, even though the team had improved every category. I then had the fun job of explaining the Yule Simpson paradox to a bunch of accountants.

Re: Simpson's paradox

#4

Post by bee_rider »

For all of the examples on Wikipedia, it seems like there was some confounding extra variable that was missed. I wonder if anybody knows of a case where it just sort of happened randomly, with no big underlying cause?

Or maybe I’m thinking of it wrong and this is impossible.

Re: Simpson's paradox

#5

Post by throwaway98797 »

r_thambapillai wrote: post #3

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold h…

it’s shocking that product mix wasn’t slide on reporting

but marketing selects for positivity not objectivity

the facts and only the facts that support what they do

Re: Simpson's paradox

#6

Post by eadler »

bee_rider wrote: post #4

For all of the examples on Wikipedia, it seems like there was some confounding extra variable that was missed. I wonder if anybody knows of a case where it just sort of happened randomly, with no big underlying cause? Or maybe I’m thinking of it wrong and this is impossible.

It can happen any time there is a mix shift in the underlying quantity of the subgroups. It's just that random changes in quantities are not likely to be studied or reported. It's easy to generate manually though.

Re: Simpson's paradox

#7

Post by pdonis »

roenxi wrote: post #2

https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error to creep in when populations are created in a biased way. These two effects explain a lot of the stupid decisions that come out of "data driven" processes. It is common for data to suggest the opposite of the truth.

> It is common for data to suggest the opposite of the truth.

Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics.

For example, in the UC Berkeley case described in the Simpson's Paradox article, the data actually doesn't tell you anything useful about "bias" in the sense of "something people are doing that they should do differently to make the admissions process fairer". It doesn't even tell you where to look for possible "bias" without knowing more about the admissions process: it is controlled primarily by departments or by the university as a whole?

Post reply on HN