Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Simpson's paradox

#31

Post by ploxiln »

Tomte wrote: post #26
jldugger wrote: post #22

Earlier quoted context omitted.

Pretty much every dataset I work with as an SRE is full of these paradoxes. One classic published example comes from Google: A network engineer took a trip to Indonesia or something (can't find the citation to confirm the exact tale), noticed the service was slow, and when asking around everyone said "that's how its always been." Basically the local cellular networks are slow and off island fiber connects are saturat…

But your example isn‘t a case of Simpson‘s Paradox (which is purely statistical), but Jevons Paradox (which is about human behaviour and economics).

If I recall the youtube slow-internet optimisation case correction, I think it is an example of Simpson's paradox. They made it faster for countries with fast internet, and faster for countries with slow internet, and then the average performance across all users/countries was slower, because now the countries with slow internet used youtube much more than before.

Re: Simpson's paradox

#32

Post by bruce511 »

CrazyStat wrote: post #15

When I taught intro stats many years ago I used to use house prices as a nice example of Simpson's Paradox (with actual data, for the students to investigate as part of a computational lab). The data I had was on US house sales from 2008, so it's 15 years out of date now--perhaps things have changed since. At the time, the average price for single-family house sales was higher for houses without central AC than for h…

Im reading this outcome to be the reverse of the examples above. Or perhaps identifying the correct stat to use based on your goals.

In other words, in this case I don't really care what the national average is. I care about my house, my street, my area.

In other cases, like in marketing, the stat that matters first is overall net profit. From there we can burrow down to understand the factors. In which case we come across business share before marketing spend.

In the networking example, the goal is usage (throughput). Not speed or latency.

Drawing the wrong stat first leads to incorrect conclusions.

Re: Simpson's paradox

#33

Post by brabel »

konstantinua00 wrote: post #28
r_thambapillai wrote: post #3

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold h…

every time I hear about examples of simpson in peactice, I don't get what lesson to learn marketting team overoptimized, so non-nutrition demand fell? drop nutrition from line of products, so that you're both efficient in products you do and overall? these metrics are insufficient and it's better to look at gross change rather than ratios? I have no idea

The article suggests an answer to your question, see the last sentence of the introduction:

"its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once."

In the case above, they failed at "keeping the parts in mind" as clearly, the different ratios between different products was crucial.

Re: Simpson's paradox

#34

Post by brabel »

kromem wrote: post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

Sorry to nitpick, but "light was made of discrete units that weighed very little and were moving very fast" is not really correct.

First of all, light has exactly zero weight (only a massless particle can travel at exactly the speed of light, and at no other speed for that matter).

Secondly, you're leaving out the wave/particle duality of light, which sort of reminds the Simpson's paradox description of "just two different ways to think about the same data", without which you simply can't fully understand the behaviour of light (or of the statistical system you're looking at).

Re: Simpson's paradox

#35

Post by [deleted] »

appplication wrote: post #25
Anon84 wrote: post #14

Earlier quoted context omitted.

It’s actually surprisingly common. You can even find it in “classical” toy datasets like Iris: https://github.com/DataForScience/Causality/blob/master/1.2%...

Covid vaccination rates and deaths were rather famously subject to it. E.g. some combination of stats like “most covid deaths were vaccinated individuals”, “vaccination reduces death rate”, and “population segment with lowest vaccination rates has lowest covid death rates.” were all true at the same time.

[deleted]

Re: Simpson's paradox

#36

Post by lordgrenville »

ploxiln wrote: post #31
Tomte wrote: post #26

Earlier quoted context omitted.

But your example isn‘t a case of Simpson‘s Paradox (which is purely statistical), but Jevons Paradox (which is about human behaviour and economics).

If I recall the youtube slow-internet optimisation case correction, I think it is an example of Simpson's paradox. They made it faster for countries with fast internet, and faster for countries with slow internet, and then the average performance across all users/countries was slower, because now the countries with slow internet used youtube much more than before.

But the improvement induced the demand, which to my mind makes this different from Simpson's Paradox.

Re: Simpson's paradox

#37

Post by pcwelder »

Encountered it recently. I had two different dataset to evaluate model performance on from different domains.

One dataset was closer to training data and the other was closer to our business use case. The hypothesis was that performance on the latter dataset would be poorer due to overfitting.

Indeed the accuracy on all categories had reduced. However, overall accuracy was much higher!

This was because the second dataset had higher frequency of easy to predict categories.

If we had just looked at overall number we would have concluded that there was no overfitting to train domain, which was not the case.

Re: Simpson's paradox

#38

Post by staunton »

pdonis wrote: post #7
roenxi wrote: post #2

https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error to creep in when populations are created in a biased way. These two effects explain a lot of the stupid decisions that come out of "data driven" processes. It is common for data to suggest the opposite of the truth.

> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case describ…

> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited

I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it's still some sort of statistics.

Re: Simpson's paradox

#39

Post by infogulch »

konstantinua00 wrote: post #28
r_thambapillai wrote: post #3

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold h…

every time I hear about examples of simpson in peactice, I don't get what lesson to learn marketting team overoptimized, so non-nutrition demand fell? drop nutrition from line of products, so that you're both efficient in products you do and overall? these metrics are insufficient and it's better to look at gross change rather than ratios? I have no idea

Maybe the lesson is to analyze different business units (product categories?) independently first, then the whole.

Re: Simpson's paradox

#40

Post by Gibbon1 »

bruce511 wrote: post #32
CrazyStat wrote: post #15

When I taught intro stats many years ago I used to use house prices as a nice example of Simpson's Paradox (with actual data, for the students to investigate as part of a computational lab). The data I had was on US house sales from 2008, so it's 15 years out of date now--perhaps things have changed since. At the time, the average price for single-family house sales was higher for houses without central AC than for h…

Im reading this outcome to be the reverse of the examples above. Or perhaps identifying the correct stat to use based on your goals. In other words, in this case I don't really care what the national average is. I care about my house, my street, my area. In other cases, like in marketing, the stat that matters first is overall net profit. From there we can burrow down to understand the factors. In which case we come…

> like in marketing, the stat that matters first is overall net profit.

I have a take on that. The stat that matters is the profit per unit of non scalable business resource. As in how much management, marketing, sales, accounting, and engineering time does the product take per unit. It's important because those are often hard to scale. You can have a low margin product that requires zip of the above and it's good business. And the reverse, high margins but requires too much of the above and it's bad.

Post reply on HN