Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Simpson's paradox

#41

Post by lern_too_spel »

Tomte wrote: post #26
jldugger wrote: post #22

Earlier quoted context omitted.

Pretty much every dataset I work with as an SRE is full of these paradoxes. One classic published example comes from Google: A network engineer took a trip to Indonesia or something (can't find the citation to confirm the exact tale), noticed the service was slow, and when asking around everyone said "that's how its always been." Basically the local cellular networks are slow and off island fiber connects are saturat…

But your example isn‘t a case of Simpson‘s Paradox (which is purely statistical), but Jevons Paradox (which is about human behaviour and economics).

Latency improved for everyone, but overall average latency increased because usage increased faster in high latency areas. That's Simpson's Paradox. Simpson's Paradox doesn't care where the subpopulations you're measuring came from.

Re: Simpson's paradox

#44

Post by kromem »

brabel wrote: post #34
kromem wrote: post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

Sorry to nitpick, but "light was made of discrete units that weighed very little and were moving very fast" is not really correct. First of all, light has exactly zero weight (only a massless particle can travel at exactly the speed of light, and at no other speed for that matter). Secondly, you're leaving out the wave/particle duality of light, which sort of reminds the Simpson's paradox description of "just two dif…

This was written in 50 BCE, nearly two thousand years before Einstein's Nobel winning work proving the discrete qualities of photons.

I'm well aware it's at best a partial description of light.

But it's leagues ahead of Plato's tiny triangles of fire in Timaeus or any other contemporary descriptions.

Also, technically zero mass is very little weight (the least, in fact). And the speed of light is very fast (the fastest). So Lucretius was correct in his statements, if just conservative in the degree to which he stated them (which was in line with the Epicurean commitment to the avoidance of false negatives).

Wave particle duality doesn't really get discussed in Western antiquity outside of a single tangent describing the beliefs of the Peratae who claim the universe has a threefold nature, with the first being continuous and infinitely divisible, the second being a near infinite number of potentialities, and the third being a formal instance. There's a bit of an Everettian quality to their thinking, but outside of its quite broad scope of thought I'm unaware of anyone saying "yeah, reality is both continuous and discrete at the same time" until physicists grappling with contradictory experimental results in the 20th century. The closest in antiquity outside of this group was arguably Plato's theory of forms where the forms were continuous and their physical manifestations discrete, though this is materially different from the idea they are both simultaneously occurring in what's around us (even if Plato's paradigm most likely influenced the much later Peratae).

Re: Simpson's paradox

#45

Post by lern_too_spel »

lordgrenville wrote: post #36
ploxiln wrote: post #31

Earlier quoted context omitted.

If I recall the youtube slow-internet optimisation case correction, I think it is an example of Simpson's paradox. They made it faster for countries with fast internet, and faster for countries with slow internet, and then the average performance across all users/countries was slower, because now the countries with slow internet used youtube much more than before.

But the improvement induced the demand, which to my mind makes this different from Simpson's Paradox.

Doesn't matter. That is not relevant to the paradox.

Re: Simpson's paradox

#46

Post by bryanrasmussen »

lordgrenville wrote: post #36
ploxiln wrote: post #31

Earlier quoted context omitted.

If I recall the youtube slow-internet optimisation case correction, I think it is an example of Simpson's paradox. They made it faster for countries with fast internet, and faster for countries with slow internet, and then the average performance across all users/countries was slower, because now the countries with slow internet used youtube much more than before.

But the improvement induced the demand, which to my mind makes this different from Simpson's Paradox.

I would say the improvement allowed the demand to be met, everybody wanted to use youtube, but few could.

Just like many people may want to eat a wide range of expensive tasty food, but have to make do with junk because it's what they can afford.

Re: Simpson's paradox

#47

Post by tetris11 »

jldugger wrote: post #22
r_thambapillai wrote: post #3

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold h…

Pretty much every dataset I work with as an SRE is full of these paradoxes. One classic published example comes from Google: A network engineer took a trip to Indonesia or something (can't find the citation to confirm the exact tale), noticed the service was slow, and when asking around everyone said "that's how its always been." Basically the local cellular networks are slow and off island fiber connects are saturat…

isn't that the "One More Lane, I Promise!" meme

Re: Simpson's paradox

#49

Post by yccs27 »

staunton wrote: post #38
pdonis wrote: post #7

Earlier quoted context omitted.

> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case describ…

> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it…

I think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.
Post reply on HN