Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Simpson's paradox

#51

Post by aldonius »

estiaan wrote: post #23

I was reading the example of UC Berkely appearing to have gender bias in the admissions and read the following: “it showed that women tended to apply to more competitive departments with lower rates of admission, even among qualified applicants (such as in the English department), whereas men tended to apply to less competitive departments with higher rates of admission (such as in the engineering department)” That’s…

I think it could be consistent.

In Australia we usually think about undergrad competitiveness in terms of the minimum ATAR rank of last years' admissions, right?

But you could also think about competitiveness in terms of admission fraction.

And I think those two metrics can be consistent if a very popular degree has a disproportionately low fraction of high-ATAR applications.

Re: Simpson's paradox

#52

Post by lupusreal »

brabel wrote: post #34
kromem wrote: post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

Sorry to nitpick, but "light was made of discrete units that weighed very little and were moving very fast" is not really correct. First of all, light has exactly zero weight (only a massless particle can travel at exactly the speed of light, and at no other speed for that matter). Secondly, you're leaving out the wave/particle duality of light, which sort of reminds the Simpson's paradox description of "just two dif…

Weight isn't mass; weight is the force acting on something due to gravity. Gravity effects light, albeit only by a little, so in this sense light has a small but nonzero weight.

Re: Simpson's paradox

#53

Post by brabel »

lupusreal wrote: post #52
brabel wrote: post #34

Earlier quoted context omitted.

Sorry to nitpick, but "light was made of discrete units that weighed very little and were moving very fast" is not really correct. First of all, light has exactly zero weight (only a massless particle can travel at exactly the speed of light, and at no other speed for that matter). Secondly, you're leaving out the wave/particle duality of light, which sort of reminds the Simpson's paradox description of "just two dif…

Weight isn't mass; weight is the force acting on something due to gravity. Gravity effects light, albeit only by a little, so in this sense light has a small but nonzero weight.

I don't believe that's a correct interpretation. The reason light bends in the presence of gravity is that space time itself is curved, and light follows a "straight line" on that curved space time.

Given weight is defined as `W=mg`, and `m` is `0` for light, light can't have any weight. I think the question is itself incorrect: you can't weigh light because light is not something you can "stop" and put on a balance.

The fact that gravity appears to "attract" light is an illusion. Light only has what is called "relativistic mass" which has very little to do with how we normally think of mass and weight.

This article explains it pretty well: https://science.howstuffworks.com/light-weigh.htm

Re: Simpson's paradox

#54

Post by dumb1224 »

r_thambapillai wrote: post #3

I once encountered this in the real world as a data analyst a long time ago. I was working at an e-commerce company, called The Hut Group, and the whole year our marketing team had been saying our marketing cost of goods sold (the percentage of our revenue we needed to spend on marketing) had been declining across every product category. But at year end, the execs were shocked to realize that our cost of goods sold h…

I thought it is pretty common to apply mixed / hierarchical linear models? I didn't study statistics but in our field of many problems of modelling biological effects we would do that.

E.g https://www.pymc.io/projects/examples/en/latest/generalized_...

Re: Simpson's paradox

#55

Post by kgwgk »

lern_too_spel wrote: post #45
lordgrenville wrote: post #36

Earlier quoted context omitted.

But the improvement induced the demand, which to my mind makes this different from Simpson's Paradox.

Doesn't matter. That is not relevant to the paradox.

How does "Average and p95 latency actually increased after shipping the work to production. How does an objectively good change make things worse?" relate to Simpson's paradox again?

Re: Simpson's paradox

#56

Post by [deleted] »

appplication wrote: post #25
Anon84 wrote: post #14

Earlier quoted context omitted.

It’s actually surprisingly common. You can even find it in “classical” toy datasets like Iris: https://github.com/DataForScience/Causality/blob/master/1.2%...

Covid vaccination rates and deaths were rather famously subject to it. E.g. some combination of stats like “most covid deaths were vaccinated individuals”, “vaccination reduces death rate”, and “population segment with lowest vaccination rates has lowest covid death rates.” were all true at the same time.

[deleted]

Re: Simpson's paradox

#57

Post by DonsDiscountGas »

estiaan wrote: post #23

I was reading the example of UC Berkely appearing to have gender bias in the admissions and read the following: “it showed that women tended to apply to more competitive departments with lower rates of admission, even among qualified applicants (such as in the English department), whereas men tended to apply to less competitive departments with higher rates of admission (such as in the engineering department)” That’s…

I was surprised to read that too. I think the answer is that here we're looking at admissions rate = number admitted / number applied, which is not the same as overall difficulty in a conceptual sense.

The only people applying to grad school in math are people who got a BS in math and did so with good grades (or perhaps some other STEM field + significant theoretical math coursework). On the arts side I suspect they draw from a larger pool (plus people tend to switch from STEM to something else a lot more than the other way around) of backgrounds. It's easier to convince oneself that a short story is great (when others may disagree) than convincing oneself a math proof is correct when it objectively is not. So there's less self-selection on the applicant side, and hence a lower admissions rate.

Re: Simpson's paradox

#58

Post by DonsDiscountGas »

bee_rider wrote: post #4

For all of the examples on Wikipedia, it seems like there was some confounding extra variable that was missed. I wonder if anybody knows of a case where it just sort of happened randomly, with no big underlying cause? Or maybe I’m thinking of it wrong and this is impossible.

In order for it to count as Simpsons paradox I think there would need to be a confounding variable. It's certainly possible for it to appear spuriously, and for something to look like a confounder when it isn't, but there would need to be some type of subgroup.

Re: Simpson's paradox

#59

Post by JacobAldridge »

roenxi wrote: post #2

https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error to creep in when populations are created in a biased way. These two effects explain a lot of the stupid decisions that come out of "data driven" processes. It is common for data to suggest the opposite of the truth.

As the aphorism goes, there are Lies, Damned Lies, and Statistics.

Re: Simpson's paradox

#60

Post by wastewastewaste »

tetris11 wrote: post #47
jldugger wrote: post #22

Earlier quoted context omitted.

Pretty much every dataset I work with as an SRE is full of these paradoxes. One classic published example comes from Google: A network engineer took a trip to Indonesia or something (can't find the citation to confirm the exact tale), noticed the service was slow, and when asking around everyone said "that's how its always been." Basically the local cellular networks are slow and off island fiber connects are saturat…

isn't that the "One More Lane, I Promise!" meme

It is, but usually the meme misrepresents induced demand. While I don't like cars and we should focus on other infrastructure, adding a lane does help.

It does not reduce congestion, but it does now serve more people at this same current congestion level. And those people have come from somewhere. Sometimes from public transport, which isn't really good, but sometimes from some backwater road.

Post reply on HN