Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Simpson's paradox

#61

Post by im3w1l »

brabel wrote: post #53
lupusreal wrote: post #52

Earlier quoted context omitted.

Weight isn't mass; weight is the force acting on something due to gravity. Gravity effects light, albeit only by a little, so in this sense light has a small but nonzero weight.

I don't believe that's a correct interpretation. The reason light bends in the presence of gravity is that space time itself is curved, and light follows a "straight line" on that curved space time. Given weight is defined as `W=mg`, and `m` is `0` for light, light can't have any weight. I think the question is itself incorrect: you can't weigh light because light is not something you can "stop" and put on a balance.…

Light bends spacetime too.

Re: Simpson's paradox

#62

Post by lmz »

wastewastewaste wrote: post #60
tetris11 wrote: post #47

Earlier quoted context omitted.

isn't that the "One More Lane, I Promise!" meme

It is, but usually the meme misrepresents induced demand. While I don't like cars and we should focus on other infrastructure, adding a lane does help. It does not reduce congestion, but it does now serve more people at this same current congestion level. And those people have come from somewhere. Sometimes from public transport, which isn't really good, but sometimes from some backwater road.

Until more people choose to live further away because the commute is now tolerable with the extra lane (and it's cheaper), and then you're back to square one.

Re: Simpson's paradox

#63

Post by ploxiln »

kgwgk wrote: post #55
lern_too_spel wrote: post #45

Earlier quoted context omitted.

Doesn't matter. That is not relevant to the paradox.

How does "Average and p95 latency actually increased after shipping the work to production. How does an objectively good change make things worse?" relate to Simpson's paradox again?

That's exactly it. After "shipping the work to production" (making it faster for everybody), the overall average and p95 got worse. Each sub-population experienced improvement: countries with fast internet got faster youtube, countries with slow internet got faster youtube. But the overall average and p95 got worse: overall average was slower youtube. Because now more users from the second sub-population bring the overall average speed down (or latency up). That's Simpson's paradox.

Re: Simpson's paradox

#64

Post by mFixman »

bryanrasmussen wrote: post #46
lordgrenville wrote: post #36

Earlier quoted context omitted.

But the improvement induced the demand, which to my mind makes this different from Simpson's Paradox.

I would say the improvement allowed the demand to be met, everybody wanted to use youtube, but few could. Just like many people may want to eat a wide range of expensive tasty food, but have to make do with junk because it's what they can afford.

It would be Simpsons' Paradox if Google services in Indonesia were initially slow because Indonesians tend to use YouTube more often than lighter services.

There wasn't an error in the conclusions of the initial measuremen. It was the solution that had problems.

Re: Simpson's paradox

#65

Post by kgwgk »

ploxiln wrote: post #63
kgwgk wrote: post #55

Earlier quoted context omitted.

How does "Average and p95 latency actually increased after shipping the work to production. How does an objectively good change make things worse?" relate to Simpson's paradox again?

That's exactly it. After "shipping the work to production" (making it faster for everybody), the overall average and p95 got worse. Each sub-population experienced improvement: countries with fast internet got faster youtube, countries with slow internet got faster youtube. But the overall average and p95 got worse: overall average was slower youtube. Because now more users from the second sub-population bring the ov…

Ah, you may be right. It's not clear in the story that "Average and p95 latency actually increased after shipping the work to production." means average of Indonesia and ex-Indonesia and not just Indonesian average.

Re: Simpson's paradox

#66

Post by topaz0 »

brabel wrote: post #53
lupusreal wrote: post #52

Earlier quoted context omitted.

Weight isn't mass; weight is the force acting on something due to gravity. Gravity effects light, albeit only by a little, so in this sense light has a small but nonzero weight.

I don't believe that's a correct interpretation. The reason light bends in the presence of gravity is that space time itself is curved, and light follows a "straight line" on that curved space time. Given weight is defined as `W=mg`, and `m` is `0` for light, light can't have any weight. I think the question is itself incorrect: you can't weigh light because light is not something you can "stop" and put on a balance.…

> the reason light bends in the presence of gravity ...

This is also why gravity bends the trajectories of massive particles, which also follow geodesics of the curved spacetime (in the absence of other forces).

Re: Simpson's paradox

#67

Post by lukas099 »

kromem wrote: post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

I'm having trouble making the connection between Simpson's paradox and the Epicureans. Can you help me out?

Re: Simpson's paradox

#68

Post by lukas099 »

marklubi wrote: post #21

What is the deal with posting (seemingly) random links to Wikipedia articles lately without context? Some of them are interesting, but most(?) of them come without details. Please provide me with some context.

Things hackers would find interesting. I think they have always been posted as long as I've been here.

Re: Simpson's paradox

#69

Post by CrazyStat »

bruce511 wrote: post #32
CrazyStat wrote: post #15

When I taught intro stats many years ago I used to use house prices as a nice example of Simpson's Paradox (with actual data, for the students to investigate as part of a computational lab). The data I had was on US house sales from 2008, so it's 15 years out of date now--perhaps things have changed since. At the time, the average price for single-family house sales was higher for houses without central AC than for h…

Im reading this outcome to be the reverse of the examples above. Or perhaps identifying the correct stat to use based on your goals. In other words, in this case I don't really care what the national average is. I care about my house, my street, my area. In other cases, like in marketing, the stat that matters first is overall net profit. From there we can burrow down to understand the factors. In which case we come…

Right, one of the interesting things about Simpson's paradox is that there's not a uniform right answer: sometimes you care about the overall average, sometimes you care about the averages of subpopulations. You have to judge that based on the situation.

One of the other comments linked [1] which includes Judea Pearl's analysis of Simpson's paradox from a causal inference point of view [2], which lays this out nicely (though maybe not easy to understand--it took me many hours of study to get comfortable with Pearl's causal inference work, even with a strong stats background).

[1] https://plato.stanford.edu/entries/paradox-simpson/

[2] https://plato.stanford.edu/entries/paradox-simpson/#ConfPear...

Re: Simpson's paradox

#70

Post by vidarh »

wastewastewaste wrote: post #60
tetris11 wrote: post #47

Earlier quoted context omitted.

isn't that the "One More Lane, I Promise!" meme

It is, but usually the meme misrepresents induced demand. While I don't like cars and we should focus on other infrastructure, adding a lane does help. It does not reduce congestion, but it does now serve more people at this same current congestion level. And those people have come from somewhere. Sometimes from public transport, which isn't really good, but sometimes from some backwater road.

The bigger problem with induced demand is that it's often poor ROI to add that lane where the demand is highest.

That is, imagine you have a big city. You can add capacity for 1m extra people to travel to the city centre, where there's lots of congestion. Or you find ways to induce demand around the other limits of town, even town current demand is low there.

Odds are you'll pick the first, because it's "obvious" and doesn't require much thinking to see it'd help. But we really ought to look at cost-benefit of the second option too, because repeatedly inducing demand in the centre keeps driving up the incremental cost of further improvements, along plenty of other undesirable second order effects.

Post reply on HN