Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson's paradox

en.wikipedia.org

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Simpson's paradox

#81

Post by klodolph »

saghm wrote: post #72
kromem wrote: post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

> Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" Isn't this a bit of a misunderstanding on their part on the meaning of the word "paradox"? The fact that they're called paradoxes is that they go against initial intuition and _seem_ contradictory, not that they necessarily are. If anything, I'd guess…

There are various categories of paradoxes and different ways people have categorized them.

Quine calls this one a “veridical paradox”, where it seems false but is true.

Example of a different types of paradox are: any proof that 1=0, Russell’s Paradox, and Zeno’s paradox. These are either false in some sense or used to illustrate fallacious reasoning.

Re: Simpson's paradox

#82

Post by freedomben »

nomilk wrote: post #43

The short animation on the wiki page is a great example of a picture being worth 1000 words: https://en.wikipedia.org/wiki/File:Simpsons_paradox_-_animat...

Hot damn, you're not kidding. I was struggling a bit with it from reading the article text, but that animation clarified it for me in seconds!

Re: Simpson's paradox

#83

Post by pdonis »

andirk wrote: post #79
pdonis wrote: post #7

Earlier quoted context omitted.

> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case describ…

Is the UC Berkeley case a good example of the importance of normalizing data before analyzing? Where things need to be put on a level playing field and handicaps applied to remove auxiliary noise.

Normalizing data doesn't fix the issue in the UC Berkeley case, because you still have to pick what to normalize over: do you normalize over the entire university, or separately over each department?

The answer to questions like that can't be found in the data. You have to go look at how the university admission process actually works, and what roles the university vs. the individual departments play in it.

Re: Simpson's paradox

#84

Post by vidarh »

otherme123 wrote: post #78
vidarh wrote: post #70

Earlier quoted context omitted.

The bigger problem with induced demand is that it's often poor ROI to add that lane where the demand is highest. That is, imagine you have a big city. You can add capacity for 1m extra people to travel to the city centre, where there's lots of congestion. Or you find ways to induce demand around the other limits of town, even town current demand is low there. Odds are you'll pick the first, because it's "obvious" and…

Adding lanes is like getting a bigger cache with the same throughput. It's obvious at the supermarket: what goes faster, a single cashier processing four short lanes of 10 people with round robin, or two cashiers processing a single lane with 40 people? Is the city center able to process 1m extra people? If not, it doesn't matter how many lanes you build.

Well you often can make it able to "process" 1m extra people: You can build overpasses, and tunnels, and taller buildings. But the cost-per-extra-person will tend to go up accordingly, to the point where you could spend an extraordinary amount attracting people out of the centre.

E.g. London's "Crossrail" / Elizabeth line cost $24 billion. Granted, it also allows some people to go through London faster, but I can't help to wonder what that money could've done if applied to attract businesses out of the centre instead. E.g. upgrading links between towns on the outskirts, upgrading town centres, and generally try to make it more attractive for businesses to be located further out.

Given the extraordinary costs it takes to do large infrastructure projects in London, I'd be very surprised if you couldn't get a higher return on investment that way, or by investing similar sums elsewhere in the UK entirely.

Re: Simpson's paradox

#85

Post by pdonis »

yccs27 wrote: post #49
staunton wrote: post #38

Earlier quoted context omitted.

> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it…

I think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.

Also, results from active experiments aren't limited to statistics. You can set up experiments to have discrete results, where no statistics is required to test a hypothesis.

For example, the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all: the two different models make contradictory predictions with no continuous variation between them.

[1] https://en.wikipedia.org/wiki/GHZ_experiment

Re: Simpson's paradox

#86

Post by missingrib »

klodolph wrote: post #81
saghm wrote: post #72

Earlier quoted context omitted.

> Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" Isn't this a bit of a misunderstanding on their part on the meaning of the word "paradox"? The fact that they're called paradoxes is that they go against initial intuition and _seem_ contradictory, not that they necessarily are. If anything, I'd guess…

There are various categories of paradoxes and different ways people have categorized them. Quine calls this one a “veridical paradox”, where it seems false but is true. Example of a different types of paradox are: any proof that 1=0, Russell’s Paradox, and Zeno’s paradox. These are either false in some sense or used to illustrate fallacious reasoning.

I don't think Zeno's paradoxes have truly been proven false.

Re: Simpson's paradox

#87

Post by jldugger »

klodolph wrote: post #81
saghm wrote: post #72

Earlier quoted context omitted.

> Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" Isn't this a bit of a misunderstanding on their part on the meaning of the word "paradox"? The fact that they're called paradoxes is that they go against initial intuition and _seem_ contradictory, not that they necessarily are. If anything, I'd guess…

There are various categories of paradoxes and different ways people have categorized them. Quine calls this one a “veridical paradox”, where it seems false but is true. Example of a different types of paradox are: any proof that 1=0, Russell’s Paradox, and Zeno’s paradox. These are either false in some sense or used to illustrate fallacious reasoning.

> “veridical paradox”, where it seems false but is true.

> any proof that 1=0

So, 1=0 seems false but is true?

Re: Simpson's paradox

#88

Post by shawabawa3 »

missingrib wrote: post #86
klodolph wrote: post #81

Earlier quoted context omitted.

There are various categories of paradoxes and different ways people have categorized them. Quine calls this one a “veridical paradox”, where it seems false but is true. Example of a different types of paradox are: any proof that 1=0, Russell’s Paradox, and Zeno’s paradox. These are either false in some sense or used to illustrate fallacious reasoning.

I don't think Zeno's paradoxes have truly been proven false.

> I don't think Zeno's paradoxes have truly been proven false.

Are you suggesting that there's a chance motion doesn't exist?

Re: Simpson's paradox

#89

Post by klodolph »

jldugger wrote: post #87
klodolph wrote: post #81

Earlier quoted context omitted.

There are various categories of paradoxes and different ways people have categorized them. Quine calls this one a “veridical paradox”, where it seems false but is true. Example of a different types of paradox are: any proof that 1=0, Russell’s Paradox, and Zeno’s paradox. These are either false in some sense or used to illustrate fallacious reasoning.

> “veridical paradox”, where it seems false but is true. > any proof that 1=0 So, 1=0 seems false but is true?

1=0 proofs are examples of a different type of paradox. It is an example of a paradox that does not seem true.

Re: Simpson's paradox

#90

Post by klodolph »

shawabawa3 wrote: post #88
missingrib wrote: post #86

Earlier quoted context omitted.

I don't think Zeno's paradoxes have truly been proven false.

> I don't think Zeno's paradoxes have truly been proven false. Are you suggesting that there's a chance motion doesn't exist?

It also has not been proven that real, correct proofs for 1=0 do not exist. Paradoxes are not all about proofs.
Post reply on HN