Live data from Hacker News

Simpson's paradox

en.wikipedia.org

101–110 of 111 posts

Re: Simpson's paradox

#101
post #97

Earlier quoted context omitted.

Sure, but that doesn't contradict what I said. From the Wikipedia article I referenced: "For specific combinations of orientations, perfect (rather than statistical) correlations between the three polarizations are predicted by both local hidden variable theory (aka "local realism") and by quantum mechanical theory, and the predictions may be contradictory." "Perfect" correlations means, as the parenthetical comment…

You said > the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all However, one needs to use statistics to even show GHZ works. That does sound contradictory to me. The correlations you get in experiments are never perfect and in this case they can be pretty far from perfect.

> one needs to use statistics to even show GHZ works

Not for the particular cases described in the quote I gave. For a complete verification of all the GHZ theorem's predictions, yes, you need to do statistics, because some of those predictions are probabilistic.

> The correlations you get in experiments are never perfect

In some cases, like the ones described in the quote I gave, it isn't a matter of correlations. You have contradictory results predicted by two different models, each prediction being 100% certain according to the model. You don't need any statistics to test that: just do one single run and see which way it comes out.

Re: Simpson's paradox

#102
post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

Interesting list of Epicurean theories. Any good starting point that addresses them together and how multiple possibility thinking is related?

I can't recommend enough straight up reading Lucretius'sNature of Things.

But one of the examples in there of how their methodology ends up successful is when he's discussing the possible reasons lighting and thunder occur at different times.

One possibility thrown out is that they are actually occurring at different times. But another is that they occur at the same time but one takes longer to reach the viewer than the other.

On its own, these two ideas don't indicate the correct answer.

But then Lucretius ties the latter to another observation - that this seems similar to how a drummer in the distance can be seen to beat the drums before you would hear the drums.

Essentially in an age without the methodology of testable predictions, they circumvented that shortcoming by considering multiple hypotheses for multiple naturally occurring observations and looking for overlaps between them.

This seems to have pointed them in the correct direction on a number of major topics, especially relative to their contemporaries who were generally arguing for a particular hypothesis with various appeals to rhetoric or principle (like Aristotle claiming the leader of a bee hive couldn't be female because it had a stinger and "the gods don't give women weapons").

The times the Epicureans completely miss the mark is generally when they disregarded their principle of avoiding false negatives and discounted things with insufficient observational evidence (for example, they had pretty bad cosmology and they rejected the Stoic pre-gravity due to their incorrect base assumption of infinite amounts of matter). The times they kept an open mind and considered how concepts overlapped, even when they were wrong about the 'why' of an initial assumption they were often correct in secondary assumptions when tying it into multiple other systems and observations.

Re: Simpson's paradox

#103
post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

I'm having trouble making the connection between Simpson's paradox and the Epicureans. Can you help me out?

Ellenberg says the way to avoid falling into Simpson's paradox is to keep multiple views of the data in mind when doing analyses.

Let's say you were in ancient Greece, and you separately observe a drummer on a hill bang a drum before you hear it.

Then on another day you see lightning before you hear thunder.

If you consider each event on its own, a perfectly logical explanation is that there's something unique to drums that slows down the sound from them so it takes longer to reach you, and that lighting and thunder occur at different points in time.

But if you consider the set of both events together, a hypothesis that solves both at the same time is that things you hear take longer to reach you over long distances than things you see.

This was actually one of the examples directly from Lucretius, who in discussing the multiple hypotheses for why lightning and thunder occur at different times tied his suggestion that they occur at the same time but have different travel speeds to his observations of drummers on hills.

It's less specifically Simpson's paradox and more the general value of Ellenberg's analytical advice on avoiding the Simpson's paradox as having been at the root of the success (in hindsight) of one of the wiser philosophy schools in antiquity.

Re: Simpson's paradox

#104

Earlier quoted context omitted.

> I don't think Zeno's paradoxes have truly been proven false. Are you suggesting that there's a chance motion doesn't exist?

It also has not been proven that real, correct proofs for 1=0 do not exist. Paradoxes are not all about proofs.

I'm fairly sure it has actually, under some axiom schemas.

Re: Simpson's paradox

#105
post #12

I absolutely love the Ellenberg quote: > Mathematician Jordan Ellenberg argues that Simpson's paradox is misnamed as "there's no contradiction involved, just two different ways to think about the same data" and suggests that its lesson "isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once." Keeping multiple possibilities in mind at once was what allow…

> that in order for free will to exist the quanta making up matter had to have multiple possible results under the same governing physical laws and conditions

That seems like a strange and out-of-place statement, unless I'm misunderstanding it.

I assume this is talking about Quantum mechanics, but I don't think this represents Quantum mechanics or free will correctly, and I doubt the Epicureans knew anything about QM at all.

Re: Simpson's paradox

#106

When I taught intro stats many years ago I used to use house prices as a nice example of Simpson's Paradox (with actual data, for the students to investigate as part of a computational lab). The data I had was on US house sales from 2008, so it's 15 years out of date now--perhaps things have changed since. At the time, the average price for single-family house sales was higher for houses without central AC than for h…

Ahh.. I've seen this pattern before, skim-read the wikipedia article and yet it didn't click.

Your comment made it all fit together. May not be the most accurate (I dont know) but it helped me. Thank you.

If you have any written material I can access, from your courses, I'd be interested to read.

Thanks for this.

Re: Simpson's paradox

#107
post #104

Earlier quoted context omitted.

It also has not been proven that real, correct proofs for 1=0 do not exist. Paradoxes are not all about proofs.

I'm fairly sure it has actually, under some axiom schemas.

Could you give an example? It would have to be something that doesn’t contain Peano numbers, due to Gödel’s incompleteness theorem…

Re: Simpson's paradox

#108
post #104

Earlier quoted context omitted.

I'm fairly sure it has actually, under some axiom schemas.

Could you give an example? It would have to be something that doesn’t contain Peano numbers, due to Gödel’s incompleteness theorem…

Well I was being a bit cheeky in my answer.

Since you didn't specify under what system we need to prove that 0=1 doesn't exist, I vaguely remembered or figured there was a simpler version of arithmetic under which that concept makes sense, but which wouldn't be strong enough to fall into incompleteness territory (so it would have to be weaker than Peano arithmetic, like you said).

So I just looked it up (thanks ChatGPT), and there's something called Presburger Arithmetic, to quote Wikipedia: (https://en.wikipedia.org/wiki/Presburger_arithmetic)

> The signature of Presburger arithmetic contains only the addition operation and equality, omitting the multiplication operation entirely. The theory is computably axiomatizable; the axioms include a schema of induction.

So a very dumbed-down version of arithmetic, but which does contain a notion like 0=1, and which is complete and consistent, so it can't contain a proof of 0=1.

Obviously, this is probably not the kind of thing you meant, hence my cheekily bringing it up :)

Re: Simpson's paradox

#110
post #76
post #23

I was reading the example of UC Berkely appearing to have gender bias in the admissions and read the following: “it showed that women tended to apply to more competitive departments with lower rates of admission, even among qualified applicants (such as in the English department), whereas men tended to apply to less competitive departments with higher rates of admission (such as in the engineering department)” That’s…

Since no one gave the obvious[1] answer: The data is for application to graduate programs. There is a ton more funding for engineering, and many/most students going for a PhD in engineering don't pay for it. There's very little funding in the humanities, and most students are not willing to pay high costs for a PhD in the humanities, so the department tightly restricts admission. As a result, it's easier to get into…

Ah yup, I definitely read that in the context of undergraduate degrees, I think you’re probably right.
Post reply on HN