Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Berkson's Paradox

twitter.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Berkson's Paradox

#31

Post by caddemon »

kgwgk wrote: post #28
caddemon wrote: post #22

Earlier quoted context omitted.

I think Berkson's paradox is more specific than just correlations arising from non-random sample selection. Correlations that are not representative of the general population could still be useful, if it's a meaningful correlation within some subgroup of interest. The problem is when the features you are correlating relate too closely to the features that were used for sample selection - then you can end up with a tr…

> I've always learned of Simpson's paradox as relating more to different sample sizes when partitioning data When you look at proportions based on binary outcomes it may be related to imbalanced groups but it's more general than that. In the context discussed here of correlations between continous variables the groups can be of similar size. See for example the chart here: https://towardsdatascience.com/simpsons-para…

Interesting, I only ever heard of Simpson's paradox in the context of comparing overall averages versus subgroup averages.

I guess this paradox could then be thought of as a special case of Simpson's paradox? Since the out group will exclude people with both traits there should also be a negative correlation there, which disappears in the overall population. But in Berkson's case it seems they're implying the subgroup correlation is spurious whereas with Simpson's it could go either way.

Re: Berkson's Paradox

#32

Post by kgwgk »

caddemon wrote: post #31
kgwgk wrote: post #28

Earlier quoted context omitted.

> I've always learned of Simpson's paradox as relating more to different sample sizes when partitioning data When you look at proportions based on binary outcomes it may be related to imbalanced groups but it's more general than that. In the context discussed here of correlations between continous variables the groups can be of similar size. See for example the chart here: https://towardsdatascience.com/simpsons-para…

Interesting, I only ever heard of Simpson's paradox in the context of comparing overall averages versus subgroup averages. I guess this paradox could then be thought of as a special case of Simpson's paradox? Since the out group will exclude people with both traits there should also be a negative correlation there, which disappears in the overall population. But in Berkson's case it seems they're implying the subgrou…

> Since the out group will exclude people with both traits there should also be a negative correlation there

Not necessarily. Imagine the traits are distributed uniformly and independently in [-1 1]. There is no correlation:

    ******
    ******
    ******
    ******
    ******
    ******
If you select people with at least one positive trait you will find negative correlation in the group + but the correlation will still be zero in the group -.

    ++++++
    ++++++
    ++++++
    ---+++
    ---+++
    ---+++

Re: Berkson's Paradox

#33

Post by caddemon »

kgwgk wrote: post #32
caddemon wrote: post #31

Earlier quoted context omitted.

Interesting, I only ever heard of Simpson's paradox in the context of comparing overall averages versus subgroup averages. I guess this paradox could then be thought of as a special case of Simpson's paradox? Since the out group will exclude people with both traits there should also be a negative correlation there, which disappears in the overall population. But in Berkson's case it seems they're implying the subgrou…

> Since the out group will exclude people with both traits there should also be a negative correlation there Not necessarily. Imagine the traits are distributed uniformly and independently in [-1 1]. There is no correlation: ****** ****** ****** ****** ****** ****** If you select people with at least one positive trait you will find negative correlation in the group + but the correlation will still be zero in the gro…

Makes sense, I was picturing more of a diagonal boundary but you're right the paradox doesn't specify the shape of the boundary. Thanks!

Re: Berkson's Paradox

#34

Post by Pyramus »

FeepingCreature wrote: post #27
junippor wrote: post #13

Earlier quoted context omitted.

> The latter appears when analyzing subgroups gives a different result than analyzing the pooled data. > The former is about correlations that appear in samples which are not representative of the general population, due to the way that those samples are selected. You just said the same thing twice. Think about it. For one you used terms like "subgroups" and "pooled data" and for the other "samples" and "general popu…

Simpson's paradox: analyzing trends per subgroup can give a different result than pooled data. Berkson's paradox: analyzing a single subgroup selected with a function aggregating two traits (additively?) will indicate an anticorrelation between the traits. Simpson's paradox says you can't judge group trends from subgroup trends. Berkson's paradox says given a group selected in a specific way , it will have a certain…

Yes and no.

Berkson's paradox is a special case of Simpson's for the two subgroups selected and non-selected.

The difference is that Berkson's paradox involves selecting the subgroup a posteriori and in a particular way, Simpson's paradox assumes a selection a priori.

Re: Berkson's Paradox

#35

Post by kgwgk »

Pyramus wrote: post #34
FeepingCreature wrote: post #27

Earlier quoted context omitted.

Simpson's paradox: analyzing trends per subgroup can give a different result than pooled data. Berkson's paradox: analyzing a single subgroup selected with a function aggregating two traits (additively?) will indicate an anticorrelation between the traits. Simpson's paradox says you can't judge group trends from subgroup trends. Berkson's paradox says given a group selected in a specific way , it will have a certain…

Yes and no. Berkson's paradox is a special case of Simpson's for the two subgroups selected and non-selected. The difference is that Berkson's paradox involves selecting the subgroup a posteriori and in a particular way, Simpson's paradox assumes a selection a priori.

Another difference is that Simpson's "paradox" involves all the subgroups that the full population is partitioned into, unlike Berkson's "paradox".

Re: Berkson's Paradox

#36

Post by junippor »

kgwgk wrote: post #35
Pyramus wrote: post #34

Earlier quoted context omitted.

Yes and no. Berkson's paradox is a special case of Simpson's for the two subgroups selected and non-selected. The difference is that Berkson's paradox involves selecting the subgroup a posteriori and in a particular way, Simpson's paradox assumes a selection a priori.

Another difference is that Simpson's "paradox" involves all the subgroups that the full population is partitioned into, unlike Berkson's "paradox".

I like how you put paradox in quotes. I also annoys me when people call these things paradoxes. They're more properly called counter-intuitive phenomena. I wonder if there's a single-word name for that.

Re: Berkson's Paradox

#37

Post by kgwgk »

junippor wrote: post #36
kgwgk wrote: post #35

Earlier quoted context omitted.

Another difference is that Simpson's "paradox" involves all the subgroups that the full population is partitioned into, unlike Berkson's "paradox".

I like how you put paradox in quotes. I also annoys me when people call these things paradoxes. They're more properly called counter-intuitive phenomena. I wonder if there's a single-word name for that.

paradox :-)

Re: Berkson's Paradox

#38

Post by junippor »

kgwgk wrote: post #37
junippor wrote: post #36

Earlier quoted context omitted.

I like how you put paradox in quotes. I also annoys me when people call these things paradoxes. They're more properly called counter-intuitive phenomena. I wonder if there's a single-word name for that.

paradox :-)

So I actually checked and... turns out you're right.

According to Wikipedia, "paradox" can either mean "logically self-contradictory statement" or a "statement that runs contrary to one's expectation". I always thought that it meant the former only.

These two concepts should really really have separate words.

Re: Berkson's Paradox

#39

Post by getlawgdon »

junippor wrote: post #38
kgwgk wrote: post #37

Earlier quoted context omitted.

paradox :-)

So I actually checked and... turns out you're right. According to Wikipedia, "paradox" can either mean "logically self-contradictory statement" or a "statement that runs contrary to one's expectation". I always thought that it meant the former only. These two concepts should really really have separate words.

You "checked Wikipedia," is that it? You're done now?

Re: Berkson's Paradox

#40

Post by junippor »

getlawgdon wrote: post #39
junippor wrote: post #38

Earlier quoted context omitted.

So I actually checked and... turns out you're right. According to Wikipedia, "paradox" can either mean "logically self-contradictory statement" or a "statement that runs contrary to one's expectation". I always thought that it meant the former only. These two concepts should really really have separate words.

You "checked Wikipedia," is that it? You're done now?

You sound like you're trying to make a point. Make a point.
Post reply on HN