Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simpson’s Paradox (2016)

forrestthewoods.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Simpson’s Paradox (2016)

#31

Post by heavenlyblue »

srean wrote: post #18

Simpson's Paradox is one of the many phenomena that shows how different applied ML is from regular software engineering. Another one is feedback loops between decomposed subproblems. In ML encapsulation, shielding away of inner details often does not work. One needs to know what is happening on the other side of the abstraction boundary. This is a problem for managers and PM coning to ML from a purely software engine…

>> In ML encapsulation, shielding away of inner details often does not work.

I call bs on this. It’s just that we haven’t yet invented a consistent type theory on top of ML.

Re: Simpson’s Paradox (2016)

#33

Post by apathy »

heavenlyblue wrote: post #31
srean wrote: post #18

Simpson's Paradox is one of the many phenomena that shows how different applied ML is from regular software engineering. Another one is feedback loops between decomposed subproblems. In ML encapsulation, shielding away of inner details often does not work. One needs to know what is happening on the other side of the abstraction boundary. This is a problem for managers and PM coning to ML from a purely software engine…

>> In ML encapsulation, shielding away of inner details often does not work. I call bs on this. It’s just that we haven’t yet invented a consistent type theory on top of ML.

“Just”

This would be like saying “it’s just that we haven’t proven P!=NP” in CS. Best of luck.

Meanwhile applied people will deal with the problem by model diagnostics and sensitivity analysis as has been done for decades. I can’t wait for the next AI winter to come. So tired of this handwaving by people who don’t seem to have practical experience.

Re: Simpson’s Paradox (2016)

#34

Post by gumby »

lettergram wrote: post #26

Idk why the 2016 needs to be in the title here. I understand for date relevant content, but this is not.

It’s not uncommon for something clear and expository to be invalidated and putting the date in the title may cause someone who knows the domain to say “oh, this must be from before this was all invalidated” and post a useful reference as a comment.

Well it’s a convention and some conventions (like this) are better applied uniformly than allowing for acidental editorialiation.

Re: Simpson’s Paradox (2016)

#35

Post by apathy »

rwilson4 wrote: post #8
sopooneo wrote: post #5

In simples case at least, such as with the kidney stones, can we reduce our risk of reaching wrong conclusions by increasing our sample size of patients and randomizing which receive each treatment?

Yes absolutely! Random assigment along with statistical power and significance considerations does indeed allow one to draw causal conclusions. It’s the gold standard for causal inference.

I You’re doing it right.

Re: Simpson’s Paradox (2016)

#36

Post by lettergram »

city41 wrote: post #32
lettergram wrote: post #26

Idk why the 2016 needs to be in the title here. I understand for date relevant content, but this is not.

Another reason to put the year is it helps people decide if they’ve read it before.

I doubt that helps. For instance, this is from 2016, had you read it before?

I’m simply suggesting it because I don’t think it adds anything to the conversation. In addition, I’ve seen this being added more often lately and I worry it makes people think it’s date relevant (as I did) or that it somehow provides less value due to some time delay.

Re: Simpson’s Paradox (2016)

#37

Post by apathy »

AnthonyMouse wrote: post #29
rwilson4 wrote: post #21

Earlier quoted context omitted.

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit. [0] Causality, Judea Pearl [1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens an…

The trouble is you can't fix large numbers of statistical confounders with more data because there is a limit for how many factors you can control for before the measurement error overwhelms the signal. To do statistical controls, you essentially sort the data by category, so that you're not just comparing black people with white people, you're comparing middle class 18 year old black female college applicants with c…

I hate to take “both sides” but in the absence of confounding by indication, you can often use propensity scoring within robust models to decrease these impacts.

Mind you, the problem with non random and undetected sampling bias is that it can be subtle. See for example https://www.nytimes.com/2018/08/06/upshot/employer-wellness-...

Re: Simpson’s Paradox (2016)

#38

Post by gok »

The last example of software optimization causing mean slowdown because users actually use the software is so true. Another example I've seen is better ML models causing accuracy to go down; users try harder things.

Re: Simpson’s Paradox (2016)

#39

Post by Matumio »

AnthonyMouse wrote: post #29
rwilson4 wrote: post #21

Earlier quoted context omitted.

You're right, and in some instances it is possible to draw causal conclusions from observational data. See [0] and [1] for two pretty different perspectives. But for this to work, you need a lot of data: both lots of units (e.g. people), and a lot of information about each individual unit. [0] Causality, Judea Pearl [1] Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Guido Imbens an…

The trouble is you can't fix large numbers of statistical confounders with more data because there is a limit for how many factors you can control for before the measurement error overwhelms the signal. To do statistical controls, you essentially sort the data by category, so that you're not just comparing black people with white people, you're comparing middle class 18 year old black female college applicants with c…

Even if you had infinite data, you are not allowed to just control for everything you measured. You still need to bring in your causal knowledge. E.g. you probably shouldn't control for body weight if it was measured a month after the treatment.

The point is, if you use your causal knowledge in a smart way, you can also draw strong conclusions from just observational data.

Re: Simpson’s Paradox (2016)

#40

Post by srean »

apathy wrote: post #33
heavenlyblue wrote: post #31

Earlier quoted context omitted.

>> In ML encapsulation, shielding away of inner details often does not work. I call bs on this. It’s just that we haven’t yet invented a consistent type theory on top of ML.

“Just” This would be like saying “it’s just that we haven’t proven P!=NP” in CS. Best of luck. Meanwhile applied people will deal with the problem by model diagnostics and sensitivity analysis as has been done for decades. I can’t wait for the next AI winter to come. So tired of this handwaving by people who don’t seem to have practical experience.

Whoah! You have quite a treasure trove in your favorites. The possibility of getting some work done vanished as soon as I found that.
Post reply on HN