Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians use a technique that leverages randomness to deal with the unknown

quantamagazine.org

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#51

Post by wdkrnls »

CrazyStat wrote: post #46
Jun8 wrote: post #17

Not one mention of the EM algorithm, which is, as far as I can understand, is being described here ( https://en.m.wikipedia.org/wiki/Expectation%E2%80%93maximiza... ). It has so many applications, among which is estimating number of clusters for a Gaussian mixture model. An ELI5 intro: https://abidlabs.github.io/EM-Algorithm/

EM can be used to impute data, but that would be single imputation. Multiple imputation as described here would not use EM since the goal is to get samples from a distribution of possible values for the missing data.

In other words, EM makes more sense. All this imputation stuff seems to me more like an effort to keep using obsolete modeling techniques.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#53

Post by CrazyStat »

wdkrnls wrote: post #51
CrazyStat wrote: post #46

Earlier quoted context omitted.

EM can be used to impute data, but that would be single imputation. Multiple imputation as described here would not use EM since the goal is to get samples from a distribution of possible values for the missing data.

In other words, EM makes more sense. All this imputation stuff seems to me more like an effort to keep using obsolete modeling techniques.

Absolutely not.

EM imputation (or single imputation in general) fails to account for the uncertainty in imputed data. You end up with artificially inflated confidence in your results (p-values too small, confidence/credible intervals too narrow, etc.).

Multiple imputation is much better.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#54

Post by disgruntledphd2 »

nyrikki wrote: post #49
disgruntledphd2 wrote: post #42

Earlier quoted context omitted.

Ok that's interesting. I profoundly disagree with your tone, but would really like to hear with you regard as good approaches to the problem of missing data (particularly where you have dropout from a study or experiment).

Perhaps looking into the issues with uncongeniality and multiple imputation may help, although I haven't looked at MI for a a long time so consider my reply as an attempt to be helpful vs authoritive. In another related intuition for a probable foot gun relates to learning linearly inseparable functions like XOR which requires MLPs. A single missing value in an XOR situation is far more challenging than participant d…

Honestly, I think that we're coming at this from very different perspectives.

Single imputation is garbage for accurate inference, as it reduces variance and thus confidence intervals as P(missing) increases.

MI is a useful method for alleviating this bias (though at the cost of a lot more compute).

That's why it gets used, and it's performed extremely well in real world analyses for basically my entire life (and I'm middle-aged now).

> especially in the unsupervised context.

I wouldn't use MI in an unsupervised context (but maybe some people do).

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#55

Post by xiaodai »

vouaobrasil wrote: post #6
xiaodai wrote: post #2

I don’t know. I find quanta articles very high noise. It’s always hyping something

I agree with that. I skip the Quanta magazine articles, mainly because the titles seem to be a little to hyped for my taste and don't represent the content as well as they should.

Same. I think it does disservice to math.
Post reply on HN