Why not interpolate the missing data points with similar patients data?
This must be about the confidence of the approach. Maybe interpolation would be overconfident too.
Statisticians use a technique that leverages randomness to deal with the unknown
31–40 of 55 posts
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#32It reminds me somewhat of dithering in signal processing.
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#33Does any living statistician come close to the level of Donald Rubin in terms of research impact? Missing data analysis, causal inference, EM algorithm, any probably more. He just walks around creating new subfields.
Also approximate Bayesian computation, principal stratification, and the Bayesian Bootstrap.
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#34Does any living statistician come close to the level of Donald Rubin in terms of research impact? Missing data analysis, causal inference, EM algorithm, any probably more. He just walks around creating new subfields.
Andrew Gelman?
Gelman has contributed to Bayesianism, hierarchial models and Stan is great, but that's not even close to what Rubin has done.
ps. Gelman was Rubin's doctoral student.
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#35Isn't this just Monte Carlo, or did I miss something?
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#36Not one mention of the EM algorithm, which is, as far as I can understand, is being described here ( https://en.m.wikipedia.org/wiki/Expectation%E2%80%93maximiza... ). It has so many applications, among which is estimating number of clusters for a Gaussian mixture model. An ELI5 intro: https://abidlabs.github.io/EM-Algorithm/
> It has so many applications, among which is estimating number of clusters for a Gaussian mixture model Any sources for that? As far as I remember, EM is used to calculate actual cluster parameters (means, covariances etc), but I'm not aware of any usage to estimate what number of clusters works best. Source: I've implemented EM for GMMs for a college assignment once, but I'm a bit hazy on the details.
you are right you still need the number of clusters
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#37Earlier quoted context omitted.
I feel like multiple imputation is fine when you have data missing at random. The problem is that data is never actually missing at random and there’s always some sort of interesting variable that confounds which pieces are missing
True true but how do you account for missing data based on variables you care about and those you don't? More specifically, how do you determine if the pattern you seem to be identifying is actually related to the phenomenon being measured and not an error in the measurement tools themselves? For example, a significant pattern of answers to "Yes / No: have you ever been assaulted?" are blank. This could be (A), respo…
> how do you determine if the pattern you seem to be identifying is actually related to the phenomenon being measured and not an error in the measurement tools themselves?
If the “which data is missing” information can be used be to compress the data that isn’t missing further than it can be compressed be alone, then the missing data is missing at least in part due to the phenomenon being measured. Otherwise, it’s not.
We’re basically just asking if K(non-missing data | which data is missing) A decent practical approximation might be to see if you can develop a model that predicts the non-missing data better when augmented with the “which information is missing” information than via self-prediction. That could be an interesting research project...
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#38Earlier quoted context omitted.
> It has so many applications, among which is estimating number of clusters for a Gaussian mixture model Any sources for that? As far as I remember, EM is used to calculate actual cluster parameters (means, covariances etc), but I'm not aware of any usage to estimate what number of clusters works best. Source: I've implemented EM for GMMs for a college assignment once, but I'm a bit hazy on the details.
you are right you still need the number of clusters
I've been out of the loop for stats for a while, but is there a viable approach for estimating ex ante the number of clusters when creating a GMM? I can think if constructing ex post metrics, i.e using a grid and goodness of fit measurements, but these feel more like brute forcing it
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#39Earlier quoted context omitted.
I don't find the language of the article full of "hype"; they describe the history of different forms of imputation from single to multiple to ML-based. The table is particularly useful as it describes what the article is all about in a way that can stick to students' minds. I'm very grateful for QuantaMagazine for its popular science reporting.
The Quanta articles usually have a gossipy style and are very low information density.
They're usually more science history than science. Who did what, when, and a basic ovnrview of why it's important.
Re: Statisticians use a technique that leverages randomness to deal with the unknown
#40Earlier quoted context omitted.
True true but how do you account for missing data based on variables you care about and those you don't? More specifically, how do you determine if the pattern you seem to be identifying is actually related to the phenomenon being measured and not an error in the measurement tools themselves? For example, a significant pattern of answers to "Yes / No: have you ever been assaulted?" are blank. This could be (A), respo…
> how do you determine if the pattern you seem to be identifying is actually related to the phenomenon being measured and not an error in the measurement tools themselves? If the “which data is missing” information can be used be to compress the data that isn’t missing further than it can be compressed be alone, then the missing data is missing at least in part due to the phenomenon being measured. Otherwise, it’s no…
There’s already a bunch of stats research on this problem. Some useful terms to look up are MCAR (missing completely at random) and MNAR (missing not at random)