Live data from Hacker News

Statisticians use a technique that leverages randomness to deal with the unknown

quantamagazine.org

41–50 of 55 posts

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#41

Earlier quoted context omitted.

you are right you still need the number of clusters

I've been out of the loop for stats for a while, but is there a viable approach for estimating ex ante the number of clusters when creating a GMM? I can think if constructing ex post metrics, i.e using a grid and goodness of fit measurements, but these feel more like brute forcing it

Unsupervised learning is hard, and the pick K problem is probably the hardest part.

For PCA or factor analysis, there's lots of ways but without some way of determining ground truth it's difficult to know if you've done a good job.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#42
post #4

Earlier quoted context omitted.

There are plenty of legit. articles that discuss/survey imputation in ML/AI: https://scholar.google.com/scholar?hl=de&as_sdt=0%2C5&q=%22m...

The prestigious journal "Artificial intelligence in medicine"? No. Just because it's on Google scholar doesn't mean it's worth anything. These are almost all trash. On the first page there's one maybe legit paper in an ok venue as far as ML is concerned (KDD; an adjacent field to ML) that's 30 years old. No. AI/ML folks don't do imputation on our datasets. I cannot think of a single major dataset in vision, nlp, or r…

Ok that's interesting. I profoundly disagree with your tone, but would really like to hear with you regard as good approaches to the problem of missing data (particularly where you have dropout from a study or experiment).

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#43

Earlier quoted context omitted.

you are right you still need the number of clusters

I've been out of the loop for stats for a while, but is there a viable approach for estimating ex ante the number of clusters when creating a GMM? I can think if constructing ex post metrics, i.e using a grid and goodness of fit measurements, but these feel more like brute forcing it

Is the question fundamentally: what's the relative likelihood of each number or clusters?

If so then estimating the marginal likelihood of each one and comparing them seems pretty reasonable?

(I mean in the sense of Jaynes chapter 20.)

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#44
Does anyone else find it maddeningly difficult to read Quanta articles on desktop, because the nav bar keeps dancing around the screen? One of my least favorite web design things is the "let's move the bar up and down the screen depending on what direction he's scrolling, that'll really mess with him." I promise I can find the nav bar on my own when I need it.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#45

Earlier quoted context omitted.

you are right you still need the number of clusters

I've been out of the loop for stats for a while, but is there a viable approach for estimating ex ante the number of clusters when creating a GMM? I can think if constructing ex post metrics, i.e using a grid and goodness of fit measurements, but these feel more like brute forcing it

There are Bayesian nonparametric methods that do this by putting a dirichlet process prior on the parameters of the mixture components. Both the prior specification and the computation (MCMC) are tricky, though.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#46
post #17

Not one mention of the EM algorithm, which is, as far as I can understand, is being described here ( https://en.m.wikipedia.org/wiki/Expectation%E2%80%93maximiza... ). It has so many applications, among which is estimating number of clusters for a Gaussian mixture model. An ELI5 intro: https://abidlabs.github.io/EM-Algorithm/

EM can be used to impute data, but that would be single imputation. Multiple imputation as described here would not use EM since the goal is to get samples from a distribution of possible values for the missing data.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#47
post #11

Does any living statistician come close to the level of Donald Rubin in terms of research impact? Missing data analysis, causal inference, EM algorithm, any probably more. He just walks around creating new subfields.

Mike Jordan, Tibshirani, Emmanuel Candes

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#48

Earlier quoted context omitted.

Very doubtful. There are plenty of drugs that get approved and are of questionable value. Plenty of procedures that turn out to be not useful. The incentives in industry are even worse because everything depends on lying with data if you can do it.

I assume by "in industry" they meant in jobs where you are doing data analysis to support decisions that your employer is making. This would be any typical "data scientist" job nowadays. There the consequences of BSing are felt by the entity that pays you, and will eventually come back around to you. The incentives in medicine are more similar to those in academia, where your job is to cook up data that convinces som…

Yes, precisely this! I’ve seen more than a few people fired for generating BS analyses that didn’t help their employer, especially in tech where scrutiny is immense when things start to fail.

Re: Statisticians use a technique that leverages randomness to deal with the unknown

#49

Earlier quoted context omitted.

The prestigious journal "Artificial intelligence in medicine"? No. Just because it's on Google scholar doesn't mean it's worth anything. These are almost all trash. On the first page there's one maybe legit paper in an ok venue as far as ML is concerned (KDD; an adjacent field to ML) that's 30 years old. No. AI/ML folks don't do imputation on our datasets. I cannot think of a single major dataset in vision, nlp, or r…

Ok that's interesting. I profoundly disagree with your tone, but would really like to hear with you regard as good approaches to the problem of missing data (particularly where you have dropout from a study or experiment).

Perhaps looking into the issues with uncongeniality and multiple imputation may help, although I haven't looked at MI for a a long time so consider my reply as an attempt to be helpful vs authoritive.

In another related intuition for a probable foot gun relates to learning linearly inseparable functions like XOR which requires MLPs.

A single missing value in an XOR situation is far more challenging than participant dropouts causing missing data.

Specifically the problem is counterintuitively non-convex, with multiple possibilities for convergence without information in the corpus to know which may be true.

That is a useful lens in my mind, where I think of the manifold being pushed down in opposite sectors as the kernel trick.

Another potential lens to think about it is that in medical studies the assumption is that there is a smooth and continuous function, while in learning, we are trying to find a smooth continuous function with minimal loss.

We can't assume that the function we need to learn is smooth, but autograd specifically limits what is learnable and simplicity bias, especially with feed forward networks is an additional concern.

One thing that is common for people to conflate is the fact that a differentiable function is probably smooth and continuous.

But the set of continuous functions that is differentiable _anywhere_ is a meger set.

Like anything in math and logic, the assumptions you can make will influence what methods work.

As ML is existential quantification, and because it is insanely good at finding efficient glitches in the matrix, within the limits of my admittedly limited knowledge, MI would need to be a very targeted solution with a lot of care to avoid set shattering from causing uncongeniality, especially in the unsupervised context.

Hopefully someone else can provide a better productive insights.

Post reply on HN