Machine learning works spectacularly well, but mathematicians aren’t sure why
1–10 of 137 posts
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#2It would be interesting to see what the purists arrive at
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#3It would be interesting to see what the purists arrive at
Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates."
Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC).
Of course, many mathematical statistical methods started off as hacks too. The invention of the L2 loss function, I imagine, must've gone a little like "alright, let's try to find the pattern here, let's draw the line through this series of points that minimizes the sum of the distance of each point to the line... hm, I don't like how this looks, let's try squaring the distances instead and then minimize that sum so we punish large deviations from our line a bit more... oh, that looks better."
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#4Note that the writer is the famed Ingrid Daubechies, who was one of the main driver behind wavelets and early works on sparse representations.
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#5It would be interesting to see what the purists arrive at
Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…
Actually the squaring was for a more practical reason. Lets try to minimize the sum of the distance, oops |y-ax| is not differentiable so we can't use calculus, lets square it instead.
We later realized this worked so well because it's max likelihood of the model y=ax+b+e where e is a gaussian error function. But that came much later.
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#6It would be interesting to see what the purists arrive at
Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…
> let's try squaring the distances instead and then minimize that sum so we punish large deviations from our line a bit more
More like so we can differentiate easy. Then if anyone asks, you wave your hands and mumble something along these lines :)
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#7The fact that the class of NN functions is universal is almost vacuous: the basic idea is that if you allow a "neuron" for every point in your input space then you can mimic any function you like (i.e. each neuron handles a single input). Obviously such representations become arbitrarily large.
Which almost immediately suggests a solution to why NN learning works: the processes that produce the types of datasets humans are interested in are produced by (effectively) polysized networks and you can probably say things like the probability of recovering a polysize function from polysize samples is high.
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#8It would be interesting to see what the purists arrive at
Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…
Regarding L2, I think it is just the most convenient way to do the maths from a set of observations (differentiability). Later one, the link with maximum likelihood-based methods was made, by Gauss.
In 'machine learning', structured learning by Vapnik and co (theory behind SVM), has a beautiful and strong mathematical underpinning. But in general, it is true we don't really have a good understanding of why learning algorithms work. The very notion of generalization to unobserved data is not well understood (I like D. Wolpert papers on that topic).
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#9Earlier quoted context omitted.
Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…
Actually the squaring was for a more practical reason. Lets try to minimize the sum of the distance, oops |y-ax| is not differentiable so we can't use calculus, lets square it instead. We later realized this worked so well because it's max likelihood of the model y=ax+b+e where e is a gaussian error function. But that came much later.
In grad school I asked that so many times trying to figure out why it was the "right" solution and this historical cheat caused so much cognitive dissonance.
Then again, on the other hand, "calculus works nicely" and "it's simple" probably are good spooky directors toward useful models anyway.
Re: Machine learning works spectacularly well, but mathematicians aren’t sure why
#10there are a number of features in machine learning that seem to have counterparts in statistics - stuff like m-like estimators as kernels (I think - correct me if wrong), information criteria for feature selection, that have existed a long time ago.
maybe it's the combination of sheer computing power and availability of data that allows better models. maybe we've never looked at algorithmically generated models because we also want a narrative (commonsensical explanation) to the model, not just a matter of algorithmically finding correlations between jelly beans and acne, say.
part of me thinks, ok the machines found something. now can we actually use that to understand the world, rather than build more recommendation algorithms? (haha). I'm not sure what an advisor will say about a doc student who says, let's just throw reams of data at a machine until we find a meaningful correlation, and then let's reason from the correlations (my guess is 'no', that's not really the scientific method is it).
I'm hopeful, and I don't think the answer to the question will come from academia.