Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

71–80 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#71

Post by mziel »

dasboth wrote: post #43
nl wrote: post #37

Earlier quoted context omitted.

It really kinda is. In the Ilya Sutskever Talking Machines podcast[1] he describes exactly how it is "magic". He talks about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained. And then they discovered how to initia…

I was just listening to that podcast yesterday and loved the way he presented this information. I know next to nothing about NNs but I could pretty much follow everything that he said (from an intuitive sense, anyway, if not a technical one). This "magic" of converging networks reminds me of how ensemble methods, such as random forests, are effective but people aren't sure why. There's certainly (AFAIK) no theoretica…

There are. It's all about variance reduction. See the Breiman's paper on Bagging. There's nothing special about random forest though (apart from the fact that a decision tree is a good learner, because of the nonlinearities for example), you can you ensemble learning with any "basic" learner.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#72

Post by eli_gottlieb »

Xcelerate wrote: post #64
eli_gottlieb wrote: post #36

Earlier quoted context omitted.

Actually, there's been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent. https://github.com/gregversteeg/CorEx http://arxiv.org/abs/1406.1222 http://arxiv.org/abs/1410.7404 Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting gr…

Those papers are very interesting. I was particularly surprised that the method exactly recovers the Big-5 components of personality from the results of the survey questions. The whole idea of "simplest model that explains most of the data" has always been very appealing to me. The concept is closely tied to reproducing kernel Hilbert spaces, which have recently experienced a revival in interest due to the represente…

>I was particularly surprised that the method exactly recovers the Big-5 components of personality from the results of the survey questions.

Why? Is that a standard dataset? They could have just cherrypicked tasks.

I also noticed that they seem to only have the actual computation working for discrete random variables (integer features) right now, which limits its applicability. They also seem to use mass-function estimators, which again can work well for discrete data while becoming statistically intractable when dealing with continuous random variables.

>The whole idea of "simplest model that explains most of the data" has always been very appealing to me.

In this case it's a bit more like, "The set of latent variables that best screen off the observables from each-other."

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#73

Post by TTPrograms »

dave_sullivan wrote: post #56

In addition to "We need a cluster for deep learning", the second most popular mostly untrue thing I hear is "We have no idea how neural networks learn". However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they're on to something . There are also many papers that discuss possible theoretical implications of experimental results. But what…

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#74

Post by argonaut »

crazypyro wrote: post #20
ilzmastr wrote: post #16

Earlier quoted context omitted.

Yea it's more like the opposite, in theory you only need 1 hidden layer to get the same capacity in the network as with more hidden layers, but in practice, since auto encoders, many hidden layers gradually decreasing in size are easier to get to from local energy minimums to globally low energy minimums

Yes, people took the Universal approximation theorem[0] as evidence that they only need 1 layer, but there is zero guarantee of efficiency. A single hidden layer may mean many magnitudes more neurons needed over a n-hidden layer, n > 1 network, which could cause an unrealistic training time. Having multiple layers can reduce this training time with an optimal structure. [0] https://en.wikipedia.org/wiki/Universal_app…

Nobody ever took that theorem seriously. Deep nets were around since the 90s.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#75

Post by shmel »

TTPrograms wrote: post #73
dave_sullivan wrote: post #56

In addition to "We need a cluster for deep learning", the second most popular mostly untrue thing I hear is "We have no idea how neural networks learn". However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they're on to something . There are also many papers that discuss possible theoretical implications of experimental results. But what…

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Mainly because in high-dimensional space saddle points are much more common and many local minima have very close fitness. Look for example http://arxiv.org/abs/1405.4604 and following papers.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#76

Post by tianlins »

it seems like a general phenomenon that our scientific understanding decreasing with the complexity of the system. in classic physics, we could have very clean equations capturing the dynamics of a system. this is not true any more in chemistry/biology/social science. modern deep learning pipeline, compared to its shallow counterparts, gains more complexity, so it's not surprising at all to me that we are not able to understand it well at the moment.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#77

Post by the_duck »

>In the last 15 years or so, researchers have created a number of tools to probe the geometry of these hidden structures. For example, you might build a model of the surface by first zooming in at many different points. At each point, you would place a drop of virtual ink on the surface and watch how it spread out.

It sounds to me like this 'ink drop' is a metaphor to explain some state-of-the-art dimensionality reduction technique. Does anyone know the common name of this technique?

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#78

Post by robotresearcher »

TTPrograms wrote: post #73
dave_sullivan wrote: post #56

In addition to "We need a cluster for deep learning", the second most popular mostly untrue thing I hear is "We have no idea how neural networks learn". However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they're on to something . There are also many papers that discuss possible theoretical implications of experimental results. But what…

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Because the solution space is convex if you've chosen your representation well.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#79

Post by HockeyPlayer »

vonklaus wrote: post #52

There was a great article called The Space Doctor's Big Idea , published in the New Yorker a few months back. It explained Einstein's theories in the top 1000s English words used in America. While sone might wave their hands and call this an eli5, it was quite well done and between that and some xkcd comics I was able to learn enough to ubderstand how orbit works and a few other awesome facts. I am interested in find…

You may enjoy The Master Algorithm: http://www.amazon.com/Master-Algorithm-Ultimate-Learning-Mac...

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#80

Post by modeless »

TTPrograms wrote: post #73
dave_sullivan wrote: post #56

In addition to "We need a cluster for deep learning", the second most popular mostly untrue thing I hear is "We have no idea how neural networks learn". However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they're on to something . There are also many papers that discuss possible theoretical implications of experimental results. But what…

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this becomes less and less likely, and for a million dimensions it's vanishingly unlikely.

What you get instead of local minima are saddle points, where some dimensions are curving up and some are curving down. Saddle points can also be problematic for optimization but they can be dealt with using fancy optimization techniques. For a more rigorous explanation, see http://arxiv.org/abs/1406.2572

Post reply on HN