Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

91–100 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#91

Post by gone35 »

j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

This is still a long way away from truly understanding the representational power of deep nets, but it's the first general progress that has been made since the early 90's.

Exactly. Someone should tell Yann LeCun this --see e.g. Section 3.2 in [1], or pretty much every time he brings up circuit complexity theory to automagically imply that "most functions representable compactly with a deep architecture would require a very large number of components if represented with a shallow one." [ibid, p.14]

[1] http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#92

Post by sunstone »

modeless wrote: post #80
TTPrograms wrote: post #73

Earlier quoted context omitted.

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this…

This is a very cool insight for the uninitiated. Thanks.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#93

Post by modeless »

bendbro wrote: post #90
modeless wrote: post #80

Earlier quoted context omitted.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this…

Wouldn't it still be likely to settle on a local minima when the deciding factors for the existence of a local minima are limited to those that contribute to the likelihood of a single output category, and not whether all functions are curving up? An example I can think of would be an absurd million input neural network, where one of the inputs only has a pronounced effect on one of the outputs. It seems like it woul…

An interesting question. A million input neural network isn't necessarily absurd. Images are very high dimensional. One could easily imagine a one megapixel input to a neural net. But in natural images no single pixel is indicative of any single image characteristic by itself. I think this isn't a coincidence but a common characteristic of "natural" high dimensional data, on which neural nets tend to work well. So yes, I'd say what you've described is not likely for a large category of "natural" high dimensional data which probably includes most of the data we care about in the real world.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#94

Post by kuschku »

apozem wrote: post #62
vonklaus wrote: post #52

There was a great article called The Space Doctor's Big Idea , published in the New Yorker a few months back. It explained Einstein's theories in the top 1000s English words used in America. While sone might wave their hands and call this an eli5, it was quite well done and between that and some xkcd comics I was able to learn enough to ubderstand how orbit works and a few other awesome facts. I am interested in find…

You may have noticed this, but that New Yorker article was written by Randall Munroe, the creator of XKCD.

Which makes sense, considering his last book is "Thing Explainer", and explains stuff like the "Up Goer Five" (Saturn V): https://xkcd.com/1133/

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#95

Post by diegoperini »

kuschku wrote: post #94
apozem wrote: post #62

Earlier quoted context omitted.

You may have noticed this, but that New Yorker article was written by Randall Munroe, the creator of XKCD.

Which makes sense, considering his last book is "Thing Explainer", and explains stuff like the "Up Goer Five" (Saturn V): https://xkcd.com/1133/

This deserves to be put here:

https://www.youtube.com/watch?v=2p_8gx-XHJo

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#96

Post by robotresearcher »

gipp wrote: post #89
robotresearcher wrote: post #78

Earlier quoted context omitted.

Because the solution space is convex if you've chosen your representation well.

This is definitely not the case for general neural networks, though.

If gradient descent is working reliably, the problem is convex. See the sibling comments for the intuition for large dimensional spaces.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#97

Post by TTPrograms »

modeless wrote: post #80
TTPrograms wrote: post #73

Earlier quoted context omitted.

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this…

By that argument gradient descent should work for any classifier with a sufficiently high number of independent parameters.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#98

Post by tjr »

amelius wrote: post #59

This is the reason I am not so much interested in this field. It is too much "let's try this and see what happens" rather than really engineering a solution. I guess more people feel that way, and I guess that is a good thing, otherwise everybody would now be working on AI, since it is such a promising field.

Some AI pioneers seem to feel that rule-based symbolic solutions still have a place, e.g.

http://dustycloud.org/blog/sussman-on-ai/

If we are depending on AI systems to make important decisions, possibly life-or-death decisions, shouldn't we understand how they come to the conclusions that they do?

I cannot deny the tremendous advances in neural networks recently, but I find Sussman's point compelling.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#99

Post by shas3 »

I think another interesting analogy (in terms of field-development trajectory) to deep learning is the whole idea of l1-regularized regression or LASSO or sparsity priors or sparse signal processing ([1] goes by many names depending on which field you work in). The whole idea is that by penalizing 'dense' solutions to a regression problem, you can 'promote' 'sparse' solutions like the ones that occur in many many applications. This had been used by various communities with some theoretical justification for years, at least since the 1970s. However, the real theoretical breakthroughs framing this problem in something close to the rigor and usefulness of Shannon's sampling theorem, came in 2004 from two papers independently: David Donoho; and Emmanuel Candes, Justin Romberg, Terence Tao. Theoretical CS community also got close to the answer in their work in the late 1990s and early 2000s in many papers on random projections and sketching. There are many connections between deep learning and ideas in the broad area of sparse regression. But one epistemological point of intersection is how theoretical results pertain to asymptotically large cases, but in applications work very well even with much smaller systems.

[1] https://en.wikipedia.org/wiki/Compressed_sensing

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#100

Post by aswanson »

vidarh wrote: post #42
algomanic wrote: post #26

Earlier quoted context omitted.

Could you explain weight matrix "spurious interactions"?

Very, very roughly a neural network consists of a bunch of connected nodes that you propagate signals through. Each of those connections carries a weight that affects how the signal gets propagated through the rest of the network. Let's say a specific problem has only one very specific set of connections that matters. You'll eventually add up with weights that reflects that, but that doesn't prevent a lot of other co…

Is that why sparsity terms have been introduced into objective functions?
Post reply on HN