Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

131–137 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#131

Post by [deleted] »

tacos wrote: post #87
gnaritas wrote: post #84

Earlier quoted context omitted.

> Age is unrelated to wisdom and I'm talking about the full experience. That's not true; age and wisdom are quite related, just not directly causal. They are correlated. Older people are generally wiser, it's just not a guarantee, nor is it impossible for young people to be wise, but it's certainly far less common. With age comes experience and with experience, wisdom has fertile ground to grow, though it doesn't alw…

Notwithstanding the fact that "age is unrelated to wisdom" was a gentle contextual device used to dismiss the commenter's odd assumption about my age, a search for "age is unrelated to wisdom" turns up actual research papers. "neither general nor personal wisdom have a positive linear relationship to age... age is not only not related to personal wisdom (as is the case for general wisdom) but even negatively related.…

[deleted]

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#132

Post by gnaritas »

tacos wrote: post #87
gnaritas wrote: post #84

Earlier quoted context omitted.

> Age is unrelated to wisdom and I'm talking about the full experience. That's not true; age and wisdom are quite related, just not directly causal. They are correlated. Older people are generally wiser, it's just not a guarantee, nor is it impossible for young people to be wise, but it's certainly far less common. With age comes experience and with experience, wisdom has fertile ground to grow, though it doesn't alw…

Notwithstanding the fact that "age is unrelated to wisdom" was a gentle contextual device used to dismiss the commenter's odd assumption about my age, a search for "age is unrelated to wisdom" turns up actual research papers. "neither general nor personal wisdom have a positive linear relationship to age... age is not only not related to personal wisdom (as is the case for general wisdom) but even negatively related.…

If you actually searched for that, then you know there's far more studies showing the opposite. You can find a research paper supporting just about any position, that doesn't make it the majority opinion of the field. You actually had to ignore a lot of stuff saying the opposite, to find something saying age and wisdom weren't related. That's OK, you're young, you'll get wiser with age.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#133

Post by disgruntledphd2 »

eli_gottlieb wrote: post #36
j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

Actually, there's been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent. https://github.com/gregversteeg/CorEx http://arxiv.org/abs/1406.1222 http://arxiv.org/abs/1410.7404 Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting gr…

That's really interesting, especially with respect to the personality trait recovery. I'm particularly amazed given that I've never been able to replicate the Big5 model in its entirety against particular datasets.

Only commenting on that portion, as it's what I know. The author doesn't appear to give any indication of which test was used, and where the sample was taken from. This makes me a little suspicious, but its probably just a disciplinary thing.

Additionally, many tests reduce the pool of available questions for a scale after testing it against against other scales + theory. This means that what the author is recovering, is the scale imposed on the items by the original authors.

Nonetheless, this has forced me to reinvestigate the Big5 model, so it's not all bad :).

Massive appreciation for the links.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#134

Post by yosyp »

shasta wrote: post #117
bendbro wrote: post #111

Earlier quoted context omitted.

Do you have any good examples of where it is likely to fail?

Take any smooth function f and consider the multi-dimensional g(x,y,z...) = f(x) + f(y) + f(z) + ... If f had m local minima, then the n-dimensional version of g has m^n local minima.

You are correct in that there will be m^n minima, but since your surface is highly symmetric, local minima are trivially easy to find. Are you simplifying the statement by skipping over the importance of intervals?

If the intervals spanned by x,y,z do not overlap, then your statement does not hold unless local minima are evenly distributed over the entire region, which defeats the purpose of searching for minima in the first place. If the intervals spanned by x,y,z are identical, then gradient descent can be reduced to just searching over f(x) by symmetry. You are right that there are m^n minima, but they are found in (nm)^1 steps. If the intervals spanned by x,y,z partially overlap, you can further reduce the search space to just f(x) and the union of the intervals.

I don't see how your example shows that gradient descent is likely to fail for any classifier with a sufficiently high number of independent parameters. Am I missing something?

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#135

Post by shasta »

yosyp wrote: post #134
shasta wrote: post #117

Earlier quoted context omitted.

Take any smooth function f and consider the multi-dimensional g(x,y,z...) = f(x) + f(y) + f(z) + ... If f had m local minima, then the n-dimensional version of g has m^n local minima.

You are correct in that there will be m^n minima, but since your surface is highly symmetric, local minima are trivially easy to find. Are you simplifying the statement by skipping over the importance of intervals? If the intervals spanned by x,y,z do not overlap, then your statement does not hold unless local minima are evenly distributed over the entire region, which defeats the purpose of searching for minima in t…

I was just tossing out an example of a higher dimensional function and counting the local minima. The reasoning that critical points tend to be saddle points, not minima, seems irrelevant if critical points are increasing exponentially for a typical higher dimension function.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#137

Post by argonaut »

crazypyro wrote: post #104
argonaut wrote: post #74

Earlier quoted context omitted.

Nobody ever took that theorem seriously. Deep nets were around since the 90s.

I've seen people with passing knowledge of NNs throw it around sometimes and its also referred to in a lot of literature as one reason for needing to parallelize neural networks (which I've been reading a lot on, due to a project), even if its not hugely important.

It's not hugely important because it tells us little of practice use. I mean, k nearest neighbors, given infinite data, can model any function as well. In practice, single layer neural nets are not very useful and don't do a good job of learning feature representations.
Post reply on HN