Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

31–40 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#31

Post by akavi »

sawwit wrote: post #29
akavi wrote: post #25

Earlier quoted context omitted.

My gut response was, I think, a less rigorous version of this: Humans care about tasks our own brains are good at. If NNs are in fact decent approximations of the way our brains work, then it makes sense they will be anomalously effective on this subset of the "actual" problem space.

You seem to be saying that ANNs are good at pattern recognition because BNNs are. That just defers it to the question why BNNs are good at it. OTOH, the parent comment suggested we should draw our attention to the processes that produce the data and find correspondences to how NNs decompose it.

No, I'm saying that what we perceive as "being good at pattern recognition" is defined by the sort of pattern recognition BNNs are good at.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#32

Post by zepolen »

Thought about how cool it would be to use machine learning to generate machine learning algorithms and how that could be the basis of artificial "life".

Maybe start with a minimal implementation 'bootstrap protozoa' that can evolve to highly complicated forms.

Then realised that since Machine Learning processes takes a long time that would take forever making auto evolving computer life forms a long way away.

Although if it were possible, in the same way we became sentient based on very simple inputs, that could also mean computers, with a sufficiently complex process, could too...

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#33

Post by nl »

tacos wrote: post #22

As for "spectacularly well" -- well, the person behind the curtain wiggling the levers retains a lot of influence. Garbage in, garbage out, remember? I'm reminded of the time Google Translate autodetected "Gesundheit" as Spanish. And Gmail kindly offering to translate "hahaha" from Portuguese, putting an ad for coconuts next to it. Data science is improving, but you might be surprised how slowly. Especially in the co…

I'm not sure how old you are, but what exactly are you expectations when you state Data science is improving, but you might be surprised how slowly.

We have machines that can categorise pictures better that humans. In 2011 that seemed completely impossible.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#34

Post by wooter »

algomanic wrote: post #26
rdlecler1 wrote: post #23

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cit…

Could you explain weight matrix "spurious interactions"?

noise, biases, collisions, etc. due to imperfect or insufficient data, representational space issues, etc.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#35

Post by nl »

princeb wrote: post #10

there are a number of features in machine learning that seem to have counterparts in statistics - stuff like m-like estimators as kernels (I think - correct me if wrong), information criteria for feature selection, that have existed a long time ago. maybe it's the combination of sheer computing power and availability of data that allows better models. maybe we've never looked at algorithmically generated models becau…

I'm not sure what an advisor will say about a doc student who says, let's just throw reams of data at a machine until we find a meaningful correlation, and then let's reason from the correlations

Currently they are saying yes, please, and desperately trying to source ever bigger volumes of data.

Norvig et. al's article "The unreasonable effectiveness of data"[1] has been cited over 400 times which is a lot for something with no formula.

[1] http://static.googleusercontent.com/media/research.google.co...

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#36

Post by eli_gottlieb »

j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

Actually, there's been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent.

https://github.com/gregversteeg/CorEx

http://arxiv.org/abs/1406.1222

http://arxiv.org/abs/1410.7404

Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting graphs with hierarchical probabilistic programs. I only wish I could've been the first!

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#37

Post by nl »

joakleaf wrote: post #15

Neural networks is advanced curve fitting -- That's why. It isn't really all that magical or mysterious.

It really kinda is.

In the Ilya Sutskever Talking Machines podcast[1] he describes exactly how it is "magic". He talks about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained.

And then they discovered how to initialize the random starting values in such a way that they do converge - for reasons no one really understands.

I've heard a lot of people claim there is no magic, but I tend to think Sutskever[2] knows a little bit more than most people about deep neural networks.

[1] http://www.thetalkingmachines.com/blog/2015/1/15/machine-lea...

[2] http://www.cs.toronto.edu/~ilya/

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#38

Post by eli_gottlieb »

cdavid wrote: post #8
stdbrouw wrote: post #3

Earlier quoted context omitted.

Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…

Regarding L2, I think it is just the most convenient way to do the maths from a set of observations (differentiability). Later one, the link with maximum likelihood-based methods was made, by Gauss. In 'machine learning', structured learning by Vapnik and co (theory behind SVM), has a beautiful and strong mathematical underpinning. But in general, it is true we don't really have a good understanding of why learning a…

>The very notion of generalization to unobserved data is not well understood (I like D. Wolpert papers on that topic).

Which papers?

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#39

Post by eli_gottlieb »

akavi wrote: post #31
sawwit wrote: post #29

Earlier quoted context omitted.

You seem to be saying that ANNs are good at pattern recognition because BNNs are. That just defers it to the question why BNNs are good at it. OTOH, the parent comment suggested we should draw our attention to the processes that produce the data and find correspondences to how NNs decompose it.

No, I'm saying that what we perceive as "being good at pattern recognition" is defined by the sort of pattern recognition BNNs are good at.

Which still doesn't explain precisely what sort of statistical inference BNNs are performing.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#40

Post by nl »

eli_gottlieb wrote: post #36
j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

Actually, there's been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent. https://github.com/gregversteeg/CorEx http://arxiv.org/abs/1406.1222 http://arxiv.org/abs/1410.7404 Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting gr…

Possibly also of interest: Recursive Neural Networks Can Learn Logical Semantics[1]

Probably not as theoretical as the work you referenced, but interesting to me because of the deeply practical outcomes in NLP.

[1] http://www.aclweb.org/anthology/W/W15/W15-40.pdf#page=22

Post reply on HN