Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

21–30 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#21

Post by tel »

senekerim wrote: post #19
tel wrote: post #9

Earlier quoted context omitted.

In grad school I asked that so many times trying to figure out why it was the "right" solution and this historical cheat caused so much cognitive dissonance. Then again, on the other hand, "calculus works nicely" and "it's simple" probably are good spooky directors toward useful models anyway.

It's not really a cheat, as it produces the MLE (i.e. most likely) estimate under certain assumptions (e.g. errors are normally distributed, which occurs naturally if the errors are large sums of many unrelated measurements errors).

Oh, I'm aware. It's just hard to always justify it and especially hard to do so in historical context.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#22

Post by tacos »

As for "spectacularly well" -- well, the person behind the curtain wiggling the levers retains a lot of influence. Garbage in, garbage out, remember?

I'm reminded of the time Google Translate autodetected "Gesundheit" as Spanish. And Gmail kindly offering to translate "hahaha" from Portuguese, putting an ad for coconuts next to it.

Data science is improving, but you might be surprised how slowly. Especially in the consumer space, because the metrics on effectiveness are so warped.

Voice recognition of numbers only, over a phone connection, can be below 40% accuracy! Much of the perceived success of these systems comes not from the core machine algorithm, but from clever human tweaks around it. Also end-users who are happy with what they get, not quite realizing how goofy it all is if they were to get a glimpse of the raw data.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#23

Post by rdlecler1 »

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cited ~130 times. There I represent an algorithm to evolve the connectivity of the network. What you find if that a network will tend to remove spurious interactions if the system is allowed to evolve. Because there will be very few network topologies that are both sparsely connected and functionally equivalent (think of how many ways you could create a minimally complex 8-bit added) there is likely only a small handful of non-isomorphic network topologies for any given function. With these sparse networks we should get a better grasp on the functional circuits that drive them. When the networks appear fully connected, at least in each layer, that circuitt does not reveal itself.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#24

Post by iandanforth »

rdlecler1 wrote: post #23

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cit…

NVIDIA NIPS paper doing just that for NN.

http://papers.nips.cc/paper/5784-learning-both-weights-and-c...

Not sure if it tells us more about why the network works, but they sure as heck get rid of a lot of connections.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#25

Post by akavi »

algomanic wrote: post #7

The fact that the class of NN functions is universal is almost vacuous: the basic idea is that if you allow a "neuron" for every point in your input space then you can mimic any function you like (i.e. each neuron handles a single input). Obviously such representations become arbitrarily large. Which almost immediately suggests a solution to why NN learning works: the processes that produce the types of datasets huma…

My gut response was, I think, a less rigorous version of this: Humans care about tasks our own brains are good at. If NNs are in fact decent approximations of the way our brains work, then it makes sense they will be anomalously effective on this subset of the "actual" problem space.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#26

Post by algomanic »

rdlecler1 wrote: post #23

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cit…

Could you explain weight matrix "spurious interactions"?

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#28

Post by algomanic »

j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

Related: "On the Expressive Power of Deep Learning: A Tensor Analysis". http://arxiv.org/abs/1509.05009

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#29

Post by sawwit »

akavi wrote: post #25
algomanic wrote: post #7

The fact that the class of NN functions is universal is almost vacuous: the basic idea is that if you allow a "neuron" for every point in your input space then you can mimic any function you like (i.e. each neuron handles a single input). Obviously such representations become arbitrarily large. Which almost immediately suggests a solution to why NN learning works: the processes that produce the types of datasets huma…

My gut response was, I think, a less rigorous version of this: Humans care about tasks our own brains are good at. If NNs are in fact decent approximations of the way our brains work, then it makes sense they will be anomalously effective on this subset of the "actual" problem space.

You seem to be saying that ANNs are good at pattern recognition because BNNs are. That just defers it to the question why BNNs are good at it.

OTOH, the parent comment suggested we should draw our attention to the processes that produce the data and find correspondences to how NNs decompose it.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#30

Post by StephenAshmore »

cschmidt wrote: post #27

Can anyone give me a pointer on what techniques the author is talking about for unsupervised learning at the end of the article?

Sure. These techniques are called Dimensionality Reduction or manifold learning. The wikipedia article actually has a good description: https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... Some of the more interesting ones: locally linear embedding, isomap, MLLE, Maximum Variance Unfolding.
Post reply on HN