Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

41–50 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#41

Post by obsim »

zepolen wrote: post #32

Thought about how cool it would be to use machine learning to generate machine learning algorithms and how that could be the basis of artificial "life". Maybe start with a minimal implementation 'bootstrap protozoa' that can evolve to highly complicated forms. Then realised that since Machine Learning processes takes a long time that would take forever making auto evolving computer life forms a long way away. Althoug…

A LSTM recurrent neural network was used to learn a learning algorithm: http://www.bioinf.jku.at/publications/older/1504.pdf

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#42

Post by vidarh »

algomanic wrote: post #26
rdlecler1 wrote: post #23

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cit…

Could you explain weight matrix "spurious interactions"?

Very, very roughly a neural network consists of a bunch of connected nodes that you propagate signals through. Each of those connections carries a weight that affects how the signal gets propagated through the rest of the network.

Let's say a specific problem has only one very specific set of connections that matters. You'll eventually add up with weights that reflects that, but that doesn't prevent a lot of other connections from having weights set during training, but that may end up being cancelled out or reduced enough to have no meaningful impact whatsoever on the end result.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#43

Post by dasboth »

nl wrote: post #37
joakleaf wrote: post #15

Neural networks is advanced curve fitting -- That's why. It isn't really all that magical or mysterious.

It really kinda is. In the Ilya Sutskever Talking Machines podcast[1] he describes exactly how it is "magic". He talks about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained. And then they discovered how to initia…

I was just listening to that podcast yesterday and loved the way he presented this information. I know next to nothing about NNs but I could pretty much follow everything that he said (from an intuitive sense, anyway, if not a technical one).

This "magic" of converging networks reminds me of how ensemble methods, such as random forests, are effective but people aren't sure why. There's certainly (AFAIK) no theoretical grounds to say that a bunch of random decision trees should yield universally good results.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#44

Post by tacos »

nl wrote: post #33
tacos wrote: post #22

As for "spectacularly well" -- well, the person behind the curtain wiggling the levers retains a lot of influence. Garbage in, garbage out, remember? I'm reminded of the time Google Translate autodetected "Gesundheit" as Spanish. And Gmail kindly offering to translate "hahaha" from Portuguese, putting an ad for coconuts next to it. Data science is improving, but you might be surprised how slowly. Especially in the co…

I'm not sure how old you are, but what exactly are you expectations when you state Data science is improving, but you might be surprised how slowly. We have machines that can categorise pictures better that humans. In 2011 that seemed completely impossible.

Age is unrelated to wisdom and I'm talking about the full experience.

A Google Image search for "Wonder Wheel" (the famous Coney Island Ferris Wheel) shows this spoked diagram within the first page of results:

http://searchengineland.com/figz/wp-content/seloads/2011/07/...

Also this year, Google Photos classified black people as gorillas.

http://www.usatoday.com/story/tech/2015/07/01/google-apologi...

Consumers are rarely exposed to the raw machine output -- for good reason. My experience building these sorts of systems is that they're pretty goofy and they fail unexpectedly. After chasing audio and video problems using custom software as well as three major toolkits, I find myself hyper-aware of the flaws in public systems.

Also common sense dictates that it's more about the data scientist on the way in and the UX person on the way out than the machine.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#45

Post by j2kun »

eli_gottlieb wrote: post #36
j2kun wrote: post #14

I can chime in for the theoretical computer scientists. Deep learning (in its simplest form) corresponds to the class of circuits whose gates are linear threshold functions. Our primary goal with such functions is not to show what problems can be solved by small circuits using linear threshold gates, but what problems cannot be solved with such circuits. Until last week [1], it was an open problem whether every funct…

Actually, there's been a few papers showing solid theoretical progress on understanding what structure in data deep networks are learning to represent. https://github.com/gregversteeg/CorEx http://arxiv.org/abs/1406.1222 http://arxiv.org/abs/1410.7404 Disclaimer: I reinvented the idea about a month or two ago, and ran into the existing papers when googling for joint entropy estimators after making some interesting gr…

I meant "deep nets" in terms of boolean circuits, which seems like a different beast from these papers.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#46

Post by micwawa »

cschmidt wrote: post #27

Can anyone give me a pointer on what techniques the author is talking about for unsupervised learning at the end of the article?

I think one of the seminal papers is by Belkin and Niyogi:

http://people.cs.uchicago.edu/~niyogi/papersps/BNcolt05.pdf

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#48

Post by darkhorn »

As far as I know no mathematician was able to graduate from statistics department as second major or (second) minor (whatever you call it in English). I mean in my university. I hope you get the idea; it looks like statistics is hard for mathematicians, at least for most of them. Thus, I don't understand why they talk about this statistics topic with mathematicians but not statisticians. Yeah, may be should ask to a statistician! Duh!

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#50

Post by wsxcde »

To take the discussion on a slight tangent, how uncommon is this phenomenon? An applied tool works really well but nobody knows why. I can give another example from the domain of formal verification: SAT solvers which are at the core of most modern verification/synthesis tools.

You can download open source SAT solvers today that work spectacularly well on "real" SAT instances with millions and millions of variables and clauses. Yes, SAT is the quintessential NP-complete problem and in fact it is pretty easy to come up with a SAT instance with only a few tens of variables/clauses that would kill these solvers. But somehow these "hard" instances almost never occur in practical problems generated in hardware/software verification/synthesis as well as ton of other applications (planning, constraint programming etc.)

So this must there is some characteristic of the problems that we generate in practice that makes them "easy" but we don't have a good understanding of what this characteristic is. All we know, for now, is that we've somehow stumbled upon a near-perfect set of heuristics that work amazingly well on the SAT instances we encounter in practice.

Post reply on HN