Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Brief Overview of Deep Learning

yyue.blogspot.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: A Brief Overview of Deep Learning

#31

Post by svantana »

amelius wrote: post #25

I found this to be insightful: > ... human neurons are slow yet humans can perform lots of complicated tasks in a fraction of a second. More specifically, it is well-known that a human neuron fires no more than 100 times per second. This means that, if a human can solve a problem in 0.1 seconds, then our neurons have enough time to fire only 10 times --- definitely not much more than that. It therefore follows that a…

This seems wrong. My understanding is that the ~200 spikes/second limit derives from the cell needing to "reload" before firing again, rather than some built in latency. Relaying a spike can be very quick indeed. A better conclusion would be that we don't have time for too many recursions in that short a time. Also, I can't think of any particularly hard problem that humans can solve in 0.1 seconds (see e.g. http://en.wikipedia.org/wiki/Hick%27s_law)

Re: A Brief Overview of Deep Learning

#32

Post by bnegreve »

svantana wrote: post #31
amelius wrote: post #25

I found this to be insightful: > ... human neurons are slow yet humans can perform lots of complicated tasks in a fraction of a second. More specifically, it is well-known that a human neuron fires no more than 100 times per second. This means that, if a human can solve a problem in 0.1 seconds, then our neurons have enough time to fire only 10 times --- definitely not much more than that. It therefore follows that a…

This seems wrong. My understanding is that the ~200 spikes/second limit derives from the cell needing to "reload" before firing again, rather than some built in latency. Relaying a spike can be very quick indeed. A better conclusion would be that we don't have time for too many recursions in that short a time. Also, I can't think of any particularly hard problem that humans can solve in 0.1 seconds (see e.g. http://e…

> Also, I can't think of any particularly hard problem that humans can solve in 0.1 seconds

Recognizing someone is a hard problem that most humans can do efficiently (except me maybe)

Re: A Brief Overview of Deep Learning

#33

Post by dharma1 »

I've been playing with Caffe for recognising images. It's kind of mind blowing how well it works. Yet the networks I tested could "only" recognise photos, not drawings or anything abstract.

A human could easily attribute meaning to a drawing, even if the drawing was very abstract or she had never seen a similar drawing before. Whereas a deep networks seem to rely on visual similarity to things it has seen in the past, on a pixel level. The networks I tried could tell something was a cartoon, but not what the cartoon depicted, even if it's something simple like a face.

The deep networks I tried also really struggled with recognising different textures. Like closeups of sand, water etc, things that a human would instantly recognise. They could classify it as a texture but not what kind of texture.

Re: A Brief Overview of Deep Learning

#34

Post by lars »

joyofdata wrote: post #30

> therefore follows that a large neural network with 10 layers can do anything a human can in 0.1 seconds. very funny ... as if ANNs are sufficiently comparable to actual neural activity. also I think it is naive to assess the "powerful"-ness of the brain to what is going on in a single neuron - it is certainly the parallel interaction which creates the human intelligence. > And if human neurons turn out to be noisy…

This post is written by Ilya Sutskever, who has co-authored some of biggest breakthroughs in machine learning the last five years. Which do you think is most likely: a) That he has a naive understanding of machine learning and neuroscience, or b) that this was written informally, and without guarding against every possible way it can be misinterpreted. Please be a little charitable when interpreting other peoples writings.

Re: A Brief Overview of Deep Learning

#35

Post by croddin »

primaryobjects wrote: post #23
fchollet wrote: post #11

Earlier quoted context omitted.

Input: a 60-dimensional vector that is the concatenation of 10 6-dimensional binary vectors encoding the binary representation of the input numbers. Output: the same, sorted. At least that's one dead simple way to formulate the problem, multiple other solutions would work as well, and some would probably work better.

I started playing with this. I take each digit and normalize it by dividing by 9. Then use each normalized digit as an input: Example sorting 987654 and 123456 Input: 1, .9, .8, .7, .6, .5, .2, .3, .4, .5, .6, .7 Expected output: .2, .3, .4, .5, .6, .7, 1, .9, .8, .7, .6, .5 You can then encode/decode the inputs and outputs accordingly. if (value I'm able to get 100% accuracy on a limited training set with 2 hidden l…

Interesting. It sounds like he used binary. I wonder how different bases affect how well the network can learn things like sorting?

Re: A Brief Overview of Deep Learning

#36

Post by joyofdata »

lars wrote: post #34
joyofdata wrote: post #30

> therefore follows that a large neural network with 10 layers can do anything a human can in 0.1 seconds. very funny ... as if ANNs are sufficiently comparable to actual neural activity. also I think it is naive to assess the "powerful"-ness of the brain to what is going on in a single neuron - it is certainly the parallel interaction which creates the human intelligence. > And if human neurons turn out to be noisy…

This post is written by Ilya Sutskever, who has co-authored some of biggest breakthroughs in machine learning the last five years. Which do you think is most likely: a) That he has a naive understanding of machine learning and neuroscience, or b) that this was written informally, and without guarding against every possible way it can be misinterpreted. Please be a little charitable when interpreting other peoples wri…

just curious - can you give an example for a big breaktrhough he co-authored?

nonetheless - some of his remarks are very specific and I don't see how informal style applies here to excuse them.

Re: A Brief Overview of Deep Learning

#37

Post by lars »

joyofdata wrote: post #36
lars wrote: post #34

Earlier quoted context omitted.

This post is written by Ilya Sutskever, who has co-authored some of biggest breakthroughs in machine learning the last five years. Which do you think is most likely: a) That he has a naive understanding of machine learning and neuroscience, or b) that this was written informally, and without guarding against every possible way it can be misinterpreted. Please be a little charitable when interpreting other peoples wri…

just curious - can you give an example for a big breaktrhough he co-authored? nonetheless - some of his remarks are very specific and I don't see how informal style applies here to excuse them.

He was second author on the AlexNet paper, wherein Alex Krizhevsky, Sutskever and Hinton blew everyone else out of the water on the ImageNet competition [2]. Their error rate was about 10 percentage points lower than others. Relatively speaking they had about 40% fewer errors than anyone else. This is possibly the biggest result in computer vision the last five years. So it seems a little silly to educate him on the basics of machine learning :)

[1] http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

[2] http://www.image-net.org/challenges/LSVRC/2012/results.html

Re: A Brief Overview of Deep Learning

#38

Post by joyofdata »

lars wrote: post #37
joyofdata wrote: post #36

Earlier quoted context omitted.

just curious - can you give an example for a big breaktrhough he co-authored? nonetheless - some of his remarks are very specific and I don't see how informal style applies here to excuse them.

He was second author on the AlexNet paper, wherein Alex Krizhevsky, Sutskever and Hinton blew everyone else out of the water on the ImageNet competition [2]. Their error rate was about 10 percentage points lower than others. Relatively speaking they had about 40% fewer errors than anyone else. This is possibly the biggest result in computer vision the last five years. So it seems a little silly to educate him on the…

well thanks for the info - but then I shift my critique to that I find it unnecessary to distort ML and biological concepts just to simplify the subject, when an accurate depiction wouldn't be much more difficult. Especially to not differentiate properly between memorization and generalization/learning is odd b/c this is one of the most prominent mistakes - it is specifically not the goal to minimize the in-sample-error! that would lead to very bad results most of the time

Re: A Brief Overview of Deep Learning

#39

Post by primaryobjects »

primaryobjects wrote: post #23
fchollet wrote: post #11

Earlier quoted context omitted.

Input: a 60-dimensional vector that is the concatenation of 10 6-dimensional binary vectors encoding the binary representation of the input numbers. Output: the same, sorted. At least that's one dead simple way to formulate the problem, multiple other solutions would work as well, and some would probably work better.

I started playing with this. I take each digit and normalize it by dividing by 9. Then use each normalized digit as an input: Example sorting 987654 and 123456 Input: 1, .9, .8, .7, .6, .5, .2, .3, .4, .5, .6, .7 Expected output: .2, .3, .4, .5, .6, .7, 1, .9, .8, .7, .6, .5 You can then encode/decode the inputs and outputs accordingly. if (value I'm able to get 100% accuracy on a limited training set with 2 hidden l…

Update: I was able to train the network to sort sets of two 3-digit numbers. I used a neural network with 2 hidden layers of 25 nodes. The training/test accuracy after 10 minutes is 78%/74%. Not bad. https://github.com/primaryobjects/nnsorting

Re: A Brief Overview of Deep Learning

#40

Post by p1esk »

joyofdata wrote: post #38
lars wrote: post #37

Earlier quoted context omitted.

He was second author on the AlexNet paper, wherein Alex Krizhevsky, Sutskever and Hinton blew everyone else out of the water on the ImageNet competition [2]. Their error rate was about 10 percentage points lower than others. Relatively speaking they had about 40% fewer errors than anyone else. This is possibly the biggest result in computer vision the last five years. So it seems a little silly to educate him on the…

well thanks for the info - but then I shift my critique to that I find it unnecessary to distort ML and biological concepts just to simplify the subject, when an accurate depiction wouldn't be much more difficult. Especially to not differentiate properly between memorization and generalization/learning is odd b/c this is one of the most prominent mistakes - it is specifically not the goal to minimize the in-sample-er…

Actually, Ilya explains his statement regarding minimizing training errors in his comment exchange with Bengio:

"Although I didn't define it in the article, generalization (to me) means that the gap between the training and the test error is small. So for example, a very bad model that has similar training and test errors does not overfit, and hence generalizes, according to the way I use these concepts. It follows that generalization is easy to achieve whenever the capacity of the model (as measured by the number of parameters or its VC-dimension) is limited --- we merely need to use more training cases than the model has parameters / VC dimension. Thus, the difficult part is to get a low training error."

Post reply on HN