Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Brief Overview of Deep Learning

yyue.blogspot.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: A Brief Overview of Deep Learning

#11

Post by fchollet »

sadkingbilly wrote: post #10

"so I implemented a small neural network and trained it to sort 10 6-bit numbers, which was easy to do to my surprise" Does anyone know what the inputs and outputs of a neural network that sorts numbers would look like?

Input: a 60-dimensional vector that is the concatenation of 10 6-dimensional binary vectors encoding the binary representation of the input numbers.

Output: the same, sorted.

At least that's one dead simple way to formulate the problem, multiple other solutions would work as well, and some would probably work better.

Re: A Brief Overview of Deep Learning

#12

Post by rdtsc »

Whatever happened to shallow learning (or you know the regular learning) everyone did before deep learning.

Anyone still doing that?

Is this like BigData. As soon as someone mentioned BigData, anyone in the world who touched data all of the sudden did BigData.

So is this something coming out of Google and Facebook and such and everyone else in Academia is happily building SVMs and 2 layer neural networks or some new discovery happend and overturned the whole ML and AI field on its head?

> Crucially, the number of units required to solve these problems is far from exponential --- on the contrary, the number of units required is often so “small” that it is even possible, using current hardware,

Number of units is not what's important. There are "only" what, 10B (100B?)neurons in the brain? But isn't the trick in the connections. And there are orders of magnitudes more connectsion (hundreds of trillions). Not exponential but even quadratic at those numbers is still quite large.

Re: A Brief Overview of Deep Learning

#13

Post by lscharen »

fchollet wrote: post #6

This post, while very interesting, attempts to draw a completely unwarranted parallel between deep nets and the human brain, as if layers of artificial neurons running on a GPU and the cortical layers of your brain were two interchangeable things. So far, there has been no evidence that the brain works anything like an artificial neural network. Maybe it does, and there are several theories in that direction, but at…

The point of drawing comparisons to the human brain is that we know how quickly humans can perform visual recognition tasks and speed of signal propagation between neurons. Combining these two properties implies that the human brain is able to solve these tasks without feedback, i.e. no loops. Thus, a DNN should be able to perform similar tasks if it can be trained (which it can).

Recurrent neural nets add feedback and are are whole different kettle of fish.

Re: A Brief Overview of Deep Learning

#14

Post by unlikelymordant »

rdtsc wrote: post #12

Whatever happened to shallow learning (or you know the regular learning) everyone did before deep learning. Anyone still doing that? Is this like BigData. As soon as someone mentioned BigData, anyone in the world who touched data all of the sudden did BigData. So is this something coming out of Google and Facebook and such and everyone else in Academia is happily building SVMs and 2 layer neural networks or some new…

deep learning is only applicable to some problems e.g. speech recognition and image classification where there is mountains of data to train many network parameters but also a certain level of complexity in the features. Most simple problems with a few thousand feature vectors for training are still solved by SVMs etc. Deep learning is getting excellent results in the problems it is good at though, better than any other classifier.

Remember SVMs were being thrown around as the ML wunderkind prior to Deep learning. After a while people figure out exactly what some things are good at and some things not so good.

Re: A Brief Overview of Deep Learning

#15

Post by gamegoblin »

fchollet wrote: post #11
sadkingbilly wrote: post #10

"so I implemented a small neural network and trained it to sort 10 6-bit numbers, which was easy to do to my surprise" Does anyone know what the inputs and outputs of a neural network that sorts numbers would look like?

Input: a 60-dimensional vector that is the concatenation of 10 6-dimensional binary vectors encoding the binary representation of the input numbers. Output: the same, sorted. At least that's one dead simple way to formulate the problem, multiple other solutions would work as well, and some would probably work better.

Could also have an 18 bit output where each 3 bit block told you the index of that input in the sorted array.

Re: A Brief Overview of Deep Learning

#16

Post by SeckinJohn »

fchollet wrote: post #11
sadkingbilly wrote: post #10

"so I implemented a small neural network and trained it to sort 10 6-bit numbers, which was easy to do to my surprise" Does anyone know what the inputs and outputs of a neural network that sorts numbers would look like?

Input: a 60-dimensional vector that is the concatenation of 10 6-dimensional binary vectors encoding the binary representation of the input numbers. Output: the same, sorted. At least that's one dead simple way to formulate the problem, multiple other solutions would work as well, and some would probably work better.

you'd be better off if you mapped those 10 numbers to a number in [0:(10!-1)] interval since there are only 10! different potential answers. So you make the job of the algorithm easier (figuring out the "structure" of the function it's trying to solve for is easier when the result space is compressed without any 'signal' loss)

another improvement could be "enhancing" of the inputs: when you figure out how you will permute the numbers to sort them, create specific and randomized variations of that specific list of numbers and feed the learning algorithm with the correct results of those permutations too. for instance if you have 5 70 2 13 as a training input, the trainer algorithm could generate the following extra inputs based on this so that the algorithm will get a better chance of figuring out the sorting for a test input like 2 15 5 65:

2 5 23 70 2 5 33 70 ... 2 5 63 70 2 5 73 70 also: 2 5 14 70 2 5 13 69 etc. also modify more than 1 number at the same time(both systematically and also randomly) to generate even more "gray"-input

Re: A Brief Overview of Deep Learning

#17

Post by wodenokoto »

rdtsc wrote: post #12

Whatever happened to shallow learning (or you know the regular learning) everyone did before deep learning. Anyone still doing that? Is this like BigData. As soon as someone mentioned BigData, anyone in the world who touched data all of the sudden did BigData. So is this something coming out of Google and Facebook and such and everyone else in Academia is happily building SVMs and 2 layer neural networks or some new…

A rather sizable part of the article talk about why large deep network can solve problems that shallow networks can't.

I won't say that the ML field got turned on its head by deep networks. I think people from the very start have wanted to try and make networks deeper, if only they'd known how.

Re: A Brief Overview of Deep Learning

#18

Post by discardorama »

anjc wrote: post #2

Good article, but I still don't understand why they're suddenly popular again. So processing is faster, but is there some development in processing which has improved this domain especially? Developments in "big data" processing? Concepts like mapreduce? What's with the resurgence :S

There are a myriad reasons why they're popular again.

1. Hardware has caught up, and is cheap. When Backprop was invented back in the 80s, you couldn't train networks with more than a couple of 1000 nodes tops. Today, with GPUs, you can train networks with billions of parameters.

2. More data is available. Back in those days, you had a few dozens (maybe a few 100s) of examples in your training set. Today, people play with sets larges than 1TB.

3. Dramatic successes. For a while, the ImageNet competition was seeing slow and stead progress. Then DL comes along, and there's a 20% jump in performance (I'm too lazy to look up the exact numbers...). If you've ever competed in such competitions, progress is painfully slow (see, for example, the Netflix competition). So a jump of that magnitude in performance in 1 step is mind-blowing. On top of that, every year since then, the performance has increased significantly.

These are just 3 that come to mind.

Re: A Brief Overview of Deep Learning

#19

Post by nightski »

chrdlu wrote: post #5
anjc wrote: post #2

Good article, but I still don't understand why they're suddenly popular again. So processing is faster, but is there some development in processing which has improved this domain especially? Developments in "big data" processing? Concepts like mapreduce? What's with the resurgence :S

I feel like cloud technology may be a contributing factor as well. At the same time, once Geoffrey Hinton used a deep neural network and participated in the ImageNet contest (2012: http://www.image-net.org/challenges/LSVRC/2012/results.html ), his results beat the next best thing by a full 10%. The results were so astounding that many people immediately began re-visiting neural networks. Shortly afterwards, people pr…

But don't kid yourself, if you have ever been to CVPR, or one of the machine learning conferences you see all of the papers trying to squeeze a few more % accuracy on a test set out of existing algorithms. Maybe one or two papers actually do something novel. Rarely will their be a new approach altogether.

The point I am trying to make is that one shouldn't get star eyed by leading researchers in the field and assume they can't contribute. Simple novel ideas have led to massive changes in the industry.

Re: A Brief Overview of Deep Learning

#20

Post by [deleted] »

discardorama wrote: post #18
anjc wrote: post #2

Good article, but I still don't understand why they're suddenly popular again. So processing is faster, but is there some development in processing which has improved this domain especially? Developments in "big data" processing? Concepts like mapreduce? What's with the resurgence :S

There are a myriad reasons why they're popular again. 1. Hardware has caught up, and is cheap. When Backprop was invented back in the 80s, you couldn't train networks with more than a couple of 1000 nodes tops. Today, with GPUs, you can train networks with billions of parameters. 2. More data is available. Back in those days, you had a few dozens (maybe a few 100s) of examples in your training set. Today, people play…

[deleted]
Post reply on HN