Deep Learning - The Biggest Data Science Breakthrough of the Decade • Hacker News Board

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#21

Post by ergodic » Thu, Mar 14, 2013, 6:53 PM UTC

lrei wrote: post #12

ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

I've heard this argument ever since Norvig's Unreasonable Effectiveness of Data. While having a ton of data available is great, it has its limits. I believe you are overestimating the effectiveness of data (as, imo, Norvig did). And here specifically, it's not the case for the hype: from the NYT article [1]: "The achievement was particularly impressive because the team decided to enter the contest at the last minute…

> it might be great to have a human readable and tweakable solution (assuming you have the resources) but for something like recognising handwritten digits from images, not so much.

Agree, but with black-box I meant not something that is opaque to my grand-mother but partially opaque to engineers that implement MLP machine learning applications and the tech-lead that takes the decisions. The thing is that even research people (or maybe specially them) tend to positively bias things they do not completely understand (so I think, maybe its just me ;)). That is what I meant with black-box delusion. As you say only time will tell.

Regarding DBNs, again, the case of ASR uses DNNs which is to say big-fat MLPs. The model is handled as a DBN only for pre-training, and layer-wise pre-training does a similar job anyway.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#22

Post by badfortrains » Thu, Mar 14, 2013, 6:54 PM UTC

ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

The thing that NNs have in their favor that other "20 year old techniques" lack is their ability to model any mathematical equation. There is no fundamental limit to the complexity of systems NNs can model (as there is with other AI techniques).

The problem with NNs is the difficulty of training them. Back propagation with random initial weights is simple, but it can easily converge on suboptimal local maximum if the learning rate is too aggressive. On the other hand, a slow learning rate requires an exponential increase in training time and data. Back propagation as a method was never really broken, it simply wasn't efficient enough to be effective in most situations. Deep belief techniques seem to remedy these inefficiencies in a significant way, while remaining a generalized solution.

Essentially deep belief networks seem to optimize NNs to the point where new problems are now approachable, and greatly improve the performance of current NN solvable problems. The complaint that "the core of the recipe itself is not very new", seems irrelevant in light of the results.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#23

Post by hsshah » Thu, Mar 14, 2013, 6:58 PM UTC

We should discourage submissions like this here on HN that require registration to view the content. So even though this topic is of great interest of mine, I will not be upvoting it. Sorry.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#24

Post by ergodic » Thu, Mar 14, 2013, 7:03 PM UTC

badfortrains wrote: post #22

ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

The thing that NNs have in their favor that other "20 year old techniques" lack is their ability to model any mathematical equation. There is no fundamental limit to the complexity of systems NNs can model (as there is with other AI techniques). The problem with NNs is the difficulty of training them. Back propagation with random initial weights is simple, but it can easily converge on suboptimal local maximum if the…

Again, please, have a look at Seide et al 2011 before commenting. Besides that I am not complaining, just saying, wait a little more before you claim the breakthrough of the decade.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#25

Post by spikels » Thu, Mar 14, 2013, 7:16 PM UTC

ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

While deep learning is a very cool technique and is currently getting the best results in a few domains I think all the hype may become a problem. I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon.

It must be the association with the human brain that just makes neural networks more exciting than other techniques. But dispite the appeal of imitating nature has this usually been the easiest way to make progress in the past? Seems like it would be harder to achieve both goals at the same time.

So far the results are looking pretty good but it is probably best to keep the hype at a reasonable level unless it is crucial of your business model. ;)

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#26

Post by stiff » Thu, Mar 14, 2013, 7:27 PM UTC

ergodic wrote: post #18

stiff wrote: post #14

Earlier quoted context omitted.

I have no idea if it is the breakthrough of the decade, but I think deep learning isn't just taking a perceptron with many hidden layers and applying backpropagation to it, as you seem to say, all the interesting things about it you summarized as "fancy stuff" and "not making a big difference", without any context, references or arguments. I do not feel competent to discuss it as I have very little experience in this…

If it was not clear enough, "fancy stuff" and "not making a big difference" refers to Seide et al 2011 mentioned in the same paragraph. Table 2 is particularly revealing to this regard. http://research.microsoft.com/apps/pubs/default.aspx?id=1531... As I said I can only speak with more or less certainty regarding ASR. I am fairly sure that the success in ASR (with Google and MS embracing DNNs for ASR) contribute sign…

There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers:

http://research.microsoft.com/pubs/157341/FeatureEngineering...

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#27

Post by spikels » Thu, Mar 14, 2013, 7:34 PM UTC

jph00 wrote: post #2

I created this talk for the Enterprise Big Data track of O'Reilly's Strata conference - so it's not a technical description of how deep learning works. Rather, it's an attempt to show why it's important, and how it fits into current data science trends. The "Biggest Data Science Breakthrough of the Decade" in the title is a rather bold claim, I know... But I think it might be justified. If there are are bigger breakt…

Not sure which decade you are talking about. If you mean the 2010s or the next 10 years we'll just have to see what the next 7 or 10 years bring.

But if you mean the past 10 years I would have to say that the "distributed storage and processing" revolution (Hadoop and others) has had a much bigger impact on data science than all of neural networks including deep networks.

Why the need to hype what is already a well publicized development? I'm starting to cringe whenever I hear "data science" or "big data" and I love this stuff.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#28

Post by rm999 » Thu, Mar 14, 2013, 7:34 PM UTC

spikels wrote: post #25

ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

While deep learning is a very cool technique and is currently getting the best results in a few domains I think all the hype may become a problem. I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon. It must be the association with the human brain that just…

>I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon.

And before 1980s style neural networks there were 1950s perceptrons. That was a much bigger mess, it took more than ten years for someone to point out how 'dumb' perceptrons were (they couldn't even model an XOR), which led to a collapse in AI funding that lasted more than 25 years.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#29

Post by stiff » Thu, Mar 14, 2013, 7:46 PM UTC

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any understanding of anything particular about deep learning, just general derogatory comments "not used in industry", "overhyped", pointers to whatever someone heard in an undergraduate ML class on older types of networks etc.

Maybe if you don't have anything on topic to say, just do not comment? You really are not obliged to have an opinion on everything.

(Waiting for the downvotes)

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#30

Post by spikels » Thu, Mar 14, 2013, 7:47 PM UTC

rm999 wrote: post #28

spikels wrote: post #25

Earlier quoted context omitted.

While deep learning is a very cool technique and is currently getting the best results in a few domains I think all the hype may become a problem. I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon. It must be the association with the human brain that just…

>I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon. And before 1980s style neural networks there were 1950s perceptrons. That was a much bigger mess, it took more than ten years for someone to point out how 'dumb' perceptrons were (they couldn't even mode…

Can we be a little more thoughtful this time and avoid the boom and bust cycle that so often leads to problems?

You would think that since it already happened with neural networks before it would be less likely to happen again. However it may be that the same factors that lead to the last cycle are still in operation and it is actually more like to happen again. Something like the reasons for the seemingly endless series of real estate bubbles.