Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning - The Biggest Data Science Breakthrough of the Decade

oreillynet.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#51

Post by warmfuzzykitten »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

After 10 minutes, not a single slide apart from the cloudera ad has been shown. This is a podcast.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#52

Post by lrei »

mshron wrote: post #34
lrei wrote: post #33

Earlier quoted context omitted.

I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?

Not the OP but: * Problem definition * Infrastructure * Data transformation * Exploratory analysis (arguably part of model work) * Results presentation Then again, this is an ongoing disagreement I have with the Kaggle folks over what constitutes "data science," where I'm pretty confident that "applied machine learning" is a better explanation of what their contests are about.

I see. Thanks.

I'd say data transformation is a part of feature engineering (commonly the bulk of the effort in a ML application). And exploratory analysis is part of model work. W/o those 2 one would be building a model out of dreams and wishes.

Data Science is probably a poorly chosen description. I'd say common use includes infrastructure work which for most of us consists in engineering work.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#53

Post by ergodic »

stiff wrote: post #41
ergodic wrote: post #31

Earlier quoted context omitted.

I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for dif…

In comparison to older MLP research, besides the new training algorithm, there is this new insight that the deep structure of the network might be efficient for generating very good encodings of the input variables, like described here: http://en.wikipedia.org/wiki/Autoencoder I am not very familiar with speech recognition, but I think what they talk about here: Instead of factorizing the networks, e.g., into a monop…

Again, the use of senones instead of monophones or diphones is just changing the output targets is not a novelty per sé.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#54

Post by ergodic »

jph00 wrote: post #36
ergodic wrote: post #31

Earlier quoted context omitted.

I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for dif…

DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets. I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinfor…

>DNNs can be thought of a stacked Restricted Boltzmann Machines

Agree, as explained in Hinton et al 2006.

http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf

But this is just for pre-training, as I said. If you look at Seides paper, they pre-train treating the MLP as a DBN and then they train it as a classic MLP with BP. Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no use of DBNs paradigms at all.

>Their structure and training is very different to traditional MLPs

I insist if we are talking of the same DNNs explained in Microsofts paper, this is not true. If we were to be talking about different DNNs please elaborate I would love to hear about that (seriously, no irony here).

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#55

Post by cscurmudgeon »

stiff wrote: post #44
cscurmudgeon wrote: post #42

Earlier quoted context omitted.

Instead of doing an obvious ad-hominem (Yes :D, this is one of the times it is genuine!), can you address what the other commenters are pointing out? I have degrees in Physics, EE and a PhD in CS. I munched axiomatic set theory and infinite ordinals during the course of my PhD. I dabbled in theoretical machine learning for 3 years. See, I can play the credentials game too. But does that address the fact that they are…

I am not playing the credentials game, quite the contrary, I remember trying to understand DBFs while at the university and failing miserably due to the complexity of the subject. I am also not defending deep learning in any way or having any stance about the subject myself. I just think in a place like HN you should not criticize technology without having practical experience or technical arguments, and of course "n…

1. Google hiring X is not the same as X's model being scientifically valid.

2. Model X for intelligence being highly technical or cool in some mathematical way is not scientific validation.

3. Google hiring X is not the same as X's model being successful in the industry. I have long switched over to DuckDuckGo for technical queries.

Anyways, what AI people should first address always is point number 2.

AI has always jumped from one cool thing to the next without answering whether that cool thing has any scientific basis.

Don't bring another AI winter ;)

It is always cool to see excitement over research in AI! (As long as it does not drown out other competitive approaches which might bear fruit in the long run.)

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#56

Post by jhartmann »

spikels wrote: post #25
ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

While deep learning is a very cool technique and is currently getting the best results in a few domains I think all the hype may become a problem. I was around for the prior round of neural network excitement and much time, effort and money was wasted. In that case it turned out that other techniques were more tractable and thus easier to use and improve upon. It must be the association with the human brain that just…

I have a machine learning startup that is using deep learning neural networks, so I'm probably biased here. I really think there is something that is worth the hype here, this is the first time we can solve significant problems without lots of feature engineering to make the neural network be able to solve the problem. While I'm sure there are going to be tons of things that deep belief neural networks can not do well even with these new capabilities and breakthroughs, there is a crapload of data out there that is begging to be analysed. Being able to get reasonable performance without a ton of feature engineering and years for a black arts team to build something that can get the data into a state where problems can be answered is SUPER exciting. The Neural Networks we are using are more specialized and more like Yann Lecun's, and we aren't using dropout like Hinton but we already have something that gets very good accuracy in our problem domain. There are some new techniques that are just coming out of Montreal, one in particular I'm very excited about called Maxout that looks like it will be another significant advance. One of the problems networks like this usually have is that the activation functions saturate above a certain level and once a neuron is in the saturated state the gradient training process will not move it anymore. Maxout is different in that it doesn't have this property, and it seems to maximize the benefit of the random selection process of dropout.

While I don't have the math credentials to match Hinton I think as more 'normal' folks like me get into the game there will also be some interesting things going on. We are trying some interesting things that seem very promising, and I'm sure there are lots of other folks beginning to play with these things that will have some interesting ideas and approaches as well.

So I personally think this is super exciting, and while it might not be applicable for every problem Deep Learning will definitely have a big impact.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#57

Post by rm999 »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

>not used in industry

Well, that ones about me. Yes, I have a plenty of experience in machine learning, including undergraduate research in neural networks, a graduate degree in machine learning, and more than five years of industry experience (including several years building some of the most utilized neural network models in industry). I have read many of the deep network papers in detail, and have played around with them on actual data.

And yes, I think your comment deserves to be downvoted; unlike those of us with insight into the issue you added nothing to the discussion other than derision. It bothers me that comments like yours end up at the top of so many threads like this.

edit: and I'd like to point out as someone in the industry I have a good reason to temper expectations. Undeserved hype leads to bubbles, and bubbles create collateral damage when they pop. The AI industry has dealt with this at least twice already, and I don't want to see it happen again. The results so far are extremely exciting, but deep networks still need to prove they deserve the hype.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#58

Post by rm999 »

stiff wrote: post #44
cscurmudgeon wrote: post #42

Earlier quoted context omitted.

Instead of doing an obvious ad-hominem (Yes :D, this is one of the times it is genuine!), can you address what the other commenters are pointing out? I have degrees in Physics, EE and a PhD in CS. I munched axiomatic set theory and infinite ordinals during the course of my PhD. I dabbled in theoretical machine learning for 3 years. See, I can play the credentials game too. But does that address the fact that they are…

I am not playing the credentials game, quite the contrary, I remember trying to understand DBFs while at the university and failing miserably due to the complexity of the subject. I am also not defending deep learning in any way or having any stance about the subject myself. I just think in a place like HN you should not criticize technology without having practical experience or technical arguments, and of course "n…

>of course "not used in industry" is not a technical argument. With Google just yesterday hiring George Hinton

Again, you're quoting me on that. Yes, 'not used much in data science' is a valid argument that it's not one of the biggest breakthroughs in data science.

And if you want to discuss the topic (while blanket criticizing people for not knowing what they're talking about) at least get the father of deep network's name right: it's Geoff Hinton, not George.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#59

Post by jph00 »

ergodic wrote: post #54
jph00 wrote: post #36

Earlier quoted context omitted.

DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets. I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinfor…

>DNNs can be thought of a stacked Restricted Boltzmann Machines Agree, as explained in Hinton et al 2006. http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf But this is just for pre-training, as I said. If you look at Seides paper, they pre-train treating the MLP as a DBN and then they train it as a classic MLP with BP. Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no us…

There's also the random knockout of neurons, as mentioned in the webinar.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#60

Post by jph00 »

warmfuzzykitten wrote: post #51
stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

After 10 minutes, not a single slide apart from the cloudera ad has been shown. This is a podcast.

Oh that's annoying. When I did the webinar I was sharing my screen and showing both slides and scribbles in onenote. Is there no button to view that in the recording? Sorry I can't check myself now, I'm on my phone.
Post reply on HN