Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning - The Biggest Data Science Breakthrough of the Decade

oreillynet.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#31

Post by ergodic »

stiff wrote: post #26
ergodic wrote: post #18

Earlier quoted context omitted.

If it was not clear enough, "fancy stuff" and "not making a big difference" refers to Seide et al 2011 mentioned in the same paragraph. Table 2 is particularly revealing to this regard. http://research.microsoft.com/apps/pubs/default.aspx?id=1531... As I said I can only speak with more or less certainty regarding ASR. I am fairly sure that the success in ASR (with Google and MS embracing DNNs for ASR) contribute sign…

There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers: http://research.microsoft.com/pubs/157341/FeatureEngineeri…

I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper.

The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for differences between DNNs and MLPs.

Could you point me to a concrete paragraph?, I would be happy to be mistaken in this regard.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#32

Post by ergodic »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

rather than being so aggressive and waiting for downvotes you could choose to be constructive ;) and answer a concise technical question to a sub-thread that you yourself started

https://news.ycombinator.com/item?id=5377101

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#33

Post by lrei »

rm999 wrote: post #13

I'm concerned deep networks are being overhyped. They're certainly exciting, but they haven't seen much use in industry yet; it's too early to make claims about how they have impacted data science. Also, data science involves a lot more than building predictive models. In my experience >95% of effort goes into something other than building a model. In kaggle contests you usually concentrate on that edit: I haven't li…

I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#34

Post by mshron »

lrei wrote: post #33
rm999 wrote: post #13

I'm concerned deep networks are being overhyped. They're certainly exciting, but they haven't seen much use in industry yet; it's too early to make claims about how they have impacted data science. Also, data science involves a lot more than building predictive models. In my experience >95% of effort goes into something other than building a model. In kaggle contests you usually concentrate on that edit: I haven't li…

I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?

Not the OP but:

* Problem definition

* Infrastructure

* Data transformation

* Exploratory analysis (arguably part of model work)

* Results presentation

Then again, this is an ongoing disagreement I have with the Kaggle folks over what constitutes "data science," where I'm pretty confident that "applied machine learning" is a better explanation of what their contests are about.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#35

Post by rm999 »

mshron wrote: post #34
lrei wrote: post #33

Earlier quoted context omitted.

I'm genuinely curious (not being snarky or wtv): what do you put 95% of your effort into?

Not the OP but: * Problem definition * Infrastructure * Data transformation * Exploratory analysis (arguably part of model work) * Results presentation Then again, this is an ongoing disagreement I have with the Kaggle folks over what constitutes "data science," where I'm pretty confident that "applied machine learning" is a better explanation of what their contests are about.

I kind of got them to say it here:

https://news.ycombinator.com/item?id=4655927

BTW, I'm a big fan of the data analysis that came out of okcupid, is that all your work?

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#36

Post by jph00 »

ergodic wrote: post #31
stiff wrote: post #26

Earlier quoted context omitted.

There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers: http://research.microsoft.com/pubs/157341/FeatureEngineeri…

I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for dif…

DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets.

I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinformatics.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#37

Post by spikels »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :)

It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but they do not a appear to used much in production because of their complexity and high training cost. This is still cutting edge technology.

In my experience most professionally trained mathematicians and statisticians are still pretty skeptical of these claims. Wouldn't you agree?

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#38

Post by jackpirate »

badfortrains wrote: post #22
ergodic wrote: post #8

Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…

The thing that NNs have in their favor that other "20 year old techniques" lack is their ability to model any mathematical equation. There is no fundamental limit to the complexity of systems NNs can model (as there is with other AI techniques). The problem with NNs is the difficulty of training them. Back propagation with random initial weights is simple, but it can easily converge on suboptimal local maximum if the…

>There is no fundamental limit to the complexity of systems NNs can model (as there is with other AI techniques).

Sure there is. For example, they will never solve the halting problem. They will also (probably) never solve NP-complete problems for very large instances.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#39

Post by anigbrowl »

zby wrote: post #9

I registered and listened for maybe 9 minutes - and nothing interesting, just talk about the talk. How I hate webcasts!

Agreed. Don't care much for podcasts either, although I can see the value for people who drive frequently.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#40

Post by mtrimpe »

rm999 wrote: post #13

I'm concerned deep networks are being overhyped. They're certainly exciting, but they haven't seen much use in industry yet; it's too early to make claims about how they have impacted data science. Also, data science involves a lot more than building predictive models. In my experience >95% of effort goes into something other than building a model. In kaggle contests you usually concentrate on that edit: I haven't li…

I think this new speech recognition improvement by Microsoft depends on deep networks and it's results are very impressive.

http://research.microsoft.com/en-us/news/features/speechreco... http://research.microsoft.com/en-us/projects/mavis/

That might just be a single application but if it extends into other domains it might end up being very valuable indeed.

It will eventually become just another tool of course just like anything else but if it brings 10-20% improvements in even a few other long-stagnant areas I would agree with saying that it is a big deal.

Post reply on HN