Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning - The Biggest Data Science Breakthrough of the Decade

oreillynet.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#41

Post by stiff »

ergodic wrote: post #31
stiff wrote: post #26

Earlier quoted context omitted.

There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers: http://research.microsoft.com/pubs/157341/FeatureEngineeri…

I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for dif…

In comparison to older MLP research, besides the new training algorithm, there is this new insight that the deep structure of the network might be efficient for generating very good encodings of the input variables, like described here:

http://en.wikipedia.org/wiki/Autoencoder

I am not very familiar with speech recognition, but I think what they talk about here:

Instead of factorizing the networks, e.g., into a monophone and a context-dependent part [5], or decomposing them hierarchically [6], CD-DNN-HMMs directly model tied context-dependent states (senones). This had long been considered ineffective, until [1] showed that it works and yields large error reductions for deep networks.

might be related to this fact. 20 years ago it wasn't known why would you pick a deep network instead of a shallow one, there was even this famous theorem of Kolmogorow that a lot of people in ML misunderstood, that a network with just one hidden layer can in theory learn any function with arbitrary precision.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#42

Post by cscurmudgeon »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

Instead of doing an obvious ad-hominem (Yes :D, this is one of the times it is genuine!), can you address what the other commenters are pointing out?

I have degrees in Physics, EE and a PhD in CS. I munched axiomatic set theory and infinite ordinals during the course of my PhD. I dabbled in theoretical machine learning for 3 years. See, I can play the credentials game too.

But does that address the fact that they are not used in the industry? AI is full of charlatans and broken promises. Sadly, by listing "deep learning" alongside Deep Blue and Watson, it seems more charlatany.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#43

Post by drucken »

hsshah wrote: post #23

We should discourage submissions like this here on HN that require registration to view the content. So even though this topic is of great interest of mine, I will not be upvoting it. Sorry.

While I agree with you, note there is already flagrant "abuse" of this on HN by the posting of newspaper links with paywalls, in particular the New York Times.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#44

Post by stiff »

cscurmudgeon wrote: post #42
stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

Instead of doing an obvious ad-hominem (Yes :D, this is one of the times it is genuine!), can you address what the other commenters are pointing out? I have degrees in Physics, EE and a PhD in CS. I munched axiomatic set theory and infinite ordinals during the course of my PhD. I dabbled in theoretical machine learning for 3 years. See, I can play the credentials game too. But does that address the fact that they are…

I am not playing the credentials game, quite the contrary, I remember trying to understand DBFs while at the university and failing miserably due to the complexity of the subject. I am also not defending deep learning in any way or having any stance about the subject myself. I just think in a place like HN you should not criticize technology without having practical experience or technical arguments, and of course "not used in industry" is not a technical argument. With Google just yesterday hiring George Hinton who lead most of the deep learning research and with Jeff Dean working on it there already[1] it is also overall a rather weak one.

[1] http://research.google.com/archive/large_deep_networks_nips2...

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#45

Post by SiVal »

ergodic wrote: post #21
lrei wrote: post #12

Earlier quoted context omitted.

I've heard this argument ever since Norvig's Unreasonable Effectiveness of Data. While having a ton of data available is great, it has its limits. I believe you are overestimating the effectiveness of data (as, imo, Norvig did). And here specifically, it's not the case for the hype: from the NYT article [1]: "The achievement was particularly impressive because the team decided to enter the contest at the last minute…

> it might be great to have a human readable and tweakable solution (assuming you have the resources) but for something like recognising handwritten digits from images, not so much. Agree, but with black-box I meant not something that is opaque to my grand-mother but partially opaque to engineers that implement MLP machine learning applications and the tech-lead that takes the decisions. The thing is that even resear…

Regarding the "black-box delusion", it's not just you. You see a magician do a trick, and it's amazing. Then he explains how it is done, and the excitement vanishes. Oh, that's all it is, no big deal.

Any sufficiently advanced technology is indistinguishable from magic, and who knows what wonders magic might accomplish? But once you understand the "trick", it's obvious that it can't do much more than what it's doing. Oh, well. The magic is gone.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#46

Post by stiff »

spikels wrote: post #37
stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :) It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but th…

It is OK to be skeptical, all I am trying to say is that most of the comments leave the impression of the poster trying to lean over backwards to say something at all related to the topic, often ending up with generic truisms. For example how is model building being a small part of practical ML criticism of deep learning? How is it on-topic at all?

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#47

Post by joe_the_user »

stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

The OP title include "The Biggest Data Science Breakthrough of the Decade", the topic has already stopped being "is this interesting" but rather "is there something to justify the hype" or, surprise, "is this 'The Biggest Data Science Breakthrough of the Decade'"

Sure, I only have a graduate math degree and only follow these latest developments casually and perhaps I just miss exact way this newest artificial neural network stuff is really that different than the older stuff. But the only thing that's being touted is a NYTimes article. As another poster said, if you'd like to add to the conversation, give us some "meat" here.

My small exposure to ML also left me feeling the whole train, test, operate cycle is a pain in the neck.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#48

Post by spikels »

stiff wrote: post #46
spikels wrote: post #37

Earlier quoted context omitted.

It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :) It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but th…

It is OK to be skeptical, all I am trying to say is that most of the comments leave the impression of the poster trying to lean over backwards to say something at all related to the topic, often ending up with generic truisms. For example how is model building being a small part of practical ML criticism of deep learning? How is it on-topic at all?

Not my comment but I think they mean that it is a relatively small part of the overall process and thus not "The Biggest Data Science Breakthrough". Maybe not a fair criticism but they have a point.

I would love to see a breakthrough in data cleansing or how about just standardized coding, labeling and formatting. Unfortunately I've used lots of 3rd party data sources and wasted more time on these brainless activities than I want to think about. Consider yourself lucky if you only work on web logs where you control what they look like.

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#49

Post by espeed »

Quoc V. Le (http://ai.stanford.edu/~quocle/) of Stanford and Google did a talk today at Univ. of Washington:

  "Scaling deep learning to 10,000 cores and beyond":
  Presentation Univ. of Washington (March 14, 2013)
  https://www.cs.washington.edu/htbin-post/mvis/mvis?ID=1338
You can see one of Quoc's previous talks online:

"Tera-scale deep learning: - Quoc V. Le from ML Lunch @ CMU http://vimeo.com/52332329

You may remember Jeff Dean's (http://research.google.com/pubs/jeff.html) post on this: https://plus.google.com/118227548810368513262/posts/PozFb134...

The corresponding research at Google...

"Building high-level features using large scale unsupervised learning"

http://research.google.com/pubs/pub38115.html

http://research.google.com/archive/unsupervised_icml2012.htm...

Previous HN discussion: https://news.ycombinator.com/item?id=4145558

--

How Many Computers to Identify a Cat? 16,000 http://www.nytimes.com/2012/06/26/technology/in-a-big-networ...;

Re: Deep Learning - The Biggest Data Science Breakthrough of the Decade

#50

Post by [deleted] »

spikels wrote: post #37
stiff wrote: post #29

I am sorry, but does anyone else have the impression that a lot of people is commenting here with so much confidence while clearly not knowing anything about the topic? It takes almost an equivalent of an undergraduate mathematics degree and than a lot of experience in ML itself to get a decent understanding how things like Deep Belief Networks work, so I don't wonder none of the comments so far hinted at any underst…

It is purely a matter of opinion whether DBNs are "overhyped" but I hope you would agree that they are currently being "hyped". And I hope you understand how this can actually damage the potential of what is likely some very good technology like say like what happened to neural nets. :) It is also a matter of opinion how widely they are being used in industry. Certainly they are being studied in many companies but th…

[deleted]
Post reply on HN