Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Scale Deep Learning – Jeff Dean [pdf]

static.googleusercontent.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#31

Post by crazypyro »

waterlesscloud wrote: post #10
nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too" That's an interesting direction from which to view things. Then AI progress can be measured by increasing that timeframe. Though I suspect there are some pretty gigantic discontinuities in there. The things humans can do in 3-4 seconds are qualitatively different from what they can do in less than 1 second, for example. Still, it's a usefu…

There's also the issue that we don't even fully understand what the brain even does, so how can we claim that anything in this incomplete set of operations is possible with a computer...

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#32

Post by kastnerkyle »

brandonb wrote: post #20
kastnerkyle wrote: post #17

Earlier quoted context omitted.

Bookwise, Yoshua Bengio, Aaron Courville, and Ian Goodfellow are nearly finished with their MIT Press book on deep learning: http://www.iro.umontreal.ca/~bengioy/dlbook/ . It is pretty strong on the true theory of what is going on in deep networks, and has fairly good intuition for how and why things work. Paired with the deep learning tutorials http://www.deeplearning.net/tutorial/ , as well as the content from UFLD…

Didn't realize Yoshua & co had a book coming! That would definitely be the one to read. BTW, for anybody who wants to learn machine learning in general, Kyle's blog also seems to be packed full of clear explanations with working demo code: http://kastnerkyle.github.io/ Very nice!

Thanks for checking it out! I am planning to add a few deep learning related posts during the holidays. The recent results for NLP, captioning and speech using encoder/decoder models are just too cool not to demo.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#33

Post by chestervonwinch »

andrewcamel wrote: post #2

Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…

There's not a lot of theoretical background beyond the basics of pattern recognition to get started understanding this stuff. Of course, pattern recognition requires some knowledge of probability, statistics, linear algebra, and vector calculus. There are books on pattern recognition that are rather friendly towards this prerequisite knowledge, though.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#34

Post by Strilanc »

nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

Whelp, Page 54 blew my mind right out the window.

> E(hotter) - E(hot) + E(big) ≈ E(bigger)

> E(Rome) - E(Italy) + E(Germany) ≈ E(Berlin)

These things are linearly separable?!

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#35

Post by agibsonccc »

agibsonccc wrote: post #28
nl wrote: post #27

Earlier quoted context omitted.

The characterisation of Glove as better than Word2Vec is controversial. I'm on mobile now, but one of the word2vec authors had a Google doc going through the claims, and pointing out that similar performance was possible from word2vec by changing the parameters word2vec is used with. Edit: a link about this. https://news.ycombinator.com/item?id=8660624

Speaking from personal experience. I get paid to do deep learning. One of skymind's biggest app areas is text. That being said: I will be benchmarking deeplearning4j's glove with word2vec here soon. Any machine learning algorithm is better when you tune it. I personally like glove due to having less knobs. The mechanics involving document statistics being part of the gradient update is also interesting. I've also mes…

I'm aware the tone was a little condescending and I won't take back what I said. I will actually back it up though ;).

My main point is just because something is controversial shouldn't stop you from trying it. That's what research is: trying new things.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#36

Post by abrichr »

Strilanc wrote: post #34
nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

Whelp, Page 54 blew my mind right out the window. > E(hotter) - E(hot) + E(big) ≈ E(bigger) > E(Rome) - E(Italy) + E(Germany) ≈ E(Berlin) These things are linearly separable?!

Deep neural networks essentially transform input data into a vector space where the data is "easier" to model. So while the input vectors may not be linearly separable in the input space, the network learns how to transform the input vectors into a space where they are.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#37

Post by olavgg »

discardorama wrote: post #22
zonzo wrote: post #13

Earlier quoted context omitted.

torch7 used by facebook and google. Fast becoming the standard industrial neural nets library. http://torch.ch Read the tutorials and you can get something working in a day

"Used by Facebook and Google" ? Citation needed. AFAIK, at least Google has an internal homebrew solution, that automatically scales to large clusters.

AMA with Yann LeCun which confirms they are using it. http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_...

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#38

Post by jskonhovd »

andrewcamel wrote: post #2

Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…

I disagree. I believe that the foundation you will learn in statistics, linear algebra, and other mathematics would be worth the effort to any computer scientist. You would enjoy the introduction Machine Learning courses on Udacity or Coursera.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#39

Post by ajtulloch »

discardorama wrote: post #22
zonzo wrote: post #13

Earlier quoted context omitted.

torch7 used by facebook and google. Fast becoming the standard industrial neural nets library. http://torch.ch Read the tutorials and you can get something working in a day

"Used by Facebook and Google" ? Citation needed. AFAIK, at least Google has an internal homebrew solution, that automatically scales to large clusters.

Facebook AI Research uses Torch heavily, and contributes improvements back to the open-source version.
Post reply on HN