Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Scale Deep Learning – Jeff Dean [pdf]

static.googleusercontent.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#21

Post by rasz_pl »

jberryman wrote: post #15
nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

> Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. Actually, I'm not sure we can do those things in quite 0.1 sec. All I know about this is from around minute 9 from http://www.radiolab.org/story/267176-never-quite-now/ . One guest on the show even esti…

maybe concious thought, most of the stuff in the brain is on autopilot and happens behind our backs, conciousness gets the results and just builds narrative (often rationalizing the results to make us feel better)

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#22

Post by discardorama »

zonzo wrote: post #13
andrewcamel wrote: post #4

Earlier quoted context omitted.

That's pretty comforting to hear. Do you know of any resources off the top of your head to learn this type of thing?

torch7 used by facebook and google. Fast becoming the standard industrial neural nets library. http://torch.ch Read the tutorials and you can get something working in a day

"Used by Facebook and Google" ? Citation needed. AFAIK, at least Google has an internal homebrew solution, that automatically scales to large clusters.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#23

Post by agibsonccc »

I know distributed neural nets are 10000 miles out there for most, but I just want to add a few nuggets for those considering it.

I work with distributed deep nets quite a bit. It's a different animal than training on a GPU.

I am working on benchmarks with my framework deeplearning4j now.

That aside, a few neat references/projects that will be digestible for people.

For those of you already in neural net land, there's a few key takeaways when doing distributed neural nets:

parameter averaging across mini batches

(depending on the algorithm) adagrad

momentum

[1] Project Adam: http://www.wired.com/2014/07/microsoft-adam/ [2]: Associated Paper: https://www.usenix.org/system/files/conference/osdi14/osdi14...

[3]: Hogwild algorithm: http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf

[4]: A variation of this I use called Iterative Reduce done by my partner Josh Patterson: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-re...

[5]: Sandblaster LBFGS by Dean and Co. http://research.google.com/archive/large_deep_networks_nips2...

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#24

Post by agibsonccc »

nl wrote: post #12
sanxiyn wrote: post #11

Earlier quoted context omitted.

> Page 60-70 talks about paragraph embedding. I haven't seen this published before. Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

Thanks. It's interesting how the state of the art is outpacing publishing. From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too. [1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf

There's also better embeddings than word2vec now:

http://nlp.stanford.edu/projects/glove/

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#26

Post by ehurrell »

brandonb wrote: post #9

For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus…

Thanks for the resources, my current hobby project is attempting to recreate deep learning in Clojure, have been burning through material, cheers!

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#27

Post by nl »

agibsonccc wrote: post #24
nl wrote: post #12

Earlier quoted context omitted.

Thanks. It's interesting how the state of the art is outpacing publishing. From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too. [1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf

There's also better embeddings than word2vec now: http://nlp.stanford.edu/projects/glove/

The characterisation of Glove as better than Word2Vec is controversial. I'm on mobile now, but one of the word2vec authors had a Google doc going through the claims, and pointing out that similar performance was possible from word2vec by changing the parameters word2vec is used with.

Edit: a link about this. https://news.ycombinator.com/item?id=8660624

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#28

Post by agibsonccc »

nl wrote: post #27
agibsonccc wrote: post #24

Earlier quoted context omitted.

There's also better embeddings than word2vec now: http://nlp.stanford.edu/projects/glove/

The characterisation of Glove as better than Word2Vec is controversial. I'm on mobile now, but one of the word2vec authors had a Google doc going through the claims, and pointing out that similar performance was possible from word2vec by changing the parameters word2vec is used with. Edit: a link about this. https://news.ycombinator.com/item?id=8660624

Speaking from personal experience. I get paid to do deep learning. One of skymind's biggest app areas is text.

That being said: I will be benchmarking deeplearning4j's glove with word2vec here soon. Any machine learning algorithm is better when you tune it.

I personally like glove due to having less knobs. The mechanics involving document statistics being part of the gradient update is also interesting.

I've also messed quite a bit with the distributed representations.

I'm not partial to any particular implementation. I'll use what works. That being said, I'm not armchair. I'll be backing this up with my own data as well.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#29

Post by benanne »

discardorama wrote: post #22
zonzo wrote: post #13

Earlier quoted context omitted.

torch7 used by facebook and google. Fast becoming the standard industrial neural nets library. http://torch.ch Read the tutorials and you can get something working in a day

"Used by Facebook and Google" ? Citation needed. AFAIK, at least Google has an internal homebrew solution, that automatically scales to large clusters.

Google DeepMind chiefly uses Torch7. I presume parts of Twitter also use it now since they acquired Clement Farabet's startup MadBits.

Facebook's AI research lab has contributed to the Torch7 project (which is unsurprising since it is lead by Yann LeCun, and Torch7 was originally developed in his group at NYU).

I wouldn't go as far as to say it's "becoming the industry standard" though. Caffe and Theano are also very popular.

Post reply on HN