Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

arxiv.org

Post reply on HN
Print view View original

11–20 of 25 posts

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#11

Post by maurits »

gbachik wrote: post #6

How do you guys understand this stuff!? The topic absolutely fascinated me, but after reading the paper... I just feel dumb. Are there any good resources I could use to better understand machine learning papers such as this? I mean I can't even comprehend the implications this paper could have? Is anyone opening to doing some mentoring? Gbachik@gmail.com

Andrew Ng has an introduction class to some deep-learning topics here:

[1]: Wiki with code, exercises and explanation

[2]: Video lecture one with a recap on back-propagation

[3]: Video lecture two on Sparse Auto Encoders

[4]: Handouts

[1]: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

[2]: http://www.stanford.edu/class/cs294a/video1.html

[3]: http://www.stanford.edu/class/cs294a/video2.html

[4]: http://www.stanford.edu/class/cs294a/handouts.html

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#12

Post by nl »

If you are interested in this, the list of papers for the 2015 International Conference on Learning Representations has been released: http://www.iclr.cc/doku.php?id=iclr2015:main#submissions

There's some pretty good stuff there. I really liked Crypto-Nets: Neural Networks over Encrypted Data[1]:

The problem we address is the following: how can a user employ a predictive model that is held by a third party, without compromising private information. For example, a hospital may wish to use a cloud service to predict the readmission risk of a patient. However, due to regulations, the patient's medical files cannot be revealed. The goal is to make an inference using the model, without jeopardizing the accuracy of the prediction or the privacy of the data.

[1] http://arxiv.org/abs/1412.6181

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#13

Post by chriswarbo »

gbachik wrote: post #6

How do you guys understand this stuff!? The topic absolutely fascinated me, but after reading the paper... I just feel dumb. Are there any good resources I could use to better understand machine learning papers such as this? I mean I can't even comprehend the implications this paper could have? Is anyone opening to doing some mentoring? Gbachik@gmail.com

Machine learning can be quite jargon-heavy, and in my experience the core ideas can often be hidden amongst a bunch of unnecessary maths.

I got frustrated by a paper yesterday which contained function definitions, summations-of-summations, products of sequences, convolutions, set theory, switching back-and-forth between unary-functions/vectors and binary-functions/matrices, converting back-and-forth between {0, 1}, {-1, 1} and {true, false}, weighting elements of a set by 0/1 instead of taking a sub-set, linear programming, etc.

What was their result? To speed up pair-wise comparisons of structured data, only do N% of the comparisons and it will only take N% of the time. To decide which comparisons to discard, see what works well on a small sample of inputs.

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#14

Post by rmcfeeley »

Why don't we reallocate some of this effort to teaching humans how to use language more effectively? Or is it better to just delegate the responsibility for truth to a seemingly "pure" logic that we like to think exists outside of our human condition?

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#15

Post by gbachik »

gbachik wrote: post #6

How do you guys understand this stuff!? The topic absolutely fascinated me, but after reading the paper... I just feel dumb. Are there any good resources I could use to better understand machine learning papers such as this? I mean I can't even comprehend the implications this paper could have? Is anyone opening to doing some mentoring? Gbachik@gmail.com

Thanks for all the resources guys! I really appreciate it! I'll start diving in!

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#17

Post by kastnerkyle »

If you see convolution as basically truncated recurrence this approach ties in very strongly to recent approaches to machine translation using recurrent nets. I guess depth should allow you to find longterm dependencies, but the fact that CNN were designed for images which have strong local structure and much weaker long term structure makes me think RNNs are better for language, where we see a lot of important long term dependencies. As an example: "The man with the long brown hair entered the saloon" - I would tie saloon and man as the key pieces of that sentence, but that dependency is pretty long and somewhat different than natural images where you don't really expect the corners of images to have any strong relationship in general.

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#18

Post by [deleted] »

[deleted] wrote: post #16
rmcfeeley wrote: post #14

Why don't we reallocate some of this effort to teaching humans how to use language more effectively? Or is it better to just delegate the responsibility for truth to a seemingly "pure" logic that we like to think exists outside of our human condition?

[deleted]

[deleted]

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#19

Post by gwern »

gojomo wrote: post #10
biomimic wrote: post #4

I see Thomas Mikolov, the creator of Google's Word2Vec ( https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/1234... ) is referenced in the paper.

Notably, Mikolev is now at Facebook. My hunch (as a total outsider) is that anything Google publishes is about 2 years behind their current best practices.

The back-and-forth on the ImageNet record between Google/Facebook/Baidu suggests that, unless they're exquisitely coordinating research release, at least some of what they're releasing is indeed close to their state of the art.

obviously writing it up does mean the specified results will be a bit behind what they can actually do in their labs, but that's true of everywhere

Re: A Convolutional Neural Network for Modelling Sentences (2014) [pdf]

#20

Post by bearzoo »

kastnerkyle wrote: post #17

If you see convolution as basically truncated recurrence this approach ties in very strongly to recent approaches to machine translation using recurrent nets. I guess depth should allow you to find longterm dependencies, but the fact that CNN were designed for images which have strong local structure and much weaker long term structure makes me think RNNs are better for language, where we see a lot of important long…

^agreed - no ANN can be treated as a 'jack of all trades' because of the success found in one domain (it seems more networks are designed to boost performance on one type of data)
Post reply on HN