Embed, encode, attend, predict: the new deep learning formula for NLP models
1–10 of 17 posts
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#2I really loved reading this article but it's always so hard to figure out exactly how these things work out in detail.
I understand matrix multiplication but it seems that (some of) these matrix to vector calculations are actually trained by/as part of the neural net... but how exactly that works I can't figure out coming at it from articles like this.
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#3I really loved reading this article but it's always so hard to figure out exactly how these things work out in detail. I understand matrix multiplication but it seems that (some of) these matrix to vector calculations are actually trained by/as part of the neural net... but how exactly that works I can't figure out coming at it from articles like this.
If you want to learn more about it, I recommend you to read neural networks and deep learning: http://neuralnetworksanddeeplearning.com
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#4I really loved reading this article but it's always so hard to figure out exactly how these things work out in detail. I understand matrix multiplication but it seems that (some of) these matrix to vector calculations are actually trained by/as part of the neural net... but how exactly that works I can't figure out coming at it from articles like this.
(Author here)
Thanks! I'm planning to make two follow up posts, on each of the systems, that go through those details. I blurred them out in this post because I wanted to get across this more abstract story about the data types and transformations.
There are lots of good posts about attention mechanisms. The WildML post is good, as is Chris Olah's post. Bidirectional RNNs are a little bit less well covered, but the idea is not too difficult to understand given a single RNN (or LSTM, GRU etc).
You should also read the papers :). This is how most people who are doing ML --- including the people building practical things, not researchers --- are staying up to date. Academia is so competitive and writing is cheap relative to experimentation. The deep learning literature is really pretty easy to follow.
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#5Here's the implementation of the first example model, the Parikh et al. (2016) textual entailment system:
https://github.com/explosion/spaCy/tree/master/examples/kera...
This got dropped during editing...Updating the post to make this more prominent.
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#6That's a nice article in that it manages to get excitement across without forgetting to balance it somewhat. Kind of a rare thing these days.
Btw, I'm interested to hear how well training with large one-hot encoded vectors scales. A paper someone pointed me to recently on HN suggested that it doesn't scale very well:
One-shot Learning with Memory-Augmented Neural Networks [https://arxiv.org/abs/1605.06065]
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#7I really like this post. So much NLP research is 'locked away' in academic papers, and making the knowledge more accessible through posts like this is very important for large-scale adoption by non-academics.
Also, really well done on the site design. Love the graphics, font, layout and 'progress bar' animation at the top. Very nice UX overall.
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#8I really loved reading this article but it's always so hard to figure out exactly how these things work out in detail. I understand matrix multiplication but it seems that (some of) these matrix to vector calculations are actually trained by/as part of the neural net... but how exactly that works I can't figure out coming at it from articles like this.
(Author here) Thanks! I'm planning to make two follow up posts, on each of the systems, that go through those details. I blurred them out in this post because I wanted to get across this more abstract story about the data types and transformations. There are lots of good posts about attention mechanisms. The WildML post is good, as is Chris Olah's post. Bidirectional RNNs are a little bit less well covered, but the i…
Really like what you're doing with SpaCy and explosionAI, good stuff :)
What do you think about dilated convolutional encoder/decoder networks [1]? Useful for NLP beyond machine translation?
[1] https://arxiv.org/abs/1610.10099, https://github.com/paarthneekhara/byteNet-tensorflow
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#9Earlier quoted context omitted.
(Author here) Thanks! I'm planning to make two follow up posts, on each of the systems, that go through those details. I blurred them out in this post because I wanted to get across this more abstract story about the data types and transformations. There are lots of good posts about attention mechanisms. The WildML post is good, as is Chris Olah's post. Bidirectional RNNs are a little bit less well covered, but the i…
Really like what you're doing with SpaCy and explosionAI, good stuff :) What do you think about dilated convolutional encoder/decoder networks [1]? Useful for NLP beyond machine translation? [1] https://arxiv.org/abs/1610.10099 , https://github.com/paarthneekhara/byteNet-tensorflow
One of the main issues with character level CNN's (irrespective of convolution type IIRC) is the inability of the model to handle unknown words, which is something that word level models do well. So if you look at applications of NLP in domains that need this to work well, you won't get much from purely char models in my experience.
Re: Embed, encode, attend, predict: the new deep learning formula for NLP models
#10typo: contradition
I just wish I understood the rest of the article...