Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring LSTMs

blog.echen.me

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Exploring LSTMs

#31

Post by mikkel »

visarga wrote: post #18

LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…

Deep mind just released a couple of papers about this:

https://deepmind.com/blog/neural-approach-relational-reasoni...

Here is an implementation of "A simple neural network module for relational reasoning" in pytorch: https://github.com/kimhc6028/relational-networks

Re: Exploring LSTMs

#32

Post by Drdrdrq »

mrplank wrote: post #17

LSTMs are on their retour in my opinion. They are a hack to make memory in recurrent networks more persistent. In practice they overfit too easy. They are being replaced with convolutional networks. Have a look at the latest paper from Facebook about translation for more details.

The way I see it, the difference is that with CNN you have fixed maximum timeframe in which knowledge about world is preserved, while LSTMs and RNNs in general do not impose such restrictions. This makes them better suited for some applications.

If I am missing something please correct me.

Re: Exploring LSTMs

#33

Post by mrdrozdov »

visarga wrote: post #18

LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…

I'd put a little more faith in LSTMs. There's a lot less evidence for what they can't do than what they can. With enough fiddling, can get LSTMs to work for most tasks.

I'm not sure they're quite as complicated as you're making them out to be. If they are, then try a GRU instead ;)

Re: Exploring LSTMs

#34

Post by mrdrozdov »

mrplank wrote: post #17

LSTMs are on their retour in my opinion. They are a hack to make memory in recurrent networks more persistent. In practice they overfit too easy. They are being replaced with convolutional networks. Have a look at the latest paper from Facebook about translation for more details.

I think LSTMs are far from being replaced. They work too well and are too simple to use.

Re: Exploring LSTMs

#35

Post by canjobear »

YeGoblynQueenne wrote: post #24

In the experiment on teaching an LSTM to count, it's useful to note that the examples it's trained on are derivations [1] from a grammar a^nb^n (with n > 0), a classic example of a Context-Freee Grammar (CFG). It's well understood that CFGs can not be induced from examples. Which accounts for the fact that LSTMs cannot learn "counting" in this manner, nor indeed can any other learning method that learns from examples…

It's not clear that the learnability results about formal grammars are useful for describing what can be learned by practical algorithms, where the grammar does not have to be identified with perfect confidence.

As an example, it is possible to induce a counting algorithm from examples with high probability: https://colala.bcs.rochester.edu/papers/piantadosi2012bootst...

Re: Exploring LSTMs

#36

Post by visarga »

LukeB42 wrote: post #20
visarga wrote: post #18

LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…

Any links to implementations?

Yes, one of the links is from DeepMind: "A simple neural network module for relational reasoning" https://arxiv.org/abs/1706.01427

Another one is from Thomas Kipf: "Graph Convolutional Matrix Completion" https://arxiv.org/abs/1706.02263

Re: Exploring LSTMs

#37

Post by avmich »

MrQuincle wrote: post #29
YeGoblynQueenne wrote: post #24

In the experiment on teaching an LSTM to count, it's useful to note that the examples it's trained on are derivations [1] from a grammar a^nb^n (with n > 0), a classic example of a Context-Freee Grammar (CFG). It's well understood that CFGs can not be induced from examples. Which accounts for the fact that LSTMs cannot learn "counting" in this manner, nor indeed can any other learning method that learns from examples…

> It's well understood that CFGs can not be induced from examples I think you mean something more specific (e.g. polynomial in a particular sense). + Automatic Learning of Context-Free Grammar (Chen et al.): http://www.aclweb.org/anthology/O06-1004 + Learning context-free grammars from structural data in polynomial time (Sakakibara, 1994): http://www.sciencedirect.com/science/article/pii/03043975909... (uses positive…

Also http://nbviewer.jupyter.org/url/norvig.com/ipython/xkcd1313.... , considering regexes are a subset of CFGs.

Re: Exploring LSTMs

#38

Post by LukeB42 »

visarga wrote: post #36
LukeB42 wrote: post #20

Earlier quoted context omitted.

Any links to implementations?

Yes, one of the links is from DeepMind: "A simple neural network module for relational reasoning" https://arxiv.org/abs/1706.01427 Another one is from Thomas Kipf: "Graph Convolutional Matrix Completion" https://arxiv.org/abs/1706.02263

Thanks for elucidating the contents of the DeepMind paper and for hipping me to the Graph Convolutional Matrix Completion paper.

Re: Exploring LSTMs

#39

Post by rjtavares »

jph00 wrote: post #26
jph00 wrote: post #25

Earlier quoted context omitted.

I hope that wasn't quite what I said :) I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.

Oh also I did say that CNNs often beat RNNs, even for time series, language, and audio. So I generally try them first. You can always stick an RNN on top of the CNN and get the best of both worlds.

I just wanted to take the chance to say thank you for the course. As someone who is interested in the topic but isn't a practicioner, just watching it all work in practice has been fascinating and highly enlightening.

Re: Exploring LSTMs

#40

Post by make3 »

inlineint wrote: post #16

I personally find recurrent highway networks (RHNs) as described in [1] to be easier to understand and remember the formulas for than the original LSTM. Because as they are generalizations of LSTM, if one understands RHNs, one can understand LSTMs as just a particular case of RHN. Instead of handwaving about "forgetting", it is IMO better to understand the problem of vanishing gradients and how can forget gates actua…

Yes, but LSTMs are fucking everywhere in the industry so understanding them is crucial
Post reply on HN