Is the code for generating the reactions from the LSTM hidden units posted anywhere? That was the best part for me and I'd love to use it in my own projects.
Exploring LSTMs
21–30 of 48 posts
Re: Exploring LSTMs
#22LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…
What you describe is very similar to the concept of Differentiable Neural Computer, from DeepMind [1]. While still experimental they already have nice results.
[1] https://deepmind.com/blog/differentiable-neural-computers/
Re: Exploring LSTMs
#23Is the code for generating the reactions from the LSTM hidden units posted anywhere? That was the best part for me and I'd love to use it in my own projects.
In Andrej Karpathy's excellent blogpost on RNNs[1], he links to some of the code he used for visualisation[2]. I have done something similar. In general: put your the activations of each cell memory over time through tanh and color it based on that.
[1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [2] http://cs.stanford.edu/people/karpathy/viscode.zip
Re: Exploring LSTMs
#24In the experiment on teaching an LSTM to count, it's useful to note that the examples it's trained on are derivations [1] from a grammar a^nb^n (with n > 0), a classic example of a Context-Freee Grammar (CFG).
It's well understood that CFGs can not be induced from examples. Which accounts for the fact that LSTMs cannot learn "counting" in this manner, nor indeed can any other learning method that learns from examples.
_______________
[1] "Strings generated from"
[2] The same goes for any formal grammars other than finite ones, as in simpler than regular.
Re: Exploring LSTMs
#25Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?
I hope that wasn't quite what I said :)
I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.
Re: Exploring LSTMs
#26Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?
I hope that wasn't quite what I said :) I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.
Oh also I did say that CNNs often beat RNNs, even for time series, language, and audio. So I generally try them first. You can always stick an RNN on top of the CNN and get the best of both worlds.
Re: Exploring LSTMs
#27[deleted]
Re: Exploring LSTMs
#28LSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because i…
Any links to implementations?
[deleted]
Re: Exploring LSTMs
#29In the experiment on teaching an LSTM to count, it's useful to note that the examples it's trained on are derivations [1] from a grammar a^nb^n (with n > 0), a classic example of a Context-Freee Grammar (CFG). It's well understood that CFGs can not be induced from examples. Which accounts for the fact that LSTMs cannot learn "counting" in this manner, nor indeed can any other learning method that learns from examples…
> It's well understood that CFGs can not be induced from examples
I think you mean something more specific (e.g. polynomial in a particular sense).
+ Automatic Learning of Context-Free Grammar (Chen et al.): http://www.aclweb.org/anthology/O06-1004
+ Learning context-free grammars from structural data in polynomial time (Sakakibara, 1994): http://www.sciencedirect.com/science/article/pii/03043975909... (uses positive skeletons)
Nice overview: http://staff.icar.cnr.it/staff/ruffolo/public_html/progetti/...
Re: Exploring LSTMs
#30Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?
Yes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance) Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can lear…
So if LSTMs are purposely forgetting, do you need less training data than a CNN?