Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch
Suggestion: do the same for transformer.
An Introduction to Recurrent Neural Networks
11–20 of 26 posts
Re: An Introduction to Recurrent Neural Networks
#12This kind of article is absolutely the thing everyone new to deep learning/neural networks should read. I wish there was one for each type of algorithm.
I do have similar articles for Neural Networks (MLP) and CNNs:
---
NN: https://victorzhou.com/blog/intro-to-neural-networks/
NN HN discussion: https://news.ycombinator.com/item?id=19320217
---
CNN: https://victorzhou.com/blog/intro-to-cnns-part-1/ https://victorzhou.com/blog/intro-to-cnns-part-2/
CNN HN discussions: https://news.ycombinator.com/item?id=19981736 https://news.ycombinator.com/item?id=20064900
Re: An Introduction to Recurrent Neural Networks
#13Re: An Introduction to Recurrent Neural Networks
#14It's worth noting that apparently (as I learned lately) RNNs are going slightly out of fashion because they are hard to parallelize and have trouble remembering important stuff at larger distances. Transformers are proposed as a possible solution - very roughly speaking, they use attention mechanisms instead of recurrent memory and can run in parallel.
I have to say that while I understand the problems with recurrent nets (which I've used many times), I haven't yet grokked the alternatives. Here are some decently looking search results for you as starting points. Warning, these can be longer and heavier reads probably not for beginners.
https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c7... (there's some sensationalism here to be fair)
https://mchromiak.github.io/articles/2017/Sep/12/Transformer...
https://www.analyticsvidhya.com/blog/2019/06/understanding-t...
https://www.tensorflow.org/beta/tutorials/text/transformer
That being said, I think that understanding RNNs is very beneficial conceptually and nowadays there are relatively easy to use implementations that should be pretty good for many use cases.
Re: An Introduction to Recurrent Neural Networks
#15Nice! I like that the author wrote the code by hand rather than leaning on some framework. It makes it a lot easier to connect the math to the code. :)
As a meta-comment on these "Introduction to _____ neural network" articles (not just this one), I wish people would spend more time talking about when their neural net isn't the right tool for the job. SVMs, kNN, even basic regression techniques aren't any less effective than they were 20 years ago. They're easier to interpret and debug, require many fewer parameters, and potentially (you may need to apply some tricks here or there) faster at both training and evaluation time.
Re: An Introduction to Recurrent Neural Networks
#16It's worth noting that apparently (as I learned lately) RNNs are going slightly out of fashion because they are hard to parallelize and have trouble remembering important stuff at larger distances. Transformers are proposed as a possible solution - very roughly speaking, they use attention mechanisms instead of recurrent memory and can run in parallel. I have to say that while I understand the problems with recurrent…
[deleted]
Re: An Introduction to Recurrent Neural Networks
#17It's worth noting that apparently (as I learned lately) RNNs are going slightly out of fashion because they are hard to parallelize and have trouble remembering important stuff at larger distances. Transformers are proposed as a possible solution - very roughly speaking, they use attention mechanisms instead of recurrent memory and can run in parallel. I have to say that while I understand the problems with recurrent…
Mainly RNNs are much slower to train than transformers.
Re: An Introduction to Recurrent Neural Networks
#18Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch
Very nicely written post. I particularly like how you attached a link to your codebase on repl.it so anyone who is interested can tinker with the code. One thing I have been wondering for some time is whether the vanilla RNN can learn negations (i.e. 'not good' == 'bad') and valence shifts (e.g. modifier words like 'very' --- they do not carry sentiment connotations themselves, but may amplify/dampen the sentiment of…
I tried a LSTM model of stock twists and it seemed reasonably good at handling negations (single and double negatives at least)
Re: An Introduction to Recurrent Neural Networks
#19Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch
In your post you initialize your weights a certain way but you said there are better ways. Do you have any resources for better ways?
Re: An Introduction to Recurrent Neural Networks
#20It's worth noting that apparently (as I learned lately) RNNs are going slightly out of fashion because they are hard to parallelize and have trouble remembering important stuff at larger distances. Transformers are proposed as a possible solution - very roughly speaking, they use attention mechanisms instead of recurrent memory and can run in parallel. I have to say that while I understand the problems with recurrent…
Mainly RNNs are much slower to train than transformers.
As well as stability issues, long range dependencies, etc..