Were RNNs all we needed?
arxiv.org
Were RNNs all we needed?
1–10 of 269 posts
Re: Were RNNs all we needed?
#2I haven’t gone through the paper in detail yet but maybe someone can answer.
If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
Re: Were RNNs all we needed?
#3I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
I only had a quick look, but it looks like they tweaked the state update so the model can be run with parallel scan instead of having to do it sequentially.
Re: Were RNNs all we needed?
#4Betteridge's law of headlines?
Re: Were RNNs all we needed?
#5Note Yoshua Bengio in the author list. This shouldn't be taken lightly.
Re: Were RNNs all we needed?
#6Betteridge's law of headlines?
For paper titles, the law is that the answer is always "yes"
Re: Were RNNs all we needed?
#7To their credit, the authors (Y. Bengio among them) end the paper with the question, not suggesting they know the answer. These models are very small even by academic standards so any finding would not necessarily extend to current LLM scales. The main conclusion is that RNN class networks can be trained as efficiently as modern alternatives but the resulting performance is only competitive at small scale.
Re: Were RNNs all we needed?
#8I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
It doesn't completely remove it, it removes certain dependencies on it so that it can be computed by parallel scan, there is still a hidden state. It bears some similarity to what was done with Mamba.
Re: Were RNNs all we needed?
#9Note Yoshua Bengio in the author list. This shouldn't be taken lightly.
And this is where science breaks down.
Re: Were RNNs all we needed?
#10I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
They didn't remove the hidden state entirely, they just removed it from the input, forget and update gates. I haven't digested the paper either, but I think that in the case of a GRU this means that the hidden state update masking (z_t and r_t in the paper's formulas) only depends on the new input, not the input plus the prior hidden state.