Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

51–60 of 269 posts

Re: Were RNNs all we needed?

#51

Post by TaurenHunter »

ants_everywhere wrote: post #46
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

> is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) (Somewhat) fun and (somewhat) related fact: there's a whole cottage industry of "is all you need" papers https://arxiv.org/search/?query=%22is+all+you+need%22&search...

Reminds me of the "Considered Harmful" articles:

https://meyerweb.com/eric/comment/chech.html

Re: Were RNNs all we needed?

#52

Post by psb217 »

mkaic wrote: post #48
trott wrote: post #47

My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. But what is an RNN to do? While the length of it…

The counterargument here is that you can just scale the size of the hidden state sufficiently such that it can hold compressed representations of whatever-length sequence you like. Ultimately, what I care about is whether RNNs could compete with transformers if FLOPs are held constant—something TFA doesn't really investigate.

Well, that's what Transformer already does... One problem with the scaling you're describing is that there would be a massive amount of redundant information stored in hidden activations during training the RNN. The hidden state at each time step t in the sequence would need to contain all info that (i) could be useful for predicting the token at time t and (ii) that could be useful for predicting tokens at times >t. (i) is obvious and (ii) is since all information about the past is transferred to future predictions through the current hidden state. In principle, Transformers can avoid storing redundant info in multiple hidden states at the cost of having to maintain and access (via attention) a larger hidden state at test/eval time.

Re: Were RNNs all we needed?

#53

Post by dheera »

xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

I mean, transformer-based LLMs are RNNs, just really really really big ones with very wide inputs that maintain large amounts of context.

Re: Were RNNs all we needed?

#54

Post by immibis »

dheera wrote: post #53
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

I mean, transformer-based LLMs are RNNs, just really really really big ones with very wide inputs that maintain large amounts of context.

No. An RNN has an arbitrarily-long path from old inputs to new outputs, even if in practice it can't exploit that path. Transformers have fixed-size input windows.

Re: Were RNNs all we needed?

#55

Post by jprete »

TaurenHunter wrote: post #51
ants_everywhere wrote: post #46

Earlier quoted context omitted.

> is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) (Somewhat) fun and (somewhat) related fact: there's a whole cottage industry of "is all you need" papers https://arxiv.org/search/?query=%22is+all+you+need%22&search...

Reminds me of the "Considered Harmful" articles: https://meyerweb.com/eric/comment/chech.html

I wonder if there's something about tech culture - or tech people - that encourages them to really, really like snowclones.

Re: Were RNNs all we needed?

#56

Post by hedgehog »

kelseyfrog wrote: post #42
baanist wrote: post #23

Earlier quoted context omitted.

Why aren't AI researchers automating the search for efficient architectures?

The search space is all off too wide, difficult to parameterize, and there is a wide gap between effective and ineffective architectures - ie: a very small change can make a network effectively DOA.

Notably architecture search was popular for small vision nets where the cost of many training runs was low enough. I suspect some of the train-then-prune approaches will come back, but even there only by the best funded teams.

Re: Were RNNs all we needed?

#58

Post by famouswaffles »

immibis wrote: post #54
dheera wrote: post #53

Earlier quoted context omitted.

I mean, transformer-based LLMs are RNNs, just really really really big ones with very wide inputs that maintain large amounts of context.

No. An RNN has an arbitrarily-long path from old inputs to new outputs, even if in practice it can't exploit that path. Transformers have fixed-size input windows.

You can't have a fixed state and have arbitrarily-long path from input. Well you can but then it's just meaningless because you fundamentally cannot keep stuffing information of arbitrary length into a fixed state. RNNs effectively have fixed-size input windows.

Re: Were RNNs all we needed?

#59

Post by immibis »

mkaic wrote: post #35

I strongly enjoy the simplicity of their "minGRU" architecture. It's basically just: class MinGRU(nn.Module): def __init__(self, token_size, hidden_state_size): self.token_to_proposal = nn.Linear(token_size, hidden_size) self.token_to_mix_factors = nn.Linear(token_size, hidden_size) def forward(self, previous_hidden_state, current_token): proposed_hidden_state = self.token_to_proposal(current_token) mix_factors = tor…

This architecture, on the surface, seems to preclude the basic function of recognizing sequences of tokens. At the very least, it seems like it should suffer from something like the pumping lemma: if [the ][cat ][is ][black ] results in the output getting close to a certain vector, [the ][cat ][is ][black ][the ][cat ][is ][black ][the ][cat ][is ][black ] should get even closer to that vector and nowhere close to a "why did you just repeat the same sentence three times" vector? Without non-linear mixing between input token and hidden state, there will be a lot of linear similarities between similar token sequences...

Re: Were RNNs all we needed?

#60

Post by immibis »

famouswaffles wrote: post #58
immibis wrote: post #54

Earlier quoted context omitted.

No. An RNN has an arbitrarily-long path from old inputs to new outputs, even if in practice it can't exploit that path. Transformers have fixed-size input windows.

You can't have a fixed state and have arbitrarily-long path from input. Well you can but then it's just meaningless because you fundamentally cannot keep stuffing information of arbitrary length into a fixed state. RNNs effectively have fixed-size input windows.

The path is arbitrarily long, not wide. It is possible for an RNN to be made that remembers the first word of the input, no longer how long the input is. This is not possible with a transformer, so we know they are fundamentally different.
Post reply on HN