Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

181–190 of 269 posts

Re: Were RNNs all we needed?

#181

Post by famouswaffles »

tsimionescu wrote: post #177
famouswaffles wrote: post #71

Earlier quoted context omitted.

It's necessary for arbitrary information processing if you can forget and have no way to "unforget". A model can decide to forget something that turns out to be important for some future prediction. A human can go back and re-read/listen etc, A transformer is always re-reading but a RNN can't and is fucked.

If the networks are to ever be a path to a closer to general intelligence, they will anyway need to be able to ask for context to be repeated, or to have separate storage where they can "choose" to replay it themselves. So this problem likely has to be solved another way anyway, both for transformers and for RNNs.

For a transformer, context is already always being repeated every token. They can fetch information that became useful anytime they want. I don't see what problem there is to solve here.

Re: Were RNNs all we needed?

#182

Post by tsimionescu »

famouswaffles wrote: post #181
tsimionescu wrote: post #177

Earlier quoted context omitted.

If the networks are to ever be a path to a closer to general intelligence, they will anyway need to be able to ask for context to be repeated, or to have separate storage where they can "choose" to replay it themselves. So this problem likely has to be solved another way anyway, both for transformers and for RNNs.

For a transformer, context is already always being repeated every token. They can fetch information that became useful anytime they want. I don't see what problem there is to solve here.

For a transformer, context is limited, so the same kind of problem applies after you exceed some size.

Re: Were RNNs all we needed?

#183

Post by immibis »

mkaic wrote: post #64
immibis wrote: post #59

Earlier quoted context omitted.

This architecture, on the surface, seems to preclude the basic function of recognizing sequences of tokens. At the very least, it seems like it should suffer from something like the pumping lemma: if [the ][cat ][is ][black ] results in the output getting close to a certain vector, [the ][cat ][is ][black ][the ][cat ][is ][black ][the ][cat ][is ][black ] should get even closer to that vector and nowhere close to a…

Counterpoint: the hidden state at the beginning of ([the][cat][is][black]) x 3 is (probably) initialized to all zeros, but after seeing those first 4 tokens, it will not be all zeros. Thus, going into the second repetition of the sentence, the model has a different initial hidden state, and should exhibit different behavior. I think this makes it possible for the model to learn to recognize repeated sequences and avo…

The new hidden state after the first repetition will just be a linear combination between zero and what the non-recurring network outputs. After more repetitions, it will be closer to what the network outputs.

Re: Were RNNs all we needed?

#184

Post by visarga »

ebalit wrote: post #171
theanonymousone wrote: post #158

I remember that, the way I understood it, Transformers solved two major "issues" of RNNs that enabled the later boom: Vanishing gradients limiting the context (and model?) size and difficulty in parallelisation limiting the size of the training data. Do we have solutions for these two problems now?

Transformers can also fetch at any moment any previous information that become useful . RNN are constantly updating and overwriting their memory. It means they need to be able to predict what is going to be useful in order to store it for later. This is a massive advantage for Transformers in interactive use cases like in ChatGPT. You give it context and ask questions in multiple turns. Which part of the context was…

You could theoretically run the input twice, allowing the model to correlate later tokens with previous ones. It would fix the problem with not knowing what information to retain. A more complicated approach would train the RNN to request replaying some earlier data when needed.

A great thing about RNNs is they can easily fork the state and generate trees, it would be possible to backtrack and work on combinatorial search problems.

Also easier to cache demonstrations for free in the initial state, a model that has seen lots of data is not using more memory than a model starting from scratch.

Re: Were RNNs all we needed?

#185

Post by f_devd »

trott wrote: post #156
f_devd wrote: post #142

Earlier quoted context omitted.

I was responding to > a Transformer layer can look back see O(X^2) numbers, while an RNN can only see O(X) numbers The thing is RNN can look back infinitely if you don't exceed the state capacity. For transformers the state it is defined semi-implicitly (you can change the hidden dims but you cannot extend the look back; ignoring transformer-xl et al.) defined by the amount of tokens, for an RNN it's defined explicit…

> The big-O here is irrelevant for the architectures since it's all in the configuration & implementation of the model; i.e. there is no relevant asymptote to compare. ?! NNs are like any other algorithm in this regard. Heck, look at the bottom of page 2 of the Were RNNs All We Needed paper. It has big-O notation there and elsewhere. > I was responding to >> a Transformer layer can look back see O(X^2) numbers, while…

That's just the state calculation which would be O(N) and O(1) respectively. The based paper is saying if you made Transformers recurrent you would have a state size of 2Nd -> O(N), while based has a state size of d*d' -> O(1).

Transformers do have O(N^2) time & memory complexity, and Based/RNN/SSM {O(N) time, O(1) mem}, with respect to sequence length if that's what you mean. The point is it doesn't really give an indication of quality.

We can choose our constant arbitrarily so the big-O you've stated only indicates memory/time-complexity not 'look-back' ability relevant to any task. If you input the entire sequence N times into an RNN, you also have perfect recall with O(N^2) but it's not exactly an efficient use of our resources.

Ideally our state memory is maximally utilized, this is the case for RNNs in the limit (although likely oversubscribed) but is not the case for transformers. The holy grail is to have an input-dependent state-size, however that is quite difficult.

Re: Were RNNs all we needed?

#186

Post by famouswaffles »

bob1029 wrote: post #154
spencerchubb wrote: post #134

Earlier quoted context omitted.

When you train a neural network, it is not search, it is descending through a curve. If you were to search for billions of parameters by brute force, you literally could not do it in the lifespan of the universe. A neural network is differentiable, meaning you can take the derivative of it. You train the parameters by taking finding gradient with respect to each parameter, and going in the opposite direction. Hence t…

A biological neural network is certainly not differentiable. If the thing we want to build is not realizable with this technique, why can't we move on from it? Gradient descent isn't the only way to do this. Evolutionary techniques can explore impossibly large, non-linear problem spaces. Being able to define any kind of fitness function you want is sort of like a super power. You don't have to think in such constrain…

>A biological neural network is certainly not differentiab

Biology is biology and has its constraints. Doesn't necessarily mean a biologically plausible optimizer would be the most efficient or correct way in silicon.

>If the thing we want to build is not realizable with this technique, why can't we move on from it?

All the biologically plausible optimizers we've fiddled with (and we've fiddled with quite a lot) just work (results wise) like gradient descent but worse. We've not "moved on" because gradient descent is and continues to be better.

>Evolutionary techniques can explore impossibly large, non-linear problem spaces.

Sure, with billions of years (and millions of concurrent experiments) on the table.

Re: Were RNNs all we needed?

#187

Post by xpe »

baanist wrote: post #23
marcosdumay wrote: post #18

R == Recurrent From theory the answer to the question should be "yes", they are Turing complete. The real question is about how to train them, and the paper is about that.

Why aren't AI researchers automating the search for efficient architectures?

Program synthesis is a generalization of this. I’m not sure that many ML researchers have thought about the connections yet.

Re: Were RNNs all we needed?

#188

Post by roboboffin »

quantadev wrote: post #73
PunchTornado wrote: post #27

To me this is further evidence that these LLMs learn only to speak English, but there is no reasoning at all in them. If you simplify a lot and obtain the same results and we know how complex the brain is.

Every LLM expert on the planet agrees LLMs are doing "reasoning". No one says they have feelings or qualia, but we all know there's definitely genuinely artificial reasoning happening. What LLMs have shown both Neuroscience and Computer Science is that reasoning is a mechanical process (or can be simulated by mechanical processes) and is not purely associated only with consciousness.

I'm not sure that's true at all. There are several well known researchers that say LLMs are in fact not doing reasoning.

Re: Were RNNs all we needed?

#189

Post by ithkuil »

falcor84 wrote: post #175
tadala wrote: post #174

Everyone wants to use less compute to fit more in, but (obviously?) the solution will be to use more compute and fit less. Attention isn't (topologically) attentive enough. All these RNN-lite approaches are doomed, beyond saving costs, they're going to get cooked by some other arch—even more expensive than transformers.

Would you mind expanding upon your thesis? If that compute and all those parameters aren't "fitting" the training examples, what is it that the model is learning, and how should that be analyzed?

I think there are two distinct areas. One is the building of the representations, which is achieved by fitting. The other area is loosely defined as "computing" which is some kind of searching for a path through representation space. All of that is wrapped in a translation layer that can turn those representations into stuff we humans can understand and interact with. All of that is achieved to some extent by current transformer architectures, but I guess some believe that they are not quite as effective at the "computation/search" stage.

Re: Were RNNs all we needed?

#190

Post by eru »

leereeves wrote: post #143
eru wrote: post #135

Earlier quoted context omitted.

So (most) neural nets can be seen as a function of a _fixed_ form with some inputs and lots and lots of parameters. In my example, a and b were the parameters. The kinds of data you can approximate well with a simple sine wave and the kinds of data you can approximate with a straight line are rather different. Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the sha…

> Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the shape of the function. It doesn't change sine into multiplication etc. It definitely can. The output will always be piecewise linear (with ReLU), but the overall shape can change completely.

Sorry, when I meant 'shape' of the function, I meant the shape of the abstract syntax tree (or something like that).

Not the shape of its graph when you draw it.

Post reply on HN