Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

141–150 of 269 posts

Re: Were RNNs all we needed?

#141

Post by imjonse »

phkahler wrote: post #31
imjonse wrote: post #7

To their credit, the authors (Y. Bengio among them) end the paper with the question, not suggesting they know the answer. These models are very small even by academic standards so any finding would not necessarily extend to current LLM scales. The main conclusion is that RNN class networks can be trained as efficiently as modern alternatives but the resulting performance is only competitive at small scale.

>> These models are very small even by academic standards so any finding would not necessarily extend to current LLM scales. Emphasis on not necessarily. >> The main conclusion is that RNN class networks can be trained as efficiently as modern alternatives but the resulting performance is only competitive at small scale. Shouldn't the conclusion be "the resulting competitive performance has only been confirmed at sma…

yes, that is clearer indeed. However S4 and Mamba class models have also performed well at small scale and started lagging with larger models and larger context sizes, or at particular tasks.

Re: Were RNNs all we needed?

#142

Post by f_devd »

trott wrote: post #133
f_devd wrote: post #109

Earlier quoted context omitted.

Transformers actually have an quantifiable state size (see https://hazyresearch.stanford.edu/static/posts/2024-06-22-ac... ) although it's anywhere between 200k and 2M floats (for 360M and 1.33B respectively iinm). So a sufficiently sized RNN could have the same state capacity as a transformer. (this is from the Based paper: https://arxiv.org/pdf/2402.18668 )

> Transformers actually have an quantifiable state size Are you griping about my writing O(X^2) above instead of precisely 2X^2, like this paper? The latter implies the former. > So a sufficiently sized RNN could have the same state capacity as a transformer. Does this contradict anything I've said? If you increase the size of the RNN, while keeping the Transformer fixed, you can match their recurrent state sizes (if…

I was responding to

> a Transformer layer can look back see O(X^2) numbers, while an RNN can only see O(X) numbers

The thing is RNN can look back infinitely if you don't exceed the state capacity. For transformers the state it is defined semi-implicitly (you can change the hidden dims but you cannot extend the look back; ignoring transformer-xl et al.) defined by the amount of tokens, for an RNN it's defined explicitly by the state size.

The big-O here is irrelevant for the architectures since it's all in the configuration & implementation of the model; i.e. there is no relevant asymptote to compare.

As an aside this was what was shown in the based paper, the fact that you can have a continuity of state (as with RNN) while have the same associative recall capability as a transformer (the main downfall of recurrent methods at that point).

Re: Were RNNs all we needed?

#143

Post by leereeves »

eru wrote: post #135
refulgentis wrote: post #131

Earlier quoted context omitted.

> Naively, you wouldn't expect I, a nave, expected this. Is multiplication versus sine in the analogy hiding it, perhaps? I've always pictured it as just "needing to learn" the function terms and the function guts are an abstraction that is learned. Might just be because I'm a physics dropout with a bunch of whacky half-remembered probably-wrong stuff about how any function can be approximated by ex. fourier series.

So (most) neural nets can be seen as a function of a _fixed_ form with some inputs and lots and lots of parameters. In my example, a and b were the parameters. The kinds of data you can approximate well with a simple sine wave and the kinds of data you can approximate with a straight line are rather different. Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the sha…

> Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the shape of the function. It doesn't change sine into multiplication etc.

It definitely can. The output will always be piecewise linear (with ReLU), but the overall shape can change completely.

Re: Were RNNs all we needed?

#145

Post by jjtheblunt »

baanist wrote: post #77
jjtheblunt wrote: post #30

Earlier quoted context omitted.

What are you saying is Turing-complete?

Neural networks are Turing complete, i.e. there is a universal neural network that can compute any effectively computable function¹. Incidentally, when this is combined with Rice's theorem² it means that safety research is essentially an unsolvable problem because any non-trivial property of a sufficiently complex neural network, e.g. one that can simulate a Turing machine, will have properties which can not be predi…

super interesting, and i'd not seen either reference. thanks very much.

Re: Were RNNs all we needed?

#146

Post by avereveard »

xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

well yes but actually no I guess: the transformers benefit at the time was that they were more stable while learning, enabling larger and larger network and dataset to be learnt.

Re: Were RNNs all we needed?

#147

Post by quantadev »

viktor_von wrote: post #139
quantadev wrote: post #128

Earlier quoted context omitted.

> What is a "linear weight"? In the context of discussing linearity v.s. non-linearity adding the word "linear" in front of "weight" is more clear, which is what my top level post on this thread was all about too. It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training,…

> It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training, to achieve genuine reasoning. When such basic perceptrons are scaled enormously, it becomes less surprising that they can achieve some level of 'genuine reasoning' (e.g., accurate next-word prediction), since th…

Yeah, no one is surprised that LLMs do what they're trained to do: predict tokens. The surprise comes from the fact that merely training to predict tokens ends up with model weights that generate emergent reasoning.

If you want to say reasoning and token prediction are just the same thing at scale you can say that, but I don't fall into that camp. I think there's MUCH more to learn, and indeed a new field of math or even physics that we haven't even discovered yet. Like a step change in mathematical understanding analogous to the invention of Calculus.

Re: Were RNNs all we needed?

#148

Post by mirekrusin »

drodgers wrote: post #101
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

> The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape I have almost the opposite take. We've had a lot of datasets for ages, but all the progress in the last decade has come from advances how curves are architected and fit to the dataset (including applying more computing power). Maybe there's some theoretical sense in which older models could have solve…

Isn’t bogosort transformer and quicksort proposed modified rnn (175 times faster training for 500 seq) here?

Re: Were RNNs all we needed?

#149

Post by quantadev »

wrs wrote: post #127
quantadev wrote: post #96

Earlier quoted context omitted.

> The non linearities are not there primarily to keep the outputs in a given range Precisely what the `Activation Function` does is to squash an output into a range (normally below one, like tanh). That's the only non-linearity I'm aware of. What other non-linearities are there? All the training does is adjust linear weights tho, like I said. All the training is doing is adjusting the slopes of lines.

With a ReLU activation function, rather than a simple linear function of the inputs, you get a piecewise linear approximation of a nonlinear function. ReLU enables this by being nonlinear in a simple way, specifically by outputting zero for negative inputs, so each linear unit can then limit its contribution to a portion of the output curve. (This is a lot easier to see on a whiteboard!)

ReLU technically has a non-linearity at zero, but in some sense it's still even MORE linear than tanh or sigmoid, so it just demonstrates even better than tanh-type squashing that all this LLM stuff is being done ultimately with straight line math. All a ReLU function does is choose which line to use, a sloped one or a zero one.

Re: Were RNNs all we needed?

#150

Post by bob1029 »

actionfromafar wrote: post #88
teruakohatu wrote: post #25

Earlier quoted context omitted.

> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale. This! Not just fastest but with the lowest resources in total. Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of p…

Unless we could build chips in 3D?

We are already doing this.
Post reply on HN