Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

151–160 of 269 posts

Re: Were RNNs all we needed?

#151

Post by quantadev »

mr_toad wrote: post #112
quantadev wrote: post #69

Earlier quoted context omitted.

Most LLMs aren't even using a "curve" yet at all, right? All they're using is a series of linear equations because the model weights are a simple multiply and add (i.e. basic NN Perceptron). Sure there's a squashing function on the output to keep it in a range from 0 to 1 but that's done BECAUSE we're just adding up stuff. I think probably future NNs will be maybe more adaptive than this perhaps where some Perceptron…

> It's astounding that we DID get the emergent intelligence from just doing this "curve fitting" onto "lines" rather than actual "curves". In Ye Olden days (the 90’s) we used to approximate non-linear models using splines or seperate slopes models - fit by hand. They were still linear, but with the right choice of splines you could approximate a non-linear model to whatever degree of accuracy you wanted. Neural netwo…

In college (BSME) I wrote a computer program to generate cam profiles from Bezier curves. It's just a programming trick to generate curves from straight lines at any level of accuracy you want just by letting the computer take smaller and smaller steps.

It's an interesting concept to think of how NNs might be able to exploit this effect in some way based on straight lines in the weights, because a very small number of points can identify avery precise and smooth curves, where directions on the curve might equate to Semantic Space Vectors.

Re: Were RNNs all we needed?

#152

Post by ziofill »

leereeves wrote: post #143
eru wrote: post #135

Earlier quoted context omitted.

So (most) neural nets can be seen as a function of a _fixed_ form with some inputs and lots and lots of parameters. In my example, a and b were the parameters. The kinds of data you can approximate well with a simple sine wave and the kinds of data you can approximate with a straight line are rather different. Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the sha…

> Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the shape of the function. It doesn't change sine into multiplication etc. It definitely can. The output will always be piecewise linear (with ReLU), but the overall shape can change completely.

You can fit any data with enough parameters. What’s tricky is to constrain a model so that it approximates the ground truth well where there are no data points. If a family of functions is extremely flexible and can fit all kinds of data very efficiently I would argue it makes it harder for those functions to have correct values out of distribution.

Re: Were RNNs all we needed?

#153

Post by [deleted] »

spencerchubb wrote: post #134
baanist wrote: post #74

Earlier quoted context omitted.

I thought the whole point of neural networks was that they were good at searching through these spaces. I'm pretty sure OpenAI is pruning their models behind the scenes to reduce their costs because that's the only way they can keep reducing the cost per token. So their secret sauce at this point is whatever pruning AI they're using to whittle the large computation graphs into more cost efficient consumer products.

When you train a neural network, it is not search, it is descending through a curve. If you were to search for billions of parameters by brute force, you literally could not do it in the lifespan of the universe. A neural network is differentiable, meaning you can take the derivative of it. You train the parameters by taking finding gradient with respect to each parameter, and going in the opposite direction. Hence t…

[deleted]

Re: Were RNNs all we needed?

#154

Post by bob1029 »

spencerchubb wrote: post #134
baanist wrote: post #74

Earlier quoted context omitted.

I thought the whole point of neural networks was that they were good at searching through these spaces. I'm pretty sure OpenAI is pruning their models behind the scenes to reduce their costs because that's the only way they can keep reducing the cost per token. So their secret sauce at this point is whatever pruning AI they're using to whittle the large computation graphs into more cost efficient consumer products.

When you train a neural network, it is not search, it is descending through a curve. If you were to search for billions of parameters by brute force, you literally could not do it in the lifespan of the universe. A neural network is differentiable, meaning you can take the derivative of it. You train the parameters by taking finding gradient with respect to each parameter, and going in the opposite direction. Hence t…

A biological neural network is certainly not differentiable. If the thing we want to build is not realizable with this technique, why can't we move on from it?

Gradient descent isn't the only way to do this. Evolutionary techniques can explore impossibly large, non-linear problem spaces.

Being able to define any kind of fitness function you want is sort of like a super power. You don't have to think in such constrained ways down this path.

Re: Were RNNs all we needed?

#156

Post by trott »

f_devd wrote: post #142
trott wrote: post #133

Earlier quoted context omitted.

> Transformers actually have an quantifiable state size Are you griping about my writing O(X^2) above instead of precisely 2X^2, like this paper? The latter implies the former. > So a sufficiently sized RNN could have the same state capacity as a transformer. Does this contradict anything I've said? If you increase the size of the RNN, while keeping the Transformer fixed, you can match their recurrent state sizes (if…

I was responding to > a Transformer layer can look back see O(X^2) numbers, while an RNN can only see O(X) numbers The thing is RNN can look back infinitely if you don't exceed the state capacity. For transformers the state it is defined semi-implicitly (you can change the hidden dims but you cannot extend the look back; ignoring transformer-xl et al.) defined by the amount of tokens, for an RNN it's defined explicit…

> The big-O here is irrelevant for the architectures since it's all in the configuration & implementation of the model; i.e. there is no relevant asymptote to compare.

?!

NNs are like any other algorithm in this regard. Heck, look at the bottom of page 2 of the Were RNNs All We Needed paper. It has big-O notation there and elsewhere.

> I was responding to

>> a Transformer layer can look back see O(X^2) numbers, while an RNN can only see O(X) numbers

In the BASED paper, in Eq. 10, sizeof(s) = 2dN. But I defined d = N = X above. Ergo, sizeof(s) = 2X^2 = O(X^2).

For minGRU, sizeof(s) = d. Ergo, sizeof(s) = X = O(X).

Re: Were RNNs all we needed?

#157

Post by f1shy »

drodgers wrote: post #101
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

> The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape I have almost the opposite take. We've had a lot of datasets for ages, but all the progress in the last decade has come from advances how curves are architected and fit to the dataset (including applying more computing power). Maybe there's some theoretical sense in which older models could have solve…

Wait! We certainly did NOT have huge datasets (like current internet) for ages. Not even decades. I’ve seen a lecture by a MIT professor (which I cannot find now) where he asserted categorically, that the advances in AI are mostly because of the huge data that we now have and we didn’t before. And that was an old video.

Re: Were RNNs all we needed?

#158

Post by theanonymousone »

I remember that, the way I understood it, Transformers solved two major "issues" of RNNs that enabled the later boom: Vanishing gradients limiting the context (and model?) size and difficulty in parallelisation limiting the size of the training data.

Do we have solutions for these two problems now?

Re: Were RNNs all we needed?

#159

Post by nutanc »

wongarsu wrote: post #50
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

One big thing that bells and whistles do is limit the training space. For example when CNNs took over computer vision that wasn't because they were doing something that dense networks couldn't do. It was because they removed a lot of edges that didn't really matter, allowing us to spend our training budget on deeper networks. Similarly transformers are great because they allow us to train gigantic networks somewhat e…

This is true. This is the reason, in many of our experiments we find that using a new algorithm, KESieve, we actually find the planes much faster than the traditional deep learning training approaches. The premise is, a neaural network builds planes which separate the data and adjusts these planes through an iterative learning process. What if we can find a non iterative method which can draw these same planes. We have been trying this and so far we have been able to replace most network layers using this approach. haven't tried for transformers though yet.

Some links if interested:

[1] https://gpt3experiments.substack.com/p/understanding-neural-...

[2] https://gpt3experiments.substack.com/p/building-a-vector-dat...

Post reply on HN