Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

81–90 of 269 posts

Re: Were RNNs all we needed?

#81

Post by acchow »

fsndz wrote: post #28
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

[flagged]

> After reading this paper, I am now

Is this your paper?

Re: Were RNNs all we needed?

#83

Post by charlescurt123 »

I find the entire field lacking when it comes to long-horizon problems. Our current, widely used solution is to scale, but we're nowhere near achieving the horizon scales even small mammal brains can handle. Our models can have trillions of parameters, yet a mouse brain would still outperform them on long-horizon tasks and efficiency. It's something small, simple, and elegant—an incredible search algorithm that not only finds near-optimal routes but also continuously learns on a fixed computational budget.

I'm honestly a bit envious of future engineers who will be tackling these kinds of problems with a 100-line Jupyter notebook on a laptop years from now. If we discovered the right method or algorithm for these long-horizon problems, a 2B-parameter model might even outperform current models on everything except short, extreme reasoning problems.

The only solution I've ever considered for this is expanding a model's dimensionality over time, rather than focusing on perfect weights. The higher dimensionality you can provide to a model, the greater its theoretical storage capacity. This could resemble a two-layer model—one layer acting as a superposition of multiple ideal points, and the other layer knowing how to use them.

When you think about the loss landscape, imagine it with many minima for a given task. If we could create a method that navigates these minima by reconfiguring the model when needed, we could theoretically develop a single model with near-infinite local minima—and therefore, higher-dimensional memory. This may sound wild, but consider the fact that the human brain potentially creates and disconnects thousands of new connections in a single day. Could it be that these connections steer our internal loss landscape between different minima we need throughout the day?

Re: Were RNNs all we needed?

#84

Post by quantadev »

OkayPhysicist wrote: post #75
quantadev wrote: post #69

Earlier quoted context omitted.

Most LLMs aren't even using a "curve" yet at all, right? All they're using is a series of linear equations because the model weights are a simple multiply and add (i.e. basic NN Perceptron). Sure there's a squashing function on the output to keep it in a range from 0 to 1 but that's done BECAUSE we're just adding up stuff. I think probably future NNs will be maybe more adaptive than this perhaps where some Perceptron…

The "squashing function" necessarily is nonlinear in multilayer nueral networks. A single layer of a neural network can be quite simply written a weight matrix, times an input vector, equalling an output vector, like so Ax = y Adding another layer is just multiplying a different set of weights times the output of the first, so B(Ax)= y If you remember your linear algebra course, you might see the problem: that can be…

Right. All the squashing is doing is keeping the output of any neuron in a range of below 1.

But the entire NN itself (Perceptron ones, which most LLMs are) is still completely using nothing but linearity to store all the knowledge from the training process. All the weights are just an 'm' in the basic line equation 'y=m*x+b'. The entire training process does nothing but adjust a bunch of slopes of a bunch of lines. It's totally linear. No non-linearity at all.

Re: Were RNNs all we needed?

#85

Post by nazgul17 »

quantadev wrote: post #84
OkayPhysicist wrote: post #75

Earlier quoted context omitted.

The "squashing function" necessarily is nonlinear in multilayer nueral networks. A single layer of a neural network can be quite simply written a weight matrix, times an input vector, equalling an output vector, like so Ax = y Adding another layer is just multiplying a different set of weights times the output of the first, so B(Ax)= y If you remember your linear algebra course, you might see the problem: that can be…

Right. All the squashing is doing is keeping the output of any neuron in a range of below 1. But the entire NN itself (Perceptron ones, which most LLMs are) is still completely using nothing but linearity to store all the knowledge from the training process. All the weights are just an 'm' in the basic line equation 'y=m*x+b'. The entire training process does nothing but adjust a bunch of slopes of a bunch of lines.…

The non linearities are fundamental. Without them, any arbitrarily deep NN is equivalent to a shallow NN (easily computable, as GP was saying), and we know those can't even solve the XOR problem.

> nothing but linearity

No, if you have non linearities, the NN itself is not linear. The non linearities are not there primarily to keep the outputs in a given range, though that's important, too.

Re: Were RNNs all we needed?

#86

Post by magicalhippo »

famouswaffles wrote: post #71
phkahler wrote: post #67

Earlier quoted context omitted.

>> A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. Which isn't necessary. If you say "translate the following to german." Instead, all it needs is to remember the task at hand and a much smaller amount of recent input. Well, and the ability to output in parallel with processing input.

It's necessary for arbitrary information processing if you can forget and have no way to "unforget". A model can decide to forget something that turns out to be important for some future prediction. A human can go back and re-read/listen etc, A transformer is always re-reading but a RNN can't and is fucked.

That's just because we twisted it's arm. One could for example feed the reversed input after, ie abc|cba where | is a special token. That would allow it to react to any part of the message.

Re: Were RNNs all we needed?

#87

Post by slashdave »

trott wrote: post #47

My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. But what is an RNN to do? While the length of it…

> the amount of information the model retains about it is bounded by whatever is in its hidden state

This is no different than a transformer, which, after all, is bound by a finite state, just organized in a different manner.

Re: Were RNNs all we needed?

#88

Post by actionfromafar »

teruakohatu wrote: post #25
islewis wrote: post #15

Earlier quoted context omitted.

> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime." I haven't fully ingested the paper yet, but it looks like it's focused more on compute optimization than the size of the dataset: > ... and (2) are fully parallelizable during training (175x faster for a sequence of length 512 Even if many types of architectures converge to the same l…

> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale. This! Not just fastest but with the lowest resources in total. Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of p…

Unless we could build chips in 3D?

Re: Were RNNs all we needed?

#89

Post by slashdave »

mkaic wrote: post #35

I strongly enjoy the simplicity of their "minGRU" architecture. It's basically just: class MinGRU(nn.Module): def __init__(self, token_size, hidden_state_size): self.token_to_proposal = nn.Linear(token_size, hidden_size) self.token_to_mix_factors = nn.Linear(token_size, hidden_size) def forward(self, previous_hidden_state, current_token): proposed_hidden_state = self.token_to_proposal(current_token) mix_factors = tor…

Log space is important if the token probabilities span a large range of values (powers). There is a reason that maximum likelihood fitting is always performed with log likelihoods.

Re: Were RNNs all we needed?

#90

Post by DoctorOetker »

phkahler wrote: post #67
trott wrote: post #47

My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. But what is an RNN to do? While the length of it…

>> A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. Which isn't necessary. If you say "translate the following to german." Instead, all it needs is to remember the task at hand and a much smaller amount of recent input. Well, and the ability to output in parallel with processing input.

Also, a lightweight network could do a first pass to identify tasks, instructions, constraints etc, and then a second pass could use the RNN.

Consider the flood fill algorithm or union-find algorithm, which feels magical upon first exposure.

https://en.wikipedia.org/wiki/Hoshen%E2%80%93Kopelman_algori...

Having 2 passes can enable so much more than a single pass.

Another alternative could be to have a first pass make notes in a separate buffer while parsing the input. The bandwidth of the note taking and reading can be much much lower than that required for fetching the billions of parameters.

Post reply on HN