Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

141–150 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#141

Post by candiodari »

pico_creator wrote: post #101
coppsilgold wrote: post #81

Earlier quoted context omitted.

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

Naming is a perpetual problem. My issue is with "hallucination", which everyone takes to be the "problem" with GPT style networks making things up. Never mind that transformers are just trying to predict the next likely token, NOT the truth PLUS that they're trained from the internet. As everyone knows, the internet is not known for correctness and truth. If you want any neural network to figure out the truth independently, it'll obviously need the ability to go out into the real world and even needs to be allowed to experiment for most things.

Hallucination used to mean the following. A basic neural network is:

f(x) = y = repeat(nonlinearity(ax[0] + bx[1] + ...))

And then you adjust a, b, c, ... until y is reasonable, according to the cost function. But look! The very same backpropagation can adjust x[0], x[1] ... with the same cost function and only a small change in the code.

This allows you to reverse the question neural networks answer. Which can be an incredibly powerful way to answer questions.

And that used to be called hallucination in Neural networks. Instead of "change these network weights to transform x into y, keeping x constant" you ask "change x to transform x into y, keeping the network weights constant".

Now it's impossible finding half the papers on the this topic. AARGH!

Re: RWKV: Reinventing RNNs for the Transformer Era

#142

Post by candiodari »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

Okay ... the answer really is "much better than other RNNs of any type". And transformers do: LSTMs and GRUs networks effectively have a context size of 1.

Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance.

In transformers, just like in RNNs, processing the nth token influences the nth token AND the (n+1)th token. So the influence of the nth token "slowly dies down". It influences the (n+1)th token a lot, the (n+2)th token a bit less and so on. The special thing about attention is that it doesn't start dying until you go past the context size.

Or with a bit of math notation. If a neural network is a function f, then:

RNN: f(x[n]) = f( f(x[n-1]), x[n] )

Transformer: f(x[n]) = f(sum(i=0...context size, f(x-i)), x[n])

The difference between LSTMs and GRUs is in the structure of f. And "sum" is, like a lot of things in here, uh, basically accurate but missing a lot of detail.

Re: RWKV: Reinventing RNNs for the Transformer Era

#143

Post by galeos »

coppsilgold wrote: post #81
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

"...every token embedding interacts with every other token embedding before it"

And, in the case of BERT, every token embedding after it too.

Re: RWKV: Reinventing RNNs for the Transformer Era

#144

Post by pico_creator »

ianbutler wrote: post #67
pico_creator wrote: post #64

Earlier quoted context omitted.

TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…

https://twitter.com/boborado/status/1659608452849897472 You might find that thread interesting, they're taking submissions for potential partnership with LambdaLabs a cloud compute company that has a few hundred H100s laying around. They have an open form and their cofounder is currently doing the rounds having meetings and this may be a good candidate. I'm not associated with them at all, just interested in the spac…

wierdly their form requires a company rep (which RWKV does not have, as its not a company) - lets see how it goes ...

Re: RWKV: Reinventing RNNs for the Transformer Era

#145

Post by cs702 »

pico_creator wrote: post #35
cs702 wrote: post #22

Alas, it doesn't appear to work well for longer contexts: https://twitter.com/arankomatsuzaki/status/16390003799784038... Has anyone here experimented with this recently to confirm?

It has already been confirmed that with the right dataset we can scale it effectively from 2k to 4K, and 4K to 8k via fine tuning (you dun even need to train a new foundation model) We believe this can be done for 16k to way beyond 100k Research in how RWKV handle the hidden state shows that it is barely used (imo: (This is actively being experimented on - we dun really know the limit yet)

Thank you. This is great to hear!

I'm going to take a closer look :-)

Re: RWKV: Reinventing RNNs for the Transformer Era

#146

Post by cs702 »

FallDead wrote: post #25
cs702 wrote: post #22

Alas, it doesn't appear to work well for longer contexts: https://twitter.com/arankomatsuzaki/status/16390003799784038... Has anyone here experimented with this recently to confirm?

one of the authors here!, I think someone in our discord did experiments to prove that it does work for longer contexts, The pace of this work moves really fast. This might have been an earlier models in the series. RWKV it needs to be trained for longer contexts lengths in order to obtain that skill a context tuning if you will. IRCC there will be a follow up paper for it.

Thank you for taking the time to comment here!

That sounds promising. Maybe scale (of model and training samples) is all you need.

And RNNs are obviously so much more efficient at inference.

I'm going to take a closer look :-)

Re: RWKV: Reinventing RNNs for the Transformer Era

#147

Post by PoignardAzur »

chrgy wrote: post #115

Here is a summary of all comments by Transformers, wonder how RNN does: RWKV is a new language model architecture that is comparable to transformers in terms of performance. RWKV is more efficient than transformers, which makes it possible to train larger models on smaller datasets. The RWKV community is open source and welcomes contributions from anyone. There are plans to create larger versions of RWKV, but this wi…

> The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including:

What? I though the chinchilla-optimal regime was something like "20 tokens per weight". That's not remotely exponential, even by the word's colloquial use.

Re: RWKV: Reinventing RNNs for the Transformer Era

#148

Post by matteogrella »

RC_ITR wrote: post #95

> We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference” It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.

As one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).

Re: RWKV: Reinventing RNNs for the Transformer Era

#149

Post by MontyCarloHall »

candiodari wrote: post #142
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

Okay ... the answer really is "much better than other RNNs of any type". And transformers do: LSTMs and GRUs networks effectively have a context size of 1. Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance. In transformers, just like in RNNs, processing the nth token influences the nth…

RNNs have a theoretically infinite context size, but in practice it’s limited by vanishing gradients due to too much recursion. That’s why the recursive units are usually LSTMs or GRUs that have explicit functionality to cut off the recursion, using a learnable threshold that’s a function of both the current and recursive inputs.

In your example, x[n-1] = f(x[n-2], x[n-1]). So really, your expression should be

RNN: x[n] = f( … f( f( f( f(k0, x[0]), x[1]), x[2]), x[3]), …, x[n-1])

where k0 is a learnable parameter.

Re: RWKV: Reinventing RNNs for the Transformer Era

#150

Post by MontyCarloHall »

teruakohatu wrote: post #14
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

>Many people could reason about the book in its entirety

That’s analogous to having War and Peace in the training set. When actually reading War and Peace, nobody‘s working memory includes a precise recall of everything they’ve read thus far, which is more analogous to an LLM’s context size.

Post reply on HN