Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

111–120 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#111

Post by pico_creator »

humbleharbinger wrote: post #8
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

Not sure if it answers your question but the paper notes something similar in its discussion of limitations: > the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compar…

Prompt design is definitely a huge one shifting to rwkv

Sadly too many folks copy and paste what works for openAI and move on when it fails

Re: RWKV: Reinventing RNNs for the Transformer Era

#112

Post by zaptrem »

pico_creator wrote: post #89
anon291 wrote: post #78

Earlier quoted context omitted.

Currently, what I'm seeing with RWKV is that attention fades of quickly. The model will start to produce output, but very quickly (a few dozen tokens), its own output tokens are suddenly taking 'precedence' over the input question and it starts to simply repeat itself. For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format.…

Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

Why would asking the question first improve quality? Is it because the model will be better aware of what info it Can and can’t throw away at each step? This seems like the opposite of transformers.

Re: RWKV: Reinventing RNNs for the Transformer Era

#113

Post by lukeplato »

pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Is there any potential improvements over transformers for interpretablity or alignment?

Re: RWKV: Reinventing RNNs for the Transformer Era

#114

Post by LoganDark »

zaptrem wrote: post #112
pico_creator wrote: post #89

Earlier quoted context omitted.

Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

Why would asking the question first improve quality? Is it because the model will be better aware of what info it Can and can’t throw away at each step? This seems like the opposite of transformers.

RWKV does not work like transformers. The "transformer" part here is the training step. RWKV is an RNN with fixed-size state, so old information slightly decays each time it reads a new token. Hence the freshest memory is of the most recent tokens.

Re: RWKV: Reinventing RNNs for the Transformer Era

#115

Post by chrgy »

Here is a summary of all comments by Transformers, wonder how RNN does:

RWKV is a new language model architecture that is comparable to transformers in terms of performance. RWKV is more efficient than transformers, which makes it possible to train larger models on smaller datasets. The RWKV community is open source and welcomes contributions from anyone. There are plans to create larger versions of RWKV, but this will require more computational resources. Here are some additional details about the chinchilla law and the dataset problem:

The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including:

The Pile: A massive dataset of text and code. The Chinchilla: A smaller dataset of text and code that is designed for training RWKV models. The Red Pajamas: A dataset of text and code that is being used to train a 65B RWKV model. These datasets are stored in a variety of locations, including:

The RWKV GitHub repository The Chinchilla website The Red Pajamas website The RWKV community is constantly updating the datasets and adding new ones. If you are interested in contributing, please visit the RWKV GitHub repository.

Re: RWKV: Reinventing RNNs for the Transformer Era

#116

Post by zwaps »

I wish this was written with more care. None of the symbols are defined.

Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single person with CNN background on the team and it makes zero sense. What if you actually have channels in your data? What then?

Re: RWKV: Reinventing RNNs for the Transformer Era

#117

Post by dcl »

jokteur wrote: post #102
zamnos wrote: post #85

Earlier quoted context omitted.

ASIOF spoiler below! > At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret. Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.

But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

Exactly.

But also, not just ASIOF is in the training set, but presumably lots of discussion about it and all the interesting events in the book.

Re: RWKV: Reinventing RNNs for the Transformer Era

#118

Post by pico_creator »

logicchains wrote: post #108
pico_creator wrote: post #104

Earlier quoted context omitted.

You mean red pajama? I believe that has already started for 1-14B (need to double check)

Yep that's the one. Curious roughly how many A100s it'd take to train a 65B RWKV on that.

Really bad napkin math as no one has attempted 65B (so +\- 50%)

8 x 8 x 8 A100, should be able to do a 100k++ tokens/s at that size

With a dataset of 1.2 trillion tokens. That’s 12 million seconds. Or 140 days

(PS: this is why everyone is training <60B, its crazy the cost, even if my math estimate is wrong by 300%, its still a crazy number)

Re: RWKV: Reinventing RNNs for the Transformer Era

#119

Post by dcl »

nl wrote: post #110
jokteur wrote: post #102

Earlier quoted context omitted.

But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

Thinking of it as "the training dataset" vs "the context window" is the wrong way of looking at it. There's a bunch of prior art for adaption techniques for getting new data into a trained model (fine tuning, RLHF etc). There's no real reason to think there won't be more techniques that turn what think of now as the context window into something that alters the weights in the model and is serialized back to disk.

It's a reasonable way to look at it given that's how pretty much all 'deployed' versions of LLM's work?

Re: RWKV: Reinventing RNNs for the Transformer Era

#120

Post by dcl »

danielbln wrote: post #96
dcl wrote: post #75

Earlier quoted context omitted.

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

Some loss to fidelity? Our memories are hugely lossy, reconstructed at recall based on a bunch of concepts. It's great, but it's also very lossy.

You got me.
Post reply on HN