[deleted]
RWKV: Reinventing RNNs for the Transformer Era
61–70 of 176 posts
Re: RWKV: Reinventing RNNs for the Transformer Era
#62Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.
If you think this is wild, see the PaLM 2 paper with 2.5 pages of 2 column attributions. https://arxiv.org/pdf/2305.10403.pdf
Page 28.
That is wild, almost like film credits
Re: RWKV: Reinventing RNNs for the Transformer Era
#63My goal this year: to understand what this is about :-)
Re: RWKV: Reinventing RNNs for the Transformer Era
#64Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve
Are there currently any plans to create a RWKV 30B or 65B? That seems to be the size at which the LLaMA transformer models become genuinely competitive with GPT3.5 for many tasks.
TLDR: please donate A100s to make this happen
Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available
Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range.
Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPUs required to make it happen
The code is already there, just need someone to run it,
Ps: I too am personally interested in how it will perform at ~60B, which I believe will be to be optimal model size for higher level of thoughts (this number is based on intuition not research)
Re: RWKV: Reinventing RNNs for the Transformer Era
#65One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…
> One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models?
We won't know before we've tried it. Reasoning by analogy with humans is not useful. In a year we'll have tried lots of things and can give a much better answer.
Re: RWKV: Reinventing RNNs for the Transformer Era
#66My goal this year: to understand what this is about :-)
If you are familiar with how transformer network works
There is RWKV in 150 lines to help understand all the nitty gritty
https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_li...
Re: RWKV: Reinventing RNNs for the Transformer Era
#67Earlier quoted context omitted.
Are there currently any plans to create a RWKV 30B or 65B? That seems to be the size at which the LLaMA transformer models become genuinely competitive with GPT3.5 for many tasks.
TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…
https://twitter.com/boborado/status/1659608452849897472
You might find that thread interesting, they're taking submissions for potential partnership with LambdaLabs a cloud compute company that has a few hundred H100s laying around. They have an open form and their cofounder is currently doing the rounds having meetings and this may be a good candidate.
I'm not associated with them at all, just interested in the space and things going on.
Re: RWKV: Reinventing RNNs for the Transformer Era
#68Earlier quoted context omitted.
The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?
Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not traini…
> it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board
To be fair, that filters the majority of models in the scoreboard.
Re: RWKV: Reinventing RNNs for the Transformer Era
#69Earlier quoted context omitted.
Are there currently any plans to create a RWKV 30B or 65B? That seems to be the size at which the LLaMA transformer models become genuinely competitive with GPT3.5 for many tasks.
TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…
Are there any estimates anywhere of how many A100s would be needed to e.g. train a 30B model in 6 months?
Re: RWKV: Reinventing RNNs for the Transformer Era
#70One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…
Taking into account the limits of how much can be stored in a hidden state.
My theory is that the model will start generalising the information stored once it starts going past its limits (like real life humans)
So if we take books as an example, we probably do not remember every single word. But we might vaguely remember which section or book events occur
Combine this with the agent model, and we may have an alternative for embeddings. Where we can ask which pages the model recall is relevant to the question. Bring those pages up again. And get the answers
(Once again like how a human might answer in real life)