Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

71–80 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#71

Post by stellaathena »

ftxbro wrote: post #56
stellaathena wrote: post #41

Earlier quoted context omitted.

Our discord servers (a primary one, a spin-off for RWKV, another spinoff for BioML, etc) have tens of thousands of people between them :) So not quite everyone. But this was a community effort with a public call for contributions

Are you the Stella from the Acknowledgements section? Are you in one of those spinoffs like BioML, or you aren't on the author list because you have some conflict with your work where you can't legally do something because of intellectual property laws or license things or NDA? Also when you talk about 'our discord servers' which ones do you mean? Is it ones run by Eleuther?

Yes I am (apparently) in the acknowledgments. I was not an author on the paper because I didn’t have time to contribute too much. I also try to err on the side of not being added to papers, as my position (I run EleutherAI) tends to encourage people to be overgenerous with offers. I anticipate having more time this coming month and being on the version that’s submitted for peer review, but we’ll see.

BlinkDL has been working on this project for two-ish years, originally in the EleutherAI discord and then created his own to house the project.

I wasn’t thinking too hard about my exact wording, but yes I was thinking of EleutherAI and its various spin-off servers. EleutherAI doesn’t /run/ any of the other servers, but we all have a close collaborative relationship. I’m sure there’s a lot of duplication of membership (e.g., I’m in all of them) but quickly adding up the membership of each server comes out to around 70,000. EleutherAI and LAION are the largest at 25k each, with the others typically having around 5k each. I would expect at least 30k of those users to be unique though.

Re: RWKV: Reinventing RNNs for the Transformer Era

#72

Post by pico_creator »

logicchains wrote: post #69
pico_creator wrote: post #64

Earlier quoted context omitted.

TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…

Are there any estimates anywhere of how many A100s would be needed to e.g. train a 30B model in 6 months?

That’s a loaded question without deciding dataset size

Re: RWKV: Reinventing RNNs for the Transformer Era

#73

Post by pas »

pico_creator wrote: post #72
logicchains wrote: post #69

Earlier quoted context omitted.

Are there any estimates anywhere of how many A100s would be needed to e.g. train a 30B model in 6 months?

That’s a loaded question without deciding dataset size

can you elaborate on the chinchilla law / dataset problem a bit? (perhaps by editing your previous comment?)

what datasets are available to the community, how big are these, are they needed to be updated from time to time, where are these stored, what are the usual cost ranges involved, ...? :o

thank you!

Re: RWKV: Reinventing RNNs for the Transformer Era

#74

Post by tanseydavid »

noduerme wrote: post #16
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

I know a guy who memorized "Fahrenheit 451" back in the '60s, and went on to memorize a dozen other books - I think it was a requirement of the anti-establishment cult he was in at the time. Anyhow, his recall is still pretty fantastic.

Sounds like your friend was sort of the Charles Babbage of Large Language Models.

Re: RWKV: Reinventing RNNs for the Transformer Era

#75

Post by dcl »

teruakohatu wrote: post #14
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books.

I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps with some loss of fidelity.

My guess is that there will need to be an advancement or two before we can can get an AI to read all of the ASIOF books so far and ask "What really happened at the Tower of Joy?"

Re: RWKV: Reinventing RNNs for the Transformer Era

#76

Post by anon291 »

noduerme wrote: post #24

I wrote a home-brew neural network around 2006, just to see what would happen. I'd read no papers on it, and just kind of made it up as I went along. The result was basically a cube of "neurons" which had stronger and weaker trigger points to their neighbors and would propagate "spark" to one or more neighbors based on the strength and direction of spark they got from their other neighbors. Each of the connections an…

> But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better? Maybe my concept of the principle behind modern LLMs is too archaic or rooted in a cartoon understanding of our own wetware that I tried to reproduce.

Realastically... the problem is a data problem. The math is mostly there. Transformers et al would have been figured out in no time had we had the sort of data tools we have today. I'm talking about the ease with which you can take gigabytes of data, throw it in S3, and analyze it in minutes.

That, combined with cheap and accessible compute (cloud) and the maturation of CUDA meant it was all a matter of time before this took place.

Re: RWKV: Reinventing RNNs for the Transformer Era

#77

Post by visarga »

m3kw9 wrote: post #6
mindwok wrote: post #3

As a (mostly) layperson, this seems like it could be a very significant paper. What are the odds we see the next few years of machine learning models based on RWKV like we have seen with transformers since the attention is all you need paper?

This model seem to be good when input context is large in comparison to OpenAI, would just wait till if someone productize it. Most likely it wine be just one paper but many new and old ideas combined

Being a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.

Re: RWKV: Reinventing RNNs for the Transformer Era

#78

Post by anon291 »

pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Currently, what I'm seeing with RWKV is that attention fades of quickly. The model will start to produce output, but very quickly (a few dozen tokens), its own output tokens are suddenly taking 'precedence' over the input question and it starts to simply repeat itself.

For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format. It starts off great. However, eventually, it seems like the beginning of the JSON list 'overtakes' the question I asked, and it starts to just produce random data that would seem plausible based on the set of things in the list. I realize this is due perhaps to the precision losses of the RNN as weights decay.

However, I feel there ought to be some way we can prevent that. Any thoughts?

Re: RWKV: Reinventing RNNs for the Transformer Era

#79

Post by visarga »

evolve7942 wrote: post #40
bigdict wrote: post #7

Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.

If you think this is wild, see the PaLM 2 paper with 2.5 pages of 2 column attributions. https://arxiv.org/pdf/2305.10403.pdf

No, the Bloom paper is wild: 2.5 pages of author names written with no spacing, I counted 472 authors after deduplication. PaLM 2 has only 181 authors. ;)

https://arxiv.org/pdf/2211.05100.pdf

Re: RWKV: Reinventing RNNs for the Transformer Era

#80

Post by p-e-w »

mlsu wrote: post #54
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you. These things are very diff…

> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go.

You're confusing your perception of how your mind works with how your mind actually works.

Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on.

This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.

Post reply on HN