Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

51–60 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#51

Post by pico_creator »

ftxbro wrote: post #45
pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Do you think there could be a kind of universality principle at work here, where once you make a good enough architecture then the details don't matter so much compared to the model size and training flops and dataset size? In other words, maybe it wasn't a coincidence that your architecture worked about as well as the transformer architecture?

There is a reasonable argument for that (heard the idea go around between multiple AI engineers, that once you go past a certain scale, it does not matter for its evals)

One of the biggest issue for testing all of this, is it takes a crap ton of GPUs to prove all the alternatives to transformers beyond 1B param.

For example I’m waiting for someone to do a 1B-14B text based diffusion network

Finally, if this is truely the case (and all that really matter is size+dataset)

We really should use an architecture that is cheaper to train and run. And that’s what RWKV represents here

You can even run the 7B quantized model reasonably on most laptops (try the rwkv-cpp / rwkv-cpp-node project)

Re: RWKV: Reinventing RNNs for the Transformer Era

#52

Post by [deleted] »

stellaathena wrote: post #41
ftxbro wrote: post #9

Earlier quoted context omitted.

I think it's just, everyone in their discord channel.

Our discord servers (a primary one, a spin-off for RWKV, another spinoff for BioML, etc) have tens of thousands of people between them :) So not quite everyone. But this was a community effort with a public call for contributions

[deleted]

Re: RWKV: Reinventing RNNs for the Transformer Era

#53

Post by pico_creator »

bluepoint wrote: post #44
pico_creator wrote: post #38

Earlier quoted context omitted.

Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not traini…

Can beginners without money with by PhDs, contribute? If so what would be the best way to start?

There are lots of really low hanging fruits

- integrating this with AI platform X/Y/Z

- setting up evals

- improving the code quality

- making a how to guide (it’s stuck on my todo list)

- helping with dataset

- doing silly experiments on how the architecture work (and if the changes give good result)

- etc etc

One of the community goals is to make this a model for EVERYONE on earth that means we need quality dataset for all the non English languages

So even on that level there are things to do

( find something that interest you on the community )

Re: RWKV: Reinventing RNNs for the Transformer Era

#54

Post by mlsu »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you.

These things are very different. The mental model I have is that they put the entire book into a big hypercube and then *flash*, instantly, out comes the next token. And likewise, when they encounter text that is 8001 tokens long, token #1 is completely erased, gone, like it never existed.

Re: RWKV: Reinventing RNNs for the Transformer Era

#55

Post by pico_creator »

chaxor wrote: post #43
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN). Hopefull…

Yea using the analogy of the eyes can see the entire document

Vs

I need to memorise everything as it’s spoken out, and then answer on it

Is a good approximate on the difference.

The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training)

It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task

Re: RWKV: Reinventing RNNs for the Transformer Era

#56

Post by ftxbro »

stellaathena wrote: post #41
ftxbro wrote: post #9

Earlier quoted context omitted.

I think it's just, everyone in their discord channel.

Our discord servers (a primary one, a spin-off for RWKV, another spinoff for BioML, etc) have tens of thousands of people between them :) So not quite everyone. But this was a community effort with a public call for contributions

Are you the Stella from the Acknowledgements section? Are you in one of those spinoffs like BioML, or you aren't on the author list because you have some conflict with your work where you can't legally do something because of intellectual property laws or license things or NDA? Also when you talk about 'our discord servers' which ones do you mean? Is it ones run by Eleuther?

Re: RWKV: Reinventing RNNs for the Transformer Era

#59

Post by pico_creator »

noduerme wrote: post #24

I wrote a home-brew neural network around 2006, just to see what would happen. I'd read no papers on it, and just kind of made it up as I went along. The result was basically a cube of "neurons" which had stronger and weaker trigger points to their neighbors and would propagate "spark" to one or more neighbors based on the strength and direction of spark they got from their other neighbors. Each of the connections an…

A large percent of the RWKV community ain’t experts. And are here doing weird, dumb or crazy homebrew experiments

So keep doing weird experiments

Re: RWKV: Reinventing RNNs for the Transformer Era

#60

Post by logicchains »

pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Are there currently any plans to create a RWKV 30B or 65B? That seems to be the size at which the LLaMA transformer models become genuinely competitive with GPT3.5 for many tasks.
Post reply on HN