RWKV: Reinventing RNNs for the Transformer Era • Hacker News Board

Re: RWKV: Reinventing RNNs for the Transformer Era

#101

Post by pico_creator » Tue, May 23, 2023, 6:29 AM UTC

coppsilgold wrote: post #81

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess)

But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

Re: RWKV: Reinventing RNNs for the Transformer Era

#102

Post by jokteur » Tue, May 23, 2023, 6:30 AM UTC

zamnos wrote: post #85

dcl wrote: post #75

Earlier quoted context omitted.

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

ASIOF spoiler below! > At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret. Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.

But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

Re: RWKV: Reinventing RNNs for the Transformer Era

#103

Post by taneq » Tue, May 23, 2023, 6:31 AM UTC

pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Is everyone still carefully not mentioning that it looks like it’s pronounced “Roku”? ;)

Re: RWKV: Reinventing RNNs for the Transformer Era

#104

Post by pico_creator » Tue, May 23, 2023, 6:34 AM UTC

logicchains wrote: post #97

pico_creator wrote: post #72

Earlier quoted context omitted.

That’s a loaded question without deciding dataset size

Would it be possible to just use the exact same dataset as LLaMA? (There's an open source project currently training a transformer on exactly that).

You mean red pajama? I believe that has already started for 1-14B (need to double check)

Re: RWKV: Reinventing RNNs for the Transformer Era

#105

Post by hiddencost » Tue, May 23, 2023, 6:35 AM UTC

EleutherAI is stunning. The only other papers I've seen with such a diversity of institutions are high energy particle physics.

Re: RWKV: Reinventing RNNs for the Transformer Era

#106

Post by pico_creator » Tue, May 23, 2023, 6:37 AM UTC

anon291 wrote: post #94

pico_creator wrote: post #89

Earlier quoted context omitted.

Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

Yeah... So I did that which is how I got it to begin correctly. This is what I mean though. I'll say "get a list of Blah from the following document in Json format like this: Example" Then I feed the document and add a spot for the answer. The model begins correctly. But usually in the middle of the Json list generation, it will veer off, and start hallucinating as if it forgot the document and the task. I'm happy to…

Hmm we might need to look into the instruct training data. Which is mostly based on gpt4all filtered and mixed with others

(You are using raven right? That’s the instruct trained varient)

Btw ping the discord if ur looking into finetuning for your usecase

Re: RWKV: Reinventing RNNs for the Transformer Era

#107

Post by heja2009 » Tue, May 23, 2023, 6:43 AM UTC

p-e-w wrote: post #98

anon373839 wrote: post #90

Earlier quoted context omitted.

While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.

Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

To the contrary there is very good evidence that we pull arbitrary explanations whenever needed. See Gazzaniga's split brain experiments where people gave all sorts of reasons when asked why they did something and just could not know.

Re: RWKV: Reinventing RNNs for the Transformer Era

#108

Post by logicchains » Tue, May 23, 2023, 6:44 AM UTC

pico_creator wrote: post #104

logicchains wrote: post #97

Earlier quoted context omitted.

Would it be possible to just use the exact same dataset as LLaMA? (There's an open source project currently training a transformer on exactly that).

You mean red pajama? I believe that has already started for 1-14B (need to double check)

Yep that's the one. Curious roughly how many A100s it'd take to train a 65B RWKV on that.

Re: RWKV: Reinventing RNNs for the Transformer Era

#109

Post by TheFragenTaken » Tue, May 23, 2023, 6:45 AM UTC

teruakohatu wrote: post #14

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

"The Magical Number Seven, Plus or Minus Two" [1] applies to many things. In this case, a book, could reasonably be reasoned about almost no matter the length as you argue, given the scope of a book is usually limited to a few topics (the French invasion of Russia). Similarly three to seven chapters could be retold, but not every 361 chapters. A "580,000" word long book about 58,000 things would be unfeasible for a human, but probably feasible for an LLM with a 580k context.

That in essence, I believe, is the difference. An LLM (while still predicting the next word, given it's context), seem to care less about the number of subjects in a given context, than humans do.

[1]: https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus...

Re: RWKV: Reinventing RNNs for the Transformer Era

#110

Post by nl » Tue, May 23, 2023, 6:45 AM UTC

jokteur wrote: post #102

zamnos wrote: post #85

Earlier quoted context omitted.

ASIOF spoiler below! > At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret. Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.

But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

Thinking of it as "the training dataset" vs "the context window" is the wrong way of looking at it.

There's a bunch of prior art for adaption techniques for getting new data into a trained model (fine tuning, RLHF etc). There's no real reason to think there won't be more techniques that turn what think of now as the context window into something that alters the weights in the model and is serialized back to disk.