Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

121–130 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#121

Post by pico_creator »

lukeplato wrote: post #113
pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Is there any potential improvements over transformers for interpretablity or alignment?

For anything past 8k context size

We are talking about over 10x reduction in GPU time for inferencing tokens and for training too

Aka it’s cheaper and faster

Alignment is frankly IMO purely a dataset design and training issue. And has nothing to do with the model

Re: RWKV: Reinventing RNNs for the Transformer Era

#122

Post by pico_creator »

Tostino wrote: post #99
pico_creator wrote: post #89

Earlier quoted context omitted.

Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

Are there any ways to train it to maintain attention on the original prompt no matter the distance from it, and selectively pay attention to its own output where relevant?

Instruction training. This is a WIP

Re: RWKV: Reinventing RNNs for the Transformer Era

#123

Post by sorz »

zwaps wrote: post #116

I wish this was written with more care. None of the symbols are defined. Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single person with CNN background on the team and it makes zero sense. What if you actually have channels in your data? What then?

They said the paper is still working in progress and will improve it.

https://twitter.com/AiEleuther/status/1660811180901019648

Re: RWKV: Reinventing RNNs for the Transformer Era

#124

Post by TeMPOraL »

dcl wrote: post #75
teruakohatu wrote: post #14

Earlier quoted context omitted.

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

> Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book.

Some of them, some of the time. This is best comparable with ChatGPT having those books in its training dataset.

The context window is more like short-term memory. GPT-4 can fit[0] ~1.5 chapters of Game of Thrones; GPT-4-32k almost six. Making space for prompt, questions and replies, say one chapter for GPT-4, and five chapters for GPT-4-32k.

Can you imagine having a whole chapter in your working memory at once? Being simultaneously aware of every word, every space, every comma, every turn of phrase, every character and every plot line mentioned in it - and then being able to take it all into account when answering questions? Humans can do it for a paragraph, a stanza, maybe half a page. Not a whole chapter in a novel. Definitely not five. Not simultaneously at every level.

I feel in this sense, LLMs already surpassed our low-level capacity - though the comparison is a bit flawed, since our short-term memory also keeps tracks of sights, sounds, smells, time, etc. and emotions. My point here isn't really to compare who has more space for short-term recall - it's to point out that answering questions about immediately read text is another narrow, focused task which machines can now do better than us.

----

[0] - 298000 words in the book (via [1]), over 72 chapters (via [2]), gives us 4139 words per chapter. Multiplying by 4/3, we get 5519 tokens per chapter. GPT-4-8k can fit 1.45x that; GPT-4-32k can fit 5.8x that.

[1] - https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula...

[2] - https://awoiaf.westeros.org/index.php/Chapters_Table_of_cont...

Re: RWKV: Reinventing RNNs for the Transformer Era

#125

Post by TeMPOraL »

coppsilgold wrote: post #81
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

> every token embedding interacts with every other token embedding before it

> it takes a fraction of every other token embedding and adds it to itself.

> every token embedding mixing information from other embeddings into itself

Noting the use of the word every. Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the entire thing.

Re: RWKV: Reinventing RNNs for the Transformer Era

#126

Post by MuteXR »

teruakohatu wrote: post #14
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

Realistically, you could not quote a passage from the chapter directly, and you don't really need to anyway.

Summarizing chapters with the very same LLM and including that as context may very well get you far.

Re: RWKV: Reinventing RNNs for the Transformer Era

#127

Post by Orange1688 »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

I think what you're saying is warranted, but you should be very careful putting machine learning architectures in the context of the human mind. They are inherently different, although their output may be similar at some points. What you're saying addressed this directly.

Machine learning is optimizing mathematical weights and biases, and the human mind is messy wetware. One of the first things you will read in any machine learning textbook is to throw away the notion of AI trying to simulate the human mind.

Re: RWKV: Reinventing RNNs for the Transformer Era

#128

Post by tlb »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

People condense information from more than several thousand tokens ago into higher level chunks. So the whole first chapter of Infinite Jest becomes “child tennis prodigy is weird and annoying”.

It’s not clear how to learn this. Transformers work so well because there are only a few layers between input tokens and output. Adding a summarization step is a whole new research project.

Re: RWKV: Reinventing RNNs for the Transformer Era

#129

Post by grafzhl »

TeMPOraL wrote: post #125
coppsilgold wrote: post #81

Earlier quoted context omitted.

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

> every token embedding interacts with every other token embedding before it > it takes a fraction of every other token embedding and adds it to itself. > every token embedding mixing information from other embeddings into itself Noting the use of the word every . Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the…

While all other tokens are considered, the attention mechanism is putting an individual weight on each one, in a way "paying more attention" to some than others.

Re: RWKV: Reinventing RNNs for the Transformer Era

#130

Post by [deleted] »

TheFragenTaken wrote: post #109
teruakohatu wrote: post #14

Earlier quoted context omitted.

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

"The Magical Number Seven, Plus or Minus Two" [1] applies to many things. In this case, a book, could reasonably be reasoned about almost no matter the length as you argue, given the scope of a book is usually limited to a few topics (the French invasion of Russia). Similarly three to seven chapters could be retold, but not every 361 chapters. A "580,000" word long book about 58,000 things would be unfeasible for a h…

[deleted]
Post reply on HN