Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

11–20 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#11

Post by foota »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The primary benefit of transformers is that they generally can reference things within the context, but RNNs have trouble with that. The trade off is that transformers have memory and compute requirements quadratic in the context length.

Re: RWKV: Reinventing RNNs for the Transformer Era

#14

Post by teruakohatu »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases

War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens long.

Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.

Re: RWKV: Reinventing RNNs for the Transformer Era

#15

Post by Buttons840 »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

To answer your question: https://news.ycombinator.com/item?id=35904773

> we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds.

Re: RWKV: Reinventing RNNs for the Transformer Era

#16

Post by noduerme »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

I know a guy who memorized "Fahrenheit 451" back in the '60s, and went on to memorize a dozen other books - I think it was a requirement of the anti-establishment cult he was in at the time. Anyhow, his recall is still pretty fantastic.

Re: RWKV: Reinventing RNNs for the Transformer Era

#19

Post by Buttons840 »

ftxbro wrote: post #9
bigdict wrote: post #7

Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.

I think it's just, everyone in their discord channel.

What channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.
Post reply on HN