Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

151–160 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#151

Post by stellaathena »

PoignardAzur wrote: post #147
chrgy wrote: post #115

Here is a summary of all comments by Transformers, wonder how RNN does: RWKV is a new language model architecture that is comparable to transformers in terms of performance. RWKV is more efficient than transformers, which makes it possible to train larger models on smaller datasets. The RWKV community is open source and welcomes contributions from anyone. There are plans to create larger versions of RWKV, but this wi…

> The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including: What? I…

It seems that the user is posting ChatGPT-generated text and not a real summary. It's complete nonsense, with about half of the sentences containing a factual error.

Re: RWKV: Reinventing RNNs for the Transformer Era

#152

Post by kaliqt »

TeMPOraL wrote: post #124
dcl wrote: post #75

Earlier quoted context omitted.

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

> Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. Some of them, some of the time. This is best comparable with ChatGPT having those books in its training dataset. The context window is more like short-term memory. GPT-4 can fit[0] ~1.5 chapters of Game of Thrones ; GPT-4-32k almost six . Ma…

If we are trying to at least match human level then all we have to do is summarize and store information for retrieval in the context window. Emphasis on summarize.

We take out key points explicitly so it's not summarized, and for the rest (less important parts) we summarize it and save it.

That would very likely fit and it would probably yield equal to or better recall and understanding than humans.

Re: RWKV: Reinventing RNNs for the Transformer Era

#153

Post by ftxbro »

pico_creator wrote: post #93
p-e-w wrote: post #84

The paper lists the first author's institutional affiliation as "RWKV Foundation". However, I cannot find anything about this supposed "foundation" online, and as far as I can tell, the term RWKV originates in this very paper. What's going on here?

It does not exists (yet maybe) Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc) So he had to fill something up i guess haha The idea of a foundation has been tossed around. So it may eventually happen Also the RWKV project predates this paper by over 2 years - nothing is truely new and revealed today on this paper Your simply just finding out about it now in a formal p…

> Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc) So he had to fill something up i guess haha

I saw there is no one listed in the author list as independent researcher or no affiliation or whatever. It seems hard to believe, if it's a lot of random contributors on a discord channel. Did they all have to make up some kind of 'DBA' (doing business as) name?

Re: RWKV: Reinventing RNNs for the Transformer Era

#154

Post by ofermend »

pico_creator wrote: post #47
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

One of the biggest issue with incredibly large context sizes is the lack of training / dataset meant specifically to use such large context sizes. And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue

Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.

Re: RWKV: Reinventing RNNs for the Transformer Era

#155

Post by p-e-w »

fauigerzigerk wrote: post #140
p-e-w wrote: post #98

Earlier quoted context omitted.

Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

I think you're confusing cognitive processes with neurological processes. Of course we can reflect on actual cognitive processes such as forgetting or learning, and of course psychologists can gather evidence on these cognitive processes. What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.

> Of course we can reflect on actual cognitive processes such as forgetting or learning

Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations?

"It's obvious" doesn't count as evidence.

Re: RWKV: Reinventing RNNs for the Transformer Era

#156

Post by pico_creator »

ftxbro wrote: post #153
pico_creator wrote: post #93

Earlier quoted context omitted.

It does not exists (yet maybe) Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc) So he had to fill something up i guess haha The idea of a foundation has been tossed around. So it may eventually happen Also the RWKV project predates this paper by over 2 years - nothing is truely new and revealed today on this paper Your simply just finding out about it now in a formal p…

> Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc) So he had to fill something up i guess haha I saw there is no one listed in the author list as independent researcher or no affiliation or whatever. It seems hard to believe, if it's a lot of random contributors on a discord channel. Did they all have to make up some kind of 'DBA' (doing business as) name?

I would not really call it random. While it was open for feedback/contributions.

There is a strong requirements for substantial contribution to the paper itself to qualify for authorship.

So unfortunately that does limit it in part to folks who are more familiar in writing such papers, have the resources for doing benchmarks, and charting, etc

And less so for folks who for eg, tinkers, contributed to dataset, porting the implementation to X lang, doing LoRa, or various other experiments that has nothing to do with the core architecture

Re: RWKV: Reinventing RNNs for the Transformer Era

#157

Post by spi »

coppsilgold wrote: post #81
andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

I'm pretty sure in GPT3.5+ models, this concept of attention holds no longer true. In [1] the author suggests they are using "Intra-tensor sparsity" from a 2021 paper from Google [2].

Details aside, math suggests they _must_ be using some sparse attention method: the memory used by attention is O(l^2 * d) (here O notation is a bit of overkill: it's exactly that number in 8 bit quantization or twice that in 16 bit floats). With l=32k, and d probably in the order of a few k (even "old" BERT model had it at 768, it only went up since), that would be of the order of a few TB. For (a piece of) _a single_ layer, they probably have dozens, if not hundreds, of it. The largest GPUs in commerce have 80GB of memory; there's no way they are really using that many GPUs for each single layer (even leaving aside the fact that runtime would probably be absolutely horrible).

Implicitly or explicitly, the transformer must learn to have both a "summarized" version of attention (that tells it where to focus on) and more "detailed" ones. It doesn't need to be specifically coded, if using sparse attention, the model might for example get different levels of attentions at different layers, somehow, but they probably have coded something a bit smarter than that.

[1] https://kir-gadjello.github.io/posts/gpt4-some-technical-hyp... [2] https://arxiv.org/abs/2111.12763

Re: RWKV: Reinventing RNNs for the Transformer Era

#158

Post by sundarurfriend »

candiodari wrote: post #141
pico_creator wrote: post #101

Earlier quoted context omitted.

Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

Naming is a perpetual problem. My issue is with "hallucination", which everyone takes to be the "problem" with GPT style networks making things up. Never mind that transformers are just trying to predict the next likely token, NOT the truth PLUS that they're trained from the internet. As everyone knows, the internet is not known for correctness and truth. If you want any neural network to figure out the truth indepen…

> My issue is with "hallucination", which everyone takes to be the "problem" with GPT style networks making things up.

It's most famous as a problem with ChatGPT specifically, which is presented as a chat interface where an agent answers your question. In that context, it makes sense to think of confident and detailed wrong answers as hallucinations.

You could say that it's still an LLM underneath, but then you'd be talking about a layer of abstraction beneath what most people interact with. Given that they tap into the mental model of "chat with an agent" heavily with their interface and interactions, having small disclaimers and saying "I'm just an LLM" from time to time aren't sufficient to counter people's intuitions and expectations.

Re: RWKV: Reinventing RNNs for the Transformer Era

#159

Post by mike_hearn »

ofermend wrote: post #154
pico_creator wrote: post #47

Earlier quoted context omitted.

One of the biggest issue with incredibly large context sizes is the lack of training / dataset meant specifically to use such large context sizes. And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue

Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.

It feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.

Re: RWKV: Reinventing RNNs for the Transformer Era

#160

Post by ftxbro »

pico_creator wrote: post #156
ftxbro wrote: post #153

Earlier quoted context omitted.

> Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc) So he had to fill something up i guess haha I saw there is no one listed in the author list as independent researcher or no affiliation or whatever. It seems hard to believe, if it's a lot of random contributors on a discord channel. Did they all have to make up some kind of 'DBA' (doing business as) name?

I would not really call it random. While it was open for feedback/contributions. There is a strong requirements for substantial contribution to the paper itself to qualify for authorship. So unfortunately that does limit it in part to folks who are more familiar in writing such papers, have the resources for doing benchmarks, and charting, etc And less so for folks who for eg, tinkers, contributed to dataset, porting…

Sorry I didn't mean to say "random" in the sense of stochastic, more like "eclectic". Usually when I saw such papers with eclectic author mix, at least a few are listed as independent researchers or unaffiliated (even though they might have a PhD or are applying to grad schools or whatever). I was wondering maybe I'm out of date and everyone in those circumstances now just has a 'doing research as' corporate alias.
Post reply on HN