Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

131–140 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#131

Post by int_19h »

pico_creator wrote: post #64
logicchains wrote: post #60

Earlier quoted context omitted.

Are there currently any plans to create a RWKV 30B or 65B? That seems to be the size at which the LLaMA transformer models become genuinely competitive with GPT3.5 for many tasks.

TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…

30B would be interesting because that's the practical ceiling for local GPUs assuming 4-bit quantization.

Is there some kind of dedicated fund for training hardware? Donating an A100 sounds unlikely, but surely they could be crowdfunded?

Re: RWKV: Reinventing RNNs for the Transformer Era

#133

Post by xiphias2 »

pico_creator wrote: post #101
coppsilgold wrote: post #81

Earlier quoted context omitted.

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

I believe the naming is perfect: the word attention clearly maches the intention of the model structure. It’s still clearly missing more efficiency, but we had to have ChatGPT / ChatGPT4 working and used to make the further research directions clear (increasing context length and decreasing hallucinations).

Re: RWKV: Reinventing RNNs for the Transformer Era

#134

Post by logicchains »

pico_creator wrote: post #118
logicchains wrote: post #108

Earlier quoted context omitted.

Yep that's the one. Curious roughly how many A100s it'd take to train a 65B RWKV on that.

Really bad napkin math as no one has attempted 65B (so +\- 50%) 8 x 8 x 8 A100, should be able to do a 100k++ tokens/s at that size With a dataset of 1.2 trillion tokens. That’s 12 million seconds. Or 140 days (PS: this is why everyone is training <60B, its crazy the cost, even if my math estimate is wrong by 300%, its still a crazy number)

Thank you! 888 is 512 A100s, that is indeed pretty expensive.

Re: RWKV: Reinventing RNNs for the Transformer Era

#135

Post by pico_creator »

xiphias2 wrote: post #133
pico_creator wrote: post #101

Earlier quoted context omitted.

Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative

I believe the naming is perfect: the word attention clearly maches the intention of the model structure. It’s still clearly missing more efficiency, but we had to have ChatGPT / ChatGPT4 working and used to make the further research directions clear (increasing context length and decreasing hallucinations).

Haha, yea - naming things is hard

Every-time someone comes up and say "this is not attention, because it does X and not Y"

My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.

Re: RWKV: Reinventing RNNs for the Transformer Era

#136

Post by pico_creator »

int_19h wrote: post #131
pico_creator wrote: post #64

Earlier quoted context omitted.

TLDR: please donate A100s to make this happen Most of the focus is in the 1-14B range. Due to constraints of the dataset sizes (chinchilla law), and GPUs available Community demand is also mostly in this range as there is a strong desire to optimise and run on local GPU. So more focus is in this range. Not representing blink directly here - but if anyone wants to see a 30B / 65B model. Reach out to contribute the GPU…

30B would be interesting because that's the practical ceiling for local GPUs assuming 4-bit quantization. Is there some kind of dedicated fund for training hardware? Donating an A100 sounds unlikely, but surely they could be crowdfunded?

weirdly enough, organisations are more willing to rent GPUs than money.

If you want to help fund RWKV, the ko-fi link is - https://ko-fi.com/rwkv_lm

IMO: this needs way more funding, just to sustain blink leading this project, let alone GPUs for training.

(Also - current tests shows this model doing really badly with 4bit quantized, but alright at Q5 and Q8)

Re: RWKV: Reinventing RNNs for the Transformer Era

#137

Post by pico_creator »

zwaps wrote: post #116

I wish this was written with more care. None of the symbols are defined. Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single person with CNN background on the team and it makes zero sense. What if you actually have channels in your data? What then?

If you have more specific feedback, like a specific digram or page, and how it can be made better. I will gladly forward that info, to improve the paper draft.

Because channel mixing, is a core component of this architecture, and that keyword "channel"is all over the place. I have no idea what is it you are critiquing specifically (i could not find the mention of "channel dimension" in the paper)

Re: RWKV: Reinventing RNNs for the Transformer Era

#138

Post by pico_creator »

grafzhl wrote: post #129
TeMPOraL wrote: post #125

Earlier quoted context omitted.

> every token embedding interacts with every other token embedding before it > it takes a fraction of every other token embedding and adds it to itself. > every token embedding mixing information from other embeddings into itself Noting the use of the word every . Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the…

While all other tokens are considered, the attention mechanism is putting an individual weight on each one, in a way "paying more attention" to some than others.

Yup, if you follow this definition of attention. It makes sense.

The mixing step, is computed with the trained weights, meaning the model does learn on its own, when/what to emphasise in this "mixing" process.

Hypothetically speaking, if a token does not matter, it mixing weights could end up being literally 0 (or something close to it)

Re: RWKV: Reinventing RNNs for the Transformer Era

#139

Post by pico_creator »

zh217 wrote: post #132
ftxbro wrote: post #12

> "Our experiments reveal that RWKV performs on par with similarly sized Transformers"

This sounds pretty bad, right? Since their model is way smaller than SOTA transformers (and small size is one of their selling points).

The paper is meant to compare architecture vs architecture with similar model size, and dataset - to inform decisions in future architecture designs

Its main benefits being presented with the above staying closely the same, is that it has significantly lower running and training cost without performance penalty

If you want to compare any <20B model with GPT 3.5 / 4 / 100B models evals, thats another paper altogether

Re: RWKV: Reinventing RNNs for the Transformer Era

#140

Post by fauigerzigerk »

p-e-w wrote: post #98
anon373839 wrote: post #90

Earlier quoted context omitted.

While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.

Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

I think you're confusing cognitive processes with neurological processes. Of course we can reflect on actual cognitive processes such as forgetting or learning, and of course psychologists can gather evidence on these cognitive processes.

What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.

Post reply on HN