Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

161–170 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#161

Post by fauigerzigerk »

p-e-w wrote: post #155
fauigerzigerk wrote: post #140

Earlier quoted context omitted.

I think you're confusing cognitive processes with neurological processes. Of course we can reflect on actual cognitive processes such as forgetting or learning, and of course psychologists can gather evidence on these cognitive processes. What we cannot do is make definitive claims about neurological processes and structures based on what we know about our cognitive processes.

> Of course we can reflect on actual cognitive processes such as forgetting or learning Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations? "It's obvious" doesn't count as evidence.

As I said, psychologists have been collecting actual evidence on cognitive processes such as forgetting and learning for a long time.

These studies confirm our perception that we can forget stuff.

Re: RWKV: Reinventing RNNs for the Transformer Era

#163

Post by p-e-w »

fauigerzigerk wrote: post #161
p-e-w wrote: post #155

Earlier quoted context omitted.

> Of course we can reflect on actual cognitive processes such as forgetting or learning Is there any actual evidence that our so-called reflections on our own thinking are anything more than hallucinations? "It's obvious" doesn't count as evidence.

As I said, psychologists have been collecting actual evidence on cognitive processes such as forgetting and learning for a long time. These studies confirm our perception that we can forget stuff.

"We can forget stuff" and "I believe I just forgot that" are very different things.

One is a general statement of fact, the other implies introspection of one's own individual cognitive processes. I have yet to see evidence that the latter is actually possible.

Re: RWKV: Reinventing RNNs for the Transformer Era

#164

Post by canjobear »

TeMPOraL wrote: post #125
coppsilgold wrote: post #81

Earlier quoted context omitted.

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

> every token embedding interacts with every other token embedding before it > it takes a fraction of every other token embedding and adds it to itself. > every token embedding mixing information from other embeddings into itself Noting the use of the word every . Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the…

Most LLM architectures use “soft attention” where some fractional amount of attention is put on every token. “Hard attention” is the term for what you describe.

Re: RWKV: Reinventing RNNs for the Transformer Era

#165

Post by stevetesla »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

Hey side question here, are there any papers or anything really I can read on the part about cheating by using a lookup that you mentioned?

Re: RWKV: Reinventing RNNs for the Transformer Era

#166

Post by nathan_compton »

p-e-w wrote: post #80
mlsu wrote: post #54

Earlier quoted context omitted.

These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you. These things are very diff…

> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no…

"Simply put, we have no idea how the human mind works." ~40,000 neuroscientists beg to differ. I'd argue that, more or less, we have a good, general, working theory about how the mind works.

Re: RWKV: Reinventing RNNs for the Transformer Era

#167

Post by coppsilgold »

TeMPOraL wrote: post #125
coppsilgold wrote: post #81

Earlier quoted context omitted.

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You c…

> every token embedding interacts with every other token embedding before it > it takes a fraction of every other token embedding and adds it to itself. > every token embedding mixing information from other embeddings into itself Noting the use of the word every . Phrased this way, calling it "attention" hardly makes sense, as attention is typically focused on something specific at any given time - not always on the…

Softmax takes a vector of arbitrary values and converts it into probabilities such that all the elements of the vector add to 1. Transformers use this to decide on the fractions.

This could be seen as "soft attention" where in theory there would be few winners for every "attention head".

It's also possible that the only purpose the softmax actually serves (or at least major component of) is that of normalization. Without it, the variance in the internal network dynamics between training samples would be fairly large (some training samples may have tokens that interact heavily, while others not). Making optimization problematic.

Re: RWKV: Reinventing RNNs for the Transformer Era

#168

Post by fauigerzigerk »

p-e-w wrote: post #163
fauigerzigerk wrote: post #161

Earlier quoted context omitted.

As I said, psychologists have been collecting actual evidence on cognitive processes such as forgetting and learning for a long time. These studies confirm our perception that we can forget stuff.

"We can forget stuff" and "I believe I just forgot that" are very different things. One is a general statement of fact, the other implies introspection of one's own individual cognitive processes. I have yet to see evidence that the latter is actually possible.

They are different things, but there is an entire scientific field that connects the two.

The phenomenon of people forgetting stuff is not subjective. It's not one person in isolation thinking they just forgot something. Forgetting is a phenomenon on which there is a lot of social feedback and repeatable experiments.

You seem to be denying the possibility of ever connecting science back to individual perception. Denying this makes any and all science completely meaningless though.

It's not limited to observations about ourselves. You could ask the question "but aren't you hallucinating?" about absolutely everything required to verify the outcome of a physics experiment for instance.

Re: RWKV: Reinventing RNNs for the Transformer Era

#169

Post by p-e-w »

nathan_compton wrote: post #166
p-e-w wrote: post #80

Earlier quoted context omitted.

> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no…

"Simply put, we have no idea how the human mind works." ~40,000 neuroscientists beg to differ. I'd argue that, more or less, we have a good, general, working theory about how the mind works.

"What I cannot create, I do not understand." - Richard Feynman

LLMs notwithstanding, I have yet to see a replication of the human mind created by humans. Therefore, I would take any claim that neuroscience 'understands' how the human mind works (a claim that I've never heard from an actual neuroscientist) with a huge amount of salt.

Re: RWKV: Reinventing RNNs for the Transformer Era

#170

Post by bee_rider »

stellaathena wrote: post #39
bee_rider wrote: post #5

Dumb arxiv question (sorry); is it possible to see what journal a paper was actually submitted to, to help find the reviewed version when it comes out?

The paper is going to be submitted to EMNLP next month. An early version is being released now to garner feedback and improve the paper before submission.

Thanks!
Post reply on HN