Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

81–90 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#81

Post by coppsilgold »

andy_xor_andrew wrote: post #4

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…

The word "attention" has been stretched pretty far to explain what is happening inside a transformer.

What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding.

You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel ("attention heads"), ~100 times in sequence (layers). As per GPT-3 model (175B).

Re: RWKV: Reinventing RNNs for the Transformer Era

#82

Post by cypress66 »

anon291 wrote: post #76
noduerme wrote: post #24

I wrote a home-brew neural network around 2006, just to see what would happen. I'd read no papers on it, and just kind of made it up as I went along. The result was basically a cube of "neurons" which had stronger and weaker trigger points to their neighbors and would propagate "spark" to one or more neighbors based on the strength and direction of spark they got from their other neighbors. Each of the connections an…

> But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better? Maybe my concept of the principle behind modern LLMs is too archaic or rooted in a cartoon understanding of our own wetware that I tried to reproduce. Realastically... the problem is a data problem. The math is mostly there. Transformers et al would have bee…

> Transformers et al would have been figured out in no time had we had the sort of data tools we have today.

I disagree. Even things that seem obvious in retrospect, take some time to be figured out.

Resnets, batch norm, dropout, etc are examples of this.

And I don't think transformers are obvious.

Re: RWKV: Reinventing RNNs for the Transformer Era

#83

Post by quickthrower2 »

pico_creator wrote: post #66
quickthrower2 wrote: post #63

My goal this year: to understand what this is about :-)

If you are familiar with how transformer network works There is RWKV in 150 lines to help understand all the nitty gritty https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_li...

Almost familiar: one more Karpathy lecture to go. Thanks!

Re: RWKV: Reinventing RNNs for the Transformer Era

#84

Post by p-e-w »

The paper lists the first author's institutional affiliation as "RWKV Foundation". However, I cannot find anything about this supposed "foundation" online, and as far as I can tell, the term RWKV originates in this very paper.

What's going on here?

Re: RWKV: Reinventing RNNs for the Transformer Era

#85

Post by zamnos »

dcl wrote: post #75
teruakohatu wrote: post #14

Earlier quoted context omitted.

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

ASIOF spoiler below!

> At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret.

Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.

Re: RWKV: Reinventing RNNs for the Transformer Era

#86

Post by soulofmischief »

pico_creator wrote: post #55
chaxor wrote: post #43

Earlier quoted context omitted.

I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN). Hopefull…

Yea using the analogy of the eyes can see the entire document Vs I need to memorise everything as it’s spoken out, and then answer on it Is a good approximate on the difference. The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training) It’s also why it is recommended by the community to ask the question first, then…

In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious.

For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory.

And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations.

And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.

Re: RWKV: Reinventing RNNs for the Transformer Era

#87

Post by pico_creator »

redox99 wrote: post #68
pico_creator wrote: post #38

Earlier quoted context omitted.

Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not traini…

> it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board To be fair, that filters the majority of models in the scoreboard.

Dun we all wish this wasn’t the case?

Where we have more OSS models to choose from without weird rule lawyering gotchas. Or needing to be from a research institute / a license to download the weights

Re: RWKV: Reinventing RNNs for the Transformer Era

#89

Post by pico_creator »

anon291 wrote: post #78
pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

Currently, what I'm seeing with RWKV is that attention fades of quickly. The model will start to produce output, but very quickly (a few dozen tokens), its own output tokens are suddenly taking 'precedence' over the input question and it starts to simply repeat itself. For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format.…

Rearrange the query.

Ask the question / explain the task first. Then give it the data you want to extract from.

Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)

Re: RWKV: Reinventing RNNs for the Transformer Era

#90

Post by anon373839 »

p-e-w wrote: post #80
mlsu wrote: post #54

Earlier quoted context omitted.

These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you. These things are very diff…

> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no…

While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.
Post reply on HN