RWKV: Reinventing RNNs for the Transformer Era • Hacker News Board

Re: RWKV: Reinventing RNNs for the Transformer Era

#171

Post by nathan_compton » Tue, May 23, 2023, 7:34 PM UTC

p-e-w wrote: post #169

nathan_compton wrote: post #166

Earlier quoted context omitted.

"Simply put, we have no idea how the human mind works." ~40,000 neuroscientists beg to differ. I'd argue that, more or less, we have a good, general, working theory about how the mind works.

"What I cannot create, I do not understand." - Richard Feynman LLMs notwithstanding, I have yet to see a replication of the human mind created by humans. Therefore, I would take any claim that neuroscience 'understands' how the human mind works (a claim that I've never heard from an actual neuroscientist) with a huge amount of salt.

Your original comment was that we have "no idea" how the brain works. This is absurd. We understand quite a lot about how the brain works. We know its functional elements (neurons, possibly glial cells) its basic mechanism of information processing (firing rates, spike timing) and we have a very general model of how those things are combined to actually process information (mutually inhibiting cortical columns, a laminar structure which represents (roughly) sequential stages of processing) etc etc etc. This hardly constitutes "no idea," and, indeed, the fact that artificial neural networks recapitulate in general terms the ability of neurons to process information, suggests that by the Feynman criteria (he said a lot of dumb stuff, in my opinion, by the way) we are beyond the stage of "[having] no idea."

Not that it matters, but for various reasons I have spent quite a lot of time around neuroscientists and I think most of them would agree that "no idea" is a pretty ridiculous way to characterize the state of the field.

Re: RWKV: Reinventing RNNs for the Transformer Era

#172

Post by nathan_compton » Tue, May 23, 2023, 7:36 PM UTC

heja2009 wrote: post #107

p-e-w wrote: post #98

Earlier quoted context omitted.

Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

To the contrary there is very good evidence that we pull arbitrary explanations whenever needed. See Gazzaniga's split brain experiments where people gave all sorts of reasons when asked why they did something and just could not know.

The fact that in extreme circumstances we pull explanations out of our asses (or even that we do so in non-extreme circumstances) doesn't mean we are incapable of genuine meta-cognition. No one is rational most of the times, but mathematicians can be rational for brief periods on limited subjects.

Re: RWKV: Reinventing RNNs for the Transformer Era

#173

Post by anon291 » Wed, May 24, 2023, 1:47 AM UTC

pico_creator wrote: post #106

anon291 wrote: post #94

Earlier quoted context omitted.

Yeah... So I did that which is how I got it to begin correctly. This is what I mean though. I'll say "get a list of Blah from the following document in Json format like this: Example" Then I feed the document and add a spot for the answer. The model begins correctly. But usually in the middle of the Json list generation, it will veer off, and start hallucinating as if it forgot the document and the task. I'm happy to…

Hmm we might need to look into the instruct training data. Which is mostly based on gpt4all filtered and mixed with others (You are using raven right? That’s the instruct trained varient) Btw ping the discord if ur looking into finetuning for your usecase

Yeah I'm using raven. Raven does work better. And I'm on the discord.

Unfortunately I really would like machine readable responses and raven is a bit too verbose.

Looking at fine-tuning right now.

Re: RWKV: Reinventing RNNs for the Transformer Era

#174

Post by noduerme » Wed, May 24, 2023, 3:28 AM UTC

TeMPOraL wrote: post #124

dcl wrote: post #75

Earlier quoted context omitted.

Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…

> Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. Some of them, some of the time. This is best comparable with ChatGPT having those books in its training dataset. The context window is more like short-term memory. GPT-4 can fit[0] ~1.5 chapters of Game of Thrones ; GPT-4-32k almost six . Ma…

Just thinking about this, I realized that as a musician I do it all the time. I can recall lyrics, chords, instrumental parts and phrasing to hundreds if not thousands of pieces of music and "play them back" in my head. Unlike a training set, though, I can usually do that after listening to a piece only a few times, and also recall what I thought of each part of each piece, and how I preferred to treat each note or phrase each time I played it, which gives me more of a catalog of possible phrasings the next time I perform it. This is much easier for me than remembering exact words I've read in prose. I suspect the relationships between all those different dimensions is what makes the memory more durable. I must also be creating intermediary dimensions and vectors to do that processing, because one side effect of it is that I associate colors with pitches.

Re: RWKV: Reinventing RNNs for the Transformer Era

#175

Post by pico_creator » Wed, May 24, 2023, 3:59 AM UTC

ftxbro wrote: post #160

pico_creator wrote: post #156

Earlier quoted context omitted.

I would not really call it random. While it was open for feedback/contributions. There is a strong requirements for substantial contribution to the paper itself to qualify for authorship. So unfortunately that does limit it in part to folks who are more familiar in writing such papers, have the resources for doing benchmarks, and charting, etc And less so for folks who for eg, tinkers, contributed to dataset, porting…

Sorry I didn't mean to say "random" in the sense of stochastic, more like "eclectic". Usually when I saw such papers with eclectic author mix, at least a few are listed as independent researchers or unaffiliated (even though they might have a PhD or are applying to grad schools or whatever). I was wondering maybe I'm out of date and everyone in those circumstances now just has a 'doing research as' corporate alias.

Ahh. As someone who has been listed as “independent researcher” before. I get what you mean - oh well, it is what it is

Re: RWKV: Reinventing RNNs for the Transformer Era

#176

Post by RC_ITR » Thu, May 25, 2023, 4:48 PM UTC

matteogrella wrote: post #148

RC_ITR wrote: post #95

> We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference” It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.

As one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).

How can something recurrent be parallelized?

> the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined.

And sure, this is the core concept of self attention, no?