91–100 of 176 posts
Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.
It's Type 10 of https://xkcd.com/2456/ .
Earlier quoted context omitted.
This model seem to be good when input context is large in comparison to OpenAI, would just wait till if someone productize it. Most likely it wine be just one paper but many new and old ideas combined
Being a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.
The paper lists the first author's institutional affiliation as "RWKV Foundation". However, I cannot find anything about this supposed "foundation" online, and as far as I can tell, the term RWKV originates in this very paper. What's going on here?
Blink is an individual and does not represent a company (aka not google, not eleuther, not , etc)
So he had to fill something up i guess haha
The idea of a foundation has been tossed around. So it may eventually happen
Also the RWKV project predates this paper by over 2 years - nothing is truely new and revealed today on this paper
Your simply just finding out about it now in a formal paper consolidating various things that was learnt in this project
Earlier quoted context omitted.
Currently, what I'm seeing with RWKV is that attention fades of quickly. The model will start to produce output, but very quickly (a few dozen tokens), its own output tokens are suddenly taking 'precedence' over the input question and it starts to simply repeat itself. For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format.…
Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)
I'll say "get a list of Blah from the following document in Json format like this:
Example"
Then I feed the document and add a spot for the answer.
The model begins correctly. But usually in the middle of the Json list generation, it will veer off, and start hallucinating as if it forgot the document and the task. I'm happy to share specifics and datasets but this is a cross cutting problem.
Rwkv is able to answer my questions when I ask simple yes/no or classification. It's the listing that throws it for a loop. Transformers do not have the same problem. Both llama and gpt are able to maintain focus.
Also, do you know where I'd find information on how the current weights were trained?
Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference”
It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.
Earlier quoted context omitted.
> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens…
Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps wit…
Earlier quoted context omitted.
Are there any estimates anywhere of how many A100s would be needed to e.g. train a 30B model in 6 months?
That’s a loaded question without deciding dataset size
Earlier quoted context omitted.
> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no…
While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.
Earlier quoted context omitted.
Currently, what I'm seeing with RWKV is that attention fades of quickly. The model will start to produce output, but very quickly (a few dozen tokens), its own output tokens are suddenly taking 'precedence' over the input question and it starts to simply repeat itself. For example, I'm currently attempting to use RWKV for named entity extraction. I ask it to analyze a piece of text and provide output in JSON format.…
Rearrange the query. Ask the question / explain the task first. Then give it the data you want to extract from. Also you may want to give a one shot example for best result (the instruct training of raven model is very limited)
Earlier quoted context omitted.
That’s a loaded question without deciding dataset size
can you elaborate on the chinchilla law / dataset problem a bit? (perhaps by editing your previous comment?) what datasets are available to the community, how big are these, are they needed to be updated from time to time, where are these stored, what are the usual cost ranges involved, ...? :o thank you!
If not, you are getting diminishing benefits for each param you add
I’m extreme cases your model can even perform worse with more param due to lack of training data
More complicated: the quality of the data matters as well
So there are 2 major directions. Build efficient models with good dataset and optimal param count for the task
Or go big on everything (aka openAI) which requires monster GPU time for every reply token
There are obviously in between as well. Hence why the question is so loaded
Ballpark: if your not setting aside a 100k for GPUs alone, to train a 60B model from scratch, your probably not ready to train one