51–60 of 86 posts
Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?
So its not "as strict"
Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?
Thanks for flagging that out, swapped it out with another map I found, that shows different colors for different thresholds So its not "as strict"
(note, approximate date) Hyped about this! This could strike a powerful balance between performance and reasonably retained low environmental/token cost impact. Would be cool with improved coverage of Scandinavian languages along with it, but I guess we'll see.
And yeah, I think a true revolution will happen (or might already be) when we realize the value of training data and how to structure and balance its content in the most optimal way for training.
Has anyone quantified that specifically? I'd love to read more details since I'd expect the concepts to start mapping between languages at some point. I.e. with enough language fluency I'd expect learning knowledge/reasoning in language to improve the result in another. But I can't find any paper talking about specifically about that.
Earlier quoted context omitted.
If later input overwrites previous input in the internal state, it means the model does have a limit to how much input it can "remember" at any given time and that limit is less than infinite.
You can think of it like your own memory. Can you remember a very important thing from 10 years ago? Can you remember every single thing since then? Some things will remain for basically infinite period, some will have a more limited scope.
It is pretty well established that very few people are able to remember details of things for any reasonable period of time. The way that we keep those memories is by recalling them and playing the events over again in our mind. This 'refreshes' them, but at the expense of 'corrupting' them. It is almost certain that things important to you that you are sure you remember correctly are wrong on many details -- you have at times gotten a bit hazy on some aspect, tried to recall it 'figured it out' and stored that as your original memory without knowing it.
To me, 'concepts', like doing math or riding a bike, on the other hand, are different in the sense that you don't know how to ride a bike, as in you couldn't explain the muscle movements needed to balance and move on a bicycle, but when you get on it, you go through the process of figuring out the process again. So even though you 'never forget how to ride a bike' you never really knew how to do it, you just got good at learning how to do it incredibly quickly every time you tried.
Can you correct me on any misconceptions I may have about either how I think memories work, or how my thoughts should coincide with how these models work?
> A common feedback we receive for the RWKV multi-lingual approach is: it hurts our English evaluation scores (...) Has anyone quantified that specifically? I'd love to read more details since I'd expect the concepts to start mapping between languages at some point. I.e. with enough language fluency I'd expect learning knowledge/reasoning in language to improve the result in another. But I can't find any paper talkin…
The interesting take away for me was that training rwkv from zero to intelligible sentences for minority language was more faster (in units of tokens trained!) than other architectures, making it more accessible for cases where large corpus like the Pile don't or can't exist.
This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…
Data has the problem of getting successively tainted by LLMs as well as the lack of open high-quality datasets.
While architecture has the problem of shifting too much focus on a flawed architecture - transformers.
Try rwkv-demo-api.recursal.ai if you want to try it and dont want to wait for gradio
Thank you! I'm not experienced with 7B models. 3 things stand out to me: - it's absolutely not useable for the kind of use cases I solve with GPT-4 (code generation, information retrieval) - it could technically swallow a 50 page PDF, but it's not able to answer questions about it (inference speed was good, but content was garbage) - it is ok for chatting and translations (how is your day?)
From my albeit quick tests, what I found is that it knows clearly less things than mistral, it hallucinates much more, it does not follow instructions, has less reasoning capabilities and asking it to translate a Japanese text into English gave me a bad translated summary instead of the full translation.
I don't see how this is soaring past transformers when clearly it's unable to do any of the useful tasks you can use a transformer model for today...
Earlier quoted context omitted.
This caught my eye as well. I interpreted it as the map only shows countries that have English as primary language. But the article was not precise enough about this. I applaud them for focusing on multilingual performance though, as that is an important area of NLP which still has lots of room for improvement.
I highly welcome the effort as well*, but I don't see why they would have to mistake first-language ability for fluency to argue for that. The difference is vast and relevant: anyone with good English reading and writing skills can take advantage of a model and might prefer it over a worse model in their native language. *: Just sceptical whether there's enough content out there which isn't just (often badly or too s…
And that is, I presume - I could be wrong -, before anyone has tried to really mine the Norwegian national library, as even much of what is online is not easily accessible for crawling.
I think there'll be plenty of content for even much smaller languages - especially anywhere with depositary laws -, but it's often going to require cumbersome collection efforts and negotiating access.
Earlier quoted context omitted.
You can think of it like your own memory. Can you remember a very important thing from 10 years ago? Can you remember every single thing since then? Some things will remain for basically infinite period, some will have a more limited scope.
I'm not sure I understand your concept of human memory. It is pretty well established that very few people are able to remember details of things for any reasonable period of time. The way that we keep those memories is by recalling them and playing the events over again in our mind. This 'refreshes' them, but at the expense of 'corrupting' them. It is almost certain that things important to you that you are sure you…