Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Eagle 7B: Soaring past Transformers

blog.rwkv.com

Post reply on HN
Print view View original

41–50 of 86 posts

Re: Eagle 7B: Soaring past Transformers

#41

Post by Y_Y »

samus wrote: post #33

Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?

English is an official language in Malta and about 70% of the population is classed as "advanced" in English[0]. That only adds up to 280k people though, you'd probably find as many native-level speakers in any big European country, though not at the same density.

[0] https://nso.gov.mt/wp-content/uploads/Skills-Preliminary.pdf

Re: Eagle 7B: Soaring past Transformers

#42

Post by samus »

ReptileMan wrote: post #39

Their map showing distribution of English speaking people is just terrible - I am fairly sure that there is at least one percent speaking English in India, Western Europe, Eastern Europe, Russia and China.

It must set a terribly high threshold (like language certificate holders or graduates of English-speaking schools) or actually report the percentage of native speakers. But one only has to be fluent enough to write chat messages to use a text model!

Re: Eagle 7B: Soaring past Transformers

#43

Post by jack_pp »

klipt wrote: post #37
mediaman wrote: post #36

Earlier quoted context omitted.

Knowledge transfer over generations is a function of the brain. Other species have much more limited ability to transfer knowledge intergenerationally, and that is because the human brain's capability for symbolic language is much more advanced than other animals', who are not able to encode knowledge nearly as efficiently.

The point is it's a function of many connected brains, not just one brain.

Sure but that's still only possible for the human brain, other species brains aren't capable of encoding knowledge and using that to collaborate with other members.

Re: Eagle 7B: Soaring past Transformers

#45

Post by zurfer »

Harrisonv wrote: post #25

Try rwkv-demo-api.recursal.ai if you want to try it and dont want to wait for gradio

Thank you! I'm not experienced with 7B models.

3 things stand out to me:

- it's absolutely not useable for the kind of use cases I solve with GPT-4 (code generation, information retrieval)

- it could technically swallow a 50 page PDF, but it's not able to answer questions about it (inference speed was good, but content was garbage)

- it is ok for chatting and translations (how is your day?)

Re: Eagle 7B: Soaring past Transformers

#46

Post by [deleted] »

zurfer wrote: post #45
Harrisonv wrote: post #25

Try rwkv-demo-api.recursal.ai if you want to try it and dont want to wait for gradio

Thank you! I'm not experienced with 7B models. 3 things stand out to me: - it's absolutely not useable for the kind of use cases I solve with GPT-4 (code generation, information retrieval) - it could technically swallow a 50 page PDF, but it's not able to answer questions about it (inference speed was good, but content was garbage) - it is ok for chatting and translations (how is your day?)

[deleted]

Re: Eagle 7B: Soaring past Transformers

#47

Post by jaster »

Szpadel wrote: post #15
coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

Here is a relevant tidbit from the RWKV paper "Limitations" section (https://arxiv.org/abs/2305.13048):

  First, the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information,it is mechanistically limited compared to full self-attention.

Re: Eagle 7B: Soaring past Transformers

#48

Post by pico_creator »

airspresso wrote: post #38
samus wrote: post #33

Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?

This caught my eye as well. I interpreted it as the map only shows countries that have English as primary language. But the article was not precise enough about this. I applaud them for focusing on multilingual performance though, as that is an important area of NLP which still has lots of room for improvement.

Yea, the map is poorly generated IMO as well - sadly there is only a few online tools online i could fine that "help me highlight, places where this list of languages is supported"

So if we want a better map, we might need to redo from scratch

That being said, depending on your sources, you can find multiple citations for 15-18% of the world population support english

With approximately 25% being native speaker (english as first language) and 75% as non-native

[PS: i realise later we were talking about different maps, the comment is for the 2nd map, for the languages we support]

Re: Eagle 7B: Soaring past Transformers

#49

Post by pico_creator »

samus wrote: post #40
airspresso wrote: post #38

Earlier quoted context omitted.

This caught my eye as well. I interpreted it as the map only shows countries that have English as primary language. But the article was not precise enough about this. I applaud them for focusing on multilingual performance though, as that is an important area of NLP which still has lots of room for improvement.

I highly welcome the effort as well*, but I don't see why they would have to mistake first-language ability for fluency to argue for that. The difference is vast and relevant: anyone with good English reading and writing skills can take advantage of a model and might prefer it over a worse model in their native language. *: Just sceptical whether there's enough content out there which isn't just (often badly or too s…

Yea, thats why I focused only on the top 25 languages, despite the model being trained for 100+ languages.

Was not confident, on the languages beyond the 25th cut-off, until we build better datasets (which we are in works on with various regional groups!)

Re: Eagle 7B: Soaring past Transformers

#50

Post by sebzim4500 »

wokwokwok wrote: post #27
Szpadel wrote: post #15

Earlier quoted context omitted.

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

One of three things has to be true. Either: a) this is false b) perfect recall is false (ie. as the internal state is overwritten, you lose information about previous entries in the context) c) the inference time scales by the context length. It’s not possible to have perfect recall over an arbitrary length in fixed time. Not hard. Totally not possible at all That would mean you can scan an infinite amount of data pe…

(b) is the sacrifice made in these linear attention type architectures.

As a mitigation, you can leave a few normal attention layers in the model but replace the rest.

Post reply on HN