Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Eagle 7B: Soaring past Transformers

blog.rwkv.com

Post reply on HN
Print view View original

31–40 of 86 posts

Re: Eagle 7B: Soaring past Transformers

#31

Post by pico_creator »

127361 wrote: post #22

They've joined the Linux Foundation, does that mean the models are going to be eventually censored to satisfy the foundation's AI safety policies? That includes ensuring the models don't generate content that's non-inclusive or against diversity policies?

Currently the main policy is only around copyright - and nothing about AI safety: https://www.linuxfoundation.org/legal/generative-ai

Also in the full power of opensource, if LF really force something the group disagree with, we will just fork

All the other alignment policies, are optional for groups to opt-in

So I would not worry so much about that - the group already has a plan in event we need to leave the Linux Foundation - for example: If USA regulates AI training (since LF is registered under USA)

Re: Eagle 7B: Soaring past Transformers

#32

Post by Al-Khwarizmi »

lhl wrote: post #29
127361 wrote: post #22

They've joined the Linux Foundation, does that mean the models are going to be eventually censored to satisfy the foundation's AI safety policies? That includes ensuring the models don't generate content that's non-inclusive or against diversity policies?

It is trivial to fine tune any model (whether a base model or an aligned model) to your preferred output preferences as long as you have access to the model weights .

Not trivial for the general public at all, and furthermore, you need much more memory for finetuning than for inference, often making it infeasible for many machine/model combinations.

Re: Eagle 7B: Soaring past Transformers

#33

Post by samus »

Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?

Re: Eagle 7B: Soaring past Transformers

#34

Post by Harrisonv »

wokwokwok wrote: post #27
Szpadel wrote: post #15

Earlier quoted context omitted.

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

One of three things has to be true. Either: a) this is false b) perfect recall is false (ie. as the internal state is overwritten, you lose information about previous entries in the context) c) the inference time scales by the context length. It’s not possible to have perfect recall over an arbitrary length in fixed time. Not hard. Totally not possible at all That would mean you can scan an infinite amount of data pe…

perfect recall is often a function of the architecture allowing for data to bleed through linkages. you can increase the perfect token recall through dialated wavenet structures, or, in the case of v5, the use of multi-head linear attention creates multiple pathways where information can skip forward in time

Re: Eagle 7B: Soaring past Transformers

#36

Post by mediaman »

klipt wrote: post #19
jack_pp wrote: post #11

Earlier quoted context omitted.

Of course it is in the brain, the brain created and evolved the language as a very powerful tool. If intelligence was in the language then other animals would be as intelligent as us

Scientific advancement requires both brains and knowledge transfer over generations. "If I have seen further, it is by standing on the shoulders of giants."

Knowledge transfer over generations is a function of the brain.

Other species have much more limited ability to transfer knowledge intergenerationally, and that is because the human brain's capability for symbolic language is much more advanced than other animals', who are not able to encode knowledge nearly as efficiently.

Re: Eagle 7B: Soaring past Transformers

#37

Post by klipt »

mediaman wrote: post #36
klipt wrote: post #19

Earlier quoted context omitted.

Scientific advancement requires both brains and knowledge transfer over generations. "If I have seen further, it is by standing on the shoulders of giants."

Knowledge transfer over generations is a function of the brain. Other species have much more limited ability to transfer knowledge intergenerationally, and that is because the human brain's capability for symbolic language is much more advanced than other animals', who are not able to encode knowledge nearly as efficiently.

The point is it's a function of many connected brains, not just one brain.

Re: Eagle 7B: Soaring past Transformers

#38

Post by airspresso »

samus wrote: post #33

Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?

This caught my eye as well. I interpreted it as the map only shows countries that have English as primary language. But the article was not precise enough about this.

I applaud them for focusing on multilingual performance though, as that is an important area of NLP which still has lots of room for improvement.

Re: Eagle 7B: Soaring past Transformers

#40

Post by samus »

airspresso wrote: post #38
samus wrote: post #33

Has anybody else noticed the map indicating that there are barely any fluent English speakers outside of the "English-speaking" countries? Or is the threshold for fluency so high that no place even in Europe except the Ireland and the UK qualify at all?

This caught my eye as well. I interpreted it as the map only shows countries that have English as primary language. But the article was not precise enough about this. I applaud them for focusing on multilingual performance though, as that is an important area of NLP which still has lots of room for improvement.

I highly welcome the effort as well*, but I don't see why they would have to mistake first-language ability for fluency to argue for that. The difference is vast and relevant: anyone with good English reading and writing skills can take advantage of a model and might prefer it over a worse model in their native language.

*: Just sceptical whether there's enough content out there which isn't just (often badly or too straightforwardly) translated from English. Not an issue for the languages with let's say >10 Mio. speakers, but for everything smaller.

Post reply on HN