Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Eagle 7B: Soaring past Transformers

blog.rwkv.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Eagle 7B: Soaring past Transformers

#61

Post by vidarh »

ReptileMan wrote: post #39

Their map showing distribution of English speaking people is just terrible - I am fairly sure that there is at least one percent speaking English in India, Western Europe, Eastern Europe, Russia and China.

Yeah, that's weird. Depending on how you count least India, Pakistan, and Nigeria has more English speakers than the UK in absolute terms (Nigeria might end up either side depending on how strict you are), and Nigeria is on its path to overtake the UK as the country with the second largest number of native English speakers, as it's increasingly often the 2nd language of parents with different 1st languages.

E.g. my ex's parents had Igbo and Yoruba as their 1st languages, but she and all her siblings has English as theirs.

Re: Eagle 7B: Soaring past Transformers

#62

Post by lhl »

Al-Khwarizmi wrote: post #32
lhl wrote: post #29

Earlier quoted context omitted.

It is trivial to fine tune any model (whether a base model or an aligned model) to your preferred output preferences as long as you have access to the model weights .

Not trivial for the general public at all, and furthermore, you need much more memory for finetuning than for inference, often making it infeasible for many machine/model combinations.

If you are running a local LLM already (which no one in the "general public is") then the bar is really not that much higher for fine-tuning (either for an individual or community member to do).

And you don't need any additional equipment at all. When I say trivial, I really do mean it - you can go to https://www.together.ai/pricing and see for yourself - a 10M token 3 epoch fine tune on a 7B model will cost you about $10-15 right now. Upload your dataset, download your fine tune weights (or serve via their infrastructure). This is only going to get easier (compare how difficult it was to inference local models last year to what you can do with plug and play solutions like Ollama, LM Studio, or Jan today).

Note also that tuning is a one-time outlay, and merges are even less resource intensive/easier to do.

To put things in perspective, tell me how much cost and effort it would be to tune a model where you don't have the weights at all in comparison.

Re: Eagle 7B: Soaring past Transformers

#63

Post by dTal »

warkdarrior wrote: post #20
Szpadel wrote: post #15

Earlier quoted context omitted.

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

If later input overwrites previous input in the internal state, it means the model does have a limit to how much input it can "remember" at any given time and that limit is less than infinite.

There's a limit to the amount, but not to the duration (in theory). It can hold on to something it considers important for an arbitrary amount of time.

Re: Eagle 7B: Soaring past Transformers

#64

Post by Al-Khwarizmi »

lhl wrote: post #62
Al-Khwarizmi wrote: post #32

Earlier quoted context omitted.

Not trivial for the general public at all, and furthermore, you need much more memory for finetuning than for inference, often making it infeasible for many machine/model combinations.

If you are running a local LLM already (which no one in the "general public is") then the bar is really not that much higher for fine-tuning (either for an individual or community member to do). And you don't need any additional equipment at all. When I say trivial, I really do mean it - you can go to https://www.together.ai/pricing and see for yourself - a 10M token 3 epoch fine tune on a 7B model will cost you abou…

Running a local LLM - downloading LM studio, installing on Windows, using the search function to search for a popular LLM, click "download", click the button to load the model, chat.

Fine-tuning - obtaining a dataset for your task (this in itself is not trivial), figuring out how the service you linked works (after figuring out that it exists at all), uploading the dataset, paying, downloading the weights - OK, now how do you load them into LM studio?

It's all subjective, of course, but for me there's a considerable difficulty jump there.

Re: Eagle 7B: Soaring past Transformers

#65

Post by YeGoblynQueenne »

>> An eagle, flying past a transformer-looking robot

That hero image is a complete mess (e.g. look at the eagle's forward paw, the "transformer robot"'s right arm or the position of the eagle's left wing).

Why is it that people put up such obviously messed-up images in their articles? Do they not see that level of detail, or do they just find it cool to have some "AI art" in their article, as a kind of an in-group code, like "we use AI"?

Re: Eagle 7B: Soaring past Transformers

#66

Post by pmx »

YeGoblynQueenne wrote: post #65

>> An eagle, flying past a transformer-looking robot That hero image is a complete mess (e.g. look at the eagle's forward paw, the "transformer robot"'s right arm or the position of the eagle's left wing). Why is it that people put up such obviously messed-up images in their articles? Do they not see that level of detail, or do they just find it cool to have some "AI art" in their article, as a kind of an in-group co…

I mean, They're an organisation creating AI models, releasing an AI model. Them using AI images isn't hugely shocking to anyone with the capacity to reason. The picture looks cool at first glance, and that is really all that matters. It's just a cool bit of eye-candy leading into the article.

Re: Eagle 7B: Soaring past Transformers

#67

Post by samus »

vidarh wrote: post #59
samus wrote: post #40

Earlier quoted context omitted.

I highly welcome the effort as well*, but I don't see why they would have to mistake first-language ability for fluency to argue for that. The difference is vast and relevant: anyone with good English reading and writing skills can take advantage of a model and might prefer it over a worse model in their native language. *: Just sceptical whether there's enough content out there which isn't just (often badly or too s…

Norwegian has ca. 5m speakers, and ChatGPT does not just do fine with both the (mutually intelligible) Norwegian written languages, but also has no problem "translating" to/from several regional dialects when I've experimented. And that is, I presume - I could be wrong -, before anyone has tried to really mine the Norwegian national library, as even much of what is online is not easily accessible for crawling. I thin…

It's an arbitrary threshold for the sake of discussion. Of course Norway, one of the most developed countries and richest countries per capita, manages to have good content in its written languages on the internet. Model training also benefits from its linguistic and cultural similarities to Swedish and Danish, and English, Dutch, and German to a lesser degree. I suspect many people from Eastern Europe find Russian models useful for the same reason.

I agree that the situation is not hopeless for languages with a thriving written culture. But for many minority languages there might only be chat messages, some literary works, and vast amounts of machine-translated websites accessible for crawlers. I hope that in the future improved model architectures and training strategies can push the required amount of raw content way down.

Re: Eagle 7B: Soaring past Transformers

#68

Post by sanxiyn »

viraptor wrote: post #54

> A common feedback we receive for the RWKV multi-lingual approach is: it hurts our English evaluation scores (...) Has anyone quantified that specifically? I'd love to read more details since I'd expect the concepts to start mapping between languages at some point. I.e. with enough language fluency I'd expect learning knowledge/reasoning in language to improve the result in another. But I can't find any paper talkin…

https://arxiv.org/abs/2306.06688 studied this. (Check out figure 3.) In short, yes, multilingual transfer does happen. But importantly, not at 7B. Being English-centric is possibly okay for LLaMA 70B, but it really is not okay for LLaMA 7B.

Re: Eagle 7B: Soaring past Transformers

#69

Post by sanxiyn »

lelag wrote: post #58
zurfer wrote: post #45

Earlier quoted context omitted.

Thank you! I'm not experienced with 7B models. 3 things stand out to me: - it's absolutely not useable for the kind of use cases I solve with GPT-4 (code generation, information retrieval) - it could technically swallow a 50 page PDF, but it's not able to answer questions about it (inference speed was good, but content was garbage) - it is ok for chatting and translations (how is your day?)

I also tried the demo and I find it pretty much useless at most things even comparing it to a small 7b transformer model like mistral. From my albeit quick tests, what I found is that it knows clearly less things than mistral, it hallucinates much more, it does not follow instructions, has less reasoning capabilities and asking it to translate a Japanese text into English gave me a bad translated summary instead of t…

As written in the post, it is a base model with light instruction tuning, i.e. Llama2, not Llama2-chat. You should evaluate it as a base model. If you evaluate it as a chat model, of course it will perform horribly.

Re: Eagle 7B: Soaring past Transformers

#70

Post by RcouF1uZ4gsC »

Sounds like this hallucinates pretty easily

From Reddit :

https://www.reddit.com/r/LocalLLaMA/comments/1ad0j63/rwkv5_1...

User: Which is larger, a chicken egg or a cow egg?

Assistant: To determine which is larger, a chicken egg or a cow egg, let's first look at their respective sizes and compare them.

Chicken Egg: The average chicken egg size ranges from 2.5 to 3 inches (6 to 8 cm) in length and 1.5 to 2 inches (3.8 to 5 cm) in diameter.[1] Cow Egg: The average cow egg size is roughly 2 inches (5 cm) in length and 1 inch (2.5 cm) in diameter.[2] Based on these measurements, we can see that the chicken egg is slightly larger than the cow egg, with dimensions of around 3 inches by 1 inch versus 2 inches by 0.75 inches respectively (length x diameter). However, as chickens are much bigger than cows on average,[3] it would be more accurate to say that a chicken egg is roughly three times bigger than a cow egg based on their respective sizes within each species.

So, the answer is: A chicken egg is three times larger than a cow egg in terms of size when taking into account their respective dimensions within each species' average range.

Post reply on HN