Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Eagle 7B: Soaring past Transformers

blog.rwkv.com

Post reply on HN
Print view View original

11–20 of 86 posts

Re: Eagle 7B: Soaring past Transformers

#11

Post by jack_pp »

visarga wrote: post #8

This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…

Of course it is in the brain, the brain created and evolved the language as a very powerful tool. If intelligence was in the language then other animals would be as intelligent as us

Re: Eagle 7B: Soaring past Transformers

#13

Post by blackoil »

visarga wrote: post #8

This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…

> maybe intelligence was not centered in the brain. It's a social process.

Is that controversial? We are stand on the shoulder of giants before us and that is why we insist on training younglings for couple of decades on past learnings before they are believed to be of any useful. Even the smartest person won't survive long if dropped in 10000 BC.

Re: Eagle 7B: Soaring past Transformers

#14

Post by Harrisonv »

coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

For linear transformers, the current metric is "perfect token recall", the ability for the model to recall a randomized sequence of data. You can find the limit of a particular model architecture by training a model of a particular size to echo randomized data, and I believe this was touched on in the zoo-ology paper.

This doesnt prevent the model from retaining sequences or information beyond this metric, as information can easily be compressed in the state, but it anything within that window can be perfectly recalled by the model.

Internal testing has placed the value for Eagle around the 2.5k ptr[perfect token recall] mark, while community fine tunes done on the partial checkpoints for long distance information gathering and memorization have been shown to easily dwarf that.

prompt processing speed benefits from the same gemm optimizations as standard transformers, with the extra benefit of those gemm optimizations working for batch inference as well (no need for vllm as memory allocation is static per agent)

Re: Eagle 7B: Soaring past Transformers

#15

Post by Szpadel »

coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

RWKV does not have context size, or in other way do look at it, it does have infinite one.

As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

Re: Eagle 7B: Soaring past Transformers

#16

Post by marmaduke »

coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

These models don't have a fixed context size and are progressively fine-tuned for longer and longer contexts. The context length also doesn't impact inference cost.

Another aspect of performance is not just how well does the trained model perform, but is it data efficient (performance per token trained)? The comparison with Pythia (an open GPT) is shown in the article.

The rwkv4 paper is quite detailed and has examples of prompt and responses on the last few pages

https://arxiv.org/abs/2305.13048

And iirc rwkv5 is very similar to retnet which is detailed here

https://arxiv.org/abs/2307.08621

Edit now that I thought more about, the data efficiency seems like a highly important aspect given their noble goal to be fully multi lingual. This is fairly interesting theoretically as well and for other applications where abundance of data is not a given

Re: Eagle 7B: Soaring past Transformers

#18

Post by gdiamos »

Szpadel wrote: post #15
coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

There’s a difference between the computation requirements of long context lengths and the accuracy of the model on long context length tasks.

Re: Eagle 7B: Soaring past Transformers

#19

Post by klipt »

jack_pp wrote: post #11
visarga wrote: post #8

This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…

Of course it is in the brain, the brain created and evolved the language as a very powerful tool. If intelligence was in the language then other animals would be as intelligent as us

Scientific advancement requires both brains and knowledge transfer over generations.

"If I have seen further, it is by standing on the shoulders of giants."

Re: Eagle 7B: Soaring past Transformers

#20

Post by warkdarrior »

Szpadel wrote: post #15
coder543 wrote: post #6

It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

If later input overwrites previous input in the internal state, it means the model does have a limit to how much input it can "remember" at any given time and that limit is less than infinite.
Post reply on HN