Attention is, after all, not what you need.
Eagle 7B: Soaring past Transformers
21–30 of 86 posts
Re: Eagle 7B: Soaring past Transformers
#22They've joined the Linux Foundation, does that mean the models are going to be eventually censored to satisfy the foundation's AI safety policies? That includes ensuring the models don't generate content that's non-inclusive or against diversity policies?
Re: Eagle 7B: Soaring past Transformers
#23It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…
RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.
In principle it has no context size limit, but (last time I checked) in practice there is one for implementation reasons.
Re: Eagle 7B: Soaring past Transformers
#24This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…
I agree that on balance, we should spend more effort on data than modeling, but it is just not true that modeling doesn't matter. Transformer-2023 is different from Trnasformer-2020 and cumulative improvement is significant. https://arxiv.org/abs/2312.00752 did such benchmark.
If choice between Transformer and RWKV doesn't seem to matter to you, the only reason is that while Transformer-2020 evolved to Transformer-2023, RWKV-v1 (which is from 2021) also evolved to RWKV-v5. If you use Transformer-2020 or RWKV-v1 today you will feel the difference.
Re: Eagle 7B: Soaring past Transformers
#25Try rwkv-demo-api.recursal.ai if you want to try it and dont want to wait for gradio
Re: Eagle 7B: Soaring past Transformers
#26Earlier quoted context omitted.
Of course it is in the brain, the brain created and evolved the language as a very powerful tool. If intelligence was in the language then other animals would be as intelligent as us
Scientific advancement requires both brains and knowledge transfer over generations. "If I have seen further, it is by standing on the shoulders of giants."
[deleted]
Re: Eagle 7B: Soaring past Transformers
#27It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link. However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion. Given the linear nat…
RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.
One of three things has to be true. Either:
a) this is false
b) perfect recall is false (ie. as the internal state is overwritten, you lose information about previous entries in the context)
c) the inference time scales by the context length.
It’s not possible to have perfect recall over an arbitrary length in fixed time.
Not hard. Totally not possible at all
That would mean you can scan an infinite amount of data perfectly in fixed time.
So… Hrm… this kind of claim rings some kind of alarm bells, when it’s combined with this kind of sweeping announcement.
It seems to good to true; either it’s not that good, or the laws of the universe no longer hold true.
Re: Eagle 7B: Soaring past Transformers
#28Earlier quoted context omitted.
RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.
If later input overwrites previous input in the internal state, it means the model does have a limit to how much input it can "remember" at any given time and that limit is less than infinite.
You can think of it like your own memory. Can you remember a very important thing from 10 years ago? Can you remember every single thing since then? Some things will remain for basically infinite period, some will have a more limited scope.
Re: Eagle 7B: Soaring past Transformers
#29They've joined the Linux Foundation, does that mean the models are going to be eventually censored to satisfy the foundation's AI safety policies? That includes ensuring the models don't generate content that's non-inclusive or against diversity policies?
It is trivial to fine tune any model (whether a base model or an aligned model) to your preferred output preferences as long as you have access to the model weights.
Re: Eagle 7B: Soaring past Transformers
#30They've joined the Linux Foundation, does that mean the models are going to be eventually censored to satisfy the foundation's AI safety policies? That includes ensuring the models don't generate content that's non-inclusive or against diversity policies?
Downvoted, because it's a very trolly way to ask this. Especially given the foundation doesn't have an AI safety policy from what I've seen. Let's be better than this...