Eagle 7B: Soaring past Transformers
blog.rwkv.com
Eagle 7B: Soaring past Transformers
1–10 of 86 posts
Re: Eagle 7B: Soaring past Transformers
#2> We are releasing RWKV-v5 Eagle 7B, licensed as Apache 2.0 license, under the Linux Foundation, and can be used personally or commercially without restrictions
great on the team to actually set up the right incentives for testing and adoption.
Re: Eagle 7B: Soaring past Transformers
#3Is it possible to try this out on something like llama.cpp? Does the different architecture make a difference there?
Re: Eagle 7B: Soaring past Transformers
#4Is it possible to try this out on something like llama.cpp? Does the different architecture make a difference there?
[deleted]
Re: Eagle 7B: Soaring past Transformers
#5It's interesting how all focus is now primarily on decoder-only next-token-prediction models. Encoders (BERT, encoder of T5) are still useful for generating embedding for tasks like retrieval or classification. While there is a lot of work on fine-tuning BERT and T5 for such tasks, it would be nice to see more research on better pre-training architectures for embedding use cases.
Re: Eagle 7B: Soaring past Transformers
#6It’s cool that progress is being made on alternative LLM architectures, and I did upvote the link.
However, I found this article somewhat frustrating. Showing the quality of the model is only half of the story, but the article suddenly ends there. If people are going to be motivated to adopt an entirely different architecture, then performance and context size deserve at least as much discussion.
Given the linear nature being shown, it seems like the primary thing people are going to want to see is the next frontier of LLMs: context sizes of ~1M tokens. The word “context” does not even appear in this article, which is disappointing. If there were a discussion of context, it would be nice to see if it passes the passkey test.
The article also appears to reuse a chart from RWKV-4 showing how awesome a linear function is compared to a quadratic one, but… cool story? It’s not even clear what this chart is truly showing. Is this chart only showing generated tokens, or is this including prompt tokens? As I have never used RWKV, I have no idea how the prompt processing speed compares to the token generation speed. Prompt processing speed has been a big problem for Mixtral, for example.
As a reader, I want to see a couple of actual examples of X prompt tokens + Y generated tokens, and the tokens/s of X and Y for RKWV-5 and for Mistral on the same hardware. On the Mistral side, it is trivial to collect this information in llama.cpp, but I don’t know how the tooling is for RWKV.
Re: Eagle 7B: Soaring past Transformers
#7Is it possible to try this out on something like llama.cpp? Does the different architecture make a difference there?
There's https://github.com/saharNooby/rwkv.cpp, which related-ish[0] to ggml/llama.cpp
Re: Eagle 7B: Soaring past Transformers
#8This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity.
And the same data when used to train humans creates modern capable people. Alone, without society and language, we would be mere shadows of ourselves. What does it say when AI acquires so many capabilities from language data? maybe intelligence was not centered in the brain. It's a social process.
Re: Eagle 7B: Soaring past Transformers
#9My experiments with RWKV-4 showed good inference speed but suuper slow tokenization speed; I’m not sure if this was RWKV specific or implementation specific: it was some time ago. Any guidance on this for rwkv-5?