11–20 of 39 posts
Earlier quoted context omitted.
They probably had them for some reason and it was cheaper not to retrain one of them again
Its just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..
Earlier quoted context omitted.
Its just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..
It also performs better on any other size.
Earlier quoted context omitted.
Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.
It seems only OK as a model? Looking at the LLM chat leaderboard it's 71st and the 14B version is worse than a lot of 7B models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Also, llama.cpp makes inference accessible for a lot of people, and it's not available for RWKV. Not to knock on the model, I'm sure it's good. I also like that it's a succesful example of citizen science. It's just not popular…
Earlier quoted context omitted.
transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training
Yeah, that's just the training stability part to my knowledge
Earlier quoted context omitted.
Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.
It seems only OK as a model? Looking at the LLM chat leaderboard it's 71st and the 14B version is worse than a lot of 7B models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Also, llama.cpp makes inference accessible for a lot of people, and it's not available for RWKV. Not to knock on the model, I'm sure it's good. I also like that it's a succesful example of citizen science. It's just not popular…
Earlier quoted context omitted.
It also performs better on any other size.
They have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
Earlier quoted context omitted.
They have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.
All the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.
Earlier quoted context omitted.
All the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.
Yes, but MQA is limited to 6B size, while "other" larger non-RNN models in table(Llama-2) are not trained on the same dataset, and Hawk and Griffin are 7B. Sorry, I don't understand your point.
Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.
They really burst onto the scene and brought back RNNs in the world of transformers. The claim that RWKV isn't paralleizable during training also seems to be refuted in their readme. I'd guess it's generalizable performance as there is a difference between doing well on benchmarks and being usable. Personally I've tried running the weights a long time ago when it was first released and the results weren't usable but I'm sure there has been considerable progress since then.