Implementation of Google's Griffin Architecture – RNN LLM
1–10 of 39 posts
Re: Implementation of Google's Griffin Architecture – RNN LLM
#2Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have.
However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it.
But the issue we should rather be concerned with is the wall time of training for a set amount of hardware.
Back in 2018, we could train medium sized RNNs, the issue was with wall time of training and training stability.
Re: Implementation of Google's Griffin Architecture – RNN LLM
#3I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin
Why wouldn't select equal size models?..
Re: Implementation of Google's Griffin Architecture – RNN LLM
#4I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin Why wouldn't select equal size models?..
They probably had them for some reason and it was cheaper not to retrain one of them again
Re: Implementation of Google's Griffin Architecture – RNN LLM
#5I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin Why wouldn't select equal size models?..
They probably had them for some reason and it was cheaper not to retrain one of them again
Its just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..
Re: Implementation of Google's Griffin Architecture – RNN LLM
#6Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training
Re: Implementation of Google's Griffin Architecture – RNN LLM
#7Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.
Re: Implementation of Google's Griffin Architecture – RNN LLM
#8Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training
Yeah, that's just the training stability part to my knowledge
Re: Implementation of Google's Griffin Architecture – RNN LLM
#9Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.
It seems only OK as a model? Looking at the LLM chat leaderboard it's 71st and the 14B version is worse than a lot of 7B models:
https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Also, llama.cpp makes inference accessible for a lot of people, and it's not available for RWKV.
Not to knock on the model, I'm sure it's good. I also like that it's a succesful example of citizen science.
It's just not popular enough to have the inference infrastructure transformers have, not established enough to attract enough money to get 60B+ models trained, and so on.
Re: Implementation of Google's Griffin Architecture – RNN LLM
#10Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…
The paper shows that the speed is comparable to transformer models, faster with smaller with "long" sequence length like 8k.