Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementation of Google's Griffin Architecture – RNN LLM

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Implementation of Google's Griffin Architecture – RNN LLM

#31

Post by logicchains »

VHRanger wrote: post #9
foota wrote: post #7

Earlier quoted context omitted.

Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.

It seems only OK as a model? Looking at the LLM chat leaderboard it's 71st and the 14B version is worse than a lot of 7B models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Also, llama.cpp makes inference accessible for a lot of people, and it's not available for RWKV. Not to knock on the model, I'm sure it's good. I also like that it's a succesful example of citizen science. It's just not popular…

>Also, llama.cpp makes inference accessible for a lot of people, and it's not available for RWKV.

It absolutely is: https://github.com/RWKV/rwkv.cpp .

Re: Implementation of Google's Griffin Architecture – RNN LLM

#33

Post by shawntan »

foota wrote: post #7
VHRanger wrote: post #2

Like RWKV and Mamba, this is mixing some RNN properties to avoid the issues transformers have. However I'm curious about their scaling claims. They have a plot that shows how the model scales in training with the FLOPs you throw at it. But the issue we should rather be concerned with is the wall time of training for a set amount of hardware. Back in 2018, we could train medium sized RNNs, the issue was with wall time…

Do you know the downside with RWKV? Based on how they present it, it seems like the best thing since sliced bread, but I would have assumed that it would have been widely adopted if that were the case.

Not sure if this is the type of answer you're looking for, but RWKV is not really recurrent the same way RNNs are recurrent. This quasi-recurrentness allows it and its comrades to use algorithms like parallel SCAN to achieve log N complexity when parallelised. But you pay for that in terms of state-tracking.

There's a cool talk here if you care to know the details:https://www.youtube.com/watch?v=4-VXe1yPDjk

Re: Implementation of Google's Griffin Architecture – RNN LLM

#34

Post by JyrkiAlakuijala »

janwas wrote: post #32

For anyone interested in a C++ implementation, our github.com/google/gemma.cpp now supports this model.

Fun fact -- gemma.cpp uses highway, an amazing high performance computation library originally developed in the JPEG XL effort.

Re: Implementation of Google's Griffin Architecture – RNN LLM

#35

Post by boywitharupee »

VHRanger wrote: post #21
spxneo wrote: post #19

im not smart enough to know the significance of this...is Griffin like MAMBA?

Yes, like RWKV and Mamba this is a new generation of models that are more like big RNNs than pure transformers we have now

and is Griffin a state space model?

Re: Implementation of Google's Griffin Architecture – RNN LLM

#36

Post by imjonse »

boywitharupee wrote: post #35
VHRanger wrote: post #21

Earlier quoted context omitted.

Yes, like RWKV and Mamba this is a new generation of models that are more like big RNNs than pure transformers we have now

and is Griffin a state space model?

No, it's a combination of RNN and Transformer.

Re: Implementation of Google's Griffin Architecture – RNN LLM

#38

Post by stri8ed »

VHRanger wrote: post #21
spxneo wrote: post #19

im not smart enough to know the significance of this...is Griffin like MAMBA?

Yes, like RWKV and Mamba this is a new generation of models that are more like big RNNs than pure transformers we have now

Isn't that how previous models were, before the attention is all you need paper?

Re: Implementation of Google's Griffin Architecture – RNN LLM

#39

Post by VHRanger »

michwilinski wrote: post #37
imjonse wrote: post #36

Earlier quoted context omitted.

No, it's a combination of RNN and Transformer.

I mean, SSMs are in fact under the hood RNNs

At the end of the day, either you carry around a hidden state, or you have a fixed window for autoregression.

You can call hidden states "RNN-like" and autoregressive windows "transformer-like", but apart from those two core paradigms I don't know of other ways to do sequence modelling.

Mamba/RWKV/Griffin are somewhere between those two extremes.

Post reply on HN