Mamba Explained: The State Space Model Taking On Transformers
kolaayonrinde.com
Mamba Explained: The State Space Model Taking On Transformers
1–10 of 101 posts
Re: Mamba Explained: The State Space Model Taking On Transformers
#2Re: Mamba Explained: The State Space Model Taking On Transformers
#3First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.
Re: Mamba Explained: The State Space Model Taking On Transformers
#4First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.
Yes, that's technically accurate. But I prefer to think of the entire LLM space as a new scientific field that started when OpenAI released ChatGPT.
In that context, all new research directions are valuable simply for the fact that they're expanding the foundation of the field. 5 years from now, who knows what the most effective models will use under the hood, but the more we can learn about them in general, the better.
Re: Mamba Explained: The State Space Model Taking On Transformers
#5Why doesn't Equation 1b use the h' defined in Equation 1a?
Re: Mamba Explained: The State Space Model Taking On Transformers
#6First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.
True, but bear in mind the Mamba preprint is less than three months old. A lot of people are probably experimenting with these ideas right now and training a completely new, large foundation model with a different architecture will take a significant amount of time.
Re: Mamba Explained: The State Space Model Taking On Transformers
#7Why doesn't Equation 1b use the h' defined in Equation 1a?
I believe h' is for the next state. y(t) is to predict the next word so it uses the current hidden state h(t).
Re: Mamba Explained: The State Space Model Taking On Transformers
#8First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.
Yes, that's technically accurate. But I prefer to think of the entire LLM space as a new scientific field that started when OpenAI released ChatGPT. In that context, all new research directions are valuable simply for the fact that they're expanding the foundation of the field. 5 years from now, who knows what the most effective models will use under the hood, but the more we can learn about them in general, the bett…
The field of research here is far older than ChatGPT's release. Neural network research has been going on for at least 50 years.
Most of the research that enabled ChatGPT was also already known. "Attention is all you need" was a 2017 paper.
It still is a fast evolving field, but not one that just kicked off.
Re: Mamba Explained: The State Space Model Taking On Transformers
#9[deleted]
Re: Mamba Explained: The State Space Model Taking On Transformers
#10> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges
The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively.