There is also this:
https://jackcook.com/2024/02/23/mamba.html
Mamba Explained: The State Space Model Taking On Transformers
71–80 of 101 posts
Re: Mamba Explained: The State Space Model Taking On Transformers
#72Earlier quoted context omitted.
I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.
As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it…
> Just that it isn't the immediate transformer killer.
What is the best/stable-ish linear alternative for transformer right now? Especially for text generation and summarization.
We have domain specific ways of over sampling and search, so we much prefer less expensive models.
Re: Mamba Explained: The State Space Model Taking On Transformers
#73The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…
That might explain the motivation for why the Δ variable is used and varied; but not the "Selectivity", which the article says is expressed by how the matrices B and C vary while consuming input. Something I've noticed is that B, C and Δ depend only on the current token. See this: https://www.kolaayonrinde.com/blog/images/mamba/ssm_algorith... -- Another thing is that I've noticed that the definition of "SSM" in the…
OK, I've noticed that the pseudo-code above is vectorised, and so there's no recursion. The SSM function is actually described at the start of the paper, and an efficient hardware-aware implementation is suggested in section 3: https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf
Re: Mamba Explained: The State Space Model Taking On Transformers
#74Earlier quoted context omitted.
I see no evidence of this in biology nor in ML. I've read those scale papers. I've worked on scale myself. I'll bet the farm that scale isn't all you need. But I won't be surprised if people say that it is all scale. If you really think it is all scale, train a 7T ResNet MLP based model for NLP. If scale is all you need, make a LLM without DPO or RLHF. If scale is all you need, make SD3 with a GAN. Or what about a VA…
I said “scale is all you need for transformers ”. That has been true since GPT1. The best way to improve our best model today still seems to be “make it larger and train it on more data”. If you disagree please suggest a better way, or at least provide evidence that scaling up no longer works for transformers.
> at least provide evidence that scaling up no longer works for transformers.
Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
Re: Mamba Explained: The State Space Model Taking On Transformers
#75Earlier quoted context omitted.
The importance (e.g. attention) needs to be dynamic, e.g. one token will be important to some other tokens but not others. tf-idf and similar heuristics are what we were using before attention came along, e.g. tf-idf weighted bag-of-words representation of word2vec embeddings. That approaches fails in so many cases.
Attention in transformers works because over time the model learns token importance based on frequency and context. If you don’t have attention and need a fast substitute for “forgetting” non important tokens, then BM25 is an intuitive hypothesis.
To use your metaphor, TF-IDF will result in ‘fixed’ weights.
Attention makes it so that the weights of each token can be different in each sequence of tokens. Same token gets different weights depending on who its ‘neighbors’ in the sequence end up being.
This property allows the models to solve a variety of natural language problems and gets ‘used’ by the model to express context-aware dependencies.
Re: Mamba Explained: The State Space Model Taking On Transformers
#76Earlier quoted context omitted.
I said “scale is all you need for transformers ”. That has been true since GPT1. The best way to improve our best model today still seems to be “make it larger and train it on more data”. If you disagree please suggest a better way, or at least provide evidence that scaling up no longer works for transformers.
> at least provide evidence that scaling up no longer works for transformers. Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
I think they went MoE purely because straight up scaling from 175B to 1.8T is just too expensive. But it’s still 10x scaling, right?
Re: Mamba Explained: The State Space Model Taking On Transformers
#77Earlier quoted context omitted.
Attention in transformers works because over time the model learns token importance based on frequency and context. If you don’t have attention and need a fast substitute for “forgetting” non important tokens, then BM25 is an intuitive hypothesis.
To use your metaphor, TF-IDF will result in ‘fixed’ weights. Attention makes it so that the weights of each token can be different in each sequence of tokens. Same token gets different weights depending on who its ‘neighbors’ in the sequence end up being. This property allows the models to solve a variety of natural language problems and gets ‘used’ by the model to express context-aware dependencies.
[deleted]
Re: Mamba Explained: The State Space Model Taking On Transformers
#78I really enjoyed this article, thanks
Re: Mamba Explained: The State Space Model Taking On Transformers
#79Earlier quoted context omitted.
For MSFT, AMZN, GOOG, the competitive advantage comes from having huge datasets (that Nvidia doesn't have). It's a symbiosis that benefits the data-rich and GPU-rich players.
Where is the symbiosis? If data is the differentiator, how do the data owners benefit from Nvidia eating into their margins?
It's sumbiosis based on a common factor they both appreciate:
- the data/processing having to be large means the data-owners have a benefit
- the data/processing having to be large means NVIDIA also has a benefit (sells more GPUs to handle all that load)
Re: Mamba Explained: The State Space Model Taking On Transformers
#80Earlier quoted context omitted.
I said “scale is all you need for transformers ”. That has been true since GPT1. The best way to improve our best model today still seems to be “make it larger and train it on more data”. If you disagree please suggest a better way, or at least provide evidence that scaling up no longer works for transformers.
> at least provide evidence that scaling up no longer works for transformers. Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
Of scale? I would think not. I would say they are evidence against scale because they are more an argument for multi agent systems. Scale is about a singular framework. What that means is debatable though (I mean anything we call a singular network can be decomposed into sub networks. It's messy), hence the other part of my comment about not scale solutions being claimed as scale.
> I said “scale is all you need for transformers”
No you didn't. What kicked this all off was
> What’s the main difference between an ape’s brain and a human brain? Scale.
Don't retcon.