Mamba outperforms transformers "everywhere we tried"
old.reddit.com
Mamba outperforms transformers "everywhere we tried"
1–10 of 27 posts
Re: Mamba outperforms transformers "everywhere we tried"
#2We might be on the cusp of dethroning the attention & transformer dynasty! Anyone have context on how the mamba architecture fares with multimodal models?
Re: Mamba outperforms transformers "everywhere we tried"
#3Here is the paper: https://arxiv.org/abs/2312.00752
Re: Mamba outperforms transformers "everywhere we tried"
#4This is pretty awesome and I'm curious to try the code out. Has anybody been able to run this locally?
Re: Mamba outperforms transformers "everywhere we tried"
#5If you want to really dive into the math and motivation for this entirely new class of models (state space models) I highly recommend reading Albert Gu’s thesis.
https://searchworks.stanford.edu/view/14784021
I tried to read the Mamba paper and I was lost but after reading his thesis it was a lot to understand it.
The part I’m still struggling with is the section on how Mamba trains so efficiently, it’s surprising since it is actually autoregressive and step by step and I thought this is why LSTMs were dropped
Re: Mamba outperforms transformers "everywhere we tried"
#6Previously: https://news.ycombinator.com/item?id=38522428
Re: Mamba outperforms transformers "everywhere we tried"
#7Title is misleading.
“Everywhere we tried” does not include actually beating the performance (or getting particularly close to) sota transformers of the same size. That said, these models were trained for less time than the sota models so there is promise.
Re: Mamba outperforms transformers "everywhere we tried"
#8Still chewing but my takeaways thus far:
SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the performance as larger scales is yet to be explored in practice.
I found my deepest understanding of the selection mechanism came from struggling with the discretization in section 2, followed by the deeper explanations of the variables involved in 3.5.2. This video gives excellent background to SSMs, along with a detailed walk through of the paper itself.[a]
I am still coming to understand S4, SSMs in general but the video suggested this annotated explainer that has been helping a lot[b].
I would also point out section 3.1 and it’s discussion of the tradeoffs between compression and effectiveness as particularly interesting.
I do wonder how many different GPUs / hardware architectures will be able to execute the optimizations that are described as critical. I think the nature of the optimizations is the part of the paper I understand least well.
The paper taken at face value looks very exciting. The promise of a very large context window with 5x throughput for inference would be huge if it proves to scale well. I do wonder if it will make sense to train SSMs without this selection mechanism for specific continuous use cases where it seems to perform better or if other architectures will prove to better serve those cases.
----
Re: Mamba outperforms transformers "everywhere we tried"
#9Title is misleading. “Everywhere we tried” does not include actually beating the performance (or getting particularly close to) sota transformers of the same size. That said, these models were trained for less time than the sota models so there is promise.
What models in particular do you think it should have been compared to? It seems like they chose some of the most understood / cited models at around 1 B parameters.
Re: Mamba outperforms transformers "everywhere we tried"
#10dead in the water.