Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba: The Easy Way

jackcook.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Mamba: The Easy Way

#41

Post by samus »

lpasselin wrote: post #8
Der_Einzige wrote: post #2

Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up. https://github.com/mamba-org/mamba Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has…

The mamba paper shows significant improvements in all model sizes, up to 1b, the largest one tested. Are there any reason why it wouldn't scale to 7b or more? Have they tried it?

That's the issue - I keep hearing that it is beyond small research group's budget to meaningfully train such a large model. You don't just need GPU time, you also need data. And just using the dregs of the internet doesn't cut it.

Re: Mamba: The Easy Way

#42

Post by denial »

pama wrote: post #40
denial wrote: post #33

Something minor I always wonder about when I read Mamba is the discretization. All of the sources I see referred to as derivations of it have a discretization of the form h_t =Ah_{t-1} + Bx_{t-1} for the first line instead of the given one of the form h_t =Ah_{t-1} + Bx_t. Does anyone know why this is?

Not sure how much detail you need but generally there exist implicit and explicit integrators for numerically solving (integrating) ODE. The implicit ones, like the one used here, tend to be more stable. The ideas behind SSM come from control theory ideas that used integrators with stability guarantees so that the rest of the neural network can focus on other aspects of the problem.

That's a helpful pointer. Thank you.

Re: Mamba: The Easy Way

#43

Post by KuriousCat »

jxmorris12 wrote: post #18

In case people are wondering why Mamba is exciting: There's this idea in AI right now that "scaling" models to be bigger and train on more data always makes them better. This has led to a science of "scaling laws" which study just how much bigger models need to be and how much data we need to train them on to make them a certain amount better. The relationship between model size, training data size, and performance t…

People have shown even CNNs can match up the peformance of the transformers.

https://openreview.net/forum?id=TKIFuQHHECj#

I believe there is a lot of herding going on due to the influence of people who had compute to play around with than deeply insightful or principled exploration of networks.

Re: Mamba: The Easy Way

#44

Post by intalentive »

Nice post. A couple things to add:

1. The Mamba co-author was also the FlashAttention lead author.

2. The secret ingredient that makes SSMs viable for deep learning is HiPPO theory. If you start with random initialization you're not going to get results. What you need is "optimal online function approximation" using Legendre polynomials, a Fourier basis, etc., in matrix form. The Mamba story starts with Legendre Memory Units.

Invariably someone comments, "How do we know that it scales?" We don't. But the lead author has backing and a new startup at cartesia.ai. Could be the next Mistral.

Re: Mamba: The Easy Way

#45

Post by whimsicalism »

samus wrote: post #39
whimsicalism wrote: post #21

Earlier quoted context omitted.

How are you not familiar with transformers yet have seen multiple explanations of FlashAttention?

The issue with Attention essentially is that it is used to relate all token of the input sequence with each other. The need to do that somehow makes sense no matter how much one understands about the internals of a transformer. The naive way to do that boils down to matrix multiplications, and a lot more people understand the performance issues implied by them.

your comment makes no sense to me, sorry. if you understand attention you understand transformers, period.

Re: Mamba: The Easy Way

#46

Post by israrkhan »

MoE (Mixture of Experts) is an effective way to scale transformers. Gemini 1.5 is already doing upto 1 million tokens. I have not seen any large scale mamba model, so not aware of its shortcomings, but I am sure there are tradeoffs.

It should be possible to combine Mamba with MoE, I wonder how that would look like... a billion token context?

Re: Mamba: The Easy Way

#47

Post by sigmoid10 »

intalentive wrote: post #44

Nice post. A couple things to add: 1. The Mamba co-author was also the FlashAttention lead author. 2. The secret ingredient that makes SSMs viable for deep learning is HiPPO theory. If you start with random initialization you're not going to get results. What you need is "optimal online function approximation" using Legendre polynomials, a Fourier basis, etc., in matrix form. The Mamba story starts with Legendre Memo…

The architecture is completely public. I would be surprised if certain other players (including but not limited to Mistral AI) are not training models yet. We'll hear soon enough if this is viable. Maybe not for official release candidates, but at least for internal testing.

Re: Mamba: The Easy Way

#48

Post by samus »

whimsicalism wrote: post #45
samus wrote: post #39

Earlier quoted context omitted.

The issue with Attention essentially is that it is used to relate all token of the input sequence with each other. The need to do that somehow makes sense no matter how much one understands about the internals of a transformer. The naive way to do that boils down to matrix multiplications, and a lot more people understand the performance issues implied by them.

your comment makes no sense to me, sorry. if you understand attention you understand transformers, period.

That's good to know :)

Re: Mamba: The Easy Way

#49

Post by intalentive »

israrkhan wrote: post #46

MoE (Mixture of Experts) is an effective way to scale transformers. Gemini 1.5 is already doing upto 1 million tokens. I have not seen any large scale mamba model, so not aware of its shortcomings, but I am sure there are tradeoffs. It should be possible to combine Mamba with MoE, I wonder how that would look like... a billion token context?

https://arxiv.org/abs/2401.04081

https://github.com/jzhang38/LongMamba

Re: Mamba: The Easy Way

#50

Post by whimsicalism »

israrkhan wrote: post #46

MoE (Mixture of Experts) is an effective way to scale transformers. Gemini 1.5 is already doing upto 1 million tokens. I have not seen any large scale mamba model, so not aware of its shortcomings, but I am sure there are tradeoffs. It should be possible to combine Mamba with MoE, I wonder how that would look like... a billion token context?

nope :) MoE does not scale transformers along sequence length
Post reply on HN