Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba: The Easy Way

jackcook.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Mamba: The Easy Way

#21

Post by whimsicalism »

magnio wrote: post #9

Fantastic blog post, thank you for this. I am not even familiar with transformers, yet the explanation is stellar clear to me, and the included references and context are a trasure trove. The explanation of FlashAttention is the best I have seen, and that is not even the focus of the article. One question I have on selectivity: footnote 4 says "the continuous A is constant, while our discretization parameter ∆ is inp…

How are you not familiar with transformers yet have seen multiple explanations of FlashAttention?

Re: Mamba: The Easy Way

#22

Post by moffkalast »

If I'm not mistaken the largest mamba model right now is 2.8B and undertrained with low quality data (the Pile only). The main problem is that it's new and unproven.

Should become very interesting once someone with both data and significant financial backing takes the plunge and trains something of notable size. Perhaps Llama-3 might already end up being that attempt, as we seem to be heavily into diminishing returns for transformers.

Re: Mamba: The Easy Way

#23

Post by espadrine »

thatguysaguy wrote: post #7
paxys wrote: post #5

From what I can tell all the large players in the space are continuing developing on transformers right? Is it just that Mamba is too new, or is the architecture fundamentally not usable for some reason?

Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers. There are some papers suggesting that transformers are better than SSMs in fundamental ways (e.g. They cannot do arbitrary key-based recall from their context: https://arxiv.org/abs/2402.01032 ). This means it's…

Another element is that Mamba required a very custom implementation down to custom fused kernels which I expect would need to be implemented in deepspeed or the equivalent library for a larger training run spanning thousands of GPUs.

Re: Mamba: The Easy Way

#24

Post by scarmig »

amelius wrote: post #6
woadwarrior01 wrote: post #4

Earlier quoted context omitted.

I use the former and have been experimenting with the latter. Fortunately, the contexts are separate enough that they never come up in the same sentence.

I was using mamba to install mamba the other day, when suddenly I had to run for a live mamba.

I had the exact same experience, and I was also using it for a web application powered by the Mamba web framework.

Re: Mamba: The Easy Way

#26

Post by jsenn »

This was really helpful, but only discusses linear operations, which obviously can’t be the whole story. From the paper it seems like the discretization is the only nonlinear step—in particular the selection mechanism is just a linear transformation. Is that right? How important is the particular form of the nonlinearity?

EDIT: from looking at the paper, it seems like even though the core state space model/selection mechanism is linear (except for discretization?), they incorporate a nonlinearity in the full “mamba block”, which is stacked up with residual connections and layer norm just like in a transformer. They describe this as combining a linear attention and an MLP into a single step, rather than alternating attention and MLP as in a transformer.

Re: Mamba: The Easy Way

#27

Post by avarun »

whimsicalism wrote: post #21
magnio wrote: post #9

Fantastic blog post, thank you for this. I am not even familiar with transformers, yet the explanation is stellar clear to me, and the included references and context are a trasure trove. The explanation of FlashAttention is the best I have seen, and that is not even the focus of the article. One question I have on selectivity: footnote 4 says "the continuous A is constant, while our discretization parameter ∆ is inp…

How are you not familiar with transformers yet have seen multiple explanations of FlashAttention?

Literally the exact question I had reading that comment haha

Re: Mamba: The Easy Way

#28

Post by 5kg »

jxmorris12 wrote: post #18

In case people are wondering why Mamba is exciting: There's this idea in AI right now that "scaling" models to be bigger and train on more data always makes them better. This has led to a science of "scaling laws" which study just how much bigger models need to be and how much data we need to train them on to make them a certain amount better. The relationship between model size, training data size, and performance t…

I'd love to see someone who has the resources train a model bigger than 2.8b and show the scaling law still holds.

Re: Mamba: The Easy Way

#29

Post by cs702 »

espadrine wrote: post #23
thatguysaguy wrote: post #7

Earlier quoted context omitted.

Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers. There are some papers suggesting that transformers are better than SSMs in fundamental ways (e.g. They cannot do arbitrary key-based recall from their context: https://arxiv.org/abs/2402.01032 ). This means it's…

Another element is that Mamba required a very custom implementation down to custom fused kernels which I expect would need to be implemented in deepspeed or the equivalent library for a larger training run spanning thousands of GPUs.

Not necessarily:

https://www.reddit.com/r/MachineLearning/comments/1amb3xu/d_...

Re: Mamba: The Easy Way

#30

Post by SekstiNi »

moffkalast wrote: post #22

If I'm not mistaken the largest mamba model right now is 2.8B and undertrained with low quality data (the Pile only). The main problem is that it's new and unproven. Should become very interesting once someone with both data and significant financial backing takes the plunge and trains something of notable size. Perhaps Llama-3 might already end up being that attempt, as we seem to be heavily into diminishing returns…

There is one trained on 600B tokens from SlimPajama [1], but that's fairly tiny compared to other recent releases (ex. stablelm-3b [2] trained on 4T tokens).

> low quality data (the Pile only)

The Pile is pretty good quality wise. It's mostly the size (300B tokens) that's limiting.

[1]: https://huggingface.co/state-spaces/mamba-2.8b-slimpj [2]: https://huggingface.co/stabilityai/stablelm-3b-4e1t

Post reply on HN