Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#41

Post by p1esk »

godelski wrote: post #36
kettleballroll wrote: post #20

Earlier quoted context omitted.

As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it…

As a member of the research community: that's nonsense. Publishing is an extremely noisy process in ML and is getting increasingly difficult for smaller non big tech collaborating labs. Reviewers' go to are: more datasets, scale, not novel. The easiest way to approach this is to work off of pretrained models. This is probably more obvious in the NLP world. I agree that Mamba doesn't solve everything and it still need…

What’s the main difference between an ape’s brain and a human brain? Scale. So that’s the train we’re riding at the moment. No roadblocks yet, aside from cost.

Re: Mamba Explained: The State Space Model Taking On Transformers

#43

Post by nyrikki »

soVeryTired wrote: post #39
nyrikki wrote: post #34

Earlier quoted context omitted.

The fact that 'removing the “quadratic bottleneck”' involves either reduced expressability compared to self attention or disproving SETH is another reason. The quadratic bottleneck is due to the lower bounds of exhaustive search. The papers on this only ever seem to reference perplexity. The fact it can append a word to "I'm going to the beach" that sounds good doesn't mean it is useful. There is no free lunch, and t…

What's SETH in this context? I googled to no avail.

Strong Exponential Time Hypothesis

Here is how it relates to attention.

https://arxiv.org/abs/2209.04881

Re: Mamba Explained: The State Space Model Taking On Transformers

#44

Post by [deleted] »

soVeryTired wrote: post #39
nyrikki wrote: post #34

Earlier quoted context omitted.

The fact that 'removing the “quadratic bottleneck”' involves either reduced expressability compared to self attention or disproving SETH is another reason. The quadratic bottleneck is due to the lower bounds of exhaustive search. The papers on this only ever seem to reference perplexity. The fact it can append a word to "I'm going to the beach" that sounds good doesn't mean it is useful. There is no free lunch, and t…

What's SETH in this context? I googled to no avail.

[deleted]

Re: Mamba Explained: The State Space Model Taking On Transformers

#45

Post by ssivark »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

It’s less compute for the same model sizes. Rest assured that there will still be a race to scale model sizes (and data) to achieve better performance.

Re: Mamba Explained: The State Space Model Taking On Transformers

#46

Post by koayon »

CuriouslyC wrote: post #35
fbdab103 wrote: post #30

Earlier quoted context omitted.

It is a really recent development. Even if this architecture is technically superior, it could take time before a model using it becomes competitive. Or maybe it does not pan out at all. We are still at the stage where people are throwing everything at the wall to see what sticks. Some promising ideas which work at small scale do not work at bigger.

This. Hyperparameter tuning and training include a lot of model specific black magic. Transformers have had time to mature, it'll take a while for other stuff to catch up even if they have a higher potential ceiling.

Definitely agree that a lot of work going into hyperparameter tuning and maturing the ecosystem will be key here!

I'm seeing the Mamba paper as the `Attention Is All You Need` of Mamba - it might take a little while before we get everything optimised to the point of a GPT-4 (it took 6 years for transformers but should be faster than that now with all the attention on ML)

Re: Mamba Explained: The State Space Model Taking On Transformers

#47

Post by koayon »

CuriouslyC wrote: post #35
fbdab103 wrote: post #30

Earlier quoted context omitted.

It is a really recent development. Even if this architecture is technically superior, it could take time before a model using it becomes competitive. Or maybe it does not pan out at all. We are still at the stage where people are throwing everything at the wall to see what sticks. Some promising ideas which work at small scale do not work at bigger.

This. Hyperparameter tuning and training include a lot of model specific black magic. Transformers have had time to mature, it'll take a while for other stuff to catch up even if they have a higher potential ceiling.

Another interesting one is that the hardware isn't really optimised for Mamba yet either - ideally we'd want more of the fast SRAM so that we can store more larger hidden states efficiently

Re: Mamba Explained: The State Space Model Taking On Transformers

#48

Post by pixl97 »

behnamoh wrote: post #32
landryraccoon wrote: post #21

Earlier quoted context omitted.

Why would Meta, Microsoft, Amazon and Google want Nvidia to remain dominant in hardware? Are you treating “big tech” like they all have one hive mind?

For MSFT, AMZN, GOOG, the competitive advantage comes from having huge datasets (that Nvidia doesn't have). It's a symbiosis that benefits the data-rich and GPU-rich players.

This still makes little sense as that scale will always matter. If you can drop the compute cost of a model by 10x it means you can increase model integrity/intelligence/speed etc beyond what your compute bound competitors have.

Simply put, for the time being huge datasets are going to be needed and those with bigger (cleaner?) datasets will have a better behaving model.

Re: Mamba Explained: The State Space Model Taking On Transformers

#49

Post by koayon »

thecolorgreen wrote: post #5

Why doesn't Equation 1b use the h' defined in Equation 1a?

Hey! OP here Great question - h' in Equation 1a refers to the derivative of h with respect to time (t). This is a differential equation which we can solve mathematically when we have x in order to get a closed-form solution for h. We would then plug in that h (the hidden state) into equation 1b.

In our case, we don't actually wait for a closed-form solution but instead compute the discrete representation (Equation 2)

Hope that helps!

Post reply on HN