Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

31–40 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#31

Post by nbardy »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

No these things just take time.

There is no conspiracy again efficient training. Companies aren’t going to lower compute budgets with more efficiency.

All the top labs are increasing efficiency, but they are using that to get more out of their large runs not spend less. Most companies have a relatively fixed training budget for their large runs and are trying to get the most out of it, bot save money,

Mamba is actually being scaled up and tested across other fields(bio) at a rapid pace compared to other architectures

Re: Mamba Explained: The State Space Model Taking On Transformers

#32

Post by behnamoh »

landryraccoon wrote: post #21
behnamoh wrote: post #16

Earlier quoted context omitted.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

Why would Meta, Microsoft, Amazon and Google want Nvidia to remain dominant in hardware? Are you treating “big tech” like they all have one hive mind?

For MSFT, AMZN, GOOG, the competitive advantage comes from having huge datasets (that Nvidia doesn't have). It's a symbiosis that benefits the data-rich and GPU-rich players.

Re: Mamba Explained: The State Space Model Taking On Transformers

#33

Post by godelski »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

Yes and no.

The thing is that Mamba is not perfect. There's no neural architecture to rule them all, if you will. I think the bigger issue is that we more act like there is and get on bandwagons. Let me give a clearer example from the past so we can see. The predecessor to DDPM (the work that kicked off the diffusion model era) was published in 2015[0], only a year after GANs[1]. Diffusion then showed promise but why did DDPM come out in 2020[2]? Because everyone was working on GANs. GANs produced far better images and diffusion (still is) was a lot more computationally intensive. Plus, all the people working on these diffusion models were in the same camp as those working on Normalizing Flows and other density based models, and fewer people are interested in understanding density estimation.

So the entire problem is that the community hopped on a singular railroad for research direction. There was still working going on in that direction but it wasn't nearly getting the attention that GANs got. It's hard to know if things were blocked from publication because they weren't as good as GANs. I can say from personal experience I had a Flow publication blocked because reviewers were concerned with its quality compared to GANs (this was 2019/2020, this paper will never be published because now it is even hard to publish a GAN work).

So yes and no because there is certainly railroading happening but there are also real critiques to Mamba. But what people often forget is that it is incredibly naive to compare new methods to existing methods on a direct one-to-one comparison. You're comparing something that has hundreds of hours to thousands of hours from a handful to a few dozen eyes against works with millions of hours and millions of eyes. Evaluation is just a really fucking hard thing to do but it is easy to just look at some benchmarks, even if they don't mean much. This is a fairly generalization notion though, so take the lesson to heart. But Mamba seems a bit different than our diffusion/GAN story, in that it is getting more attention than diffusion did in the 2016-2019.

[0] https://arxiv.org/abs/1503.03585

[1] https://arxiv.org/abs/1406.2661

[2] https://arxiv.org/abs/2006.11239

Re: Mamba Explained: The State Space Model Taking On Transformers

#34

Post by nyrikki »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

The fact that 'removing the “quadratic bottleneck”' involves either reduced expressability compared to self attention or disproving SETH is another reason.

The quadratic bottleneck is due to the lower bounds of exhaustive search.

The papers on this only ever seem to reference perplexity.

The fact it can append a word to "I'm going to the beach" that sounds good doesn't mean it is useful.

There is no free lunch, and this project hasn't shown that the costs are acceptable.

"I'm going to the beach" + house

Doesn't help if what you needed was

"I'm going to the beach" + tomorrow

I do hope that there is more information on the costs, or that they have disproven SETH soon.

Re: Mamba Explained: The State Space Model Taking On Transformers

#35

Post by CuriouslyC »

fbdab103 wrote: post #30
behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

It is a really recent development. Even if this architecture is technically superior, it could take time before a model using it becomes competitive. Or maybe it does not pan out at all. We are still at the stage where people are throwing everything at the wall to see what sticks. Some promising ideas which work at small scale do not work at bigger.

This. Hyperparameter tuning and training include a lot of model specific black magic. Transformers have had time to mature, it'll take a while for other stuff to catch up even if they have a higher potential ceiling.

Re: Mamba Explained: The State Space Model Taking On Transformers

#36

Post by godelski »

kettleballroll wrote: post #20
behnamoh wrote: post #16

Earlier quoted context omitted.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it…

As a member of the research community: that's nonsense. Publishing is an extremely noisy process in ML and is getting increasingly difficult for smaller non big tech collaborating labs. Reviewers' go to are: more datasets, scale, not novel. The easiest way to approach this is to work off of pretrained models. This is probably more obvious in the NLP world.

I agree that Mamba doesn't solve everything and it still needs work. But I disagree with the logic that there isn't an issue of railroading.

Re: Mamba Explained: The State Space Model Taking On Transformers

#37

Post by godelski »

nbardy wrote: post #31
behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

No these things just take time. There is no conspiracy again efficient training. Companies aren’t going to lower compute budgets with more efficiency. All the top labs are increasing efficiency, but they are using that to get more out of their large runs not spend less. Most companies have a relatively fixed training budget for their large runs and are trying to get the most out of it, bot save money, Mamba is actual…

> There is no conspiracy

Fwiw, the OP isn't suggesting conspiracy. The notion is more about convergent thinking.

Re: Mamba Explained: The State Space Model Taking On Transformers

#38

Post by CuriouslyC »

Der_Einzige wrote: post #19
CrypticShift wrote: post #10

> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively. [1] https://news.ycombinator.com/item?id=39414532

Whoever is downvoting this post needs to stop. The concepts behind control vectors, i.e. "representation engineering" are not especially new and have been highly effective in the diffusion space. I always find it entertaining when LLM folks act like they're discovering stuff that waifu stable diffusion folks knew for 6 months + about - like "concept slider loras".

You are right that playing with AI image generation models is really good for building intuition about AI models in general, even if they seem superficially different. It's kind of like surveying a battlefield from the air.

Re: Mamba Explained: The State Space Model Taking On Transformers

#39

Post by soVeryTired »

nyrikki wrote: post #34
behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

The fact that 'removing the “quadratic bottleneck”' involves either reduced expressability compared to self attention or disproving SETH is another reason. The quadratic bottleneck is due to the lower bounds of exhaustive search. The papers on this only ever seem to reference perplexity. The fact it can append a word to "I'm going to the beach" that sounds good doesn't mean it is useful. There is no free lunch, and t…

What's SETH in this context? I googled to no avail.
Post reply on HN