Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#21

Post by landryraccoon »

behnamoh wrote: post #16
p1esk wrote: post #14

Earlier quoted context omitted.

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

Why would Meta, Microsoft, Amazon and Google want Nvidia to remain dominant in hardware? Are you treating “big tech” like they all have one hive mind?

Re: Mamba Explained: The State Space Model Taking On Transformers

#22

Post by refulgentis »

Der_Einzige wrote: post #19
CrypticShift wrote: post #10

> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively. [1] https://news.ycombinator.com/item?id=39414532

Whoever is downvoting this post needs to stop. The concepts behind control vectors, i.e. "representation engineering" are not especially new and have been highly effective in the diffusion space. I always find it entertaining when LLM folks act like they're discovering stuff that waifu stable diffusion folks knew for 6 months + about - like "concept slider loras".

I don't know what you mean, can you help me?

I'm familiar with our intrepid stable diffusion sailors.

I don't know why you think the post is being downvoted.

I don't know why it would be verboten to downvote it, or indicative of the downvoter being an LLM fanatic who thinks they discovered everything.

I am puzzled by the post because it claims RAG can be replaced by control vectors.

I'm also puzzled because it claims prompts can be replaced by control vectors.

I get that if system prompts were only to shift output tone, control vectors could replace that case, but that seems narrow compared to the full set of things prompt input enables (inter alia, the in-context learning)

Re: Mamba Explained: The State Space Model Taking On Transformers

#23

Post by lamson »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

I don't really think big tech have that much control. They are aiming for optimal profit solution, thing likes AI monopoly with huge self-supervised learning just popped up recently when ChatGPT performs really well, a couple years ago, people still believed modular and supervised learning is the key to AI application. So simply current scaling deep learning/llm is most promising and it works while tradional methods don't. If there is something that works as good as current solution and requires less resource, they will go for it very fast, see the implementation of Flash attention as an example.

Re: Mamba Explained: The State Space Model Taking On Transformers

#24

Post by refulgentis »

p1esk wrote: post #13
refulgentis wrote: post #11

Earlier quoted context omitted.

Can control vectors replace RAG? i.e. if I want the model to give me a summary of the news today, and the model was trained before today, can control vectors help?

No technique can get you the news other than actually searching for and then parsing the published news.

Can a control vector replace system prompts?

i.e. can it do in-context learning without the context?

Re: Mamba Explained: The State Space Model Taking On Transformers

#25

Post by lettergram »

digdugdirk wrote: post #4
Der_Einzige wrote: post #3

First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.

Yes, that's technically accurate. But I prefer to think of the entire LLM space as a new scientific field that started when OpenAI released ChatGPT. In that context, all new research directions are valuable simply for the fact that they're expanding the foundation of the field. 5 years from now, who knows what the most effective models will use under the hood, but the more we can learn about them in general, the bett…

lol I think in general, LLM research traces its origins back to all the standard deep learning techniques: NNs, CNNs, LSTMs, RNNs, etc.

In 2018, with the release of transformers (via google) it enabled much more rapid training of models and more generalization with less data. 100% of the LLMs (as you’d probably thing of them)trace their origins to BERT.

That said, my team was working with hundred million to low billions of parameter LSTMs & CNNs back in 2016-2017 that were comparable to some lighter weight LLMs today.

In my opinion, the greatest strides in the space has less to do with the underlying architecture, and more to do with improved data formatting, accessibility and compute improvements.

Re: Mamba Explained: The State Space Model Taking On Transformers

#26

Post by imjonse »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

Low adoption is primarily caused by it being relatively recent and there are no 7B or larger public Mamba-based models to start a comparison in earnest with widely used transformer based LLMs.

Re: Mamba Explained: The State Space Model Taking On Transformers

#27

Post by imjonse »

Der_Einzige wrote: post #3

First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.

Most (all?) open-ish 7B+ models today are finetunes of proprietary/semi-closed/bigbudget LLMs. There is no such foundation model for Mamba yet.

Re: Mamba Explained: The State Space Model Taking On Transformers

#30

Post by fbdab103 »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

It is a really recent development. Even if this architecture is technically superior, it could take time before a model using it becomes competitive.

Or maybe it does not pan out at all. We are still at the stage where people are throwing everything at the wall to see what sticks. Some promising ideas which work at small scale do not work at bigger.

Post reply on HN