Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

11–20 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#11

Post by refulgentis »

CrypticShift wrote: post #10

> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively. [1] https://news.ycombinator.com/item?id=39414532

Can control vectors replace RAG?

i.e. if I want the model to give me a summary of the news today, and the model was trained before today, can control vectors help?

Re: Mamba Explained: The State Space Model Taking On Transformers

#12

Post by behnamoh »

Can the low adoption of Mamba be attributed to what is being discussed today on HN (https://news.ycombinator.com/item?id=39491863)?

Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute.

Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them the most as they are the only players in town with huge amounts of data. If the AI community would find entirely different approaches to AGI (maybe not even learning), who do you think would suffer the most from the implications?

Re: Mamba Explained: The State Space Model Taking On Transformers

#13

Post by p1esk »

refulgentis wrote: post #11
CrypticShift wrote: post #10

> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively. [1] https://news.ycombinator.com/item?id=39414532

Can control vectors replace RAG? i.e. if I want the model to give me a summary of the news today, and the model was trained before today, can control vectors help?

No technique can get you the news other than actually searching for and then parsing the published news.

Re: Mamba Explained: The State Space Model Taking On Transformers

#14

Post by p1esk »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

Re: Mamba Explained: The State Space Model Taking On Transformers

#16

Post by behnamoh »

p1esk wrote: post #14
behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach.

If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

Re: Mamba Explained: The State Space Model Taking On Transformers

#18

Post by jahewson »

behnamoh wrote: post #12

Can the low adoption of Mamba be attributed to what is being discussed today on HN ( https://news.ycombinator.com/item?id=39491863 )? Basically, Nvidia et al. don't want the AI research to move in a direction that requires less GPU compute, less training data, and less inference compute. Someone on HN (I don't remember the name) mentioned that the idea of deep learning is backed by big tech because it benefits them t…

This early in the game? No. If LLMs become vastly cheaper and faster then adoption (and model size) will increase in line with that.

Re: Mamba Explained: The State Space Model Taking On Transformers

#19

Post by Der_Einzige »

CrypticShift wrote: post #10

> In other words, you can drag and drop downloaded states into your model, like literal plug-in cartridges The same could be said of "control vectors" [1]. Both ideas are still experimental, but is seems to me IINM that they could replace "system prompts" and "RAG" respectively. [1] https://news.ycombinator.com/item?id=39414532

Whoever is downvoting this post needs to stop.

The concepts behind control vectors, i.e. "representation engineering" are not especially new and have been highly effective in the diffusion space. I always find it entertaining when LLM folks act like they're discovering stuff that waifu stable diffusion folks knew for 6 months + about - like "concept slider loras".

Re: Mamba Explained: The State Space Model Taking On Transformers

#20

Post by kettleballroll »

behnamoh wrote: post #16
p1esk wrote: post #14

Earlier quoted context omitted.

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it isn't the immediate transformer killer. It remains to be seen if something comes from it, eventually.
Post reply on HN