Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

81–90 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#81

Post by anon291 »

behnamoh wrote: post #16
p1esk wrote: post #14

Earlier quoted context omitted.

This doesn’t make sense - there are literally thousands of academic AI research labs who are severely limited by compute resources. If anything could work better than transformers and require less compute they would be all over that.

I guess the argument is that most AI research is supported by the big tech, and they have heavily invested in the deep learning approach. If the fundings were funneled to research groups working on alternative approaches, maybe we'd see the same amount of progress in AI only using another approach.

As someone who's worked at several NVIDIA competitors, including Groq, I can guarantee you that, based on my knowledge, of existing products, they would be able to make much more money should they have lower memory footprint models. Given the amount of VC capital deployed for this (on the order of 100s of millions), I don't believe this is a reasonable take.

Sure, NVIDIA et al may not want that (although, again I don't see why... they too can't produce chips fast enough so being able to provide models for customers now ought to be good), but there's so much money out there that does...

Re: Mamba Explained: The State Space Model Taking On Transformers

#82

Post by frozenseven »

godelski wrote: post #52
p1esk wrote: post #41

Earlier quoted context omitted.

What’s the main difference between an ape’s brain and a human brain? Scale. So that’s the train we’re riding at the moment. No roadblocks yet, aside from cost.

> What’s the main difference between an ape’s brain and a human brain? Scale. This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture. There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total su…

Scary how someone can be so confident in their wrong information.

> An African Elephant has roughly 3x the number of neurons humans have.

An African elephant's brain is not a scaled up chimp brain in any way. African elephants have less cortical neurons than a chimp, and roughly a third of the amount that humans have.

> Dolphins beat humans in total surface area.

Animals even less related to humans and chimps, with no prehensile appendages, living in an environment where building stuff is exceedingly difficult. And of course their brains are obviously different from any great ape.

> Neanderthals are estimated to have had larger brains too!

And were just as smart as us and also had a scaled up chimp brain.

Re: Mamba Explained: The State Space Model Taking On Transformers

#83

Post by nelsondev »

binarymax wrote: post #59
Straw wrote: post #53

The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…

I hadn’t heard of Mamba before reading this article, and I was wondering if anyone has tried setting importance of a token as a TF-IDF or BM25 lookup. Requires a first pass to construct the token index but otherwise it seems like it would address the big issue that all these architectures have - they don’t know how “important” a token is. Interestingly this seems to be the crux of Mamba - deciding what tokens to forg…

Not exactly related, but in the same vein - Deep Impact - deep learning to find term impacts in the context of their document.

https://arxiv.org/abs/2104.12016

Re: Mamba Explained: The State Space Model Taking On Transformers

#84

Post by Straw »

logicchains wrote: post #56
Straw wrote: post #53

The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…

Are there any fundamental differences between Mamba, Retnet and RWKV, or are they all variants of this same architecture?

No, all of these use the same fundamental architecture with minor tweaks, such as the dynamic gate for mamba or an outer product paramterization of the values for RWKV-v5

Re: Mamba Explained: The State Space Model Taking On Transformers

#86

Post by pama »

Straw wrote: post #84
logicchains wrote: post #56

Earlier quoted context omitted.

Are there any fundamental differences between Mamba, Retnet and RWKV, or are they all variants of this same architecture?

No, all of these use the same fundamental architecture with minor tweaks, such as the dynamic gate for mamba or an outer product paramterization of the values for RWKV-v5

A dynamic gate is a pretty distinct feature from previous SSM architectures in my opinion. In a sense, the overall fundamental architecture of mamba is still that of the transformer but with attention replaced by an SSM with dynamic gating. All of deep learning uses closely related ideas, but the SSM class of models took advantage of stability guarantees from integrators in control theory and created a class of RNN that don’t have to worry about exploding gradients. Mamba is one of the ways to make these SSM models much more expressive.

Re: Mamba Explained: The State Space Model Taking On Transformers

#87

Post by littlestymaar »

curious_cat_163 wrote: post #75
binarymax wrote: post #70

Earlier quoted context omitted.

Attention in transformers works because over time the model learns token importance based on frequency and context. If you don’t have attention and need a fast substitute for “forgetting” non important tokens, then BM25 is an intuitive hypothesis.

To use your metaphor, TF-IDF will result in ‘fixed’ weights. Attention makes it so that the weights of each token can be different in each sequence of tokens. Same token gets different weights depending on who its ‘neighbors’ in the sequence end up being. This property allows the models to solve a variety of natural language problems and gets ‘used’ by the model to express context-aware dependencies.

Given that GP explicitly said “if you don't have attention”, and we're in a thread about a language model whose main characteristics is not to use attention, I don't understand why you insist in talking about attention …

Re: Mamba Explained: The State Space Model Taking On Transformers

#88

Post by godelski »

frozenseven wrote: post #82
godelski wrote: post #52

Earlier quoted context omitted.

> What’s the main difference between an ape’s brain and a human brain? Scale. This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture. There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total su…

Scary how someone can be so confident in their wrong information. > An African Elephant has roughly 3x the number of neurons humans have. An African elephant's brain is not a scaled up chimp brain in any way. African elephants have less cortical neurons than a chimp, and roughly a third of the amount that humans have. > Dolphins beat humans in total surface area. Animals even less related to humans and chimps, with n…

animal :: cortical neurons (b) :: total neurons (b)

Human :: 16 :: 86

Gorilla :: 9.1 :: 33

Chimp :: 6 :: 22

African Elephant :: 5.6 :: 251

Chimps are generally considered more intelligent than gorillas.

Bottlenose Dolphins have 11-15b cortical neurons while humans are in the range 14-18 (range is measurement uncertainty). It's also worth noting these dolphins have a larger brain mass (1.6 kg) and larger cortical surface (3700 cm2) than humans (1.3 kg and 2400 cm2, respectively).

> with no prehensile appendages, living in...

So more than scale. Glad we agree. Seems you also agree that architecture matters too.

Re: Mamba Explained: The State Space Model Taking On Transformers

#89

Post by curious_cat_163 »

littlestymaar wrote: post #87
curious_cat_163 wrote: post #75

Earlier quoted context omitted.

To use your metaphor, TF-IDF will result in ‘fixed’ weights. Attention makes it so that the weights of each token can be different in each sequence of tokens. Same token gets different weights depending on who its ‘neighbors’ in the sequence end up being. This property allows the models to solve a variety of natural language problems and gets ‘used’ by the model to express context-aware dependencies.

Given that GP explicitly said “if you don't have attention”, and we're in a thread about a language model whose main characteristics is not to use attention, I don't understand why you insist in talking about attention …

I mean, if we are going to get past attention (very much on board with the idea!), then it might help to know what it is really contributing to a model.

My response was trying to clarify some confusion.

I am all for alternatives to attention. I don’t think BM25 cuts it. I don’t think anything that samples tokens based on BM25 weights (the idea in this subthread) would cut it.

Re: Mamba Explained: The State Space Model Taking On Transformers

#90

Post by fauigerzigerk »

coldtea wrote: post #79
fauigerzigerk wrote: post #51

Earlier quoted context omitted.

Where is the symbiosis? If data is the differentiator, how do the data owners benefit from Nvidia eating into their margins?

It's sumbiosis based on a common factor they both appreciate: - the data/processing having to be large means the data-owners have a benefit - the data/processing having to be large means NVIDIA also has a benefit (sells more GPUs to handle all that load)

Data owners are benefitting from having access to data while others don't.

High processing cost is not a benefit to them at all. It's just a cost eating into their margins.

Post reply on HN