Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#91

Post by frozenseven »

godelski wrote: post #88
frozenseven wrote: post #82

Earlier quoted context omitted.

Scary how someone can be so confident in their wrong information. > An African Elephant has roughly 3x the number of neurons humans have. An African elephant's brain is not a scaled up chimp brain in any way. African elephants have less cortical neurons than a chimp, and roughly a third of the amount that humans have. > Dolphins beat humans in total surface area. Animals even less related to humans and chimps, with n…

animal :: cortical neurons (b) :: total neurons (b) Human :: 16 :: 86 Gorilla :: 9.1 :: 33 Chimp :: 6 :: 22 African Elephant :: 5.6 :: 251 Chimps are generally considered more intelligent than gorillas. Bottlenose Dolphins have 11-15b cortical neurons while humans are in the range 14-18 (range is measurement uncertainty). It's also worth noting these dolphins have a larger brain mass (1.6 kg) and larger cortical surf…

> Chimps are generally considered more intelligent than gorillas.

And chimps are genetically more similar to humans than gorillas. A chimp brain is more similar to a human brain than a gorilla brain.

> So more than scale. Glad we agree.

We absolutely do not agree. Notice how nobody suggested that a human brain is a scaled up version of an axolotl's brain? Yeah, that didn't happen. I do wonder why?

Re: Mamba Explained: The State Space Model Taking On Transformers

#93

Post by torginus »

Straw wrote: post #53

The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…

Is this analogous to digital filters, where Transformers are the FIR filters that operate on the history of input, and IIR filters, which take past inputs into account with an exponentially decaying importance?

Re: Mamba Explained: The State Space Model Taking On Transformers

#94

Post by binarymax »

curious_cat_163 wrote: post #89
littlestymaar wrote: post #87

Earlier quoted context omitted.

Given that GP explicitly said “if you don't have attention”, and we're in a thread about a language model whose main characteristics is not to use attention, I don't understand why you insist in talking about attention …

I mean, if we are going to get past attention (very much on board with the idea!), then it might help to know what it is really contributing to a model. My response was trying to clarify some confusion. I am all for alternatives to attention. I don’t think BM25 cuts it. I don’t think anything that samples tokens based on BM25 weights (the idea in this subthread) would cut it.

What confusion? I know exactly how BM25 works and how Transformers work. I stated a hypothesis and asked if anyone has tried it. You say it won’t work. That’s just your opinion. Do you have proof or evidence? This is science. Dismissal of ideas without evidence goes against scientific principles.

Re: Mamba Explained: The State Space Model Taking On Transformers

#95

Post by hackerlight »

kgeist wrote: post #85

If Mamba selectively forgets "unnecessary" details, can it repeat the input verbatim (if asked)?

If you ask at the end of the prompt then it may have already deliberately tossed the information it deemed irrelevant prior to the question. These aren't transformers. In general the recall for arbitrary information will be worse.

Re: Mamba Explained: The State Space Model Taking On Transformers

#97

Post by kgeist »

hackerlight wrote: post #95
kgeist wrote: post #85

If Mamba selectively forgets "unnecessary" details, can it repeat the input verbatim (if asked)?

If you ask at the end of the prompt then it may have already deliberately tossed the information it deemed irrelevant prior to the question. These aren't transformers. In general the recall for arbitrary information will be worse.

So the questions should come before the content and it might work?

I think that's how also RWKV works.

Re: Mamba Explained: The State Space Model Taking On Transformers

#98

Post by nickpsecurity »

Der_Einzige wrote: post #3

First it was longformer, and linear attention models. Then it was RWKV and now it's Mamba. So many bombastic claims of improved architectural performance - and no open source models that beat the thing they purport to beat. The proof is always in the pudding, and these models will remain a curiosity for most until their weights are being benchmarked favorably on LLM leaderboards.

GPT3-176B cost $30 million dollars in compute plus millions in design, preprocessing, and operations. Then, it was able to perform as much better than prior architectures as it does today. You might want to include that in your challenge for competing models.

Let’s rephrase it. If their architecture is superior, and they have $30 million dollars, and similar preparation for training, and similar operational teams during training, then we can see if they can beat the model they’re comparing themselves to. Except, the alternatives don’t have tens of millions of dollars with the best support teams. So, the proof you seek hasn’t had a chance to happen due to severe lack of resources.

Hence, comparisons to GPT2 and small versions of GPT3. Even that might not be fair given the money and teams behind even small GPT3’s. Execution of the project is as critical for success as the model architecture.

Re: Mamba Explained: The State Space Model Taking On Transformers

#99

Post by hackerlight »

kgeist wrote: post #97
hackerlight wrote: post #95

Earlier quoted context omitted.

If you ask at the end of the prompt then it may have already deliberately tossed the information it deemed irrelevant prior to the question. These aren't transformers. In general the recall for arbitrary information will be worse.

So the questions should come before the content and it might work? I think that's how also RWKV works.

It's known to help although I wouldn't expect it to be perfect recall unless the network is big enough.

The network will read the data token by token. So if you put the question at the beginning it will know what information it needs to pay attention to inside the rest of your context. Of course, if the network is too small, it still won't be perfect recall for a sufficiently complicated/large question/context.

Re: Mamba Explained: The State Space Model Taking On Transformers

#100

Post by Straw »

pama wrote: post #86
Straw wrote: post #84

Earlier quoted context omitted.

No, all of these use the same fundamental architecture with minor tweaks, such as the dynamic gate for mamba or an outer product paramterization of the values for RWKV-v5

A dynamic gate is a pretty distinct feature from previous SSM architectures in my opinion. In a sense, the overall fundamental architecture of mamba is still that of the transformer but with attention replaced by an SSM with dynamic gating. All of deep learning uses closely related ideas, but the SSM class of models took advantage of stability guarantees from integrators in control theory and created a class of RNN t…

Its distinct, but not very- its an EMA without assuming uniform time. The stability of EMA has nothing to do with integrators in control theory and neither do these models.

These models aren't really RNNs- they have only a linear gate which cannot depend on previous tokens at this layer, so they cant update their state in a way which depends on the current state very much.

Post reply on HN