Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained: The State Space Model Taking On Transformers

kolaayonrinde.com

Post reply on HN
Print view View original

61–70 of 101 posts

Re: Mamba Explained: The State Space Model Taking On Transformers

#61

Post by godelski »

p1esk wrote: post #55
godelski wrote: post #52

Earlier quoted context omitted.

> What’s the main difference between an ape’s brain and a human brain? Scale. This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture. There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total su…

Other animals with larger brains might have other bottlenecks preventing them from reaching full potential of their intelligence. Neanderthals might have been smarter than us, but went extinct for reasons not related to intelligence. But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.

> Other animals with larger brains might have other bottlenecks

>>> What’s the main difference between an ape’s brain and a human brain? Scale.

Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.

Re: Mamba Explained: The State Space Model Taking On Transformers

#62

Post by halflings »

binarymax wrote: post #59
Straw wrote: post #53

The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…

I hadn’t heard of Mamba before reading this article, and I was wondering if anyone has tried setting importance of a token as a TF-IDF or BM25 lookup. Requires a first pass to construct the token index but otherwise it seems like it would address the big issue that all these architectures have - they don’t know how “important” a token is. Interestingly this seems to be the crux of Mamba - deciding what tokens to forg…

The importance (e.g. attention) needs to be dynamic, e.g. one token will be important to some other tokens but not others.

tf-idf and similar heuristics are what we were using before attention came along, e.g. tf-idf weighted bag-of-words representation of word2vec embeddings. That approaches fails in so many cases.

Re: Mamba Explained: The State Space Model Taking On Transformers

#63

Post by p1esk »

godelski wrote: post #61
p1esk wrote: post #55

Earlier quoted context omitted.

Other animals with larger brains might have other bottlenecks preventing them from reaching full potential of their intelligence. Neanderthals might have been smarter than us, but went extinct for reasons not related to intelligence. But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.

> Other animals with larger brains might have other bottlenecks >>> What’s the main difference between an ape’s brain and a human brain? Scale. Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.

Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from.

When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.

Re: Mamba Explained: The State Space Model Taking On Transformers

#64

Post by godelski »

p1esk wrote: post #63
godelski wrote: post #61

Earlier quoted context omitted.

> Other animals with larger brains might have other bottlenecks >>> What’s the main difference between an ape’s brain and a human brain? Scale. Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.

Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from. When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.

I see no evidence of this in biology nor in ML. I've read those scale papers. I've worked on scale myself. I'll bet the farm that scale isn't all you need. But I won't be surprised if people say that it is all scale.

If you really think it is all scale, train a 7T ResNet MLP based model for NLP. If scale is all you need, make a LLM without DPO or RLHF. If scale is all you need, make SD3 with a GAN. Or what about a VAE, Normalizing Flow, HMM? Do it with different optimizers. Do it with gradient free methods. Do it with different loss functions.

The bitter lesson wasn't "scale is all you need." That's just a misinterpretation.

Edit: It's fine to disagree. We can compete on the ideas and methods. That's good for our community. So continue down yours, and I'll continue down mine. All I ask is that since your camp is more popular, you don't block those on my side. If you're right, we'll get to AGI soon. If we're right, we still might. But if we're right and you all block us, we'll get another AI winter in between. If we're right and you all don't block us, we can progress without skipping a beat. Just don't put all your eggs in one basket. It isn't good for anyone.

Re: Mamba Explained: The State Space Model Taking On Transformers

#65

Post by [deleted] »

Straw wrote: post #53

The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy. SSMs are doing exponentially weighted moving averages (EMA). That's it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount…

[deleted]

Re: Mamba Explained: The State Space Model Taking On Transformers

#67

Post by p1esk »

godelski wrote: post #64
p1esk wrote: post #63

Earlier quoted context omitted.

Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from. When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.

I see no evidence of this in biology nor in ML. I've read those scale papers. I've worked on scale myself. I'll bet the farm that scale isn't all you need. But I won't be surprised if people say that it is all scale. If you really think it is all scale, train a 7T ResNet MLP based model for NLP. If scale is all you need, make a LLM without DPO or RLHF. If scale is all you need, make SD3 with a GAN. Or what about a VA…

I said “scale is all you need for transformers”. That has been true since GPT1. The best way to improve our best model today still seems to be “make it larger and train it on more data”.

If you disagree please suggest a better way, or at least provide evidence that scaling up no longer works for transformers.

Re: Mamba Explained: The State Space Model Taking On Transformers

#69

Post by sja »

imjonse wrote: post #28

Explaining Mamba is a rite of passage, like the monad tutorials of yore.

Or Balks[0]:

BALK RULES! IMPORTANT! 1. You can’t just be up there and just doin’ a balk like that.

1a. A balk is when you

1b. Okay well listen. A balk is when you balk the

1c. Let me start over

1c-a. The pitcher is not allowed to do a motion to the, uh, batter, that prohibits the batter from doing, you know, just trying to hit the ball. You can’t do that.

1c-b. Once the pitcher is in the stretch, he can’t be over here and say to the runner, like, “I’m gonna get ya! I’m gonna tag you out! You better watch your butt!” and then just be like he didn’t even do that.

1c-b(1). Like, if you’re about to pitch and then don’t pitch, you have to still pitch. You cannot not pitch. Does that make any sense?

1c-b(2). You gotta be, throwing motion of the ball, and then, until you just throw it.

1c-b(2)-a. Okay, well, you can have the ball up here, like this, but then there’s the balk you gotta think about.

1c-b(2)-b. Fairuza Balk hasn’t been in any movies in forever. I hope she wasn’t typecast as that racist lady in American History X.

1c-b(2)-b(i). Oh wait, she was in The Waterboy too! That would be even worse.

1c-b(2)-b(ii). “get in mah bellah” – Adam Water, “The Waterboy.” Haha, classic…

1c-b(3). Okay seriously though. A balk is when the pitcher makes a movement that, as determined by, when you do a move involving the baseball and field of

2. Do not do a balk please.

[0]: https://justinbee.tumblr.com/post/15309101943/best-explanati...

Re: Mamba Explained: The State Space Model Taking On Transformers

#70

Post by binarymax »

halflings wrote: post #62
binarymax wrote: post #59

Earlier quoted context omitted.

I hadn’t heard of Mamba before reading this article, and I was wondering if anyone has tried setting importance of a token as a TF-IDF or BM25 lookup. Requires a first pass to construct the token index but otherwise it seems like it would address the big issue that all these architectures have - they don’t know how “important” a token is. Interestingly this seems to be the crux of Mamba - deciding what tokens to forg…

The importance (e.g. attention) needs to be dynamic, e.g. one token will be important to some other tokens but not others. tf-idf and similar heuristics are what we were using before attention came along, e.g. tf-idf weighted bag-of-words representation of word2vec embeddings. That approaches fails in so many cases.

Attention in transformers works because over time the model learns token importance based on frequency and context.

If you don’t have attention and need a fast substitute for “forgetting” non important tokens, then BM25 is an intuitive hypothesis.

Post reply on HN