Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained

thegradient.pub

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Mamba Explained

#11

Post by aCoreyJ »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term

Re: Mamba Explained

#13

Post by etbebl »

Anyone else keep seeing articles about Mamba and thinking it's about Python/Conda? It's annoying when the new cool thing picks the same name as something else you like that deserves attention.

Re: Mamba Explained

#14

Post by sp332 »

So in an effective Mamba query the question goes at the end, after input data? I thought that the question should go at the beginning, so it can decide which information in the data is relevant.

Re: Mamba Explained

#15

Post by ragebol »

etbebl wrote: post #13

Anyone else keep seeing articles about Mamba and thinking it's about Python/Conda? It's annoying when the new cool thing picks the same name as something else you like that deserves attention.

> attention

I see what you did there

Re: Mamba Explained

#16

Post by y42 »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

>> Is this a problem or a strength?

I was wondering the same thing. I understand, why the initial developers of this method declared it as a strength. Still I think it's a problem, too:

If the Tranformer reads this sentence:

A equals B

It understands, that B comes after A and therefore A equals B. But how does it learn that after A comes B and therefore B equals A.

I am referring to the logical problems, that most (all?) modern language models suffer of.

Re: Mamba Explained

#17

Post by incrudible »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

> It's not like we're hitting a wall with quadratic attention. It's absurdly more expensive than SSMs, but GPUs certainly aren't getting slower.

We are not hitting a wall, but a slope. Hardware improvements will not make up for it indefinitely. Software will have to make up for it, but the problem is that it costs millions of dollars to hit compile.

Re: Mamba Explained

#18

Post by sigmoid10 »

y42 wrote: post #16
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

>> Is this a problem or a strength? I was wondering the same thing. I understand, why the initial developers of this method declared it as a strength. Still I think it's a problem, too: If the Tranformer reads this sentence: A equals B It understands, that B comes after A and therefore A equals B. But how does it learn that after A comes B and therefore B equals A. I am referring to the logical problems, that most (a…

I see many people get confused by this due to the widely spread (and false) "stochastic parrot" theme. But these models are much more than mere senzence-repeaters. In a way, the model is not learning that after A comes B. I mean, it could. With a lack of additional training data it probably would, too. But with enough data, this kind of sentence completion based purely on existing sentences no longer works because it would saturate parameters. So to retain and improve accuracy during training, it will have to come up with a compression that essentially forms a model of the real world. Or at least the world that the training corpus describes [1]. In that sense, it no longer "knows" that B comes after A (except for the input context), but it would have learned that there is a special relation between A and B. In can then also apply this kind of learned logic to new concepts that appear first in the context during inference. With all that happening internally, it only has to morph this state back into a natural language output. But with billions of parameters and countless layers, there is more than enough computational room for this to happen. In fact, recent models have shown that even small models can get pretty good at logic if you only get the training data right.

[1] https://arxiv.org/abs/2210.13382

Re: Mamba Explained

#19

Post by tippytippytango »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

It's a tradeoff to be managed depending on the application rather than a problem.

Re: Mamba Explained

#20

Post by timschmidt »

aCoreyJ wrote: post #11
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term

Ah, but we've just begun stacking transistors in the third dimension.
Post reply on HN