Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba Explained

thegradient.pub

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Mamba Explained

#41

Post by jimmySixDOF »

fisian wrote: post #37
password4321 wrote: post #6

Links to more about Mamba (selective state space models) on HN yesterday: https://news.ycombinator.com/item?id=39853958#39855430

This submission has the same content as the link here (submitted to HN about a month ago): https://news.ycombinator.com/item?id=39501982 https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html

Yes and its the same author this time published on the Gradient (the link before was to the personal blog). The Gradient by the way are amazing curators of AI news in general and have one of the better podcasts I am aware of interviewing developers in the trenches.

Adding: this resurgence in Mamba in general is also due to some actual sota progress with SSM like the new AI21 lab released this week [1] and likely to see others merging different architecture layers (this is a 52B MoE with 12B params active during inference blending both Mamba and transformers)

>As the first production-grade model based on Mamba architecture, Jamba achieves an unprecedented 3X throughput and fits 140K context on a single GPU.

[1] https://www.ai21.com/jamba

Re: Mamba Explained

#42

Post by moffkalast »

aCoreyJ wrote: post #11
andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term

Well consumer hardware can run something in the order of ~50B quantized at a "reasonable" price today, we'd need about 5 or 6 doublings to run something that would be GPT 4 tier at 1T+. So, it would need to continue for roughly a decade at least?

Current models are horrendously inefficient though, so with architectural improvements we'll have something of that capability far sooner on weaker hardware.

Re: Mamba Explained

#43

Post by Semionilo »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

I don't think it's weird or broken to think and compare on what LLM do vs what our brain do.

It shows more than not that we are also parrots

Re: Mamba Explained

#44

Post by mannykannot »

rdedev wrote: post #36
anon291 wrote: post #30

Earlier quoted context omitted.

Yes transformers are obviously more capable than humans in my opinion. Claude can ingest dozens of pages in seconds and -- in a single shot -- write a summary bringing in relevant passages. The innovation is not the speed, but the lack of recursion or iteration. Humans, even accomplished ones, have to reread sections and really 'internalize' ideas before being able to summarize and very few humans can -- in a single…

I view transformers as like the language center of the brain. When we write or speak, especially when it's critical to get things right, we have this ability to think "that doesn't make sense" and start over. I view this recursion as more of a strength than weakness. You can get an LLM to generate an answer and when asked about the validity of the answer it would acknowledge that it got it wrong. This begs the questi…

I agree, and also, when I'm writing, I am working towards a hierarchy of goals at the level of sentence, paragraph and beyond, and I'm also wondering if what I have written and plan to write could be confusing or misunderstood.

I think it's fair to ask whether these are essential techniques for improving precision and clarity, or just a way to compensate for not being able to see the whole picture all at once - but if the latter is the case, there's still room for improvement in LLMs (and me, for that matter.) I notice that experts on a topic are often able to pick out what matters most without any apparent hesitation.

Re: Mamba Explained

#45

Post by scarmig »

andy_xor_andrew wrote: post #5

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like tra…

> Lately I've been wondering... is this a problem, or a strength?

It probably depends. But an idea I've been playing with: because transformers have such a strong ability for recall during inference, they might be introducing a strong inductive bias for memorization as opposed to generalization. Why bother to build a complete world model when you can just attend to the answer? The global minimum in loss (at least for the training dataset) would use those memorizing and interpolating circuits over those that generalize well. This seems consistent with LLMs as they exist today: superhuman at recall, very mediocre at reasoning. Though, for what it's worth, existing SSSMs haven't yet shown they can outperform (or even match) transformers when it comes to reasoning.

If this hypothesis were true, you might expect to see grokking in state space models more quickly than in transformer models.

(Even if it's hard to train transformers to generalize, superhuman recall is still incredibly valuable, and likely a hybrid system would offer the best of both worlds.)

Re: Mamba Explained

#46

Post by ein0p »

This is more human like, and people will complain that it doesn’t have photographic memory. That is, it’s not superhuman in that regard. But there are many tasks where superhuman recall is not required. We know this because those tasks are currently performed by humans.

Re: Mamba Explained

#47

Post by anon291 »

rdedev wrote: post #36
anon291 wrote: post #30

Earlier quoted context omitted.

Yes transformers are obviously more capable than humans in my opinion. Claude can ingest dozens of pages in seconds and -- in a single shot -- write a summary bringing in relevant passages. The innovation is not the speed, but the lack of recursion or iteration. Humans, even accomplished ones, have to reread sections and really 'internalize' ideas before being able to summarize and very few humans can -- in a single…

I view transformers as like the language center of the brain. When we write or speak, especially when it's critical to get things right, we have this ability to think "that doesn't make sense" and start over. I view this recursion as more of a strength than weakness. You can get an LLM to generate an answer and when asked about the validity of the answer it would acknowledge that it got it wrong. This begs the questi…

> I view this recursion as more of a strength than weakness

Sure, it's a strength given that transformers are currently limited by compute budget, but theoretically, if we were to have a way to overcome this, it seems obvious to me that transformer's 'one-shot' ability makes them better.

That being said the recursive aspect you're referencing can be built into a transformer as well. This is a sampling and training problem.

Post reply on HN