Bamba: An open-source LLM that crosses a transformer with an SSM

Post by tmalsburg2 » Wed, Apr 30, 2025, 10:13 AM UTC

quantadev wrote: post #21

jmward01 wrote: post #13

This type of architecture is definitely the future. Unlimited attn is a dead end. As a human you don't need to scan an entire book just to guess what the next word will be and LLMs shouldn't need that either.

Not be contrarian, but if the next word prediction happens to be someone's name or a place or something discussed multiple places in the book then often, yes, a knowledge of the full plot of the book is "required" just to predict the next word, as you get to the middle or end of a book. For example you could never fill in the last chapter of any good book without having knowledge of every previous chapter. Not highly…

Post by cubefox » Wed, Apr 30, 2025, 10:22 AM UTC

anentropic wrote: post #60

> they added another trillion tokens and shrank the model from 18 GB to 9 GB through quantization, reducing its bit width from Mamba2’s 16-bit floating-point precision to 8-bits. This sounds like what they call "Bamba-9B" is actually an 18B model quantised to 8 bits. I thought generally we were naming models "nB" by their number of params and treating quantisation as a separate concern. Are there any other models tha…

Post by famouswaffles » Wed, Apr 30, 2025, 12:50 PM UTC

tmalsburg2 wrote: post #61

quantadev wrote: post #21

Earlier quoted context omitted.

Not be contrarian, but if the next word prediction happens to be someone's name or a place or something discussed multiple places in the book then often, yes, a knowledge of the full plot of the book is "required" just to predict the next word, as you get to the middle or end of a book. For example you could never fill in the last chapter of any good book without having knowledge of every previous chapter. Not highly…

Isn't this exactly the point of this model? No need to memorize everything (which makes transfomers expensive), just keep the relevant info. SSM are essentially recurrent models.

Post by tmalsburg2 » Wed, Apr 30, 2025, 6:40 PM UTC

famouswaffles wrote: post #65

tmalsburg2 wrote: post #61

Earlier quoted context omitted.

Isn't this exactly the point of this model? No need to memorize everything (which makes transfomers expensive), just keep the relevant info. SSM are essentially recurrent models.

You can't always know what will be "relevant info" in the future. Even humans can't do this but whenever that's an issue, we just go back and re-read, re-watch etc. None of these modern recurrent architecture have a way to do this.

Post by boroboro4 » Wed, Apr 30, 2025, 7:46 PM UTC

littlestymaar wrote: post #57

boroboro4 wrote: post #49

Earlier quoted context omitted.

I think in a way it makes transformers superior to humans, their short term memory is much more powerful =) Supporting extra long contexts also make transformers super human. Because, again, human's short term memory is exactly this - short term. And much shorter than millions of tokens we expect from models nowadays. As for SSMs - I think they compress model memory state way too much. Mixed global/local attention la…

> And much shorter than millions of tokens we expect from models nowadays. Yet all current model still suck above 32k. (Yes some can do needle in a haystack fine, but they still fail at anything even slightly more complex over a long context). 32k is still much higher than humans' though, so I agree with you that it gives them some kind of super human abilities over moderately long context, but they are still disappo…