Mamba: The Easy Way • Hacker News Board

Re: Mamba: The Easy Way

#61

Post by hansonw » Sat, Feb 24, 2024, 8:51 AM UTC

shikon7 wrote: post #58

hansonw wrote: post #34

Earlier quoted context omitted.

“RNN-mode inference” is also extremely exciting because you can precompute the hidden state of any prompt prefix (i.e. a long system prompt, or statically retrieved context) and continued generations pay the same cost irrespective of the prefix length.

But this also means that possible information retained is constant irrespective of the prefix length. This might be a problem if the prefix is composed of essentially uncompressable data.

Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!

Re: Mamba: The Easy Way

#62

Post by cztomsik » Sun, Feb 25, 2024, 7:13 AM UTC

> Importantly, these recurrent and convolutional forms, which I like to call “RNN mode” and “CNN mode,” are mathematically equivalent. This allows S4 to shape-shift depending on what you need it to do, with no difference in its outputs.

Is this really true? Because it seems to ignore hardware and data type precisions entirely. I mean computing same math thing in a different way with floating points often leads to different results.

Re: Mamba: The Easy Way

#63

Post by cztomsik » Sun, Feb 25, 2024, 7:18 AM UTC

3abiton wrote: post #59

sigmoid10 wrote: post #47

Earlier quoted context omitted.

The architecture is completely public. I would be surprised if certain other players (including but not limited to Mistral AI) are not training models yet. We'll hear soon enough if this is viable. Maybe not for official release candidates, but at least for internal testing.

Nonetheless, this is extremely exciting, unlike RWKV and Retention Network

Why? From what I read those architectures have many similarities (and same weaknesses)