Mamba: The Easy Way
jackcook.com
Mamba: The Easy Way
1–10 of 63 posts
Re: Mamba: The Easy Way
#2Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up.
https://github.com/mamba-org/mamba
Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has not happened yet
Re: Mamba: The Easy Way
#3Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up. https://github.com/mamba-org/mamba Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has…
> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
Re: Mamba: The Easy Way
#4Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up. https://github.com/mamba-org/mamba Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has…
I use the former and have been experimenting with the latter. Fortunately, the contexts are separate enough that they never come up in the same sentence.
Re: Mamba: The Easy Way
#5From what I can tell all the large players in the space are continuing developing on transformers right? Is it just that Mamba is too new, or is the architecture fundamentally not usable for some reason?
Re: Mamba: The Easy Way
#6Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up. https://github.com/mamba-org/mamba Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has…
I use the former and have been experimenting with the latter. Fortunately, the contexts are separate enough that they never come up in the same sentence.
I was using mamba to install mamba the other day, when suddenly I had to run for a live mamba.
Re: Mamba: The Easy Way
#7From what I can tell all the large players in the space are continuing developing on transformers right? Is it just that Mamba is too new, or is the architecture fundamentally not usable for some reason?
Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers.
There are some papers suggesting that transformers are better than SSMs in fundamental ways (e.g. They cannot do arbitrary key-based recall from their context: https://arxiv.org/abs/2402.01032). This means it's not just a no-brainer to switch over.
Re: Mamba: The Easy Way
#8Very annoying namespace conflict since a package called "mamba" (faster reimplementation of the python conda package manager) already existed for awhile before this architecture was even dreamed up. https://github.com/mamba-org/mamba Beyond that, I'll care about an alternative to transformers when it shows superior performance with an open source 7b-34b model compared to transformer model competitors. So far this has…
The mamba paper shows significant improvements in all model sizes, up to 1b, the largest one tested.
Are there any reason why it wouldn't scale to 7b or more? Have they tried it?
Re: Mamba: The Easy Way
#9Fantastic blog post, thank you for this. I am not even familiar with transformers, yet the explanation is stellar clear to me, and the included references and context are a trasure trove. The explanation of FlashAttention is the best I have seen, and that is not even the focus of the article.
One question I have on selectivity: footnote 4 says "the continuous A is constant, while our discretization parameter ∆ is input-dependent." What is the effect of varying the discretization instead of the (main, as I understand it) state A? My gut says it simplifies training and provides stability, but I feel A carries most of the behavior of the model, so it should have more wiggle room throughout training.
Re: Mamba: The Easy Way
#10Earlier quoted context omitted.
I use the former and have been experimenting with the latter. Fortunately, the contexts are separate enough that they never come up in the same sentence.
I was using mamba to install mamba the other day, when suddenly I had to run for a live mamba.
While chewing a Mamba?