Viewing profile — koayon

And given that the compute is O(n^2) with context window, it's a very real tradeoff, at least in the short term

This is a very fair point! If we had infinite compute then it's undeniable that transformers (i.e. full attention) would be better (exactly as you characterise it) But that's the e…

Hey! OP here Great question - h' in Equation 1a refers to the derivative of h with respect to time (t). This is a differential equation which we can solve mathematically when we ha…

Another interesting one is that the hardware isn't really optimised for Mamba yet either - ideally we'd want more of the fast SRAM so that we can store more larger hidden states ef…

Definitely agree that a lot of work going into hyperparameter tuning and maturing the ecosystem will be key here! I'm seeing the Mamba paper as the `Attention Is All You Need` of M…