31–40 of 59 posts
Earlier quoted context omitted.
Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.
I mean I guess but the diffusion objective and the ability to do simultaneous decode both dictate pretty different architectures in practice.
I'm looking forward to comparing this to Inception 2 (the text diffusion model) which in my experience is very fast and reasonably high quality.
That's completely different. That's like saying you want to compare the Nvidia 5090 GPU to the latest Call of Duty.
Earlier quoted context omitted.
Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.
Not wrong, but I think it's more accurate to say: Mamba is an architecture for the middle layers of the network (the trunk) which assumes decoding takes place through an autoregressive sequence (popping out tokens in order). This is the SSM they talk about. Diffusion is an alternative to the autoregressive approach where decoding takes place through iterative refinement on a batch of tokens (instead of one at a time…
Earlier quoted context omitted.
They don't say anything about dropping training speed.
> a departure from Mamba-2, which optimized for training speed. ?
Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…
I don’t know why you’re being downvoted. As a longtime editor your version is immensely better. Looks like the original was probably not human-written.
Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of performance, as the linear context mechanism can work for the tasks its well suited for while allowing attention to play to its strengths.
The recent Nemotron 3 Nano and Super models from NVIDIA are hybrid architectures this way, with most of their context layers as Mamba while retaining enough attention to continue to be competitive on the more complex tasks that require the quadratic attention.
See https://magazine.sebastianraschka.com/i/168650848/18-nemotro... for some discussion on this architecture