This is really promising. Are they now going to scale this up to hundreds of billions of parameters? Why stop at 1.5B if they found a potentially SOTA architecture?
Mamba-3
51–59 of 59 posts
Re: Mamba-3
#52Earlier quoted context omitted.
Why would execs need latency? I can see it for engineering - coding with slow ai is painful
Impatient execs can also be painful. EDIT: Writing this while I am waiting for Codex to complete, so I may enjoy slow AI more than the usual developer ;-)
The economic effect of latency is measured not by the incremental productivity itself, but by the combined economic downforce of thousands of resulting HN and Reddit comments :)
Re: Mamba-3
#53This is really promising. Are they now going to scale this up to hundreds of billions of parameters? Why stop at 1.5B if they found a potentially SOTA architecture?
Probably constrained by training resources. It's much easier to experiment with a smaller architecture. You may need many training runs to figure out hyperparameters for example. If each run needs multiple GPUs for a week the cost adds up quickly. I think it makes a lot of sense to start small.
Re: Mamba-3
#54Re: Mamba-3
#55Earlier quoted context omitted.
Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only…
> Well, the actual inference providers put each expert on its own single GPU. Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert o…
Correct, but note that's exactly what inference providers do.
https://arxiv.org/pdf/2412.19437
> The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320.
EP320 means expert parallelism, each on 320 GPUs.
Re: Mamba-3
#56I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…
Focusing on needs of providers isn't a very good long term strategy if you believe compute will eventually move to self hosted and on premises solutions where large batch sizes aren't needed.
That's a foolish take.
That's like gamers thinking most of Nvidia's revenue coming from gaming GPUs, so Nvidia should prioritize gamers.
Inference is ruled by inference providers, not local. Local inference is a rounding error, and will remain as such unless there is economic incentive otherwise.
Re: Mamba-3
#57Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows
Because something linear like Mamba doesn't perform as well; so you'd have a performance cliff, where suddenly the model would get more dumb and forget a lot of what was going on. Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of perform…
I am curious of the tradeoff of hybrid approaches, it sounds too good to be true.
Re: Mamba-3
#58I'm looking forward to the fifth iteration of this model.
Mamboooo no. #5
Re: Mamba-3
#59Earlier quoted context omitted.
Because something linear like Mamba doesn't perform as well; so you'd have a performance cliff, where suddenly the model would get more dumb and forget a lot of what was going on. Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of perform…
I am curious of the tradeoff of hybrid approaches, it sounds too good to be true.
It mostly trades some potential performance loss for speed, especially at longer contexts.
Nemotron 3 Super doesn't perform quite as well on benchmarks as the similarly sized Qwen3.5 122B A10B model, but it goes faster and is cheaper to run.
https://artificialanalysis.ai/?models=gpt-oss-120b%2Cmistral...
Now, you're not exactly comparing apples to apples there, since the training process (mix of data for pre-training, and the fine tuning stages of instruction turning, RLVR, etc) could have as much or more impact on how well it does as the architecture itself. Nemotron 3 Super does get better scores on performance than GPT-OSS 120B and Mistral Small 4, both also similarly sized open weights models.