Mamba-3 • Hacker News Board

Re: Mamba-3

#41

Post by joefourier » Sat, Mar 21, 2026, 2:26 PM UTC

gyrovagueGeist wrote: post #18

cubefox wrote: post #7

Earlier quoted context omitted.

Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.

Not wrong, but I think it's more accurate to say: Mamba is an architecture for the middle layers of the network (the trunk) which assumes decoding takes place through an autoregressive sequence (popping out tokens in order). This is the SSM they talk about. Diffusion is an alternative to the autoregressive approach where decoding takes place through iterative refinement on a batch of tokens (instead of one at a time…

Mamba doesn't assume auto-regressive decoding, and you can use absolutely use it for diffusion, or pretty much any other common objective. Same with a conventional transformer. For a discrete diffusion language model, the output head is essentially the same as an autoregressive one. But yes, the training/objective/inference setup is different.

Re: Mamba-3

#42

Post by notnullorvoid » Sat, Mar 21, 2026, 2:26 PM UTC

jychang wrote: post #17

I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…

Focusing on needs of providers isn't a very good long term strategy if you believe compute will eventually move to self hosted and on premises solutions where large batch sizes aren't needed.

Re: Mamba-3

#43

Post by zozbot234 » Sat, Mar 21, 2026, 2:30 PM UTC

jychang wrote: post #23

zozbot234 wrote: post #19

Earlier quoted context omitted.

> Everyone groups all the requests into a batch, and the GPU computes them together. You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV…

Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only…

> Well, the actual inference providers put each expert on its own single GPU.

Experts are usually chosen on a per-layer basis, not just by token, so I'd think this requires having lots of GPU's to make it worthwhile. You could do it with a single physical GPU by switching expert-layer mixes in a round-robin fashion after the batch for any single expert-layer mix is completed (essentially a refined version of expert offloading). But still, not easy.

Re: Mamba-3

#44

Post by sroussey » Sat, Mar 21, 2026, 2:51 PM UTC

jychang wrote: post #17

I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…

Local has a batch size of 1. If you are already memory bound then you leave compute on the table. Why not use it?

Not sure they target local though…

Re: Mamba-3

#45

Post by anentropic » Sat, Mar 21, 2026, 3:27 PM UTC

More here https://news.ycombinator.com/item?id=47423208

https://arxiv.org/abs/2603.15569

Re: Mamba-3

#46

Post by jeffhwang » Sat, Mar 21, 2026, 3:56 PM UTC

I'm glad I clicked through bc I thought the article was about Mamba, the package manager I associate with Python (similar to conda).

https://github.com/mamba-org/mamba

Re: Mamba-3

#47

Post by 0xbadcafebee » Sat, Mar 21, 2026, 5:03 PM UTC

Havoc wrote: post #28

Is there a reason we don’t switch halfway through? ie start with a classic LLM and switch to something linear like mamba as context grows

They did do that, 2 years ago. The problems are that 1) mamba makes accuracy worse as context size grows, 2) Nvidia GPUs are designed for transformers, and 3) all the software out there is also designed for transformers. It's still useful in some applications but it doesn't beat regular transformers if you have the gear

Re: Mamba-3

#48

Post by i000 » Sat, Mar 21, 2026, 5:46 PM UTC

robofanatic wrote: post #2

> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…

Agreed. What you wrote was probably the input, what we see is the LLM output with the directive to "make us sound smart, put gratuitous em-dash"

Re: Mamba-3

#49

Post by manlymuppet » Sat, Mar 21, 2026, 5:56 PM UTC

I'm looking forward to the fifth iteration of this model.

Re: Mamba-3

#50

Post by roger_ » Sat, Mar 21, 2026, 5:58 PM UTC

Can anyone explain why Mamba models start with a continuous time SSM (and discretize) vs discrete time?

I know the step isn’t fixed, also not sure why that’s important. Is that the only reason? There also seems to be a parameterization advantage too with the continuous formulation.