21–30 of 59 posts
Earlier quoted context omitted.
This is sort of what their first sentence states? Except your line implies that they are fast in training and inference, they imply they are focusing on inference and are dropping training speed for it. It's a nice opening as it is imo
They don't say anything about dropping training speed.
?
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…
Their latency measurements comparing Mamba-2 and Mamba-3 are done with a batch size of 128. It doesn't seem like Mamba-2 was compute-bound even at that batch size.
https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…
> Everyone groups all the requests into a batch, and the GPU computes them together. You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV…
Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is about 7GB at max context) and ssm/conv1d cache is even smaller- IIRC Qwen 3.5 is 146MB per token regardless of context size. Not sure about how Mamba-3 works, but I suspect read-only parameters are still a significant amount of memory bandwidth.
I guess the question isn't whether compute is 1:1 with memory, but rather if you run out of compute before you run out of vram adding more users.
> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…
Throughput is indeed king for the standard-tier mindshare-capture play. But there are many who would pay multiple times the current cost for agentic systems for engineers and executives, if it meant a meaningful reduction in latency. The economics could work extremely well.
I can see it for engineering - coding with slow ai is painful
I'm looking forward to comparing this to Inception 2 (the text diffusion model) which in my experience is very fast and reasonably high quality.
Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.
Earlier quoted context omitted.
Throughput is indeed king for the standard-tier mindshare-capture play. But there are many who would pay multiple times the current cost for agentic systems for engineers and executives, if it meant a meaningful reduction in latency. The economics could work extremely well.
Why would execs need latency? I can see it for engineering - coding with slow ai is painful