Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mamba-3

together.ai

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Mamba-3

#11

Post by qeternity »

arendtio wrote: post #8
robofanatic wrote: post #2

> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…

I don't get the downvotes, as I had trouble understanding the intro as well. It seems it was written for a very specific audience.

Yes, it is written for a specific audience.

That is not a reason for snark.

As other commenters have noted, it’s well written.

Re: Mamba-3

#12

Post by oersted »

robofanatic wrote: post #6
E-Reverance wrote: post #3

Earlier quoted context omitted.

The first sentence basically does though, no?

Of course my only objection was the language. LLMs are now old enough to leave the jargon behind and talk in simple easy to understand terms.

I’d argue the opposite, the terminology is fairly mainstream by now and “inference” has a much more specific sense than “making predictions”.

Re: Mamba-3

#13

Post by jychang »

nl wrote: post #4

I'm looking forward to comparing this to Inception 2 (the text diffusion model) which in my experience is very fast and reasonably high quality.

That's completely different. That's like saying you want to compare the Nvidia 5090 GPU to the latest Call of Duty.

Re: Mamba-3

#14

Post by cubefox »

esquire_900 wrote: post #5
robofanatic wrote: post #2

> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…

This is sort of what their first sentence states? Except your line implies that they are fast in training and inference, they imply they are focusing on inference and are dropping training speed for it. It's a nice opening as it is imo

They don't say anything about dropping training speed.

Re: Mamba-3

#15

Post by magicalhippo »

arendtio wrote: post #8
robofanatic wrote: post #2

> Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal — a departure from Mamba-2, which optimized for training speed. The key upgrades are a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Why can’t they simply say - Mamba-3 focuses on being faster and more…

I don't get the downvotes, as I had trouble understanding the intro as well. It seems it was written for a very specific audience.

> I don't get the downvotes

Because the blog post is a technical one and the intro contains very common jargon, and the proposed alternative was wrong.

Re: Mamba-3

#17

Post by jychang »

I'm not sure that I buy their conclusion that more compute during inference is good.

Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together.

With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the batch, at the same time.

If they increase the amount of compute required per token, that just reduces the maximum batch size a GPU can handle. In practice, yes this does mean each GPU can serve less users. Providers aren't leaving GPU cores idle normally during inference.

Re: Mamba-3

#18

Post by gyrovagueGeist »

cubefox wrote: post #7
nl wrote: post #4

I'm looking forward to comparing this to Inception 2 (the text diffusion model) which in my experience is very fast and reasonably high quality.

Mamba-3 is an architecture while diffusion is, I believe, a type of objective. So these are not mutually exclusive and therefore not comparable.

Not wrong, but I think it's more accurate to say:

Mamba is an architecture for the middle layers of the network (the trunk) which assumes decoding takes place through an autoregressive sequence (popping out tokens in order). This is the SSM they talk about.

Diffusion is an alternative to the autoregressive approach where decoding takes place through iterative refinement on a batch of tokens (instead of one at a time processing and locking each one in only looking forward). This can require different architectures for the trunk, the output heads, and modifications to the objective to make the whole thing trainable. Could mamba like ideas be useful in diffusion networks...maybe but it's a different problem setup.

Re: Mamba-3

#19

Post by zozbot234 »

jychang wrote: post #17

I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…

> Everyone groups all the requests into a batch, and the GPU computes them together.

You're only saving on fetching read-only parameters, and not even on that if you're using MoE models where each inference in the batch might require a different expert (unless you rearrange batches so that sharing experts becomes more likely, but that's difficult since experts change per-token or even per-layer). Everything else - KV-cache, activations - gets multiplied by your batch size. You scale both compute and memory pressure by largely the same amount. Yes, GPUs are great at hiding memory fetch latency, but that applies also to n=1 inference.

Re: Mamba-3

#20

Post by yorwba »

jychang wrote: post #17

I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the requests into a batch, and the GPU computes them together. With a fused kernel, that means the GPU streams the tensors from VRAM, and does a bunch of compute on different conversations in the…

Their latency measurements comparing Mamba-2 and Mamba-3 are done with a batch size of 128. It doesn't seem like Mamba-2 was compute-bound even at that batch size.
Post reply on HN