Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

cerebras.ai

Post reply on HN
Print view View original

121–130 of 160 posts

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#121

Post by sailingparrot »

aurareturn wrote: post #108
sailingparrot wrote: post #105

Earlier quoted context omitted.

There is no way not to use SRAM on a GPU/Cerebras/most accelerators. This is where the cores fetch the data. But that doesn’t mean you are only using SRAM, that would be impractical. Just like using a CPU just by storing stuff in the L3 cache and never going to the RAM. Unless I am missing something from the original link, I don’t know how you got to the conclusion that they only used SRAM.

I know Groq chips load the entire model into SRAM. That's why it can be so fast. So if Cerebras uses HBM to store the model but stream weights into SRAM, I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. The whole point of having a wafer chip is that you limit the need to reach out to external parts for memory since that's the slow part.

> I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM.

I don’t think you can look at those things binarily. 44GB of SRAM is still a massive amount. You don’t need infinite SRAM to get better performances. There is a reason NVidia is increasing the L2 cache size with every generation rather than just sticking with 32MB if it really changed nothing to have a bit more. The more SRAM you have the more you are able to mask communication behind computation. You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#123

Post by makestuff »

aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

I agree there will be some breakthrough (maybe by Nvidia or maybe someone else) that allows these models to run insanely cheap and even locally on a laptop. I could see a hardware company coming out with some sort of specialized card that is just for consumer grade inference for common queries. That way the cloud can be used for sever side inference and training.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#124

Post by coolspot »

twothreeone wrote: post #118
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

I think you're missing an important aspect: how many users do you want to support? > For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. That would be enough to support a single user. If you want to host a service that provides this to 10k users in parallel your cost per user scales linearly with th…

> That would be enough to support a single user. If you want to host a service that provides this to 10k users in parallel your cost per user scales linearly with the GPU costs you posted.

No. Magic of batching allows you to handle multiple user requests in parallel using the same weights with little VRAM overhead per user.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#125

Post by nroets »

aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

1500 tokens/s is 5.4 million per hour. According to the document it costs $1.20 x 5.4 = $6.48 per hour.

Which is not enough to even pay the interest on one $3m chip.

What am I missing here ?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#126

Post by impossiblefork »

jszymborski wrote: post #60
doubtfuluser wrote: post #49

Very impressive speed. A bit OT: what is the current verdict on Qwen, Kimi et al. When it comes to censorship / bias concerning narratives not allowed in the origin country?

The Qwen models are, anecdotally, probably some of the best open weight models, particularly the MoE models. They are also, anecdotally, super scary censored. Asking it if anything "interesting has happened in Tianamen Square?" And then refining with "any notable protests?" And finally "maybe something to do with a tank"... All you get is vague allusions to the square being a beautiful place with a rich history.

Do you think it's done so carefully that you suspect that they have perhaps even removed texts mentioning the Tiananmen square massacre from the training set?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#127

Post by UltraSane »

Adfeldman wrote: post #99
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

Our chips don't cost $3M. I'm not sure where you got that number but its wildly incorrect.

Is it actually $4M?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#128

Post by IshKebab »

sailingparrot wrote: post #105
aurareturn wrote: post #102

Earlier quoted context omitted.

That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would get 16k GB200 aka ~600M$, obviously way off. Yes but Cerebras achieves its speed by using SRAM.

There is no way not to use SRAM on a GPU/Cerebras/most accelerators. This is where the cores fetch the data. But that doesn’t mean you are only using SRAM, that would be impractical. Just like using a CPU just by storing stuff in the L3 cache and never going to the RAM. Unless I am missing something from the original link, I don’t know how you got to the conclusion that they only used SRAM.

> Just like using a CPU just by storing stuff in the L3 cache and never going to the RAM. Unless I am missing something from the original link, I don’t know how you got to the conclusion that they only used SRAM.

That's exactly how Graphcore's current chips work, and I wouldn't be surprised if that's how Cerebras's wafer works. It's probably even harder for Cerebras to use DRAM because each chip in the wafer is "landlocked" and doesn't have an easy way to access the outside world. You could go up or down, but down is used for power input and up is used for cooling.

You're right it's not a good way to do things for memory hungry models like LLMs, but all of these chips were designed before it became obvious that LLMs are where the money is. Graphcore's next chip (if they are even still working on it) can access a mountain of DRAM with very high bandwidth. I imagine Cerebras will be working on that too. I wouldn't be surprised if the abandon WSI entirely due to needing to use DRAM.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#129

Post by vlovich123 »

sailingparrot wrote: post #121
aurareturn wrote: post #108

Earlier quoted context omitted.

I know Groq chips load the entire model into SRAM. That's why it can be so fast. So if Cerebras uses HBM to store the model but stream weights into SRAM, I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. The whole point of having a wafer chip is that you limit the need to reach out to external parts for memory since that's the slow part.

> I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM. I don’t think you can look at those things binarily. 44GB of SRAM is still a massive amount. You don’t need infinite SRAM to get better performances. There is a reason NVidia is increasing the L2 cache size with every generation rather than just sticking with 32MB if it really changed nothing to have a bit mor…

> You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).

You would have to have an insanely fast bus to prevent I/O stalls with this. With a 235B fp16 model you’d be streaming 470GiB of data every graph execution. To do that 1000tok/s, you’d need a bus that can deliver a sustained ~500 TiB/s. Even if you do a 32 wide MoE model, that’s still about 15 TiB/s of bandwidth you’d need from the HBM to avoid stalls at 1000tok/s.

It would seem like this either isn’t fp16 or this is indeed likely running completely out of SRAM.

Of course Cerebas doesn’t use a dense representation so these memory numbers could be way off and maybe that is SRAM+DRAM combo

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#130

Post by vlovich123 »

smcleod wrote: post #43
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

What quantization formats are these? All the OSS ones from GGML apply a uniform quantization
Post reply on HN