Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

cerebras.ai

Post reply on HN
Print view View original

51–60 of 160 posts

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#51

Post by imtringued »

sailingparrot wrote: post #37
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

> With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would ge…

Yeah sure, but if you do that you are heavily dropping the token/s for a single user. The only way to recover from that is continuous batching. This could still be interesting if the KV caches of all users fit in SRAM though.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#52

Post by scosman »

0vermorrow wrote: post #14

I'm eagerly awaiting for Qwen 3 coder being available on Cerebras. I run plenty of agent loops and the speed makes a somewhat interesting difference in time "compression". Having a Claude 4 Sonnet-level model running at 1000-1500 tok/s would be extremely impressive. To FEEL THE SPEED, you can either try it yourself on Cerebras Inference page, through their API, or for example on Mistral / Le Chat with their "Flash An…

Exactly. I can see my efficiency going up a ton with this kind of speed. Every time I'm waiting for agents my mind looses some focus and context. Running parallel agents gets more speed but at the cost of focus. Near instant iteration loops in Cursor would feel magical (even more magical?).

It will also impact how we work: interactive IDEs like Cursor probably make more sense than CLI tools like Claude code when answers are nearly instant.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#54

Post by stingraycharles »

lordofgibbons wrote: post #29
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

Almost everyone runs LLM inference at fp8 - for all of the open models anyway. You only see performance drop off below fp8.

Isn’t usually mixed? I understood that Apple even uses fp1 or fp2 on their hardware embedded models they ship on their phones, but as far as I know it’s typically a whole bunch of different precisions.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#55

Post by Zenst »

throwawaymaths wrote: post #44
sailingparrot wrote: post #37

Earlier quoted context omitted.

> With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would ge…

what are the bandwidth/latency of memoryX? those are the key parameters for inference

Well MemoryX compared to H100 HBM3 the key details are that MemoryX has lower latency, but also far lower bandwidth. However the memory on Cerebras is scales a lot more over NVidia. You need a cluster of H100's to create a model, as only way to scale the memory, Cerbras is more suited to that aspect, Nvidia do their scaling in tooling, with Cerbras doing theirs in design via there silicon approach.

That's my take on it all, not many apples to oranges comparisons to work from on these two system for even rolling down the same slope.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#56

Post by nhecker »

smcleod wrote: post #43
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

> dynamically assign precision to the layers that need them

Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describing?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#57

Post by llm_nerd »

stingraycharles wrote: post #54
lordofgibbons wrote: post #29

Earlier quoted context omitted.

Almost everyone runs LLM inference at fp8 - for all of the open models anyway. You only see performance drop off below fp8.

Isn’t usually mixed? I understood that Apple even uses fp1 or fp2 on their hardware embedded models they ship on their phones, but as far as I know it’s typically a whole bunch of different precisions.

Small bit of pedantry: While there are 1 and 2-bit quantized types used in some aggressive schemes, they aren't floating point so it's inaccurate to preface them with FP. They are int types.

The smallest real floating point type is FP4.

EDIT: Who knew that correctness is controversial. What a weird place HN has become.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#58

Post by sailingparrot »

imtringued wrote: post #51
sailingparrot wrote: post #37

Earlier quoted context omitted.

> With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. That's not how you would do it with Cerebras. 44GB is SRAM, so on chip memory, not HBM memory where you would store most of the params. For reference one GB200 has only 126MB of SRAM, if you tried to estimate how many GB200 you would need for a 2TB model just by looking at the L2 cache size you would ge…

Yeah sure, but if you do that you are heavily dropping the token/s for a single user. The only way to recover from that is continuous batching. This could still be interesting if the KV caches of all users fit in SRAM though.

> but if you do that you are heavily dropping the token/s for a single user.

I don’t follow what you are saying and what “that” is specifically. Assuming it’s referencing using HBM and not just SRAM, this is not optional on a GPU, SRAM is many order of magnitudes too small. Data is constantly flowing between HBM and SRAM by design, and to get data in/out of your GPU you have to go through HBM first, you can’t skip that.

And while it is quite massive on a Cerebras system it is also still too small for very large models.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#59

Post by HumanOstrich »

throwawaymaths wrote: post #47
pjs_ wrote: post #18

Cerebras is truly one of the maddest technical accomplishments that Silicon Valley has produced in the last decade or so. I met Andy seven or eight years ago and I thought they must have been smoking something - a dinner plate sized chip with six tons of clamping force? They made it real, and in retrospect what they did was incredibly prescient

nah, it was designed for hpc and raw flops. llm inference really requires memory bandwidth.

Memory bandwidth, eh? You should learn the basics about what Cerebras does. https://www.cerebras.ai/chip

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#60

Post by jszymborski »

doubtfuluser wrote: post #49

Very impressive speed. A bit OT: what is the current verdict on Qwen, Kimi et al. When it comes to censorship / bias concerning narratives not allowed in the origin country?

The Qwen models are, anecdotally, probably some of the best open weight models, particularly the MoE models.

They are also, anecdotally, super scary censored. Asking it if anything "interesting has happened in Tianamen Square?" And then refining with "any notable protests?" And finally "maybe something to do with a tank"... All you get is vague allusions to the square being a beautiful place with a rich history.

Post reply on HN