Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

cerebras.ai

Post reply on HN
Print view View original

131–140 of 160 posts

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#131

Post by mikewarot »

It's quite possible this is getting near the upper limit that is possible with current architectures for compute. Let's say the limit were 10k tokens/second with Qwen3-235B.

There's always going to be some latency in any compute architecture. Assume some insane billionaire cast the entire Qwen3-235B model into silicon, so it all ran in parallel, tokens going in one end, and the next token coming out the other end. This wafer (or likely, stack of interconnected wafers) would likely add up to a latency from end to end of 10 to 100 milliseconds.

If you then added pipelining, the latency might actually increase a millsecond or two, but the aggregate throughput would be N times the number of pipeline stages.

If you could increase N to the point that the clock cycle were a nanosecond... what would the economic value of this thing be? 100,000 separate streams at 10,000 tokens per second, multiplexing through it.

If you change it from cast in silicon, to a program to configure the silicon (line an FPGA, but far less clunky), I believe you get the future of LLM compute. Ever faster and wider lanes between compute and RAM is a dead end, a premature optimization.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#132

Post by cgdl »

aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

Exactly what I was thinking.

What sort of latency do you think one would get with 8x B200 Blackwell chips? Do you think 1500 tokens/sec would be achievable in that setup?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#133

Post by cgdl »

Adfeldman wrote: post #99
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

Our chips don't cost $3M. I'm not sure where you got that number but its wildly incorrect.

Do you distinguish betwen "chips" and the wafer-scale system? Is the wafer-scale system significantly less than 3MM?

EDIT: online it seems TSMC prices are about 25K-30K per wafer. So even 10Xing that a wafer-scale system should be about 300K.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#134

Post by int_19h »

vlovich123 wrote: post #130
smcleod wrote: post #43

Earlier quoted context omitted.

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

What quantization formats are these? All the OSS ones from GGML apply a uniform quantization

GGML hasn't been a thing for some time, and GGUF (its successor) has features such as "importance matrix" quantization that is all about quantizing adaptively. Then there's all the stuff that Unsloth does, e.g.: https://unsloth.ai/blog/dynamic-v2

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#135

Post by cgdl »

nroets wrote: post #125
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

1500 tokens/s is 5.4 million per hour. According to the document it costs $1.20 x 5.4 = $6.48 per hour. Which is not enough to even pay the interest on one $3m chip. What am I missing here ?

Indeed, and even if the cost per wafer was 300K, since about say 20-50 wafers are needed, its still 6MM to 15MM for the system. So likely it would appear this is VC subsidized.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#136

Post by cgdl »

throwawaymaths wrote: post #47
pjs_ wrote: post #18

Cerebras is truly one of the maddest technical accomplishments that Silicon Valley has produced in the last decade or so. I met Andy seven or eight years ago and I thought they must have been smoking something - a dinner plate sized chip with six tons of clamping force? They made it real, and in retrospect what they did was incredibly prescient

nah, it was designed for hpc and raw flops. llm inference really requires memory bandwidth.

I'd say llm inference requires both memory capacity and bandwidth. Cerebras provides bandwidth with on-chip SRAM, but not capacity (an entire wafer has only 44GB SRAM).

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#137

Post by p0w3n3d »

I'm looking for a setup for local development with local qwen on my macbook. I tried localforge with mlx_lm.server but it failed to communicate (I saw a proof of concept on their page but now it seems to fail on "empty response" which in reality is not empty)

Anyone could recommend a solution?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#138

Post by zhobbs »

p0w3n3d wrote: post #137

I'm looking for a setup for local development with local qwen on my macbook. I tried localforge with mlx_lm.server but it failed to communicate (I saw a proof of concept on their page but now it seems to fail on "empty response" which in reality is not empty) Anyone could recommend a solution?

Possible I'm misunderstanding what you're trying to do, but ollama works well for me for local inference with qwen on my Macbook Pro (32GB).

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#139

Post by nateb2022 »

zhobbs wrote: post #138
p0w3n3d wrote: post #137

I'm looking for a setup for local development with local qwen on my macbook. I tried localforge with mlx_lm.server but it failed to communicate (I saw a proof of concept on their page but now it seems to fail on "empty response" which in reality is not empty) Anyone could recommend a solution?

Possible I'm misunderstanding what you're trying to do, but ollama works well for me for local inference with qwen on my Macbook Pro (32GB).

Yup, also using Ollama and on a Macbook Pro. Ollama is #1

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#140

Post by vadepaysa »

While the speeds are great, in my experience with Cerebras, its really hard to get any actual production level rate limits or token quantity allocations. We cannot design systems around them and we use other vendors.

We've spoken to their sales teams, and we've been told no.

Post reply on HN