Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

Post by andruby » Wed, Jul 23, 2025, 1:36 PM UTC

stingraycharles wrote: post #40

aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

So, does that mean that in general for the most modern high end LLM tools, to generate ~1500 tokens per seconds you need around $500k in hardware? Checking: Anthropic charges $70 per 1 million output tokens. @1500 tokens per second that would be around 10 cents per second, or around $8k per day. The $500k sounds about right then, unless I’m mistaken.

Post by simonw » Wed, Jul 23, 2025, 1:44 PM UTC

mehdibl wrote: post #8

It seem this news is "outdated" as it's from Jul 8 and might picked up confusing this model with yesterday Qwen 3 coder 405B release that is different in specs.

Post by adamtaylor_13 » Wed, Jul 23, 2025, 1:45 PM UTC

HumanOstrich wrote: post #59

throwawaymaths wrote: post #47

Earlier quoted context omitted.

nah, it was designed for hpc and raw flops. llm inference really requires memory bandwidth.

Memory bandwidth, eh? You should learn the basics about what Cerebras does. https://www.cerebras.ai/chip

Post by jscheel » Wed, Jul 23, 2025, 1:47 PM UTC

mohsen1 wrote: post #16

K2 is also now available on Groq https://console.groq.com/docs/model/moonshotai/kimi-k2-instr... very fun to see agents using those backends

Post by irthomasthomas » Wed, Jul 23, 2025, 1:58 PM UTC

nhecker wrote: post #56

smcleod wrote: post #43

Earlier quoted context omitted.

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

> dynamically assign precision to the layers that need them Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describin…