Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

Post by aitchnyu » Wed, Jul 23, 2025, 2:00 PM UTC

simonw wrote: post #63

mehdibl wrote: post #8

It seem this news is "outdated" as it's from Jul 8 and might picked up confusing this model with yesterday Qwen 3 coder 405B release that is different in specs.

I initially thought this was about the Qwen release from two days ago, Qwen3-235B-A22B-Instruct-2507 - https://simonwillison.net/2025/Jul/22/qwen3-235b-a22b-instru... - but that's a no-reasoning model and the Cerebras announcement talks about reasoning, which tipped me off that this was Qwen's Qwen3-235B-A22B from April. (These model names are so confusing.)

Post by clownpenis_fart » Wed, Jul 23, 2025, 2:10 PM UTC

irthomasthomas wrote: post #69

nhecker wrote: post #56

Earlier quoted context omitted.

> dynamically assign precision to the layers that need them Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describin…

Last I checked they ran some sort of evals before and after quantisation and measured the effect. E.g Exllama-v2 measures the loss while reciting Wikipedia articles.

Post by xtracto » Wed, Jul 23, 2025, 2:13 PM UTC

llm_nerd wrote: post #57

stingraycharles wrote: post #54

Earlier quoted context omitted.

Isn’t usually mixed? I understood that Apple even uses fp1 or fp2 on their hardware embedded models they ship on their phones, but as far as I know it’s typically a whole bunch of different precisions.

Small bit of pedantry: While there are 1 and 2-bit quantized types used in some aggressive schemes, they aren't floating point so it's inaccurate to preface them with FP. They are int types. The smallest real floating point type is FP4. EDIT: Who knew that correctness is controversial. What a weird place HN has become.

Post by xtracto » Wed, Jul 23, 2025, 2:14 PM UTC

xtracto wrote: post #75

llm_nerd wrote: post #57

Earlier quoted context omitted.

Small bit of pedantry: While there are 1 and 2-bit quantized types used in some aggressive schemes, they aren't floating point so it's inaccurate to preface them with FP. They are int types. The smallest real floating point type is FP4. EDIT: Who knew that correctness is controversial. What a weird place HN has become.

I wonder if the fact that we use "floating point" is itself a bottleneck that can be improved. Remembering my CS classes, storing an FP value requires the base and the exponent; that's a design decision. Also remembering some assembler classes, Int arithmetic is way faster than FP. Could there be a better "representation " for the numbers needed in NN that would provide the accuracy of floating point but provide fast…

Post by iyerbalaji » Wed, Jul 23, 2025, 2:19 PM UTC