Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

cerebras.ai

Post reply on HN
Print view View original

141–150 of 160 posts

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#141

Post by westurner »

> Qwen3-235B uses an efficient mixture-of-experts architecture that delivers exceptional compute efficiency, enabling Cerebras to offer the model at $0.60 per million input tokens and $1.20 per million output tokens—less than one-tenth the cost of comparable closed-source models. 

  $ 0.60/million input tokens
  $ 1.20/million output tokens
How many minutes of 4K YouTube HDR video is that equivalent to in kWh of energy usage?

> Concurrent with this launch, Cerebras has quadrupled its context length support from 32K to 131K tokens—the maximum supported by Qwen3-235B.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#142

Post by h3lp »

xtracto wrote: post #75
llm_nerd wrote: post #57

Earlier quoted context omitted.

Small bit of pedantry: While there are 1 and 2-bit quantized types used in some aggressive schemes, they aren't floating point so it's inaccurate to preface them with FP. They are int types. The smallest real floating point type is FP4. EDIT: Who knew that correctness is controversial. What a weird place HN has become.

I wonder if the fact that we use "floating point" is itself a bottleneck that can be improved. Remembering my CS classes, storing an FP value requires the base and the exponent; that's a design decision. Also remembering some assembler classes, Int arithmetic is way faster than FP. Could there be a better "representation " for the numbers needed in NN that would provide the accuracy of floating point but provide fast…

A common FP4 layout is 1 sign bit, 3 exponent bits, 0 mantissa bits. There's just not that much difference in complexity between that and a 4-bit integer---the ALU can just be a simple table lookup, for both FP and integer.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#143

Post by zamadatix »

nisten wrote: post #76

"Full 131k" context , actually the full context is double that at 262144 context and with 8x yarn mutiplier it can go up to 2million. It looks like even full chip scale Cerebras has trouble with context length, well, this is a limitation of the transformer architechture itself where memory requirements scale ~linearly and compute requirements roughly quadratically with the increase in kv cache. Anyway, YOU'RE NOT SER…

The model page says 32,768 natively with performance validated for up to 4x YaRN https://huggingface.co/Qwen/Qwen3-235B-A22B#processing-long-...

That would seem to align with the 131k number?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#144

Post by smcleod »

vlovich123 wrote: post #130
smcleod wrote: post #43

Earlier quoted context omitted.

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

What quantization formats are these? All the OSS ones from GGML apply a uniform quantization

No they don't. GGML is non-uniform. Each layer gets its own level of quantisation.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#145

Post by smcleod »

nhecker wrote: post #56
smcleod wrote: post #43

Earlier quoted context omitted.

There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.

> dynamically assign precision to the layers that need them Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describin…

Within the GGUF (and some other formats) you'll see each layer gets its own quantisation, for example embeddings layers are usually more sensitive to quantisation and as such are often kept at Q8 or FP16. If you run GGUF-dump or click on the GGUF icon on a model in huggingface you'll see.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#146

Post by sneilan1 »

So I installed litellm proxy, pointed it at the new Cerebras API with Qwen-235B and hooked up Aider to litellm. This is not as good as claude code yet but it's so much faster. I even tried using the leaked claude code prompt into Aider but it doesn't do what I expect. Still worth trying but I learned that claude code's prompt is very specific to claude. I think this is very promising however! Aider basically spat out a bunch of text, installed some stuff, made some web calls & exited. WAS REALLY FAST LOL.

you can repeat my experiment quickly with the following>

config.yaml for litellm ``` model_list: - model_name: qwen3-235b litellm_params: model: cerebras/qwen-3-235b-a22b api_key: os.environ/CEREBRAS_API_KEY api_base: https://api.cerebras.ai/v1 ```

run litellm with ``` litellm --config config.yaml --port 4000 --debug ``` (may need to install litellm[proxy])

start aider with ``` aider --model cerebras/qwen-3-235b-a22b --openai-api-base http://localhost:4000 --openai-api-key fake-key --no-show-model-warnings --auto-commits --system-file ./prompt.txt --yes ```

install whatever you need with pip etc. prompt.txt contains the leaked claude code prompt which you can find yourself on the internet.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#147

Post by ahmadyan »

Adfeldman wrote: post #99
aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

Our chips don't cost $3M. I'm not sure where you got that number but its wildly incorrect.

Congrats on Qwen3 launch, also ty for the exploration tier. Makes our life a lot easier.

Any plan/ETA on launching it's big-brother (Qwen3-code)?

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#148

Post by rbanffy »

lionkor wrote: post #115
rbanffy wrote: post #96

Earlier quoted context omitted.

> start throwing away constraints and convincing themselves of things that are tangential or opposite to the task Funny that, when given too much brainpower, AIs manifest ADHD symptoms…

Validating if you have ADHD, but still an issue that is somehow glanced over by everyone who uses AI daily(?)

I don't feel validated, and I don't feel better in any way because of this.

This will all end up in tears.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#149

Post by recursivecaveat »

aurareturn wrote: post #19

If this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's no…

The metric of run/not-run is too simplistic. You have to divide out the total throughout the system gives to all concurrent users (which we don't know). Like a golf-cart can get you from New York to LA same as a train, but the unit economics of the train are a lot more favorable, despite its increased cost. The minimum deployment scale is not irrelevant, it may make it infeasible to run an on-prem solution for most customers for eg, but if you are selling tokens via a big cloud API it doesn't really matter.

Re: Cerebras launches Qwen3-235B, achieving 1.5k tokens per second

#150

Post by bredren »

sneilan1 wrote: post #146

So I installed litellm proxy, pointed it at the new Cerebras API with Qwen-235B and hooked up Aider to litellm. This is not as good as claude code yet but it's so much faster. I even tried using the leaked claude code prompt into Aider but it doesn't do what I expect. Still worth trying but I learned that claude code's prompt is very specific to claude. I think this is very promising however! Aider basically spat out…

Thanks for the report. Can this be hooked up to claude code via a proxy?
Post reply on HN