Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

121–130 of 239 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#121

Post by wild_egg »

Aurornis wrote: post #113
nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute.

I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time.

Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#122

Post by dd8601fn »

olivermuty wrote: post #62
nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

Cerebras the tech is awesome, cerebras the company is a trainwreck

Is this the chatjimmy asic approach with a bigger model?

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#123

Post by gerdesj »

devy wrote: post #119
conception wrote: post #116

Earlier quoted context omitted.

150k by account. At 1.5k a second you hit it very quickly.

Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.

I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware.

I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve.

To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#124

Post by amelius »

wild_egg wrote: post #121
Aurornis wrote: post #113

Earlier quoted context omitted.

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute. I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time. Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was s…

Can't you do something with multiple accounts?

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#125

Post by gerdesj »

Aurornis wrote: post #113
nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

128k context is not a limit of the model, that's a limit of implementation:

"Context Length: 262,144 natively and extensible up to 1,000,000 tokens."

https://huggingface.co/Qwen/Qwen3.8-27B

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#126

Post by scratchyone »

zackangelo wrote: post #31
jasongill wrote: post #16

It would be great if they made their inference capacity for this model available via OpenRouter; the fastest provider on OpenRouter right now is at ~80tps https://openrouter.ai/qwen/qwen3.8-27b#providers They do appear to host other models on OpenRouter so maybe Qwen3.8 will be there soon: https://openrouter.ai/provider/cerebras

We're serving it around 150-200tok/s (uses our new speculative decoding implementation on a DFlash2 draft model). https://mixlayer.com , LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.

any way to see the tok/s for all the models listed on your homepage? curious which has the best speed/quality tradeoff for me

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#127

Post by perching_aix »

gpugreg wrote: post #76
d2p wrote: post #70

Earlier quoted context omitted.

> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?

Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.

then it's basically useless lol, wtf, this has to be a defect

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#128

Post by sandworm101 »

amelius wrote: post #124
wild_egg wrote: post #121

Earlier quoted context omitted.

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute. I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time. Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was s…

Can't you do something with multiple accounts?

Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#129

Post by [deleted] »

Aurornis wrote: post #113
nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

[deleted]

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#130

Post by lowbloodsugar »

pllbnk wrote: post #48

Just a couple days ago I learned about ninfer ( https://github.com/Neroued/ninfer ) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.

Ok, I need to try that. I'm getting 45tok/s with vLLM on my 6000. >600tok/s concurrent, but 45tok/s single request.
Post reply on HN