Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

141–150 of 241 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#141

Post by jychang »

amelius wrote: post #124
wild_egg wrote: post #121

Earlier quoted context omitted.

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute. I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time. Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was s…

Can't you do something with multiple accounts?

You would lose caching (if they cache)

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#142

Post by ma2kx »

sandworm101 wrote: post #128
amelius wrote: post #124

Earlier quoted context omitted.

Can't you do something with multiple accounts?

Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.

Thats not the point if you choose Cerebras as provider.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#144

Post by ma2kx »

codazoda wrote: post #111
low_tech_punk wrote: post #107

Earlier quoted context omitted.

No. You buy a minimum of $10 worth of credit, then use it at $1.49/M rate. There is no recurring charge. There is a separate subscription based plan, which is sold out now.

Got it. But, they also charge the same for cached tokens, so that probably closes the gap on Foundation models quite a bit.

I guess Cerebras didnt intend the model for agentic coding but rather for small one shot task like title generation. At least thats why I use the free tier for.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#145

Post by kristjansson »

wild_egg wrote: post #121
Aurornis wrote: post #113

Earlier quoted context omitted.

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute. I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time. Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was s…

They made the coding plan a bit better toward the end, but it was pretty tough to use throughout.

Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#146

Post by kristjansson »

jacquesm wrote: post #136
gerdesj wrote: post #123

Earlier quoted context omitted.

I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware. I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve. To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.

How fast is it?

With MTP and FP4 I max out at 30ish t/s on mine. Without MTP or in regimes where the drafter performs poorly it’s about 10 t/s. FP8 is about half that

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#147

Post by selcuka »

gerdesj wrote: post #125
Aurornis wrote: post #113

Earlier quoted context omitted.

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

128k context is not a limit of the model, that's a limit of implementation: "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." https://huggingface.co/Qwen/Qwen3.8-27B

TPM means Tokens per Minute.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#148

Post by selcuka »

perching_aix wrote: post #127
gpugreg wrote: post #76

Earlier quoted context omitted.

Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.

then it's basically useless lol, wtf, this has to be a defect

It's PR:

https://news.ycombinator.com/item?id=49556302

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#149

Post by collin »

baegi wrote: post #95
collin wrote: post #92

Earlier quoted context omitted.

This was my experience a year ago on some other model they could run super fast. Routine coding tasks would hit the per-minute token limits. Just the math there... 150k TPM... and 15k TPS means... you can run for 10 seconds every minute? The basic math boggles the mind.

Not sure how the rate limiting works, but it's 1.5k TPS, not 15k, so you could run it for 100s/min, which seems good enough to me

ah, yes, that seems right

I was using it quite a while back, different model, different quotas, but for coding tasks it routinely hit quotas which made it quite difficult to actually use.

100s/min seems pretty poor actually with sub-agents etc.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#150

Post by conception »

gerdesj wrote: post #123
devy wrote: post #119

Earlier quoted context omitted.

Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.

I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware. I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve. To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.

The problem is most providers hit tok/sec limits really fast. 1m/min is the default and the only place I can get 10m+ is from first party providers without a lot of upfront cash.
Post reply on HN