Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

211–220 of 235 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#211

Post by mrngld »

apatheticonion wrote: post #178

I just want an API that takes these crazy small / cost effective open weight models and charges peanuts for access. Think, DeepSeek Flash (before the price hikes) prices. If I can run this on a 32gb card while they have a datacenter with wholesale electricity prices, why are we not seeing "cents per billion tokens" pricing?

Because your 32gb card isn't running this at 1500 token/s. Serving these things at scale with the enormous context windows real use demands and doing some with usable performance takes a lot of expensive hardware. Yes, their margin on straight inference is allegedly really high, but that's severely offset by high capital costs.

If you want to spend a new car worth of money and still not serve as fast as Cerebras because you can't simply buy their mammoth custom chips, then yes you too can self host a huge Deepseek or GLM model.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#212

Post by mrngld »

karim79 wrote: post #134

Tokens are the new latest and greatest nonsensical shit on the planet. It's amusing. I can't wait to see the world in 1-2 years and the hilarity of looking back on this day.

Someone on horseback probably said that after seeing a car broke down on the side of the road for the first time.

Well, cars only got better.

We're going to be talking about how to leverage this new tool for many years to come.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#213

Post by mrngld »

nicce wrote: post #79
srcreigh wrote: post #67

How many years until chips like this are available to consumers?

Many. Too lucrative for certain companies and even governments to allow that to happen

You're right, but no need to be conspiratorial, demand is just so high if you ordered one today you might be lucky to see it in 2028.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#214

Post by mncharity »

latentsea wrote: post #185

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

> 1500 t/s [...] all you can eat tokens as fast as you can eat them

Pfft, you can eat tokens far far faster than that. Many orders of magnitude more. Just switch from frontier-style bespoke artisanal pets to cloud burst-parallel, latent subspace exploring/exploiting/searching, mass ensembles of cow herds.

There's N-Version Programming. Work the problem in English, in Chinese, in Haskell, Lisp, Rust, etc. Then work ports to the target lang.

There's design space sampling. Work the problem emphasizing performance, or security, or monitoring, readability, etc. Then work a synthesis.

There's non-determinism sampling. Work the problem order 10 or 100 times. Then work to combine the best bits from each.

There's sample synthesis. NP-hard aggregation of insights.

There's genetic exploration. Work populations of trees of work variants under selective pressure.

There's repo quantum superpositions of implementation space. The unspecified remains indeterminate - state space collapse occurs not upon each edit/commit, but as JIT-synthesized fuzzing/search upon each execution.

There's maintaining a pretty dev UI, but that >>10k tok/s is trivial, because like symbiotic adversary cocreation, fine-grain agent swarms, scenario analysis/forecasting, etc, etc, it is unlike the preceding items... which scale combinatorially.

"All you need is 1500 t/s"? "All you need is 640k RAM" is only 5 orders of magnitude off from 64 GB. It takes "All you need is a single Intel 3101's 64 bits", to get 9 orders of magnitude from 64 GB. Then datacenters...

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#215

Post by lukewarm707 »

nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

use 3rd party marketplaces. cerebras is resold on vercel, openrouter and huggingface.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#218

Post by wongarsu »

eveningtree wrote: post #169
gpugreg wrote: post #61

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…

The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low? This doesn't work for my use case at all (code generation). These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't s…

Might be usable for short-context utility workloads? Generate the title of your chat session based on the first three messages at the speed of light

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#219

Post by Roark66 »

Don't they have very low limits?

What do people use these tiny limits for?

I started measuring my Claude Max x5 use and last week (they gave me 50% more) I used 1.3B input tokens. Some 130M were cache writes, rest was cached. And 5M output.

This puts things in perspective. We're taking thousands of bucks weekly even if I managed to switch to Kimi K3.

What is the majority of this use? Infrastructure upgrades, troubleshooting and so on. Ingesting quite a bit of documentation at beginning of each session.

Sessions run from few hours to a month long and 1M context usually hovers near 30-60%.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#220

Post by bitexploder »

nicce wrote: post #186
bitexploder wrote: post #114

Earlier quoted context omitted.

But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)

You can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.

I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.
Post reply on HN