Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

191–200 of 242 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#191

Post by freehorse »

blaesus wrote: post #167
freehorse wrote: post #33

I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.

May I ask what you used Gemma 31B for? Last time I try it wasn't bad but then it wasn't particularly good either.

It has been used in kagi's "quick assistant", so basically summarisation tasks (gets context from a bunch of webpages and finds the answer to a question in them, spits all the relevant to the query information, or similar).

It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#192

Post by egorfine »

zackangelo wrote: post #31
jasongill wrote: post #16

It would be great if they made their inference capacity for this model available via OpenRouter; the fastest provider on OpenRouter right now is at ~80tps https://openrouter.ai/qwen/qwen3.8-27b#providers They do appear to host other models on OpenRouter so maybe Qwen3.8 will be there soon: https://openrouter.ai/provider/cerebras

We're serving it around 150-200tok/s (uses our new speculative decoding implementation on a DFlash2 draft model). https://mixlayer.com , LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.

Trying now. History question: got 96 t/s. Code review task: 130 t/s. Nice!

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#194

Post by kilroy123 »

latentsea wrote: post #185

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

This too is my dream. But I want fable level intelegence. Not Qwen or deepseek flash.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#195

Post by imtringued »

irthomasthomas wrote: post #75
eli wrote: post #74

I just did a little anecdotal test. Had pi + cerebras review a recent commit and asked a few quick followups on it. Worked great. The Cerebras session cost me $1.60 and took a total of 5.1 mins. I did get a few brief 429 rate limit errors in there. The p50 speed was 890 tok/s and 0.64s TTFT. Using OpenRouter averages, that would've cost $0.29 (no cache discount at Cerebras!) and would've taken about 14.4 minutes. So…

Thanks! Is there something about their platform that prevents caching? Or are they just not passing on the discount?

It sounds to me that they don't have enough capacity and they want to discourage people from using the service.

There is nothing about their architecture that prevents reusing the KV cache other than the opportunity cost of keeping the memory occupied.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#196

Post by brookst »

eveningtree wrote: post #169
gpugreg wrote: post #61

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…

The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low? This doesn't work for my use case at all (code generation). These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't s…

Reminiscent of race to sleep: not suitable for sustained workloads, but for bursts ones it’s a good approach.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#197

Post by jurgenburgen »

gpugreg wrote: post #61

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…

I don’t think I understand. Why would faster token generation burn more tokens? The LLM should not be generating anything in between tool calls so the only difference should be that the human waits less between turns.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#198

Post by imtringued »

walrus01 wrote: post #171
tacone wrote: post #14

Noticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon. For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.

128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.

I have done the vast majority of my agentic coding work under 128k on a 256k context model and when I cross 128k, all I do is just ask for a handoff prompt to feed back into the next session. I do this because I was told the quality degrades as the context fills up and I fundamentally dislike random compaction, I don't even know what compaction does, it is completely intransparent.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#199

Post by imtringued »

gilgoomesh wrote: post #190
fulafel wrote: post #181

Earlier quoted context omitted.

128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all. I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.

It is common for the agent tools, MCP tool, skills, project context and system prompt to take between 5k and 25k tokens. This depends heavily on your agent and tool setup. But that's before the prompt. Then after the prompt, every tool call the agent makes adds to the context. Longer turns can easily consume 50k-100k tokens between the agent and various tool calls (reading the filesystem, reading files, reading compi…

I actually decided to run YOLO mode once on GPT 5.6 Luna setting it to the fastest speed and I struggled to fill up the context while I kept sending several dozen feature prompts in the same session.
Post reply on HN