Qwen 3.8 27B available on Cerebras at 1500 tokens/s • Hacker News Board

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#181

Post by fulafel » Fri, Sep 04, 2026, 7:41 AM UTC

walrus01 wrote: post #175

fulafel wrote: post #174

Earlier quoted context omitted.

Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc. I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator. 128k is pretty much only good for chat/conversational/question asking (including tool calls…

128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all.

I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#182

Post by LoganDark » Fri, Sep 04, 2026, 7:43 AM UTC

gpugreg wrote: post #179

LoganDark wrote: post #164

Earlier quoted context omitted.

GPUs can't reach these speeds. You could build a supercomputing cluster and still not reach these speeds.

MiMo-V2.5-Pro-UltraSpeed gets pretty close with over 1000 TPS on 8x B200. It has 1.02T total parameters and 42B active, compared to 27B total/active for Qwen3.8-27B. Also, B300 are out now. I think 1500 TPS for Qwen3.8-27B should be doable.

That model uses a lot of tricks to achieve 1000 t/s. I would not use raw parameter counts alone for such comparisons, in general.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#183

Post by naklitechie » Fri, Sep 04, 2026, 7:49 AM UTC

Has reliability improved? It's flaky in Production.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#184

Post by walrus01 » Fri, Sep 04, 2026, 7:58 AM UTC

fulafel wrote: post #181

walrus01 wrote: post #175

Earlier quoted context omitted.

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator. 128k is pretty much only good for chat/conversational/question asking (including tool calls…

128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all. I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.

Tokens are also occupied by the internal reuslts of 'thinking', for instance, using the latest opencode, give qwen 3.8-flash-next with 'high' thinking mode 50KB total of python to read in six different files, count the expected tokens you'd expect it to occupy in just the size of the python. Then count the actual token count taken up after it's finished thinking about the contents of the python and analyzing it before you give it a single directive.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#185

Post by latentsea » Fri, Sep 04, 2026, 8:03 AM UTC

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#186

Post by nicce » Fri, Sep 04, 2026, 8:47 AM UTC

bitexploder wrote: post #114

nicce wrote: post #100

Earlier quoted context omitted.

They seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.

But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)

You can run it with 2x r9700 with 150-200 tokens per second. It is intelligent enough if you just point the docs / whatever for it.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#187

Post by register » Fri, Sep 04, 2026, 8:47 AM UTC

I tried it. Not impressed. The gain in development speed is only marginal as there are other bottlenecks that affect overall development speed. Probably gains might be more significant for pure content generation tasks rather than software development

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#188

Post by gilgoomesh » Fri, Sep 04, 2026, 8:50 AM UTC

latentsea wrote: post #185

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

Easy. I just need... 50 times more speed.

Better download some more RAM.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#189

Post by liamwire » Fri, Sep 04, 2026, 8:56 AM UTC

ericd wrote: post #165

vel0city wrote: post #155

Earlier quoted context omitted.

I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning. Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.

Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs. But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models. They may have had a little flexibility by suppor…

To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#190

Post by gilgoomesh » Fri, Sep 04, 2026, 8:57 AM UTC

fulafel wrote: post #181

walrus01 wrote: post #175

Earlier quoted context omitted.

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator. 128k is pretty much only good for chat/conversational/question asking (including tool calls…

128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all. I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.

It is common for the agent tools, MCP tool, skills, project context and system prompt to take between 5k and 25k tokens. This depends heavily on your agent and tool setup.

But that's before the prompt.

Then after the prompt, every tool call the agent makes adds to the context. Longer turns can easily consume 50k-100k tokens between the agent and various tool calls (reading the filesystem, reading files, reading compiler output, reading memories).

Then each "turn" with the agent stays in context and is fed into the next turn. Two or three turns and you're up near 250k.