Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

171–180 of 240 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#171

Post by walrus01 »

tacone wrote: post #14

Noticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon. For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.

128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#172

Post by wincy »

jacquesm wrote: post #136
gerdesj wrote: post #123

Earlier quoted context omitted.

I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware. I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve. To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.

How fast is it?

Parent already responded but just for reference an RTX 5090 with Ninfer hits 160 tokens/second with qwen 3.8 27B which is very usable.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#173

Post by zxexz »

selcuka wrote: post #147
gerdesj wrote: post #125

Earlier quoted context omitted.

128k context is not a limit of the model, that's a limit of implementation: "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." https://huggingface.co/Qwen/Qwen3.8-27B

TPM means Tokens per Minute.

GP is referring to GGP’s last paragraph. 150k t/m, yes, and 128k context.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#174

Post by fulafel »

walrus01 wrote: post #171
tacone wrote: post #14

Noticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon. For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.

128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.

Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc.

I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#175

Post by walrus01 »

fulafel wrote: post #174
walrus01 wrote: post #171

Earlier quoted context omitted.

128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.

Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc. I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator.

128k is pretty much only good for chat/conversational/question asking (including tool calls for searching things and spitting back/parsing a set of results) or human interactive agent purposes.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#177

Post by Aurornis »

gerdesj wrote: post #125
Aurornis wrote: post #113

Earlier quoted context omitted.

> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit shared by an entire team at once? 150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find. I think the 128K context…

128k context is not a limit of the model, that's a limit of implementation: "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." https://huggingface.co/Qwen/Qwen3.8-27B

We're talking about the Cerebras implementation, which is limited to 128K.

It's in the link.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#178

Post by apatheticonion »

I just want an API that takes these crazy small / cost effective open weight models and charges peanuts for access.

Think, DeepSeek Flash (before the price hikes) prices.

If I can run this on a 32gb card while they have a datacenter with wholesale electricity prices, why are we not seeing "cents per billion tokens" pricing?

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#179

Post by gpugreg »

LoganDark wrote: post #164
0xbadcafebee wrote: post #90

Earlier quoted context omitted.

Yeah, their public service isn't a serious/competitive offering. They don't have the capacity to serve all the customers who might want to use them at that speed. The public service exists so they get some users on OpenRouter, and that shows them as #1 on speed, which proves their tech is very fast, which gets them billions in hardware sales/licensing. If you have big enough pockets they can probably dedicate capacit…

GPUs can't reach these speeds. You could build a supercomputing cluster and still not reach these speeds.

MiMo-V2.5-Pro-UltraSpeed gets pretty close with over 1000 TPS on 8x B200. It has 1.02T total parameters and 42B active, compared to 27B total/active for Qwen3.8-27B. Also, B300 are out now. I think 1500 TPS for Qwen3.8-27B should be doable.
Post reply on HN