Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

231–240 of 240 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#231

Post by ericd »

liamwire wrote: post #189
ericd wrote: post #165

Earlier quoted context omitted.

Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs. But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models. They may have had a little flexibility by suppor…

To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.

Yeah, I'm guessing this isn't unique, but I remember the first time I used ChatJimmy, I missed the fact that it had responded because I was still hitting the enter key, and getting ready to see tokens stream in, but they were already all sitting there, and I'd missed registering the visual diff somehow.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#232

Post by gabri200 »

I tried to get Qwen3.8 27B working properly under high concurrency, and while the quality level is spectacular for the size, the performance wasn't the best, even with MTP. Unless you have a very big infrastructure, it's difficult to run a dense model concurrently with high throughput.I suppose that's why almost all large models are now MoE. On the other hand, 1500 tok/s is an impressive speed, and that speed is very important for agent tasks, so a service like this instead of local infrastructure might make sense, although it also depends on your busines constraints.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#233

Post by iamleppert »

gabri200 wrote: post #232

I tried to get Qwen3.8 27B working properly under high concurrency, and while the quality level is spectacular for the size, the performance wasn't the best, even with MTP. Unless you have a very big infrastructure, it's difficult to run a dense model concurrently with high throughput.I suppose that's why almost all large models are now MoE. On the other hand, 1500 tok/s is an impressive speed, and that speed is very…

Completely unusable last time I tried it. You got hit with rate limits after the first few minutes of using it. It's fast but cannot sustain its claimed speed before it immediately hits its rate limit. What good is it if you can't finish a task? It's like having a sports car that can only drive 180 mph for 5 seconds every minute and then has to cool down for an hour.

Borderline fraudulent to advertise to developers when it's completely impractical to use. Their only support is their company Slack channel.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#234

Post by gabri200 »

gpugreg wrote: post #61

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…

To get a good coding agentic system you need to use big context (Specs and conversation context can't be condensed every minute), so you need to use prefix caching, and the price for the hit cache tokens can't be the same that miss cache or the final price could be insane.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#235

Post by aizk »

pllbnk wrote: post #48

Just a couple days ago I learned about ninfer ( https://github.com/Neroued/ninfer ) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.

Is there an equivalent but for 4090s?

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#236

Post by apitman »

codazoda wrote: post #115
ecshafer wrote: post #83

I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"

I want a Qwen 3.8 27B hosted locally but I don't quite have the RAM for it. And, I don't want to buy the RAM until I prove I can use it. Yesterday I did have success with Gemma-4-12b with 128k context. It fits in my RAM and it's relatively fast on my hardware. I had to give it prompts that are quite a bit different from the way I use foundation models, but I did get it to work quite well. I feel like I could learn it…

You could try renting a GPU or VPS

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#237

Post by latentsea »

mncharity wrote: post #214
latentsea wrote: post #185

What we need is 1500 t/s locally. No rate limits. No rent seeking middlemen. Pure, unadulterated, all you can eat tokens as fast as you can eat them.

> 1500 t/s [...] all you can eat tokens as fast as you can eat them Pfft, you can eat tokens far far faster than that. Many orders of magnitude more. Just switch from frontier-style bespoke artisanal pets to cloud burst-parallel, latent subspace exploring/exploiting/searching, mass ensembles of cow herds. There's N-Version Programming. Work the problem in English, in Chinese, in Haskell, Lisp, Rust, etc. Then work po…

I too know words.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#238

Post by pllbnk »

aizk wrote: post #235
pllbnk wrote: post #48

Just a couple days ago I learned about ninfer ( https://github.com/Neroued/ninfer ) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.

Is there an equivalent but for 4090s?

The repository has many forks, suggesting that folks are trying to (vibe) code support for different GPUs. Might be worth a shot.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#239

Post by 70rd »

eveningtree wrote: post #169
gpugreg wrote: post #61

I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…

The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low? This doesn't work for my use case at all (code generation). These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't s…

Throughput is useful if you want to generate a lot of transcripts for RL. It's for making Qwen better, not for actually using Qwen.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#240

Post by gerdesj »

Aurornis wrote: post #177
gerdesj wrote: post #125

Earlier quoted context omitted.

128k context is not a limit of the model, that's a limit of implementation: "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." https://huggingface.co/Qwen/Qwen3.8-27B

We're talking about the Cerebras implementation, which is limited to 128K. It's in the link.

Yes, and I said its a limit of the implementation and not the model.
Post reply on HN