Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B available on Cerebras at 1500 tokens/s

inference-docs.cerebras.ai

Post reply on HN
Print view View original

201–210 of 242 posts

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#201

Post by imtringued »

walrus01 wrote: post #175
fulafel wrote: post #174

Earlier quoted context omitted.

Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc. I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.

128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator. 128k is pretty much only good for chat/conversational/question asking (including tool calls…

One of the most boring tasks to give to a subagent is to make it run cargo test and summarize the result so that the main agent doesn't get polluted by the irrelevant tests.

You have to have a pretty inefficient use case for a subagent to think that 128k is not enough.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#203

Post by FeepingCreature »

pyrolistical wrote: post #138
ecshafer wrote: post #83

I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"

I run it locally at q4_k_xl on a r9700 with kv cache bf16 and while it thinks a lot, it’s still fast enough to do the task. This model had its knowledge replaced with reasoning ability. The chain of thought what makes this reasoning effective. So this is why you need to let it think and don’t quantize the kv cache.

Or at least use a modern llama.cpp with KV activation rotation.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#204

Post by ricardobeat »

conception wrote: post #152
ricardobeat wrote: post #101

Earlier quoted context omitted.

What kind of coding tasks would you expect to hit that limit? In my setup, on a very large codebase, it takes each agent 3-4 minutes at minimum to go past 100k tokens. (note it's 150k uncached tokens , the total limit is 450k/min)

So that’s about 400 tok/sec. Times that by 3, you get 100k in under a minute. That’s doing nothing special and just using your current setup.

150k is 2500 tokens/sec.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#205

Post by low_tech_punk »

codazoda wrote: post #111
low_tech_punk wrote: post #107

Earlier quoted context omitted.

No. You buy a minimum of $10 worth of credit, then use it at $1.49/M rate. There is no recurring charge. There is a separate subscription based plan, which is sold out now.

Got it. But, they also charge the same for cached tokens, so that probably closes the gap on Foundation models quite a bit.

my hunch is that their hardware can't cache

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#206

Post by interactivecode »

sandworm101 wrote: post #128
amelius wrote: post #124

Earlier quoted context omitted.

Can't you do something with multiple accounts?

Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.

The whole point is the speed

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#207

Post by jacquesm »

kristjansson wrote: post #146
jacquesm wrote: post #136

Earlier quoted context omitted.

How fast is it?

With MTP and FP4 I max out at 30ish t/s on mine. Without MTP or in regimes where the drafter performs poorly it’s about 10 t/s. FP8 is about half that

Thank you, always nice to see real world performance figures.

We run a pretty large rig, 10 GPUs right now (this goes up and down with various experiments, getting this many GPUs to play nice at x16 GEN4 with any motherboard is a challenge), 240G VRAM in total. 256G RAM and a TR PRO. For small models the comms overhead is larger than the gains so there I have to reduce the number of active GPUs. On this machine I'm getting between 150 and 200 tg/s with FP8, but it took a lot of time and tweaking to get to that, and not all of the improvements held up when combined with other improvements. I've been playing with this stuff for a while now and it is interesting how fast the frontier is moving and how much you can now do on your own hardware. For larger models the communications overhead is low enough that we can run them on bigger groups of GPUs, and using hacked drivers to give us p2p capabilities on some of our GPUs also boosts performance considerably once you start to hit communications limits. Typically we get 50G/second in p2p mode (full duplex, half that one way).

From a cost perspective running locally is not interesting, but it allows us to do experiments that model providers would likely balk at, gives us censorship free access and allows us to work with data that we would not want to share with model providers (or can't share due to NDAs).

I will look into running ninfer, I was aware of them but had not yet gotten around to using it.

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#208

Post by vidarh »

nostrebored wrote: post #54

150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…

[dead]

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#209

Post by jasongill »

the_duke wrote: post #38
jasongill wrote: post #9

Earlier quoted context omitted.

It appears that they do support Prompt Caching: https://inference-docs.cerebras.ai/capabilities/prompt-cachi...

It doesn't reduce the price though.

Good catch, I guess I got lost in the marketing speak of the page!

Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s

#210

Post by Tuna-Fish »

vel0city wrote: post #155
ericd wrote: post #132

Earlier quoted context omitted.

No, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.

I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning. Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.

Yes and no. A single chip cannot have it's weights adjusted, once it's out, it is what it is.

But also, the model weights are in a single mask rom layer, high up in the metal stack. They could manufacture the die specialized for a given geometry of a model up to that layer, wait for updated weights, and then get the final product out in weeks after they got the weights, instead of many months which is what it would take to redesign the whole chip for the new weights.

Post reply on HN