131–140 of 238 posts
Just a couple days ago I learned about ninfer ( https://github.com/Neroued/ninfer ) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.
Earlier quoted context omitted.
Cerebras the tech is awesome, cerebras the company is a trainwreck
Is this the chatjimmy asic approach with a bigger model?
Earlier quoted context omitted.
Can't you do something with multiple accounts?
Or just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.
Earlier quoted context omitted.
150k by account. At 1.5k a second you hit it very quickly.
Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.
Earlier quoted context omitted.
Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.
I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware. I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve. To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.
Earlier quoted context omitted.
It appears that they do support Prompt Caching: https://inference-docs.cerebras.ai/capabilities/prompt-cachi...
> How are cached tokens priced? > There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model. Well, talk about flipping the narrative.
I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"
This model had its knowledge replaced with reasoning ability. The chain of thought what makes this reasoning effective.
So this is why you need to let it think and don’t quantize the kv cache.
Earlier quoted context omitted.
It appears that they do support Prompt Caching: https://inference-docs.cerebras.ai/capabilities/prompt-cachi...
> How are cached tokens priced? > There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model. Well, talk about flipping the narrative.
Just tried it on a medium size coding/debug problem on an existing codebase, observations: - Input doesn't look faster than other models, it spends a lot of time reading Read about 5M tokens - Output is awesome, super fast as you expect from the 1500t/sec I think that's correct - Tool call is failing more than say DS4, which leads to time wasted on retries (complex tools like browser control for example) - Shell comm…
I can't believe this situation has not improved in years. Is cerebras' main business selling the hardware, then?