161–170 of 240 posts
I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"
150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…
Cerebras the tech is awesome, cerebras the company is a trainwreck
Earlier quoted context omitted.
They don't have cache (e.g. KV cache). But they write down what you sent earlier to say they cached it! To still bill the same as uncached later (because they didn't actually cache it)!
More precisely they can't cache it.
Say avg model response length is 1024 tok. At 50 tok/s normal providers do your turn will only take 20s (vs Cerebras 101s) and will cost 20x less. That time and cost is per single tool call.
150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…
Yeah, their public service isn't a serious/competitive offering. They don't have the capacity to serve all the customers who might want to use them at that speed. The public service exists so they get some users on OpenRouter, and that shows them as #1 on speed, which proves their tech is very fast, which gets them billions in hardware sales/licensing. If you have big enough pockets they can probably dedicate capacit…
Earlier quoted context omitted.
No, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.
I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning. Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.
But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.
They may have had a little flexibility by supporting finetuning via LoRAs.
150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…
all the above. They just simply do not care about non enterprise customers. Today they announced qwen, guess what - it's also the same day they pulled Gemma off their shared tier. No migration notice and all developers are scrambling as we speak trying to migrate. They gave a soft head-ups on discord a week ago and when folks complained about zero-day migration they started saying 'you aren't suppose to build product…
I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.
150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool. Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information. ``` Billing access restricted Self-serve billi…
This was my experience a year ago on some other model they could run super fast. Routine coding tasks would hit the per-minute token limits. Just the math there... 150k TPM... and 15k TPS means... you can run for 10 seconds every minute? The basic math boggles the mind.
It’s one order of magnitude less TPS, but still, that’s the limit with just one user…
I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…
This doesn't work for my use case at all (code generation).
These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't see myself using llms for them in the future too.
Paying for external inference for a much larger model like qwen 3.8-flash-next Q8 with full context makes a lot more sense, since the model consumes something like 188GB RAM when fully loaded into an inference engine.
Yeah I guess this is cool and all that it runs at some ridiculous token/s rate but if the actual usage of it is highly limited... What's the point? I'd rather have a much slower tok/s rate that can chew on things 24x7.