41–50 of 295 posts
Whoa. This looks both powerful and expensive. My prediction is that, this time next year, top developers outside ai labs will be spending 50k USD+ on inference. Within labs, I've heard spend is already far beyond this per developer.
If someone subsidize maybe, but if the companies need to pay no way, unless there is hard evidence of the return.
Earlier quoted context omitted.
The stake in the side of cerebras has always been that the economics are pretty poor. Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.
Can anyone explain why Cerberus needs to be _fast_ instead of _cheap_? I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s. Is the limitation, even on cerberus, still that the cache can only serve so many concurrent sessions over time? Is there no scaling advantage? I genuinely do not understand how any of this works.
So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.
https://newsletter.semianalysis.com/p/cerebras-faster-tokens... goes quite in-depth.
> Compared with output speeds reported by Artificial Analysis GPT-5.6 Sol on Ultrafast mode runs 11x faster than Fable 5, and 5x faster than Opus 4.8 on Fast mode. Awesome work. I'm personally very excited for faster models/inference. I think speed is underrated to some degree in the current conversation. For a while, I was using Cursor's Composer quite a lot, even over frontier models, just because of how darn fast…
What do you need speed for? That's a genuine question, I feel like the limiting factor already is my creativity, attention span and budget. And I'm not even yet optimizing cost by batching things like review to slow local models over night, or schedule tasks to take full advantage of my subscriptions.
IME waiting for an agent to work through a problem is a detriment to attention span; your mind drifts to other things while you wait. Maybe you can steer several agents in a round robin instead, but then there's a cognitive tax from context switching. Faster models mean fewer gaps in focus.
Unless I have read over it, besides the animation in the intelligence vs speed graph which only mentions internal data and not whether they truly reran the AA suite, there is no actually solid statement on the important aspect of performance. Neither the Cerebras or OpenAI post [0] outright state that this performs exactly the same as regular 5.6 Sol. I feel if this was 1:1 just Sol but much faster, they'd (rightfull…
"delivering up to 750 output tokens per second and without any quality compromise" seems pretty definitive.
Same for massive performance differences in the way providers like Cerebras, Groq, etc. have deployed models including K2.6 on Cereberas specifically. Massive deltas in tool call and overall quality despite there being far more clarity in open weight vs proprietary model deployment.
The AA suite graph with that animation is the only time in either post that absolute parity is being asserted and I'd be amazed if that was the case, but am doubtful why their phrasing is so cagey.
Why not assert full parity in writing? It "performs the same (within run-to-run variance) across all evals that Sol has been tested with" is very different to "no quality compromise/degradation", the later allowing for a lot more wiggle room and interpretation in what evals you use to assess that, what quality truly means, etc., the former meaning identical in all situations.
Could also be a language barrier here in fairness, maybe this phrasing is more iron clad than I give them credit, but especially with OpenAI, I have seen enough checkpoints asserted as unchanged in "quality" to where I am skeptical. Ironically, I never saw that with Anthropic (which has gotten far more heat for degradation accusations) while a model was deployed with one exception in mid-late April this year. Pure speculation, but believe it wasn't noticed much before "agentic coding" became more popular, because chat output is far more subjective without a rating framework vs code passing which can be an objective metric with more potential for frustration.
Earlier quoted context omitted.
The stake in the side of cerebras has always been that the economics are pretty poor. Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.
Can anyone explain why Cerberus needs to be _fast_ instead of _cheap_? I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s. Is the limitation, even on cerberus, still that the cache can only serve so many concurrent sessions over time? Is there no scaling advantage? I genuinely do not understand how any of this works.
There's some technical hypotheses about it that other people are offering.
But also from a business perspective, it totally makes sense not to go any sort of batching play. It's really valuable and very clear to consumers to make your pitch entirely about lower latency rather than higher bandwidth.
There are so many scenarios that are latency-constrained that will be difficult or even impossible for someone even with fleets of high-bandwidth compute to compete with you on.
Very easy pitch to sell a customer who asks what differentiates you from other companies: you pay us a premium for lower latency than anyone else.
I'd just like to point out that the largest model Cerebras has ever served is Kimi K2.6 which is 1T parameters, so that either means that theyve had a breakthrough on the hardware engineering side of things, or GPT-5.6 Sol is likely a lot smaller than people think. If it truly is only ~1-2T parameters, then this kinda kills 2 narratives for me. 1. all the handwringing about open source catching up via Kimi K3 (3T par…
Isn’t the fact Fable is more expensive than Sol-Max by multiples already an indication that Sol is way smaller?
Earlier quoted context omitted.
Maybe then LLM's will switch to outputting raw machine code?
No. This conveys a fundamental misunderstanding of how anything pertaining to programming works. This will never happen, ever . For example, take... printf("Hello, world"); vs. a plausible illustration of how it might be compiled down to machine code... 48 65 6C 6C 6F 2C 20 77 6F 72 6C 64 48 83 EC 28 48 8D 0D F5 0F 00 00 E8 F0 00 00 00 33 C0 48 83 C4 28 C3 The latter now takes up 10x as many tokens (= 10x the cost/ti…
Earlier quoted context omitted.
The stake in the side of cerebras has always been that the economics are pretty poor. Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.
Can anyone explain why Cerberus needs to be _fast_ instead of _cheap_? I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s. Is the limitation, even on cerberus, still that the cache can only serve so many concurrent sessions over time? Is there no scaling advantage? I genuinely do not understand how any of this works.
With the tool calls that can be done, you're not pricing this against an executive assistant or pocket analyst - you're pricing this against the ability to have an entire Bourne Identity style analysis room at your disposal. The limited inventory will go to the people for whom money is no object.
Earlier quoted context omitted.
"delivering up to 750 output tokens per second and without any quality compromise" seems pretty definitive.
No quality compromise/degradation is something I have had this industry, including especially OpenAI, claim multiple times in the past and I have more than once been able to verify that it was in fact not the case. Examples being gpt-3.5-turbo vs text-davinci-003, GPT-4-Turbo and all the other post training checkpoints they had under one name (which was a major bug bear for me back then witnessing degradations with n…