Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating GPT-5.6 Sol Ultrafast

cerebras.ai

Post reply on HN
Print view View original

231–240 of 295 posts

Re: Accelerating GPT-5.6 Sol Ultrafast

#231

Post by energy123 »

anthonypasq wrote: post #46
manmal wrote: post #29

Earlier quoted context omitted.

Isn’t the fact Fable is more expensive than Sol-Max by multiples already an indication that Sol is way smaller?

no, because closed sourced model pricing has no relationship to its size. Thats what im saying. the inference margins are crazy, but people think the fonrtiner models must be 10T params or something because theyre expensive

> closed sourced model pricing has no relationship to its size.

That's too strong. Only in an actual monopoly for a product with no substitutes that has price inelastic demand can pricing fully disconnect from costs. Frontier model serving is only maybe a soft version of that, where costs and moat both contribute to pricing.

Re: Accelerating GPT-5.6 Sol Ultrafast

#232

Post by DaSHacka »

sashank_1509 wrote: post #204
navorad772 wrote: post #109

Earlier quoted context omitted.

Working with agents right now is like working on huge legacy C++ codebase (feel free to insert your nightmares). You change 10 lines of code and then wait 20 minutes before it compiles and runs tests (or write prompt and wait 20 minues for code). External distractions and huge waiting times breaks your flow. Working long time in the flow is what makes mythical 10x programmer. IMHO 10x programmers are not superior in…

Strong disagree, there’s nothing better about an agent generating a 1000 lines in 1 second or 1 minute. I can’t read it that quickly anyway.

But with the 1 second model you can get to reading the code immediately, and with the 1 minute model you have to sit doing 'nothing' for a minute.

Rinse and repeat for each change/iteration

Re: Accelerating GPT-5.6 Sol Ultrafast

#233

Post by krzyk »

dannyw wrote: post #169
aetherspawn wrote: post #130

Earlier quoted context omitted.

I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed be…

We’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company. The amount of workloads we can shift with an advisor model pattern continues to grow. It’s seriously amazing.

By "advisor model pattern" you mean https://claude.com/blog/the-advisor-strategy ?

How do you implement that outside of claude code?

Re: Accelerating GPT-5.6 Sol Ultrafast

#234

Post by beng-nl »

piyh wrote: post #16
iamcoder18 wrote: post #13

I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration. > In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a s…

Feels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.

I wish, but I don’t think this qualifies as single thread performance. Cerebras is whole wafer computation though right? So you get the result sooner, but the same amount of computation has occurred, just with huge parallelism.

Re: Accelerating GPT-5.6 Sol Ultrafast

#235

Post by Squarex »

vb-8448 wrote: post #41
owentbrown wrote: post #25

Whoa. This looks both powerful and expensive. My prediction is that, this time next year, top developers outside ai labs will be spending 50k USD+ on inference. Within labs, I've heard spend is already far beyond this per developer.

50k per month?! If someone subsidize maybe, but if the companies need to pay no way, unless there is hard evidence of the return.

Probably per year, still a crazy amount.

Re: Accelerating GPT-5.6 Sol Ultrafast

#236

Post by sixtyj »

JohnBooty wrote: post #207
TacticalCoder wrote: post #187

Earlier quoted context omitted.

> Never heard of it before, that's fucking insane. They've been acquired by AMD. Those saying the model sucks are completely missing the point: it was a proof-of-concept. The question is: what happens to a model like Anthropic's Fable 5 that does, what, 70 tokens/s (and requires lots of output tokens) when the latest open-weights model is etched on silicon and does 14 000 tokens/s? Shall the better model still have t…

Shall the better model still have the upper hand or will the raw speed compensate? At 14,000 tokens/sec there's just so much ridiculous stuff that might be possible. Let's assume that this POC proves they can take the next step, and can eventually etch a capable ~27B model into silicon. Let's call it Fred. Ralph loops automatically get real real interesting again. 200x the iteration speed. This is such a clear win I…

Yes, imho 14k TPS is just a beginning.

This Gas Town? https://github.com/gastownhall/gastown

Re: Accelerating GPT-5.6 Sol Ultrafast

#237

Post by UltraSane »

black_knight wrote: post #225
UltraSane wrote: post #69

Earlier quoted context omitted.

I'm very curios what the code is doing.

It is a normalisation procedure for inductive and coinductive data types. Basically, the code is a function which takes in an expression where you can use generic data structures as variables and then some specific data structures, and it plugs them in for the variables. It then computes the structure of the resulting data type. So, admittedly not a trivial task – hence the choice of Fable as the model. Also, this wo…

What language?

Re: Accelerating GPT-5.6 Sol Ultrafast

#238

Post by raylad »

datadrivenangel wrote: post #70
thraway3837 wrote: post #9

This is really cool. Someone here commented about similarity between this and hardware advancements for AV encode/decode. I think it's only a matter of time before miniaturization can have a thumbnail sized user-replaceable accessory that contains the LLM built onto the hardware. I admit I don't know how any of that works, but would be amazing to experience. Fully local, fully offline, ultra fast local inference bett…

In 5-10 years nice smartphones will be able to run ChatGPT (~gpt3-4) class models. A memory rich laptop (highend mac/framework) can run GPT-OSS:120b or full Gemma4 at very interactive speeds. High end phones can already run the smaller models at enough speed to be probably useful, especially for background/overnight photo tagging and curation and things like that.

In 5 years the models will probably be so much better and more compact that phones will be running models equivalent at least to Opus 4.6 if not Fable, at least within the areas they are tuned for (which probably won't include coding).

Re: Accelerating GPT-5.6 Sol Ultrafast

#239

Post by msdz »

senordevnyc wrote: post #163
msdz wrote: post #100

Earlier quoted context omitted.

> At such output speed, I wouldn’t expect reasoning. As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.

Yeah, I thought reasoning was literally just chain of thought in the output token stream, with the model itself adding delimiters to indicate what part of the output is internal reasoning, and what part is an answer to the user. Is that wrong?

No, as the sibling comment mentioned, your understanding was correct there.

What’s more, the only technical difference in speeds could be, and likely also is with the HC1 chip, between prefill (prompt processing) and decode (text generation) speeds. I don’t know whether it’s the case with Taalas’ chip, but in the “software-based” LLMs we typically see and use so far, those two stages hit different parts of a computer (processing/compute-bound vs. memory/bandwidth-bound).

Re: Accelerating GPT-5.6 Sol Ultrafast

#240

Post by sixtyj »

mips_avatar wrote: post #124
sixtyj wrote: post #72

Earlier quoted context omitted.

Output from Cerebras with GPT model is 750 tokens per second. Don’t blink. (Chatjimmy has 14,200 TPS.)

Unfortunately AMD bought them, so I don't think we will get to see another release from them.

Aha, thanks, that’s fresh; press release is from Aug 6

https://ir.amd.com/news-events/press-releases/detail/1296/am...

Post reply on HN